Assessing Statistical Significance in Machine Translation Evaluation

At a Glance

• Statistical significance testing determines whether differences between machine translation systems reflect real improvements or random variation.
• COMET generally correlates more closely with human judgments than traditional metrics such as BLEU.
• Bootstrap resampling is one of the most widely used methods for testing statistical significance in machine translation evaluation.
• Confidence intervals and p-values help determine whether observed metric differences are statistically meaningful.
• Reliable MT evaluation combines modern metrics with statistical significance testing rather than relying on metric scores alone.

Machine translation systems are often compared using automatic evaluation metrics such as BLEU and COMET. However, a higher metric score does not necessarily mean that one translation system is genuinely better than another. Small differences in evaluation scores may simply result from the particular test dataset rather than reflecting a real improvement in translation quality.

To determine whether the observed differences are meaningful, researchers use statistical significance testing. Methods such as bootstrap resampling help distinguish genuine improvements from random variation, making translation system evaluation more reliable and scientifically sound.

In this article, we explain why statistical significance matters in machine translation evaluation, compare the strengths of BLEU and COMET, and show how bootstrap significance testing can be used to compare translation systems with confidence.

How to Compare Machine Translation Systems Using Statistical Significance

When comparing machine translation systems, a higher BLEU or COMET score is only the starting point. To determine whether one system genuinely outperforms another, follow these steps:

1. Evaluate both systems on the same test dataset using a consistent evaluation protocol.
2. Measure translation quality with automatic metrics such as BLEU or, preferably, COMET.
3. Apply statistical significance testing, typically using bootstrap resampling, to estimate the reliability of the observed score differences.
4. Interpret the results using confidence intervals and p-values to determine whether the differences are statistically significant.
5. Draw conclusions only when the statistical analysis confirms that the observed improvement is unlikely to have occurred by chance.

The following sections explain each step in detail, helping you understand how to evaluate machine translation systems and interpret evaluation results with confidence.

Why Statistical Significance Matters

Automatic metrics such as BLEU and COMET are widely used to evaluate machine translation systems. Metric scores should therefore be interpreted with caution, as observed differences may reflect characteristics of the evaluation dataset rather than genuine improvements in translation quality.

Statistical significance testing helps determine whether the observed differences are likely to reflect real performance gains or random variation. This makes machine translation evaluation more reliable and supports objective comparisons between translation systems.

BLEU vs. COMET

BLEU and COMET are the two most widely used automatic metrics for machine translation evaluation, but they measure translation quality in different ways. Understanding these differences is essential when comparing translation systems and interpreting evaluation results.

BLEU

BLEU (Bilingual Evaluation Understudy) measures translation quality by comparing n-gram overlap between machine-generated translations and one or more human reference translations. It is fast, reproducible, and remains one of the most widely reported metrics in machine translation research.

Despite its popularity, BLEU relies primarily on lexical matching and may not accurately reflect semantic similarity or overall translation quality.

COMET

COMET (Crosslingual Optimized Metric for Evaluation of Translation) is a neural evaluation metric trained on human quality judgments. Instead of relying only on word overlap, it evaluates semantic similarity and contextual meaning, resulting in a stronger correlation with human assessments across many translation tasks.

Recent research consistently shows that COMET provides more reliable rankings of machine translation systems than traditional lexical metrics such as BLEU.

These findings are also reflected in the Workshop on Machine Translation (WMT) Metrics Shared Tasks, the leading annual benchmark for machine translation evaluation, where COMET has consistently ranked among the strongest automatic evaluation metrics in terms of correlation with human judgments.

Why Metric Differences Can Be Misleading

Even measurable improvements in BLEU or COMET should be interpreted carefully until statistical analysis confirms that they are unlikely to have occurred by chance. Small improvements may fall within the natural variability of the evaluation process and disappear when tested on a different sample of translations.

For this reason, metric scores should be interpreted together with statistical significance tests, which determine whether the observed differences are likely to represent genuine improvements rather than random variation.

Comparing Translation Systems with COMET

To determine which automatic metric best reflects human judgments, researchers from Microsoft conducted a comprehensive evaluation of machine translation metrics in the study “To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation”.

Professional translators independently evaluated translation quality, and the results were compared with the rankings produced by automatic evaluation metrics. Among all tested metrics, COMET demonstrated the strongest agreement with human judgments and the highest accuracy when ranking translation systems.

As shown in Figure 1, COMET consistently outperformed traditional lexical metrics such as BLEU across multiple evaluation settings. These findings have contributed to COMET becoming one of the most widely adopted metrics for machine translation quality evaluation.

Figure 1. Ranking accuracy of automatic machine translation evaluation metrics compared with human judgments.

Bootstrap Significance Testing

Bootstrap resampling is one of the most widely used methods for statistical significance testing in machine translation evaluation. Instead of relying on a single metric score, it estimates the uncertainty of evaluation results by repeatedly generating new samples from the original test dataset.

In machine translation research, paired bootstrap resampling is one of the most widely used approaches for statistical significance testing. Unlike standard bootstrap sampling, it resamples the same translation segments for both systems simultaneously, preserving their correspondence and enabling reliable pairwise comparisons on a shared test dataset.

This approach allows researchers to calculate confidence intervals and p-values, making it possible to determine whether differences between translation systems are statistically significant rather than the result of random variation.

Bootstrap Sampling

Bootstrap sampling generates multiple new datasets by randomly sampling translation segments from the original evaluation set with replacement. The evaluation metric is recalculated for each bootstrap sample using the resampled segments, allowing the statistical reliability of the overall system-level evaluation to be estimated. Each bootstrap sample contains the same number of sentences as the original dataset, although individual sentences may appear multiple times or not at all.

Repeating this process hundreds or thousands of times produces a distribution of metric scores that forms the basis for statistical significance testing.

Figure 2. Bootstrap resampling procedure for machine translation evaluation.

Confidence Intervals

The distribution of bootstrap scores is used to estimate confidence intervals for each evaluation metric or for the difference between two translation systems. A confidence interval defines the range within which the true value is expected to lie with a specified level of confidence, typically 95%.

If the confidence interval for the difference between two systems includes zero, there is insufficient evidence to conclude that one system performs better than the other.

P-value

The p-value measures how likely it is to observe the obtained difference if the two translation systems actually have the same performance. A small p-value indicates that the observed difference is unlikely to have occurred by chance.

In machine translation research, a threshold of p < 0.05 is commonly used to indicate statistical significance, although stricter thresholds such as p < 0.01 or p < 0.001 may be applied in large-scale benchmark evaluations.

Hypothesis Testing

Statistical significance testing is a form of hypothesis testing, a statistical framework used to determine whether observed differences between machine translation systems reflect genuine performance improvements or are likely to result from random variation.

The process begins with the null hypothesis (H₀), which assumes that there is no real difference between the translation systems being compared. In other words, any observed difference in BLEU, COMET, or another evaluation metric is assumed to result from random variation in the evaluation dataset.

The p-value measures how compatible the observed results are with this assumption. If the p-value falls below a predefined significance threshold (commonly 0.05), researchers reject the null hypothesis and conclude that the difference is statistically significant.

Practical Example of Comparing Two Translation Systems

Consider two machine translation systems evaluated on the same test dataset using the COMET metric.

• System A: COMET = 82.3;
• System B: COMET = 82.8.

At first glance, System B appears to perform better because it achieves a higher COMET score. However, this difference alone does not indicate whether the improvement is statistically significant.

After applying bootstrap significance testing, the resulting p-value is 0.27. Because this value is greater than the commonly used significance threshold of 0.05, the null hypothesis cannot be rejected. As a result, there is insufficient evidence to conclude that System B consistently outperforms System A, and the observed difference may simply reflect random variation in the evaluation dataset.

This example illustrates why automatic metric scores should always be interpreted together with statistical significance tests rather than in isolation.

Practical Implementation

Statistical significance testing does not need to be implemented from scratch. The open-source COMET framework developed by Unbabel includes built-in support for bootstrap resampling, allowing researchers and developers to evaluate whether differences between machine translation systems are statistically significant.

Using the same test dataset, COMET calculates quality scores for multiple translation systems, performs paired bootstrap resampling, and estimates confidence intervals and p-values. This enables statistically robust comparisons by measuring the reliability of observed metric differences.

By combining accurate neural evaluation with statistical significance testing, COMET provides a reliable framework for comparing machine translation systems in both research and production environments.

Best Practices

Reliable machine translation evaluation requires more than comparing metric scores. Following established evaluation practices helps ensure that reported improvements are both meaningful and reproducible.

• Use the same evaluation dataset. Compare all translation systems on an identical test set to eliminate differences caused by varying evaluation data.
• Prefer modern evaluation metrics. Neural metrics such as COMET generally provide a stronger correlation with human judgments than lexical metrics alone, although BLEU remains an important benchmark.
• Verify improvements with statistical significance testing. Small differences in metric scores should be confirmed using methods such as paired bootstrap resampling before drawing conclusions.
• Report uncertainty, not only scores. Confidence intervals and p-values provide essential context for interpreting evaluation results and assessing the reliability of observed differences.
• Ensure reproducible evaluation. Use standardized evaluation tools, fixed test datasets, and consistent evaluation protocols so that results can be independently reproduced and fairly compared.

Together, these practices help produce evaluation results that are not only statistically sound but also easier to interpret, reproduce, and compare across different machine translation systems.

Conclusion

Automatic evaluation metrics such as BLEU and COMET are essential for measuring machine translation quality, but metric scores alone are not sufficient for comparing translation systems. Reliable comparison of machine translation systems requires both accurate evaluation metrics and rigorous statistical analysis.

Statistical significance testing provides the additional evidence needed to determine whether observed differences are meaningful. By combining modern evaluation metrics such as COMET with bootstrap significance testing, researchers and developers can compare machine translation systems more accurately, make better-informed decisions, and report results with greater confidence.

References

1. EMNLP (2020), COMET: A Neural Framework for MT Evaluation.
2. ACL Anthology (2022), COMET-22: Unbabel-IST 2022 Submission for the Metrics Shared Task.
3. ACL Anthology (2024), Are LLMs Breaking MT Metrics? Results of the WMT24 Metrics Shared Task.
4. Arxiv (2025), COMET-poly: Machine Translation Metric Grounded in Other Candidates.
5. Arxiv (2025), Long-context Reference-based MT Quality Estimation.
6. Arxiv (2025), Bridging the Linguistic Divide: A Survey on Leveraging Large Language Models for Machine Translation.

FAQ

• What is statistical significance in machine translation evaluation? Statistical significance indicates whether the observed difference between two machine translation systems is likely to represent a real improvement rather than random variation. It helps determine if higher evaluation scores can be considered reliable.
• Why isn't a higher COMET score always better? A higher COMET score does not automatically mean that one translation system outperforms another. Small score differences may occur because of the specific evaluation dataset and may disappear when tested on different samples. Statistical significance testing is required to verify whether the improvement is meaningful.
• Why is bootstrap resampling used for machine translation evaluation? Bootstrap resampling estimates the uncertainty of evaluation metrics by repeatedly sampling the test dataset and recalculating metric scores. It is widely used to calculate confidence intervals and p-values when comparing translation systems.
• Is COMET better than BLEU? BLEU remains a widely used benchmark for machine translation evaluation, but recent research shows that COMET generally correlates more closely with human judgments. For this reason, COMET is often preferred for evaluating and comparing modern machine translation systems.
• Which statistical significance test is commonly used for machine translation? Bootstrap resampling is one of the most widely adopted statistical significance tests in machine translation research. It is implemented in evaluation frameworks such as COMET and is commonly used to compare the performance of translation systems.

END FAQ