The goal of this report is to compare translation quality between old and new language models. New models have not only improved quality but performance and memory usage. We used the BLEU metric and primarily Flores 101 test set in the report.
BLEU is the most popular metrics in the world for machine translation evaluation. Flores 101 test set was released by Facebook Research and has the biggest language pair coverage.
QUALITY METRICS DESCRIPTION
BLEU
BLEU is an automatic metric based on n-grams. It measures the precision of n-grams of the machine translation output compared to the reference, weighted by a brevity penalty to punish overly short translations. We use a particular implementation of BLEU, called sacreBLEU. It outputs corpus scores, not segment scores.
References
- Papineni, Kishore, S. Roukos, T. Ward and Wei-Jing Zhu. “Bleu: a Method for Automatic Evaluation of Machine Translation.” ACL (2002).
- Post, Matt. “A Call for Clarity in Reporting BLEU Scores.” WMT (2018).
COMET
COMET (Crosslingual Optimized Metric for Evaluation of Translation) is a metric for automatic evaluation of machine translation that calculates the similarity between a machine translation output and a reference translation using token or sentence embeddings. Unlike other metrics, COMET is trained on predicting different types of human judgments in the form of post-editing effort, direct assessment, or translation error analysis.
References
- COMET - https://machinetranslate.org/comet
- COMET: High-quality Machine Translation Evaluation - https://unbabel.github.io/COMET/html/index.html#comet-high-quality-machine-translation-evaluation
On-premise Private Software Updates
New version - 1.29.0.
Changes in functionality:
- Added support of additional models for Speech Recognizer.
- Improved document translation quality.
New version - 1.28.0.
Changes in functionality:
- Improved translation quality.
- Improved the Slack bot service.
- Updated dependencies.
New version - 1.27.0.
Changes in functionality:
- Improved translation quality.
- Added support for glossary.
- Improved the Slack bot service.
- Improved DOCX, DOC translation quality.
- Improved work with alternative translation variants.
New version - 1.26.0.
Changes in functionality:
- Improved alternative translation variants feature.
- Improved translation quality.
- Added normalizer and denoiser for Speech Recognizer.
LANGUAGE PAIRS
Note: The lower size of models on the hard drive means the lower consumption of GPU memory which leads to decreased deployment costs. Lower model size has better performance in translation time. The approximate usage of GPU memory is calculated as hard drive model size x 1.2
| Language Pair | Current Model's Size, mb | Test Data | Previous Model's BLEU | Current Model's BLEU | Difference | Previous Model's COMET | Current Model's COMET | Difference |
|---|---|---|---|---|---|---|---|---|
| English - Arabic | 190,63 | Flores 101 | 33,21 | 33,40 | +0,19 | 87,81 | 88,27 | +0,46 |
| Greek - English | 184,00 | Lingvanex | 62,93 | 64,45 | +1,52 | 91,86 | 92,39 | +0,53 |
| Lithuanian - English | 113,91 | Flores 101 | 34,06 | 34,96 | +0,90 | 85,74 | 86,24 | +0,50 |
| English - Croatian | 184,00 | Flores 101 | 31,18 | 34,95 | +3,77 | 89,03 | 91,09 | +2,06 |
| Russian - Kazakh | 190,63 | Lingvanex | 38,10 | 38,39 | +0,29 | 92,06 | 92,13 | +0,07 |
| Kazakh - Russian | 190,63 | Flores 200 | 22,75 | 22,79 | +0,04 | 87,59 | 88,12 | +0,53 |
| Catalan - English | 113,91 | Flores 101 | 46,44 | 47,54 | +1,10 | 88,05 | 88,55 | +0,50 |
| Hmong - English | 113,91 | NTREX | 20,36 | 21,30 | +0.94 | 60,89 | 61,95 | +1,06 |
| German - English | 190,65 | NTREX | 38,69 | 40,96 | +2,27 | 87,52 | 87,98 | +0,46 |
| English - Spanish | 184,02 | Lingvanex | 62,82 | 63,04 | +0,22 | 93,44 | 93,50 | +0,06 |
| Nepali - English | 113,91 | Flores 101 | 33,64 | 41,67 | +8,03 | 88,48 | 89,94 | +1,46 |
| Tajik - English | 113,91 | Flores 101 | 32,19 | 33,74 | +1,55 | 76,10 | 77,46 | +1,36 |
| English - Lithuanian | 113,91 | Flores 101 | 30,84 | 31,28 | +0,44 | 89,61 | 90,11 | +0,50 |
| English - Estonian | 113,91 | Flores 101 | 30,93 | 31,48 | +0,55 | 91,09 | 91,64 | +0,55 |
| Ukrainian - English | 184,00 | Flores 101 | 41,15 | 41,54 | +0,39 | 86,92 | 86,98 | +0,06 |
| English - Hebrew | 184,11 | Flores 101 | 35,42 | 36,00 | +0,91 | 87,87 | 88,53 | +0,66 |
| English - Malay | 184,11 | Flores 101 | 44,12 | 44,63 | +0,51 | 89,41 | 89,77 | +0,36 |
| Estonian - English | 113,91 | Flores 101 | 39,19 | 41,07 | +1,88 | 88,81 | 88,33 | +0,52 |
| Japanese - English | 190,63 | Flores 101 | 29,59 | 31,05 | +1,46 | 87,28 | 88,08 | +0,80 |
| English - Ukrainian | 184,00 | Flores 101 | 29,59 | 34,30 | +4,72 | 87,06 | 89,88 | +2,82 |
| French - English | 190,65 | Flores 101 | 48,35 | 48,82 | +0,47 | 89,31 | 89,46 | +0,15 |
| Herbew - English | 184,11 | Flores 101 | 45,01 | 46,31 | +1,30 | 87,82 | 88,82 | +0,50 |
| Albanian - English | 113,91 | Lingvanex | 55,43 | 56,43 | +1,00 | 86,63 | 87,83 | +1,20 |
| English - Hmong | 113,91 | Lingvanex | 42,26 | 60,99 | +18,73 | 75,48 | 77,35 | +1,87 |
