Rapport sur la qualité de la traduction de Lingvanex
February 27, 2024
L’objectif de ce rapport est de présenter la qualité de traduction des modèles linguistiques de Lingvanex selon les deux mesures d’évaluation de la traduction automatique les plus populaires.
Flores est un ensemble de tests de données open source et accessible au public qui a été publié par Facebook Research et qui offre la plus grande couverture de paires de langues.
Description des mesures de qualité
BLEU
BLEU est une métrique automatique basée sur les n-grammes. Elle mesure la précision des n-grammes de la traduction automatique par rapport à la référence, pondérée par une pénalité de concision pour punir les traductions trop courtes. Nous utilisons une implémentation particulière de BLEU, appelée sacreBLEU. Elle génère des scores de corpus, et non des scores de segments.
Références
- Papineni, Kishore, S. Roukos, T. Ward and Wei-Jing Zhu. “Bleu: a Method for Automatic Evaluation of Machine Translation.” ACL (2002).
- Post, Matt. “A Call for Clarity in Reporting BLEU Scores.” WMT (2018).
COMET
COMET (Crosslingual Optimized Metric for Evaluation of Translation) est une mesure d'évaluation automatique de la traduction automatique qui calcule la similarité entre une traduction automatique et une traduction de référence à l'aide d'incorporations de jetons ou de phrases. Contrairement à d'autres mesures, COMET est formé pour prédire différents types de jugements humains sous la forme d'efforts de post-édition, d'évaluation directe ou d'analyse des erreurs de traduction.
Références
- COMET - https://machinetranslate.org/comet
- COMET: High-quality Machine Translation Evaluation - https://unbabel.github.io/COMET/html/index.html#comet-high-quality-machine-translation-evaluation
Paires de langues
Remarque : la taille plus petite des modèles sur le disque dur signifie une consommation plus faible de mémoire GPU, ce qui entraîne une diminution des coûts de déploiement. Une taille de modèle plus petite offre de meilleures performances en termes de temps de traduction. L'utilisation approximative de la mémoire GPU est calculée comme la taille du modèle de disque dur x 1,2
| Paire de langues | Modèles Taille, Mo | Test Données | BLEU Métrique | COMET Métrique |
| Afrikaans - English | 113,91 | Flores 101 | 52,33 | 86,43 |
| English - Afrikaans | 113,91 | Flores 101 | 42,52 | 86,92 |
| Albanian - English | 113,91 | Lingvanex | 55,31 | 86,68 |
| English - Albanian | 113,91 | Lingvanex | 55,84 | 87,88 |
| Amharic - English | 113,91 | Flores 101 | 30,42 | 83,78 |
| English - Amharic | 113,91 | Flores 101 | 14,13 | 87,4 |
| Arabic - English | 190,63 | Flores 101 | 44,83 | 87,65 |
| English - Arabic | 190,63 | Flores 101 | 33,21 | 87,81 |
| Armenian - English | 113,91 | Flores 101 | 37,47 | 86,93 |
| English - Armenian | 113,91 | Flores 101 | 22,09 | 89,32 |
| Armenian - Russian | 190,63 | Flores 200 | 20,97 | 85,06 |
| Russian - Armenian | 190,63 | Flores 200 | 15,71 | 87,00 |
| Azerbaijani - English | 113,91 | Flores 101 | 23,76 | 85,88 |
| English - Azerbaijani | 113,91 | Flores 101 | 17,79 | 84,93 |
| Azerbaijani - Russian | 190,63 | Flores 200 | 16,95 | 85,23 |
| Russian - Azerbaijani | 190,63 | Flores 200 | 13,10 | 83,64 |
| Belarusian - English | 113,91 | Flores 101 | 19,77 | 80,89 |
| English - Belarusian | 113,91 | Flores 101 | 15,54 | 85,16 |
| Bengali - English | 113,91 | Flores 101 | 33,93 | 87,86 |
| English - Bengali | 113,91 | Flores 101 | 20,98 | 86,42 |
| Bosnian - English | 113,91 | Flores 101 | 41,62 | 87,65 |
| English - Bosnian | 113,91 | Flores 101 | 35,57 | 90,61 |
| Bulgarian - English | 184,00 | Flores 101 | 42,16 | 87,45 |
| English - Bulgarian | 184,00 | Flores 101 | 44,1 | 89,99 |
| English - Basque | 113,91 | Flores 101 | 26,52 | 83,88 |
| Catalan - English | 113,91 | Lingvanex | 46,31 | 87,97 |
| English - Catalan | 113,91 | Flores 101 | 45,62 | 87,67 |
| Cebuano - English | 113,91 | Flores 101 | 41.17 | 79,24 |
| English - Cebuano | 113,91 | Flores 101 | 35,46 | 71,8 |
| Chichewa - English | 113,91 | Flores 101 | 22,86 | 69,22 |
| English - Chichewa | 113,91 | Flores 101 | 17,44 | 64,71 |
| Chinese (Simplified) - English | 190,63 | Flores 101 | 27,96 | 84,88 |
| English - Chinese (Simplified) | 190,63 | Flores 101 | 38,47 | 85,47 |
| Chinese (Traditional) - English | 190,63 | Flores 101 | 25,11 | 83,55 |
| English - Chinese (Traditional) | 190,63 | Lingvanex | 30,25 | 85,67 |
| Corsican - English | 113,91 | Lingvanex | 56,71 | 83,19 |
| English - Corsican | 113,91 | Flores 101 | 54,21 | 76,34 |
| Croatian - English | 184,00 | Flores 101 | 37,1 | 86,84 |
| English - Croatian | 184,00 | Flores 101 | 31,03 | 88,9 |
| Czech - English | 184,00 | Flores 101 | 41,83 | 88,33 |
| English - Czech | 184,00 | Flores 101 | 35,96 | 90,77 |
| Danish - English | 184,00 | Flores 101 | 49,75 | 89,88 |
| English - Danish | 184,00 | Flores 101 | 46,29 | 89,46 |
| Dutch - English | 184,00 | Flores 101 | 34,04 | 87,2 |
| English - Dutch | 184,00 | Flores 101 | 29,71 | 88,1 |
| Estonian - English | 113,91 | Flores 101 | 39,11 | 88,79 |
| English - Estonian | 113,91 | Flores 101 | 30,74 | 91,03 |
| Finnish - English | 184,00 | Flores 101 | 36,87 | 89,76 |
| English - Finnish | 184,00 | Flores 101 | 27,06 | 90,96 |
| French - English | 195,05 | Flores 101 | 47,82 | 89,17 |
| English - French | 195,05 | Flores 101 | 53,25 | 88,4 |
| Frisian - English | 113,91 | Lingvanex | 63,19 | 83,68 |
| English - Frisian | 113,91 | Lingvanex | 54,23 | 80,62 |
| Georgian - Russian | 190,63 | Flores 200 | 18,6 | 85,8 |
| Russian - Georgian | 190,63 | Flores 200 | 13,85 | 85,77 |
| English - Georgian | 113,91 | Flores 101 | 16,56 | 87,13 |
| German - English | 190,65 | Flores 101 | 46,54 | 89,07 |
| English - German | 184,02 | Flores 101 | 43,09 | 88,15 |
| Greek - English | 184,02 | Flores 101 | 36,52 | 87,22 |
| English - Greek | 184,02 | Flores 101 | 28,69 | 88,44 |
| English - Gujarati | 113,91 | Flores 101 | 28,00 | 88,61 |
| Gaelic-English | 113,91 | Flores 101 | 25,24 | 71,42 |
| English-Gaelic | 113,91 | Flores 101 | 24,72 | 80,92 |
| Galician - English | 113,91 | Flores 101 | 40,58 | 87,52 |
| English - Galician | 113,91 | Flores 101 | 41,08 | 87,03 |
| English - Hausa | 113,91 | Flores 101 | 28,93 | 80,76 |
| Hawaiian - English | 113,91 | Lingvanex | 45,12 | 71,92 |
| English - Hawaiian | 113,91 | Lingvanex | 64,27 | 78,56 |
| Hebrew - English | 113,91 | Flores 101 | 44,92 | 87,78 |
| English - Hebrew | 113,91 | Flores 101 | 35,36 | 87,8 |
| Hindi - English | 113,91 | Lingvanex | 38,78 | 87,66 |
| English - Hindi | 113,91 | Flores 101 | 38,86 | 78,84 |
| Hungarian - English | 184,00 | Flores 101 | 35,15 | 87,27 |
| English - Hungarian | 184,00 | Flores 101 | 28,48 | 87,44 |
| Igbo - English | 113,91 | Flores 101 | 21,85 | 64,78 |
| English - Igbo | 113,91 | Flores 101 | 19,01 | 71,94 |
| Indonesian - English | 184,00 | Flores 101 | 45,8 | 89,14 |
| English - Indonesian | 184,37 | Flores 101 | 50,51 | 92,04 |
| Italian - English | 184,00 | Flores 101 | 34,74 | 87,89 |
| English - Italian | 184,00 | Flores 101 | 32,57 | 32,57 |
| Irish - English | 113,91 | Flores 101 | 44,15 | 85,29 |
| English - Irish | 113,91 | Flores 101 | 38,86 | 81,81 |
| Japanese - English | 190,63 | Flores 101 | 29,16 | 86,71 |
| English - Japanese | 190,63 | Flores 101 | 35,86 | 89,8 |
| Khmer - English | 113,91 | Flores 101 | 26,12 | 81,08 |
| English - Khmer | 113,91 | Flores 101 | 4,77 | 82,37 |
| Kinyarwanda - English | 113,91 | Flores 101 | 28,36 | 70,85 |
| English - Kinyarwanda | 113,91 | Flores 101 | 25,84 | 66,67 |
| Korean - English | 113,91 | Flores 101 | 29,68 | 86,22 |
| English - Korean | 113,91 | Flores 101 | 32,42 | 88,94 |
| Kurdish - English | 113,91 | Flores 101 | 22,94 | 74,97 |
| English - Kurdish | 113,91 | Flores 101 | 12,85 | 80,84 |
| Kyrgyz - English | 113,91 | Flores 101 | 24,29 | 84,79 |
| English - Kyrgyz | 113,91 | Flores 101 | 16,65 | 88,14 |
| English - Kannada | 113,91 | Flores 101 | 25,33 | 85,00 |
| Kazakh - Russian | 190,63 | Flores 200 | 22,6 | 87,36 |
| Russian - Kazakh | 190,63 | Flores 200 | 19,05 | 89,42 |
| Kyrgyz - Russian | 190,63 | Flores 200 | 16,38 | 86,32 |
| Russian - Kyrgyz | 190,63 | Flores 200 | 13,43 | 87,62 |
| Lao - English | 113,91 | Flores 101 | 31,45 | 84,11 |
| English - Lao | 113,91 | Flores 101 | 59,54 | 83,24 |
| Latin - English | 113,91 | Lingvanex | 24,24 | 74,81 |
| English - Latin | 113,91 | Lingvanex | 14,83 | 77,64 |
| Latvian - English | 113,91 | Flores 101 | 36,74 | 86,88 |
| English - Latvian | 113,91 | Flores 101 | 36,05 | 89,45 |
| Lithuanian - English | 113,91 | Flores 101 | 34,16 | 85,51 |
| English - Lithuanian | 113,91 | Flores 101 | 30,84 | 89,61 |
| Luxembourgish - English | 113,91 | Flores 101 | 38,69 | 74,82 |
| English - Luxembourgish | 113,91 | Flores 101 | 27,28 | 57,29 |
| Malay - English | 113,91 | Flores 101 | 42,73 | 87,6 |
| English - Malay | 113,91 | Flores 101 | 44,07 | 89,38 |
| English - Malayalam | 113,91 | Flores 101 | 23,38 | 88,92 |
| Maltese - English | 113,91 | Flores 200 | 53,11 | 81,83 |
| English - Maltese | 113,91 | Flores 200 | 46,23 | 74,05 |
| Maori - English | 113,91 | Flores 101 | 22,93 | 64,7 |
| English - Maori | 113,91 | Flores 101 | 17,44 | 61,51 |
| Marathi - English | 113,91 | Flores 101 | 37,94 | 87,49 |
| English - Marathi | 113,91 | Flores 101 | 21,42 | 75,73 |
| Mongolian- English | 113,91 | Lingvanex | 31,4 | 85,43 |
| English - Mongolian | 113,91 | Flores 101 | 20,07 | 89,00 |
| Myanmar - English | 113,91 | Flores 101 | 23,13 | 82,43 |
| English - Myanmar | 113,91 | Flores 101 | 55,72 | 87,25 |
| Malagasy - English | 113,91 | Lingvanex | 38,6 | 83,1 |
| Norwegian - English | 184,00 | Flores 101 | 43,74 | 88,27 |
| English - Norwegian | 184,02 | Flores 101 | 35,03 | 89,4 |
| Odia - English | 113,91 | Flores 101 | 32,34 | 86,77 |
| English - Odia | 113,91 | Flores 101 | 21,09 | 81,84 |
| Persian - English | 190,63 | Flores 101 | 38,6 | 87,17 |
| English - Persian | 190,63 | Flores 101 | 26,76 | 87,7 |
| Polish - English | 184,00 | Flores 101 | 30,94 | 85,72 |
| English - Polish | 184,00 | Flores 101 | 24,24 | 89,21 |
| Pushto - English | 113,91 | Flores 101 | 27,05 | 79,68 |
| English - Pushto | 113,91 | Flores 101 | 15,04 | 77,94 |
| Portuguese (Brazil) - English | 184,00 | Lingvanex | 65,11 | 92,95 |
| English - Portuguese (Brazil) | 184,02 | Lingvanex | 62,75 | 93,84 |
| Portuguese - English | 184,02 | Flores 101 | 51,38 | 89,71 |
| English - Portuguese | 184,02 | Flores 101 | 50,93 | 89,27 |
| Romanian - English | 184,00 | Flores 101 | 43,02 | 88,68 |
| English - Romanian | 179,58 | Flores 101 | 41,01 | 88,84 |
| Russian - English | 190,65 | Flores 101 | 37,86 | 86,63 |
| English - Russian | 190,65 | Flores 101 | 34,64 | 89,37 |
| Russian - Belarusian | 190,63 | Lingvanex | 57,35 | 25,78 |
| Serbian - English | 184,00 | Flores 101 | 43,06 | 86,68 |
| English - Serbian | 184,00 | Flores 101 | 36,87 | 88,49 |
| Sesotho - English | 113,91 | Flores 101 | 32,91 | 72,64 |
| English - Sesotho | 113,91 | Flores 101 | 20,36 | 66,18 |
| Sinhala - English | 113,91 | Flores 101 | 36,73 | 87,92 |
| English - Sinhala | 113,91 | Flores 101 | 21,42 | 87,65 |
| Slovak - English | 113,91 | Flores | 38,18 | 86,85 |
| English - Slovak | 113,91 | Flores | 35,75 | 88,34 |
| Slovenian - English | 113,91 | Flores 101 | 34,13 | 86,55 |
| English - Slovenian | 113,91 | Flores 101 | 30,66 | 87,36 |
| Spanish - English | 184,02 | Flores 101 | 29,91 | 86,35 |
| English - Spanish | 184,00 | Flores 101 | 29,43 | 86,31 |
| Sundanese - English | 113,91 | Lingvanex | 17,17 | 74,84 |
| English - Sundanese | 113,91 | Lingvanex | 11,58 | 77,33 |
| Swedish - English | 184,00 | Flores 101 | 51,89 | 90,13 |
| English - Swedish | 184,00 | Flores 101 | 49,39 | 91,24 |
| Tajik - English | 113,91 | Flores 101 | 32,61 | 76,9 |
| English - Tajik | 113,91 | Flores 101 | 24,97 | 76,28 |
| Tajik - Russian | 190,63 | Flores 200 | 23,15 | 79,69 |
| Russian - Tajik | 190,63 | Flores 200 | 19,38 | 75,59 |
| Tamil - English | 113,91 | Flores 101 | 35,17 | 86,48 |
| English - Tamil | 113,91 | Flores 101 | 23,12 | 89,55 |
| Tatar - English | 113,91 | Flores 101 | 29,91 | 79,25 |
| English - Tatar | 113,91 | Lingvanex | 18,69 | 69,09 |
| Telugu-English | 113,91 | Flores 101 | 42,02 | 87,74 |
| English-Telugu | 113,91 | Flores 101 | 28,3 | 87,6 |
| Thai - English | 184,02 | Flores 101 | 31,63 | 87,71 |
| English - Thai | 113,91 | Flores 101 | 63,68 | 88,57 |
| Turkish - English | 184,00 | Flores 101 | 41,85 | 89,63 |
| English - Turkish | 184,00 | Flores 101 | 32,48 | 88,68 |
| Turkmen - English | 113,91 | Flores 101 | 32,68 | 78,45 |
| English - Turkmen | 113,91 | Flores 101 | 22,91 | 67,00 |
| Ukrainian - English | 190,63 | Flores 101 | 41,15 | 86,92 |
| English - Ukrainian | 184,00 | Flores 101 | 30,2 | 86,99 |
| Ukrainian - Russian | 190,63 | Flores 200 | 28,07 | 91,66 |
| Russian - Ukrainian | 190,63 | Flores 200 | 25,62 | 92,16 |
| Urdu - English | 113,91 | Flores 101 | 31,09 | 84,46 |
| English - Urdu | 113,91 | Flores 101 | 24,8 | 81,71 |
| Uyghur - English | 113,91 | Flores 101 | 24,97 | 83,96 |
| English - Uyghur | 113,91 | Flores 101 | 22,66 | 86,21 |
| Uzbek - Russian | 190,63 | Flores 200 | 22,11 | 85,4 |
| Russian - Uzbek | 190,63 | Flores 200 | 16,86 | 87,37 |
| Vietnamese - English | 184,00 | Flores 101 | 38,13 | 87,3 |
| English - Vietnamese | 184,00 | Flores 101 | 43,84 | 86,95 |
| Xhosa - English | 113,91 | Flores 101 | 34,34 | 76,53 |
| English - Xhosa | 113,91 | Flores 101 | 18,9 | 77,05 |
| Yiddish - English | 113,91 | Flores 101 | 41,89 | 76,88 |
| English - Yiddish | 113,91 | Flores 101 | 10,88 | 67,67 |
| Yoruba - English | 113,91 | Flores 101 | 14,06 | 56,35 |
| English - Yoruba | 113,91 | Lingvanex | 3,57 | 55,39 |
| Zulu - English | 113,91 | Flores 101 | 36,83 | 78,00 |
| English - Zulu | 113,91 | Flores 101 | 21,06 | 78,12 |
Demander plus d'informations
Complété
Votre demande a été envoyée avec succès.
