Poročilo o kakovosti prevodov Lingvanex
Cilj tega poročila je prikazati kakovost prevoda jezikovnih modelov Lingvanex glede na dve najbolj priljubljeni metriki ocenjevanja strojnega prevajanja.
Flores je odprtokodni in javno dostopen testni niz podatkov, ki ga je izdal Facebook Research in ima največjo pokritost jezikovnih parov.
Opis metrike kakovosti
BLEU
BLEU je avtomatska metrika, ki temelji na n-gramih. Meri natančnost n-gramov izhoda strojnega prevoda v primerjavi z referenco, ponderirano s kaznijo za kratkost, da kaznuje prekratke prevode. Uporabljamo posebno izvedbo BLEU, imenovano sacreBLEU. Izpiše rezultate korpusa, ne rezultatov segmentov.
Reference
- Papineni, Kishore, S. Roukos, T. Ward and Wei-Jing Zhu. “Bleu: a Method for Automatic Evaluation of Machine Translation.” ACL (2002).
- Post, Matt. “A Call for Clarity in Reporting BLEU Scores.” WMT (2018).
COMET
COMET (Crosslingual Optimized Metric for Evaluation of Translation) je metrika za samodejno vrednotenje strojnega prevajanja, ki izračuna podobnost med izhodom strojnega prevoda in referenčnim prevodom z uporabo vdelav žetonov ali stavkov. Za razliko od drugih meritev je COMET usposobljen za napovedovanje različnih vrst človeških presoj v obliki napora po urejanju, neposredne ocene ali analize napak pri prevodu.
Reference
- COMET - https://machinetranslate.org/comet
- COMET: High-quality Machine Translation Evaluation - https://unbabel.github.io/COMET/html/index.html#comet-high-quality-machine-translation-evaluation
Jezikovni pari
Opomba: Manjša velikost modelov na trdem disku pomeni manjšo porabo pomnilnika GPE, kar vodi do nižjih stroškov uvajanja. Manjša velikost modela ima boljšo zmogljivost v času prevajanja. Približna uporaba pomnilnika GPE je izračunana kot velikost modela trdega diska x 1,2
Jezikovni par | Modelov Velikost, mb | Test podatki | BLEU metrika | COMET metrika |
Afrikaans - English | 113,91 | Flores | 52,33 | 86,43 |
English - Afrikaans | 113,91 | Flores | 42,52 | 86,92 |
Albanian - English | 113,91 | Lingvanex | 56,43 | 87,83 |
English - Albanian | 113,91 | Lingvanex | 57,04 | 89,03 |
Amharic - English | 113,91 | Flores | 30,42 | 83,78 |
English - Amharic | 113,91 | Flores | 14,13 | 87,4 |
Arabic - English | 184,11 | Flores | 45,79 | 87,95 |
English - Arabic | 190,63 | Flores | 33,45 | 88,28 |
Armenian - English | 113,91 | Flores | 37,47 | 86,93 |
English - Armenian | 113,91 | Flores | 22,09 | 89,32 |
Armenian - Russian | 190,63 | Flores | 20,97 | 85,06 |
Russian - Armenian | 190,63 | Flores | 15,71 | 87,00 |
Azerbaijani - English | 113,91 | Flores | 23,76 | 85,88 |
English - Azerbaijani | 113,91 | Flores | 17,79 | 84,93 |
Azerbaijani - Russian | 190,63 | Flores | 16,95 | 85,23 |
Russian - Azerbaijani | 190,63 | Flores | 13,10 | 83,64 |
Basque - English | 113,91 | Flores | 32,34 | 86,40 |
English - Basque | 113,91 | Flores | 21,69 | 86,99 |
Belarusian - English | 113,91 | Flores | 19,77 | 80,89 |
English - Belarusian | 113,91 | Flores | 15,54 | 85,16 |
Bengali - English | 113,91 | Flores | 33,93 | 87,86 |
English - Bengali | 239,56 | Flores | 23,41 | 86,92 |
Bosnian - English | 113,91 | Flores | 41,62 | 87,65 |
English - Bosnian | 113,91 | Flores | 35,57 | 90,61 |
Bulgarian - English | 184,00 | Flores | 44,97 | 88,28 |
English - Bulgarian | 184,00 | Flores | 45,90 | 91,22 |
Catalan - English | 113,91 | Flores | 47,54 | 88,55 |
English - Catalan | 113,91 | Flores | 45,62 | 87,67 |
Cebuano - English | 113,91 | Flores | 41,17 | 79,24 |
English - Cebuano | 113,91 | Flores | 35,46 | 71,80 |
Chichewa - English | 113,91 | Flores | 22,86 | 69,22 |
English - Chichewa | 113,91 | Flores | 17,44 | 64,71 |
Chinese (Simplified) - English | 184,11 | Flores | 29,91 | 86,27 |
English - Chinese (Simplified) | 184,11 | Flores | 42,29 | 87,70 |
Chinese (Traditional) - English | 190,63 | Flores | 30,28 | 86,51 |
English - Chinese (Traditional) | 190,63 | Flores | 36,06 | 88,67 |
Corsican - English | 113,91 | Lingvanex | 59,90 | 83,86 |
English - Corsican | 113,91 | Lingvanex | 54,21 | 76,34 |
Croatian - English | 184,00 | Flores 101 | 39,80 | 88,08 |
English - Croatian | 184,00 | Flores | 34,95 | 91,09 |
Czech - English | 184,00 | Flores | 41,83 | 88,33 |
English - Czech | 184,00 | Flores | 36,53 | 90,98 |
Danish - English | 184,00 | Flores | 51,21 | 90,25 |
English - Danish | 184,02 | Flores | 49,88 | 91,27 |
Dutch - English | 184,00 | Flores | 34,04 | 87,20 |
English - Dutch | 184,00 | Flores | 29,71 | 88,10 |
Esperanto - English | 113,91 | Flores | 41,60 | 87,49 |
English - Esperanto | 113,91 | Flores | 32,02 | 88,92 |
Estonian - English | 113,91 | Flores | 41,07 | 89,33 |
English - Estonian | 113,91 | Flores | 31,48 | 91,64 |
Filipino - English | 113,91 | Flores | 47,41 | 86,85 |
English - Filipino | 113,91 | Flores | 37,75 | 84,61 |
Finnish - English | 184,00 | Flores | 36,87 | 89,76 |
English - Finnish | 184,00 | Flores | 27,06 | 90,96 |
French - English | 190,65 | Flores | 48,82 | 89,46 |
English - French | 190,65 | Flores | 53,25 | 88,40 |
Frisian - English | 113,91 | Lingvanex | 65,06 | 84,99 |
English - Frisian | 113,91 | Lingvanex | 54,82 | 81,09 |
Gaelic-English | 113,91 | Flores | 29,60 | 76,02 |
English-Gaelic | 113,91 | Flores | 24,72 | 80,92 |
Galician - English | 113,91 | Flores | 40,58 | 87,52 |
English - Galician | 113,91 | Flores | 37,71 | 87,03 |
Georgian - Russian | 190,63 | Flores | 18,60 | 85,80 |
Russian - Georgian | 190,63 | Flores | 13,85 | 85,77 |
Georgian - English | 113,91 | Flores | 22,09 | 89,32 |
English - Georgian | 113,91 | Flores | 16,56 | 87,13 |
German - English | 190,65 | Flores | 46,76 | 89,34 |
English - German | 184,02 | Flores | 43,09 | 88,15 |
Greek - English | 184,00 | Flores | 36,19 | 87,38 |
English - Greek | 184,00 | Flores | 29,54 | 89,02 |
Gujarati-English | 113,91 | Flores | 142,09 | 89,68 |
English - Gujarati | 113,91 | Flores | 28,00 | 88,61 |
Haitian Creole - English | 113,91 | Flores | 30,08 | 72,24 |
English - Haitian Creole | 113,91 | Flores | 28,27 | 67,41 |
Hausa - English | 113,91 | Flores | 33,64 | 78,80 |
English - Hausa | 113,91 | Flores | 28,70 | 80,76 |
Hawaiian - English | 113,91 | Lingvanex | 45,12 | 71,92 |
English - Hawaiian | 113,91 | Lingvanex | 64,27 | 78,56 |
Hebrew - English | 184,11 | Flores | 46,31 | 88,32 |
English - Hebrew | 184,11 | Flores | 36,00 | 88,53 |
Hindi - English | 113,91 | Flores | 38,18 | 88,29 |
English - Hindi | 113,91 | Flores | 67,62 | 81,99 |
English - Hmong | 113,91 | Lingvanex | 60,99 | 77,35 |
Hmong - English | 184,00 | Lingvanex | 45,87 | 75,22 |
Hungarian - English | 184,00 | Flores | 38,74 | 88,37 |
English - Hungarian | 184,00 | Flores | 31,40 | 89,97 |
Icelandic - English | 113,91 | Flores | 37,27 | 85,92 |
English - Icelandic | 113,91 | Flores | 31,90 | 86,34 |
Igbo - English | 113,91 | Flores | 26,60 | 68,85 |
English - Igbo | 113,91 | Flores | 19,01 | 71,94 |
Indonesian - English | 184,02 | Flores | 46,65 | 89,60 |
English - Indonesian | 184,02 | Flores | 50,77 | 91,90 |
Italian - English | 184,00 | Flores | 34,74 | 87,89 |
English - Italian | 184,00 | Flores | 32,57 | 88,19 |
Irish - English | 113,91 | Flores | 44,15 | 85,29 |
English - Irish | 113,91 | Flores | 38,86 | 81,81 |
Japanese - English | 190,63 | Flores | 31,05 | 88,08 |
English - Japanese | 190,63 | Flores | 39,62 | 91,56 |
Javanese - English | 113,91 | Flores | 30,29 | 76,20 |
English - Javanese | 113,91 | Flores | 28,79 | 86,59 |
Kannada - English | 113,91 | Flores | 36,01 | 87,38 |
English - Kannada | 113,91 | Flores | 65,83 | 86,33 |
Kazakh - English | 113,91 | Flores | 36,25 | 87,61 |
English - Kazakh | 113,91 | Flores | 26,48 | 90,35 |
Kazakh - Russian | 190,63 | Flores | 22,79 | 88,12 |
Russian - Kazakh | 190,63 | Flores | 19,18 | 89,57 |
Khmer - English | 113,91 | Flores | 33,19 | 85,77 |
English - Khmer | 113,91 | Flores | 4,77 | 82,37 |
Kinyarwanda - English | 113,91 | Flores | 32,98 | 73,73 |
English - Kinyarwanda | 113,91 | Flores | 25,84 | 66,67 |
Korean - English | 113,91 | Flores | 32,85 | 88,09 |
English - Korean | 113,91 | Flores | 33,66 | 89,67 |
Kurdish - English | 113,91 | Flores | 28,81 | 79,90 |
English - Kurdish | 113,91 | Flores | 12,85 | 80,84 |
Kyrgyz - English | 113,91 | Flores | 24,29 | 84,79 |
English - Kyrgyz | 113,91 | Flores | 16,65 | 88,14 |
Kyrgyz - Russian | 190,63 | Flores | 16,38 | 86,32 |
Russian - Kyrgyz | 190,63 | Flores | 13,43 | 87,62 |
Lao - English | 113,91 | Flores | 31,45 | 84,11 |
English - Lao | 113,91 | Flores | 59,54 | 83,24 |
Latin - English | 113,91 | Lingvanex | 24,24 | 74,81 |
English - Latin | 113,91 | Lingvanex | 14,83 | 77,64 |
Latvian - English | 113,91 | Flores | 38,95 | 87,67 |
English - Latvian | 113,91 | Flores | 37,88 | 90,49 |
Lithuanian - English | 113,91 | Flores | 34,96 | 86,24 |
English - Lithuanian | 113,91 | Flores | 31,28 | 90,11 |
Luxembourgish - English | 113,91 | Flores | 45,56 | 80,29 |
English - Luxembourgish | 113,91 | Flores | 27,28 | 57,29 |
Macedonian - English | 113,91 | Flores | 44,33 | 87,43 |
English - Macedonian | 113,91 | Flores | 38,19 | 89,28 |
Malagasy - English | 113,91 | Lingvanex | 38,60 | 83,10 |
English - Malagasy | 113,91 | Lingvanex | 41,74 | 85,25 |
Malay - English | 184,11 | Flores | 46,09 | 88,84 |
English - Malay | 184,11 | Flores | 44,63 | 89,77 |
Malayalam - English | 113,91 | Flores | 39,26 | 88,28 |
English - Malayalam | 113,91 | Flores | 23,19 | 88,92 |
Maltese - English | 113,91 | Flores | 53,11 | 81,83 |
English - Maltese | 113,91 | Flores | 47,57 | 79,52 |
Maori - English | 113,91 | Flores | 29,13 | 69,87 |
English - Maori | 113,91 | Flores | 17,44 | 61,51 |
Marathi - English | 113,91 | Flores | 37,94 | 87,49 |
English - Marathi | 113,91 | Flores | 21,42 | 75,73 |
Mongolian- English | 113,91 | Flores | 31,40 | 85,43 |
English - Mongolian | 113,91 | Flores | 20,07 | 89,00 |
Myanmar - English | 113,91 | Flores | 23,13 | 82,43 |
English - Myanmar | 113,91 | Flores | 55,72 | 87,25 |
Nepali - English | 113,91 | Flores | 41,67 | 89,94 |
English - Nepali | 113,91 | Flores | 64,92 | 83,27 |
Norwegian - English | 184,00 | Flores | 44,20 | 88,78 |
English - Norwegian | 184,02 | Flores | 35,85 | 90,25 |
Odia - English | 113,91 | Flores | 32,34 | 86,77 |
English - Odia | 113,91 | Flores | 21,09 | 81,84 |
Persian - English | 184,02 | Flores | 39,04 | 87,71 |
English - Persian | 184,11 | Flores | 27,24 | 87,82 |
Polish - English | 184,00 | Flores | 30,94 | 85,72 |
English - Polish | 184,00 | Flores | 24,24 | 89,21 |
Portuguese (Brazil) - English | 184,02 | Flores | 51,42 | 89,66 |
English - Portuguese (Brazil) | 184,02 | Flores | 51,67 | 89,40 |
Portuguese - English | 184,02 | Flores | 50,93 | 89,27 |
English - Portuguese | 184,02 | Flores | 51,38 | 89,71 |
Punjabi - English | 113,91 | Flores | 38,15 | 87,74 |
English - Punjabi | 113,91 | Flores | 28,91 | 84,28 |
Pushto - English | 113,91 | Flores | 27,05 | 79,68 |
English - Pushto | 113,91 | Flores | 15,04 | 77,94 |
Romanian - English | 184,00 | Flores | 45,22 | 89,30 |
English - Romanian | 179,58 | Flores | 43,68 | 90,29 |
Russian - English | 190,65 | Flores | 37,86 | 86,63 |
English - Russian | 190,65 | Flores | 34,64 | 89,37 |
Russian - Belarusian | 190,63 | Lingvanex | 57,35 | 95,38 |
Samoan - English | 113,91 | Flores | 27,81 | 68,62 |
English - Samoan | 113,91 | Flores | 30,07 | 67,20 |
Serbian - English | 184,00 | Flores | 43,06 | 86,68 |
English - Serbian | 184,00 | Flores | 36,87 | 88,49 |
Sesotho - English | 113,91 | Flores | 32,91 | 72,64 |
English - Sesotho | 113,91 | Flores | 20,36 | 66,18 |
Shona - English | 113,91 | Flores | 23,05 | 70,04 |
English - Shona | 113,91 | Flores | 13,74 | 62,88 |
Sindhi - English | 113,91 | Flores | 31,09 | 79,66 |
English - Sindhi | 113,91 | Flores | 25,90 | 80,37 |
Sinhala - English | 113,91 | Flores | 36,73 | 87,92 |
English - Sinhala | 113,91 | Flores | 21,42 | 87,65 |
Slovak - English | 184,11 | Flores | 42,32 | 88,33 |
English - Slovak | 184,11 | Flores | 40,08 | 91,01 |
Slovenian - English | 113,91 | Flores | 38,25 | 87,76 |
English - Slovenian | 113,91 | Flores | 35,39 | 90,20 |
Somali - English | 113,91 | Flores | 29,89 | 77,25 |
English - Somali | 113,91 | Flores | 14,56 | 80,85 |
Spanish - English | 184,02 | Flores | 31,48 | 86,92 |
English - Spanish | 184,02 | Flores | 30,17 | 86,72 |
Sundanese - English | 113,91 | Flores | 34,02 | 81,42 |
English - Sundanese | 113,91 | Flores | 20,23 | 79,65 |
Swahili - English | 113,91 | Flores | 44,95 | 84,91 |
English - Swahili | 113,91 | Flores | 41,14 | 85,40 |
Swedish - English | 184,00 | Flores | 51,89 | 90,13 |
English - Swedish | 184,00 | Flores | 49,39 | 91,24 |
Tajik - English | 113,91 | Flores | 33,74 | 77,46 |
English - Tajik | 113,91 | Flores | 24,97 | 76,28 |
Tajik - Russian | 190,63 | Flores | 23,15 | 79,69 |
Russian - Tajik | 190,63 | Flores | 19,38 | 75,59 |
Tamil - English | 113,91 | Flores | 35,17 | 86,48 |
English - Tamil | 113,91 | Flores | 23,12 | 89,55 |
Tatar - English | 113,91 | Flores | 30,37 | 79,43 |
English - Tatar | 113,91 | Flores | 19,01 | 69,26 |
Telugu-English | 113,91 | Flores | 42,02 | 87,74 |
English-Telugu | 113,91 | Flores | 28,30 | 87,60 |
Thai - English | 184,02 | Flores | 31,63 | 87,71 |
English - Thai | 113,91 | Flores | 63,68 | 88,57 |
Turkish - English | 184,00 | Flores | 41,85 | 89,63 |
English - Turkish | 184,00 | Flores | 35,33 | 90,82 |
Turkmen - English | 113,91 | Flores | 37,10 | 80,90 |
English - Turkmen | 113,91 | Flores | 22,91 | 67,00 |
Ukrainian - English | 184,00 | Flores | 41,54 | 86,98 |
English - Ukrainian | 184,00 | Flores | 34,30 | 89,88 |
Ukrainian - Russian | 190,63 | Flores | 28,07 | 91,66 |
Russian - Ukrainian | 190,63 | Flores | 25,62 | 92,16 |
Urdu - English | 113,91 | Flores | 31,09 | 84,46 |
English - Urdu | 113,91 | Flores | 24,80 | 81,71 |
Uyghur - English | 113,91 | Flores | 28,17 | 86,26 |
English - Uyghur | 113,91 | Flores | 22,66 | 86,21 |
Uzbek - English | 113,91 | Flores | 37,42 | 87,64 |
English - Uzbek | 113,91 | Flores | 23,04 | 90,32 |
Uzbek - Russian | 190,63 | Flores | 22,11 | 85,40 |
Russian - Uzbek | 190,63 | Flores | 16,86 | 87,37 |
Vietnamese - English | 184,00 | Flores | 38,65 | 87,53 |
English - Vietnamese | 184,11 | Flores | 46,28 | 89,16 |
Welsh - English | 113,91 | Flores | 61,36 | 89,32 |
English - Welsh | 113,91 | Flores | 57,55 | 88,83 |
Xhosa - English | 113,91 | Flores | 36,65 | 78,10 |
English - Xhosa | 113,91 | Flores | 18,90 | 77,05 |
Yiddish - English | 113,91 | Flores | 41,89 | 76,88 |
English - Yiddish | 113,91 | Flores | 10,88 | 67,67 |
Yoruba - English | 113,91 | Flores | 17,54 | 61,56 |
English - Yoruba | 113,91 | Flores | 3,57 | 55,39 |
Zulu - English | 113,91 | Flores | 36,83 | 78,00 |
English - Zulu | 113,91 | Flores | 21,06 | 78,12 |
Zahtevajte več informacij
Dokončano
Vaša zahteva je bila uspešno poslana.