Ulat sa Kalidad ng Pagsasalin ng Lingvanex

February 27, 2024

Ang layunin ng ulat na ito ay ipakita ang kalidad ng pagsasalin ng mga modelo ng wikang Lingvanex ayon sa dalawang pinakasikat na sukatan ng pagsusuri sa pagsasalin ng makina.

Ang Flores ay isang open-source at available na pampublikong data test set na inilabas ng Facebook Research at may pinakamalaking saklaw ng pares ng wika.

Paglalarawan ng mga sukatan ng kalidad

BLEU

Ang BLEU ay isang awtomatikong sukatan batay sa n-grams. Sinusukat nito ang katumpakan ng n-grams ng output ng pagsasalin ng makina kumpara sa sanggunian, na binibigyang timbang ng maikling parusa upang parusahan ang masyadong maiikling pagsasalin. Gumagamit kami ng partikular na pagpapatupad ng BLEU, na tinatawag na sacreBLEU. Naglalabas ito ng mga marka ng corpus, hindi mga marka ng segment.

Mga sanggunian

Papineni, Kishore, S. Roukos, T. Ward and Wei-Jing Zhu. “Bleu: a Method for Automatic Evaluation of Machine Translation.” ACL (2002).
Post, Matt. “A Call for Clarity in Reporting BLEU Scores.” WMT (2018).

COMET

Ang COMET (Crosslingual Optimized Metric para sa Pagsusuri ng Pagsasalin) ay isang sukatan para sa awtomatikong pagsusuri ng machine translation na kinakalkula ang pagkakapareho sa pagitan ng machine translation output at isang reference na pagsasalin gamit ang token o sentence embeddings. Hindi tulad ng ibang mga sukatan, sinanay ang COMET sa paghula ng iba't ibang uri ng mga paghatol ng tao sa anyo ng pagsisikap pagkatapos ng pag-edit, direktang pagtatasa, o pagsusuri ng error sa pagsasalin.

Mga sanggunian

COMET - https://machinetranslate.org/comet
COMET: High-quality Machine Translation Evaluation - https://unbabel.github.io/COMET/html/index.html#comet-high-quality-machine-translation-evaluation

Mga pares ng wika

Tandaan: Ang mas mababang laki ng mga modelo sa hard drive ay nangangahulugan ng mas mababang pagkonsumo ng GPU memory na humahantong sa mas mababang mga gastos sa pag-deploy. Ang mas mababang laki ng modelo ay may mas mahusay na pagganap sa oras ng pagsasalin. Ang tinatayang paggamit ng GPU memory ay kinakalkula bilang laki ng modelo ng hard drive x 1.2

Pares ng Wika	Mga modelo Sukat, mb	Pagsubok Data	BLEU Sukatan	COMET Sukatan
Afrikaans - English	113,91	Flores 101	52,33	86,43
English - Afrikaans	113,91	Flores 101	42,52	86,92
Albanian - English	113,91	Lingvanex	55,31	86,68
English - Albanian	113,91	Lingvanex	55,84	87,88
Amharic - English	113,91	Flores 101	30,42	83,78
English - Amharic	113,91	Flores 101	14,13	87,4
Arabic - English	190,63	Flores 101	44,83	87,65
English - Arabic	190,63	Flores 101	33,21	87,81
Armenian - English	113,91	Flores 101	37,47	86,93
English - Armenian	113,91	Flores 101	22,09	89,32
Armenian - Russian	190,63	Flores 200	20,97	85,06
Russian - Armenian	190,63	Flores 200	15,71	87,00
Azerbaijani - English	113,91	Flores 101	23,76	85,88
English - Azerbaijani	113,91	Flores 101	17,79	84,93
Azerbaijani - Russian	190,63	Flores 200	16,95	85,23
Russian - Azerbaijani	190,63	Flores 200	13,10	83,64
Belarusian - English	113,91	Flores 101	19,77	80,89
English - Belarusian	113,91	Flores 101	15,54	85,16
Bengali - English	113,91	Flores 101	33,93	87,86
English - Bengali	113,91	Flores 101	20,98	86,42
Bosnian - English	113,91	Flores 101	41,62	87,65
English - Bosnian	113,91	Flores 101	35,57	90,61
Bulgarian - English	184,00	Flores 101	42,16	87,45
English - Bulgarian	184,00	Flores 101	44,1	89,99
English - Basque	113,91	Flores 101	26,52	83,88
Catalan - English	113,91	Lingvanex	46,31	87,97
English - Catalan	113,91	Flores 101	45,62	87,67
Cebuano - English	113,91	Flores 101	41.17	79,24
English - Cebuano	113,91	Flores 101	35,46	71,8
Chichewa - English	113,91	Flores 101	22,86	69,22
English - Chichewa	113,91	Flores 101	17,44	64,71
Chinese (Simplified) - English	190,63	Flores 101	27,96	84,88
English - Chinese (Simplified)	190,63	Flores 101	38,47	85,47
Chinese (Traditional) - English	190,63	Flores 101	25,11	83,55
English - Chinese (Traditional)	190,63	Lingvanex	30,25	85,67
Corsican - English	113,91	Lingvanex	56,71	83,19
English - Corsican	113,91	Flores 101	54,21	76,34
Croatian - English	184,00	Flores 101	37,1	86,84
English - Croatian	184,00	Flores 101	31,03	88,9
Czech - English	184,00	Flores 101	41,83	88,33
English - Czech	184,00	Flores 101	35,96	90,77
Danish - English	184,00	Flores 101	49,75	89,88
English - Danish	184,00	Flores 101	46,29	89,46
Dutch - English	184,00	Flores 101	34,04	87,2
English - Dutch	184,00	Flores 101	29,71	88,1
Estonian - English	113,91	Flores 101	39,11	88,79
English - Estonian	113,91	Flores 101	30,74	91,03
Finnish - English	184,00	Flores 101	36,87	89,76
English - Finnish	184,00	Flores 101	27,06	90,96
French - English	195,05	Flores 101	47,82	89,17
English - French	195,05	Flores 101	53,25	88,4
Frisian - English	113,91	Lingvanex	63,19	83,68
English - Frisian	113,91	Lingvanex	54,23	80,62
Georgian - Russian	190,63	Flores 200	18,6	85,8
Russian - Georgian	190,63	Flores 200	13,85	85,77
English - Georgian	113,91	Flores 101	16,56	87,13
German - English	190,65	Flores 101	46,54	89,07
English - German	184,02	Flores 101	43,09	88,15
Greek - English	184,02	Flores 101	36,52	87,22
English - Greek	184,02	Flores 101	28,69	88,44
English - Gujarati	113,91	Flores 101	28,00	88,61
Gaelic-English	113,91	Flores 101	25,24	71,42
English-Gaelic	113,91	Flores 101	24,72	80,92
Galician - English	113,91	Flores 101	40,58	87,52
English - Galician	113,91	Flores 101	41,08	87,03
English - Hausa	113,91	Flores 101	28,93	80,76
Hawaiian - English	113,91	Lingvanex	45,12	71,92
English - Hawaiian	113,91	Lingvanex	64,27	78,56
Hebrew - English	113,91	Flores 101	44,92	87,78
English - Hebrew	113,91	Flores 101	35,36	87,8
Hindi - English	113,91	Lingvanex	38,78	87,66
English - Hindi	113,91	Flores 101	38,86	78,84
Hungarian - English	184,00	Flores 101	35,15	87,27
English - Hungarian	184,00	Flores 101	28,48	87,44
Igbo - English	113,91	Flores 101	21,85	64,78
English - Igbo	113,91	Flores 101	19,01	71,94
Indonesian - English	184,00	Flores 101	45,8	89,14
English - Indonesian	184,37	Flores 101	50,51	92,04
Italian - English	184,00	Flores 101	34,74	87,89
English - Italian	184,00	Flores 101	32,57	32,57
Irish - English	113,91	Flores 101	44,15	85,29
English - Irish	113,91	Flores 101	38,86	81,81
Japanese - English	190,63	Flores 101	29,16	86,71
English - Japanese	190,63	Flores 101	35,86	89,8
Khmer - English	113,91	Flores 101	26,12	81,08
English - Khmer	113,91	Flores 101	4,77	82,37
Kinyarwanda - English	113,91	Flores 101	28,36	70,85
English - Kinyarwanda	113,91	Flores 101	25,84	66,67
Korean - English	113,91	Flores 101	29,68	86,22
English - Korean	113,91	Flores 101	32,42	88,94
Kurdish - English	113,91	Flores 101	22,94	74,97
English - Kurdish	113,91	Flores 101	12,85	80,84
Kyrgyz - English	113,91	Flores 101	24,29	84,79
English - Kyrgyz	113,91	Flores 101	16,65	88,14
English - Kannada	113,91	Flores 101	25,33	85,00
Kazakh - Russian	190,63	Flores 200	22,6	87,36
Russian - Kazakh	190,63	Flores 200	19,05	89,42
Kyrgyz - Russian	190,63	Flores 200	16,38	86,32
Russian - Kyrgyz	190,63	Flores 200	13,43	87,62
Lao - English	113,91	Flores 101	31,45	84,11
English - Lao	113,91	Flores 101	59,54	83,24
Latin - English	113,91	Lingvanex	24,24	74,81
English - Latin	113,91	Lingvanex	14,83	77,64
Latvian - English	113,91	Flores 101	36,74	86,88
English - Latvian	113,91	Flores 101	36,05	89,45
Lithuanian - English	113,91	Flores 101	34,16	85,51
English - Lithuanian	113,91	Flores 101	30,84	89,61
Luxembourgish - English	113,91	Flores 101	38,69	74,82
English - Luxembourgish	113,91	Flores 101	27,28	57,29
Malay - English	113,91	Flores 101	42,73	87,6
English - Malay	113,91	Flores 101	44,07	89,38
English - Malayalam	113,91	Flores 101	23,38	88,92
Maltese - English	113,91	Flores 200	53,11	81,83
English - Maltese	113,91	Flores 200	46,23	74,05
Maori - English	113,91	Flores 101	22,93	64,7
English - Maori	113,91	Flores 101	17,44	61,51
Marathi - English	113,91	Flores 101	37,94	87,49
English - Marathi	113,91	Flores 101	21,42	75,73
Mongolian- English	113,91	Lingvanex	31,4	85,43
English - Mongolian	113,91	Flores 101	20,07	89,00
Myanmar - English	113,91	Flores 101	23,13	82,43
English - Myanmar	113,91	Flores 101	55,72	87,25
Malagasy - English	113,91	Lingvanex	38,6	83,1
Norwegian - English	184,00	Flores 101	43,74	88,27
English - Norwegian	184,02	Flores 101	35,03	89,4
Odia - English	113,91	Flores 101	32,34	86,77
English - Odia	113,91	Flores 101	21,09	81,84
Persian - English	190,63	Flores 101	38,6	87,17
English - Persian	190,63	Flores 101	26,76	87,7
Polish - English	184,00	Flores 101	30,94	85,72
English - Polish	184,00	Flores 101	24,24	89,21
Pushto - English	113,91	Flores 101	27,05	79,68
English - Pushto	113,91	Flores 101	15,04	77,94
Portuguese (Brazil) - English	184,00	Lingvanex	65,11	92,95
English - Portuguese (Brazil)	184,02	Lingvanex	62,75	93,84
Portuguese - English	184,02	Flores 101	51,38	89,71
English - Portuguese	184,02	Flores 101	50,93	89,27
Romanian - English	184,00	Flores 101	43,02	88,68
English - Romanian	179,58	Flores 101	41,01	88,84
Russian - English	190,65	Flores 101	37,86	86,63
English - Russian	190,65	Flores 101	34,64	89,37
Russian - Belarusian	190,63	Lingvanex	57,35	25,78
Serbian - English	184,00	Flores 101	43,06	86,68
English - Serbian	184,00	Flores 101	36,87	88,49
Sesotho - English	113,91	Flores 101	32,91	72,64
English - Sesotho	113,91	Flores 101	20,36	66,18
Sinhala - English	113,91	Flores 101	36,73	87,92
English - Sinhala	113,91	Flores 101	21,42	87,65
Slovak - English	113,91	Flores	38,18	86,85
English - Slovak	113,91	Flores	35,75	88,34
Slovenian - English	113,91	Flores 101	34,13	86,55
English - Slovenian	113,91	Flores 101	30,66	87,36
Spanish - English	184,02	Flores 101	29,91	86,35
English - Spanish	184,00	Flores 101	29,43	86,31
Sundanese - English	113,91	Lingvanex	17,17	74,84
English - Sundanese	113,91	Lingvanex	11,58	77,33
Swedish - English	184,00	Flores 101	51,89	90,13
English - Swedish	184,00	Flores 101	49,39	91,24
Tajik - English	113,91	Flores 101	32,61	76,9
English - Tajik	113,91	Flores 101	24,97	76,28
Tajik - Russian	190,63	Flores 200	23,15	79,69
Russian - Tajik	190,63	Flores 200	19,38	75,59
Tamil - English	113,91	Flores 101	35,17	86,48
English - Tamil	113,91	Flores 101	23,12	89,55
Tatar - English	113,91	Flores 101	29,91	79,25
English - Tatar	113,91	Lingvanex	18,69	69,09
Telugu-English	113,91	Flores 101	42,02	87,74
English-Telugu	113,91	Flores 101	28,3	87,6
Thai - English	184,02	Flores 101	31,63	87,71
English - Thai	113,91	Flores 101	63,68	88,57
Turkish - English	184,00	Flores 101	41,85	89,63
English - Turkish	184,00	Flores 101	32,48	88,68
Turkmen - English	113,91	Flores 101	32,68	78,45
English - Turkmen	113,91	Flores 101	22,91	67,00
Ukrainian - English	190,63	Flores 101	41,15	86,92
English - Ukrainian	184,00	Flores 101	30,2	86,99
Ukrainian - Russian	190,63	Flores 200	28,07	91,66
Russian - Ukrainian	190,63	Flores 200	25,62	92,16
Urdu - English	113,91	Flores 101	31,09	84,46
English - Urdu	113,91	Flores 101	24,8	81,71
Uyghur - English	113,91	Flores 101	24,97	83,96
English - Uyghur	113,91	Flores 101	22,66	86,21
Uzbek - Russian	190,63	Flores 200	22,11	85,4
Russian - Uzbek	190,63	Flores 200	16,86	87,37
Vietnamese - English	184,00	Flores 101	38,13	87,3
English - Vietnamese	184,00	Flores 101	43,84	86,95
Xhosa - English	113,91	Flores 101	34,34	76,53
English - Xhosa	113,91	Flores 101	18,9	77,05
Yiddish - English	113,91	Flores 101	41,89	76,88
English - Yiddish	113,91	Flores 101	10,88	67,67
Yoruba - English	113,91	Flores 101	14,06	56,35
English - Yoruba	113,91	Lingvanex	3,57	55,39
Zulu - English	113,91	Flores 101	36,83	78,00
English - Zulu	113,91	Flores 101	21,06	78,12

Tagasalin para sa

Ulat sa Kalidad ng Pagsasalin ng Lingvanex

Paglalarawan ng mga sukatan ng kalidad

BLEU

Mga sanggunian

COMET

Mga sanggunian

Mga pares ng wika

Humiling ng karagdagang impormasyon

Tagasalin para sa

Ulat sa Kalidad ng Pagsasalin ng Lingvanex

Paglalarawan ng mga sukatan ng kalidad

BLEU

Mga sanggunian

COMET

Mga sanggunian

Mga pares ng wika

Humiling ng karagdagang impormasyon

Nakumpleto