Lingvanex Tranalator

Translator for

ChatGPT Translator VS Lingvanex which one is a better translator

We’ve seen many articles and rave reviews claiming that ChatGPT’s translation capabilities are on par with DeepL and Google, and sometimes even surpass them. As a company who have been developing our own translation solutions for the past six years, we became curious about how true all of this is and how our solution compares to ChatGPT. Should we be worried about such a strong competitor?

To compare the quality of translation, we prepared test datasets for seven language pairs:

  • English-Spanish
  • English-German
  • English-Russian
  • English-French
  • English-Italian
  • English-Portuguese
  • English-Finnish

Each test dataset contains around 2500 lines and includes sentences of various themes, lengths, styles, and formatting, to eliminate text selection bias for a specific translator.

ChatGPT offers the API version 4 for limited access. At the moment, access is only available to previously created accounts that have already paid for version 3.5. According to reviews, version 4 has made significant advancements in terms of quality compared to version 3.5. We will check that too!

For testing, we will use two metrics: BLEU and COMET.

BLEU — a widely recognized standard for testing the quality of translation. By default, we will use the Sacre Bleu version. This version is used in the MT machine translation conference and various international competitions. In this metric, the comparison of translations is based on the number of n-grams (combinations of words) that follow one another. The goal of the metric is to find the maximum matching combinations between the translation made by a human and the one made by a machine. The comparison begins with clusters of four words. If there are none, it searches for three-word n-grams. If further matches are not found, it can go down to one n-gram. Points are awarded for each sequence of words (tokens) that the program finds. The drawback of the metric is that it does not account for synonyms, and if the translation accurately conveys the thought but with different words, it will show 0.

COMET — A metric designed to solve the problem of comparing synonyms, which metrics based on the symbolic comparison of two strings cannot handle. If the result of the translation is a semantically similar phrase, but described with different words, the metric will indicate comparable results. It should be noted that its result will also depend on the diversity of the language corpus used to construct the comparison classifier. This metric is widely used as an alternative to the BLEU metric.

Prompts we used for ChatGPT translation:

You are TranslateGPT. You translate user messages from English to Italian (Finnish / French / German / Portuguese / Russian / Spanish). You are the most accurate English to X translator in the world.

Below are graphs with test results:

English-Finnish translation

This language pair shows noticeable improvement in the translation quality of ChatGPT 4 compared to version 3.5. According to the COMET metric, ChatGPT4 slightly outperforms Lingvanex.

English-French translation

When it comes to translating into German, the situation is the same as with French. However, Lingvanex’s lag in the COMET metric is minimal.

English-German translation

When it comes to translating into German, the situation is the same as with French. However, Lingvanex’s lag in the COMET metric is minimal.

Let’s compile all the differences in a table. In red font, we’ll indicate where ChatGPT falls short of Lingvanex. In green font, we’ll mark where it surpasses. The data is relevant as of July 31, 2023.

The Lingvanex translation price was calculated based on the cost of a month’s rent for a basic GPU server (150 dollars) + the monthly price for a Lingvanex language model translation (from 100 dollars), and the number of characters that can be translated in a month with this configuration.

Conclusion:

The test results show that while ChatGPT 3.5 is mostly inferior to Lingvanex in translation quality, according to the COMET metric, ChatGPT4 often matches Lingvanex. It’s worth noting that currently, translating large volumes of text with ChatGPT4 is very expensive. In order to perform the tests for this article and translate roughly 20,000 lines through ChatGPT4, a total of $45 was expended. The calculation of the translation cost can be confusing, as it’s difficult to estimate in advance, in tokens, how much you’ll end up paying for the translation.

At the moment, the translation speed of ChatGPT 4 is unstable, most likely depending on the current load on their servers. We had to take 3–4 second breaks between requests. Overall, the translation speed with the test dataset was about 8 words per second. Our solution enables the translation of several thousand words per second, even on weaker servers. In addition, we observed censorship in the translations: if a line contains obscene language, ChatGPT will not proceed to translate the entire sentence.

Therefore, ChatGPT is better used for stylistic translation of small volumes of text without special security requirements. Furthermore, styles and themes can be altered on the go. By carefully selecting prompts, you can achieve improved quality tailored to a specific task, but this requires going through a significant number of prompts.

Lingvanex translation solutions are better used where large volumes of translation are required at a low cost, with safety, speed, and stability.

We’ll admit that for some language pairs, the difference in translation quality may be different, but testing all possible pairs is lengthy and costly.

In general, solutions from ChatGPT and Lingvanex are designed for different purposes and should be chosen depending on the task.


More fascinating reads await

Join the race: Machine translation in the automotive industry

Join the race: Machine translation in the automotive industry

June 04, 2024

How to translate a website?

How to translate a website?

April 30, 2024

Simplified Technical English (STE)

Simplified Technical English (STE)

April 29, 2024

Request a free trial

✓ Valid
* Indicates required field

Your privacy is of utmost importance to us, your data will be used solely for contact purposes

Completed

Your request has been sent successfully

Customize Consent Preferences

We use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.

The cookies that are categorized as "Necessary" are stored on your browser as they are essential for enabling the basic functionalities of the site.

We also use third-party cookies that help us analyze how you use this website, store your preferences, and provide the content and advertisements that are relevant to you. These cookies will only be stored in your browser with your prior consent.

You can choose to enable or disable some or all of these cookies but disabling some of them may affect your browsing experience.

Always Active

Necessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.

No cookies to display.

Always Active

Functional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.

No cookies to display.

Always Active

Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.

No cookies to display.

Always Active

Performance cookies are used to understand and analyze the key performance indexes of the website which helps in delivering a better user experience for the visitors.

No cookies to display.

Always Active

Advertisement cookies are used to provide visitors with customized advertisements based on the pages you visited previously and to analyze the effectiveness of the ad campaigns.

No cookies to display.