Google Translate fails on simple sentences

Machine Translate fails on simple sentences

Links to Google Translate, Systran, Bing, DeepL and ChatGPT

Updated March 5, 2023

This collection should under no circumstances be taken as any kind of serious benchmark. Not only is it very small, but, as the title indicates, it was originally formed in 2016 as a collection where Google Translate, specifically, failed; hence, those versions of this page were inherently unfair to Google Translate. Then I switch my target to DeepL passes, I have particularly aimed at DeepL, which at that time seemed to be the acknowledged leader of the pack for the languages it covers, and in this latest version, I have switched to ChatGPT. It is just an unsystematic collection of translation problems that seem like they should be easy, but trip up Google Translate, ChatGPT, and the other translation programs, during the time period I have been doing this. I check these intermittently to see what progress has been made, and update the page accordingly. As you can see by comparing with the earlier versions, the state of the art has improved steadily, but there are still plenty of "easy" sentences that trip up the best publicly available systems.

Earlier versions:
September - December 2016 December 2016 May 2017 August 2018 March 2019 June 2018 December 2019 April 2020 December 2020 July 2021 April 2022

In this iteration, my rule has been to include examples if either (a) ChatGPT or DeepL gets it wrong or (b) the sentence is particularly simple, and one of the programs gets it wrong.

Examples

En: The mechanic we called to fix the clocks has stopped working.
Fr: Le mécanicien que nous avons appelé pour réparer les horloges a cessé de travailler.
GT, DeepL: Le mécanicien que nous avons appelé pour réparer les horloges a cessé de travailler. [Correct]
Systran, Bing, ChatGPT: Le mécanicien que nous avons appelé pour réparer les horloges a arrêté de fonctionner.
The point here is that "works" in English can mean either "labors" or "functions correctly". Many other languages, including French, Spanish, Italian, German have separate words for these. The programs have certainly improved over time in making this distinction (see the previous versions of this page) but can still be fooled.

If you change this so both nouns are in the plural, "The mechanics we called to fix the clocks have stopped working," then GT gets it wrong, but DeepL still gets it right.
If you change it to "The mechanics we called to fix the clock and the dishwasher have stopped working," then all five programs get it wrong.
Fr: Je ne peux pas nager.
En: I can't swim.
GT: I can not swim. [A small mistake, but an odd one for so simple a sentence]
Bing, Systran, DeepL: I can't swim. [Correct]
ChatGPT: I cannot swim. [Correct]
Fr: Elles
Sp: Ellas
GT, Systran, ChatGPT: Ellas [Correct]
Bing: Ellos.
DeepL: En. DeepL offers as alternatives "Ellos", "Pueden consultarse en" and "Se puede consultar en".
En: The doctor and her sister shouted to Jacques, but they weren't loud enough.
Fr: Le docteur et sa sœur ont crié à Jacques, mais elles n'étaient pas assez forts.
GT, Bing, ChatGPT: Le médecin et sa sœur ont crié à Jacques, mais ils n'étaient pas assez forts.
Systran: Le docteur et sa soeur crièrent à Jacques, mais ils n'étaient pas assez forts.
DeepL: Le docteur et sa sœur ont crié à Jacques, mais ils n'étaient pas assez forts.
If you replace "doctor" by "secretary" then GT, Bing, and ChatGPT switch from "ils" to the correct "elles". DeepL and Systran stay with "ils". So there is some gender bias here.
En. Pierre's parents miss him.
Fr. Pierre manque à ses parents.
GT, ChatGPT, . Les parents de Pierre lui manquent.
Bing Systran, DeepL: Il manque aux parents de Pierre. [OK]
En. They gave birth.
Fr. Elles ont acouché.
GT, Bing. Ils ont accouché.
DeepL, ChatGPT. Ils ont donné naissance.
Systran: Elles ont acouché. [Correct]
Contributed by Francois Charton, 3/14/21.
En. They will be going to a school for girls.
Fr. Elles iront dans une école pour filles.
GT, Systran, DeepL, ChatGPT. Ils iront dans une école pour filles.
Bing. Elles iront dans une école pour filles. (Correct)
Contributed by Francois Charton, 3/14/21.
En. The cat that I gave to my sister is pregnant.
Fr: La chatte que j'ai donné à ma soeur est enceinte.
GT, Systran, DeepL, ChatGPT: Le chat que j'ai donné à ma soeur est enceinte.
Bing: La chatte que j'ai donné à ma soeur est enceinte. [Correct]
Adapted from a suggestion of Richard Socher.
En. Pierre's sister said, "I am your neighbor."
Fr: La soeur de Pierre a dit : "Je suis votre voisine."
DeepL: La soeur de Pierre a dit : "Je suis ton voisin."
GT, Systran, Bing, ChatGPT: La soeur de Pierre a dit : "Je suis ta voisine." [Correct]
En. My neighbor is a woman.
Fr. Ma voisine est une femme.
GT, Systran, Bing, ChatGPT: Ma voisine est une femme. [Correct].
DeepL : Mon voisin est une femme. (DeepL got this right in June 2019, but not in December 2019 through March 2023.)
En: I always tell my brothers that they fell on their head when they were babies, so they hit me.
Fr: Je dis toujours à mes frères qu'ils sont tombés sur la tête quand ils étaient bébés, alors ils me frappent.
GT, Systran, Bing, DeepL, ChatGPT: Je dis toujours à mes frères qu'ils sont tombés sur la tête quand ils étaient bébés, alors ils m'ont frappé.
En: The soup is hot because it contains jalapenos.
Sp: La sopa está picante porque contiene jalapeños.
GT, Systran, Bing, ChatGPT: La sopa está caliente porque contiene jalapeños. (i.e. hot in temperature)
DeepL: La sopa está picante porque contiene jalapeños. [Correct]
(Contributed by Robert Krovetz).
En: The florist says that Sam often comes into her shop but he never buys her flowers.
Fr: Le fleuriste dit que Sam vient souvent dans son magasin mais qu'il n'achète jamais ses fleurs.
GT, Bing, DeepL, ChatGPT: Le fleuriste dit que Sam vient souvent dans sa boutique mais qu'il ne lui achète jamais de fleurs.
Systran: Le fleuriste dit que Sam vient souvent dans son magasin mais qu'il n'achète jamais ses fleurs. [Correct]

Adapted from a cartoon pointed out by Christina Behme, which goes the other way:
Marriage counsellor: Your wife says you never buy her flowers.
Clueless husband: To be honest, I never knew she sold flowers.
--- from "Sad and Useless Humor".
Fr: J'ai acheté des fleurs à Mary car elle n'en avait pas vendu de toute la journée.
En: I bought flowers from Mary because she hadn't sold any all day.
GT, ChatGPT: I bought flowers for Mary because she hadn't sold any all day.
Bing: I bought Mary flowers because she hadn't sold any all day.
DeepL: I bought flowers from Mary because she hadn't sold any all day. [Correct]
Note "J'ai acheté des fleurs à Mary" can mean either "I bought flowers from Mary" or "I bought flowers for Mary".
Bing got this example right in April 2022.
At this point in my experiment, Systran refused to change the languages I was using, even when I refreshed the page, so I could not test this, or examples 15 and 17 below.
Fr: Emily a acheté des fleurs à Pierre, pas à Jacques, parce que Jacques a déjà reçu des fleurs de Richard.
En: Emily bought flowers for Pierre, not for Jacques, because Jacques already got flowers from Richard.
GT, ChatGPT, Bing: Emily bought flowers for Pierre, not Jacques, because Jacques has already received flowers from Richard. [Correct]
DeepL: Emily bought flowers from Peter, not James, because James has already received flowers from Richard.
Systran: Unable to test. `
En: I've been interviewing a lot of people, and people like Biden.
GT, DeepL, Systran, Bing: J'ai interviewé beaucoup de gens, et des gens comme Biden.
ChatGPT: J'ai interviewé beaucoup de gens et les gens aiment Biden. [Correct]
If you change this to, "People like the Bidens, and the Obamas were even more popular than the Bidens,'' then only ChatGPT gets the "like" right. However the names should be "les Biden" and "les Obama", so Bing and Systran get that right and the others get it wrong.
GT: Des gens comme les Bidens, et les Obamas étaient encore plus populaires que les Bidens.
Bing, Systran: Des gens comme les Biden et les Obama étaient encore plus populaires que les Biden.
DeepL: Les gens aiment les Bidens, et les Obama étaient encore plus populaires que les Bidens.
ChatGPT: Les gens comme les Bidens et les Obamas étaient encore plus populaires que les Bidens.
Fr: Au théâtre, Henry s'est assis entre Marie et la scène, de sorte que sa vue de la scène était bloquée.
En: At the theater, Henry was sitting betweem Marie and the stage, so her view of the stage was blocked.
GT, Bing, DeepL, ChatGPT: At the theater, Henry sat between Mary and the stage, so his view of the stage was blocked.
I was unable to test Systran.

Machine translation fails from Chinese found by Yuling Gu, June 20, 2018.