Note: Systran is
here.
Bing is
here.
DeepL is
here
This collection should under no circumstances be taken as any kind of serious benchmark. Not only is it very small, but, as the title indicates, it was originally formed as a collection where Google Translate, specifically, failed; hence, it is inherently unfair to Google Translate. It is just an unsystematic collection of translation problems that seem like they should be easy, but trip up Google Translate and other machine translation programs as of fall 2016. I check these intermittently to see what progress has been made, and update the page accordingly. As you can see by comparing with the earlier versions, the state of the art has improved steadily, but there are still plenty of "easy" sentences that trip up the best publicly available systems.
Earlier versions:
September - December 2016
December 2016
May 2017
August 2018
Note: I don't actually speak French, so I am grateful to Pascal Amsili for his corrections on a number of items.
Updated 6/2/2019.
Hannah Bast approves the translations "Die Uhr funktioniert nicht mehr" or "Die Uhr geht nicht mehr". Christina Behme suggests "Die Uhr ist stehengeblieben". The translations given by GT, Bing, and Systran are characterized as understandable, but no one would ever say them. DeepL's translation is in the past tense, where as the translation given by Hannah Bast is in the present, but either is correct.
If this seems unfair, note that if you turn the sentence around, "Darling, can you close the door", GT correctly give "Chéri, peux-tu fermer la porte?". So GT is capable of detecting the relation between the endearment and the familiar pronoun one way, but not the other.
If you change "sent" to "he sent", then DeepL gets it right, but none of the others do.
However, in a larger context, DeepL also gets it wrong:
Ger: Man sollte groben Unfug bleiben lassen.
Correct translation: One shouldn't engage in gross mischief.
DeepL: You shouldn't do that to gross nonsense. Alternative: You shouldn't do
that.
(Contributed by Christina Behme.)
Marriage counsellor: Your wife says you never buy her flowers.The Stanford parser also gives the wrong parse for this sentence.
Clueless husband: To be honest, I never knew she sold flowers.
--- from "Sad and Useless Humor".
GT:
Fr: Les stars des foulards couture dansent. (That is, the stars of
couture scarves dance.)
Ger: Stars in Couture-Schals tanzen. (Correct.)
Sp: Baile de estrellas en pañuelos
de alta costura. (That is, the dance of
stars in couture scarves -- a noun phrase rather than a sentence.)
It: Le stelle in sciarpe couture ballano. (Verb is in wrong place.)
Bing:
Fr: Étoiles dans la danse de foulards de couture. [Stars in the dance of
couture scarves].
Ger: Sterne in Couture-Schals tanzen. [Correct]
Sp: Estrellas en el baile bufandas de alta costura. (Not sure what this would
mean, but clearly not right, since "dance" is a noun.)
It: Stelle in sciarpe Couture danza. (Singular verb; should be plural).
Systran:
Fr: Les étoiles dans les foulards de couture dansent. [Correct].
Ger: Sterne in Couture-Tüchern tanzen. [Correct]
Sp: Las estrellas en pañuelos de couture bailan. [Correct]
It: Le stelle nelle sciarpe di moda ballano. (Verb is in the wrong place.)
DeepL:
Fr: Les stars des foulards de couture dansent. (Again, the stars of
couture scarves)
Ger: Stars in der Couture tanzen Schals. ("Scarves" has become the object of
"dance").
Sp: Las estrellas de las bufandas de alta costura bailan. (Again, this is the
stars of couture scarves.)
It: Danzano stelle in sciarpe couture. (Verb in the wrong place).
Incidentally, the Stanford Parser gives the wrong parse for this sentence (test 3/16/2019)
(ROOT
(S
(NP
(NP (NNP Stars))
(PP (IN in)
(NP (NN couture))))
(VP (VBZ scarves)
(NP (NN dance)))
(. .)))
Machine translation fails from Chinese found by Yuling Gu, June 20, 2018.