Google, Bing and Babelfish: some preliminary results

In a recent post, I mentioned a study that Chinese translator Ethan Shen is conducting to find out which of three major free machine translation platforms is best.

Yesterday I received the following message from Ethan, about some preliminary results from the study. He also reiterates his invitation to take part in the survey (you can participate in the survey at: Which Engine Translates Best?).

With his permission, I’m reposting here the message Ethan sent me:

Thanks for helping me promote my research project. We’ve reached the half-way point of our research period and I’ve made some quick observations of the data trends we’re seeing so far. I’ve made some recent changes to the survey engine to eliminate brand bias and first-result bias, if you think your readers would be interested in the below early results, I’d love to make one more publicity push to help reach our 10,000 vote goal. I’ll keep you up to date!

1. At the highest level, it appears that survey takers prefer Google Translate's results across the board.
   
   
   • In a few languages (Arabic, Polish, Dutch) the preference is overwhelming with votes for Google doubling its nearest competitor
     
   
   
2. However, once you remove voters that have self defined their fluency in the source or target language as “limited”, the contest becomes closer for some of the heavily trafficked languages
   
   
   • Bing Translator leads in German
   • Babelfish leads in Chinese
   • Google maintains its lead in Spanish, Japanese, and French
     
   
   
3. Observing just the self defined “Limited fluency” voter reveals a strong brand bias. If your fluency in the target translation language is limited, it would stand to reason your ability to assess the quality of the translation is very limited. And yet…
   
   
   • Limited fluency voters choose Google over Bing by 2 to 1
   • They also chose Google over Yahoo Babelfish by 5 to 1
     
   
   
4. As I had guessed in my hypothesis, Systran’s and Microsoft’s hybrid RBMT model performs better on shorter passages
   
   
   • For phrases below 50 characters, Google’s lead in Spanish, Japanese, and French disappear. And Microsoft’s lead in German widens
   • Beyond 50 characters, Google’s relative performance seems to improve across the board.
   • For passages that are only one sentence, the same effect is seen, though to a lesser extent than under 50 characters.
     
   
   
5. After March 4th, we’ve implemented changes to our survey-taking platform to hide the brands and randomize the positions of the results before voting. There has not been enough data collected since then to draw conclusions, but Yahoo Babelfish seems to be receiving the biggest boost, perhaps showing the effects of the recent neglect of that tool.

Ethan Shen