Google Assistant’s new bilingual support can recognize & switch between two languages

Abner Li | Aug 30 2018 - 10:00 am PT

Google announced bilingual support for Assistant back in February, and at IFA 2018 it’s beginning to roll out the functionality in six languages. Users can speak to Assistant in two default languages with phones and smart speakers able to understand and reply back in either.

Bilingual support reflects the growing number of multilingual households and helps smart speakers become a device accessible to everyone. At MWC 2018, Google framed the functionality as a way to help Assistant become a more international service with users able to essentially set two default languages.

Technologically, this is not an easy task, with Google setting out years ago to solve the problem and introduce this highly requested feature. There are three aspects in bilingual support, starting with identifying multiple languages.

Humans can recognize a language without having to understand it by paying attention to intonation, phonetic registry, and other acoustics. Since 2013, Google has developed LangID models capable of distinguishing “between pairs of languages in over 2000 alternative language pairs.”

The next step is understanding multiple languages, with Assistant determining the language and what is being said by the time a user stops speaking.

This is accomplished with the help of an additional algorithm that ranks the transcription hypotheses provided by each of the two speech recognition systems using the probabilities of the candidate languages produced by LangID, our confidence on the transcription and the user’s preferences (such as favorite artists, for example).

To optimize the experience, Google uses various tricks like switching to a single monolingual speech recognizer if early identification of the language is possible. Contextual signals are also leveraged to arrive at a quick decision.

In a video, Google shows the feature helping young kids interact in whatever language is natural for them, as well as being used by those living abroad. Meanwhile, on phones, the feature can be leveraged by users who speak one language at work and another at home.

Users will have to set the two desired languages from Assistant settings, with the functionality limited to six language pairs today: English, German, French, Spanish, Italian, and Japanese. It is rolling out starting today with a new “Assistant languages” menu under Preferences in Assistant settings.

In the coming months, the multilingual functionality will support more languages. The end goal is to support even more types of users, with Google hoping “to tackle trilingual support next.”