Google Lens getting big MUM upgrade that lets you take a photo and ask questions about it

Abner Li | Sep 29 2021 - 11:04 am PT

At I/O in May, Google announced Multitask Unified Model (MUM) as a big improvement to Search. MUM is now set to come to Google Lens “early next year.”

After taking a picture in Lens or analyzing an existing image/screenshot, swiping up will continue to show “Visual matches.” There will soon be a new “Add questions” button at the very top of the sheet that opens a very slick UI. These “Related searches” or any entered query let you refine the search.

Google showed off two demos at Search On 2021 today. Both are intended to let you combine visual and text input.

The most impressive example involves taking a picture of a broken bike part and asking, “how do you fix this.” Google will correctly identify what exactly needs to be fixed and how to do so through videos, forum results, and other help articles on the web.

By combining images and text into a single query, we’re making it easier to search visually and express your questions in more natural ways.

Another example involves capturing an image of a style — e.g. on a shirt — and asking for other pieces of clothing (“socks with this pattern”) that share the same design.

MUM in Google Lens will be available in the coming months. The company is actively testing and conducting “rigorous testing and evaluation” of the AI model before launching.

Other Search On 2021 announcements today include: