Waymo Content Search can find billions of objects encountered over 20M miles in seconds

As of early 2020, Waymo self-driving vehicles have driven 20 million miles on public roads. That makes for vast amounts of recorded sensor data to improve the autonomous system. Waymo today detailed its Content Search tool and database with billions of encountered road objects.

Waymo cars today classify road features and objects to assign corresponding behavior. All that data is stored and called upon to train machine learning systems. In order to find examples, Waymo previously relied on heuristic methods that looked for an object’s estimated speed and height.

For instance, to locate examples of people riding scooters, we might have looked through our log data for objects of a certain height traveling between 0 and 20 mph.

That approach was too broad, with Waymo now leveraging Content Search. Created with Google Research a year ago, the underlying technology used to recognize objects in Google Photos and Image Search has been applied to index the 20 million miles of driving data. The entire search process takes seconds to complete.

Content Search lookups can be conducted in one of three manners. Similarity search can be fed images already in the database or from the internet to find near-identical objects. This works by converting every object in Waymo’s database into embeddings that can be ranked based on similarity.

There’s also the ability to search by category. For example, road debris can include everything from plastic bags to tire scraps.

This deep level of understanding opens up the ability to perform extraordinarily niche searches on objects that share a particular trait such as the make and model of a car, or even specific breeds of dogs.

Lastly, Content Search can query text that appears in images by leveraging optical character recognition. This allows Waymo to read road signs, emergency vehicles, and other cars and trucks with signage.

In practice, Content Search has allowed Waymo to “exponentially increase the speed and quality of data we send for labeling.”

The ability to accelerate labeling has contributed to many improvements across our system, from detecting school buses with children about to step onto the sidewalk or people riding electric scooters to a cat or a dog crossing a street.