Lunar New Year Google Doodle uses AI, front-facing camera to teach shadow puppetry

The traditional Chinese calendar, which incorporates 12 zodiac animals, leverages moon cycles to mark the new year. In 2019, this date falls on February 5th and begins the Year of the Pig. To celebrate, Tuesday’s Google Doodle incorporates an AI Experiment that novelly uses your device’s front-facing camera to teach shadow puppetry.

Chinese shadow puppetry is an ancient art form for storytelling that uses silhouette figures and music to pass down cultural history. Cut-out paper or hand figures are placed in front of a light source and projected on a translucent screen.

Last year, Google created an interactive art installation that leverages TensorFlow and TPUs to recognize a person’s hand gestures and transform the shadow figures into digital animations of the 12 Chinese zodiac animals.

For the 2019 Lunar Year, Google has ported that experience into a browser-based game that teaches users how to contort hands into shadow puppets. Shadow Art uses TensorFlow.js to run the machine learning models directly in the browser on any phone or laptop that features a front-facing camera.

You use your hands to form one of twelve zodiac animals from the lunar cycle in front of your laptop or phone camera, trying to match your hand to the diagram on the screen. The “shadow” of your hands on the screen then transform into a shadow puppet animal.

Google added a competitive element where users are allotted 20 seconds to form each animal, with the goal of going through the full lunar cycle as fast as possible.

Shadow Art is available in 11 language variants, with the accompanying Google Doodle appearing in the U.S., Canada, Europe, and throughout Asia on Tuesday, February 5th. Users can directly visit the game now before it appears in Google Search.

English, Chinese, Thai, Bahasa Indonesia, Malay, Japanese, Korean, Spanish, and Portuguese