Skip to main content

Google AI details how the Pixel 3 captures and selects the Top Shot

Top Shot is one of the many AI-powered camera features Google introduced with the Pixel 3. Google AI is now detailing how the smart feature works and what qualities your phone is looking for when it suggests an alternate frame.

At a high-level, Top Shot saves and analyzes image frames 1.5 seconds before and after you tap the shutter button. Up to 90 images are captured, with the Pixel 3 selecting up to two alternative shots to save in high-resolution.

The shutter frame is processed and saved first. The best alternative shots are saved afterwards. Google’s Visual Core on Pixel 3 is used to process these top alternative shots as HDR+ images with a very small amount of extra latency, and are embedded into the file of the Motion Photo.

Work on Google Clips inspired the Pixel 3 feature, with the company creating a computer vision model to recognize three key attributes associated with the “best moments.”

  1. Functional qualities, like lighting
  2. Objective attributes (are the subject’s eyes open? Are they smiling?)
  3. Subjective qualities, like emotional expressions

Our neural network design detects low-level visual attributes in early layers, like whether the subject is blurry, and then dedicates additional compute and parameters toward more complex objective attributes like whether the subject’s eyes are open, and subjective attributes like whether there is an emotional expression of amusement or surprise.

According to Google, Top Shot prioritizes face analysis, but the company also worked to identify “good moments in which faces are not the primary subject.” It created additional metrics for the overall frame quality score:

  • Subject motion saliency score — the low-resolution optical flow between the current frame and the previous frame is estimated in ISP to determine if there is salient object motion in the scene.
  • Global motion blur score — estimated from the camera motion and the exposure time. The camera motion is calculated from sensor data from the gyroscope and OIS (optical image stabilization).
  • “3A” scores — the status of auto exposure, auto focus, and auto white balance, are also considered.

All the individual scores are used to train a model predicting an overall quality score, which matches the frame preference of human raters, to maximize end-to-end product quality.

During the development process, Google took into consideration what users perceive as the best shot. It collected data from hundreds of volunteers, and asked which frames looked best. Other steps taken include improvements aimed at avoiding blur and handling multiple faces.

More about Pixel 3:


Check out 9to5Google on YouTube for more news:

FTC: We use income earning auto affiliate links. More.

You’re reading 9to5Google — experts who break news about Google and its surrounding ecosystem, day after day. Be sure to check out our homepage for all the latest news, and follow 9to5Google on Twitter, Facebook, and LinkedIn to stay in the loop. Don’t know where to start? Check out our exclusive stories, reviews, how-tos, and subscribe to our YouTube channel

Comments

Author

Avatar for Abner Li Abner Li

Editor-in-chief. Interested in the minutiae of Google and Alphabet. Tips/talk: abner@9to5g.com

Manage push notifications

notification icon
We would like to show you notifications for the latest news and updates.
notification icon
You are subscribed to notifications
notification icon
We would like to show you notifications for the latest news and updates.
notification icon
You are subscribed to notifications