Google AI details how Pixel 4 is using dual-cameras and dual-pixels for Portrait Mode

Thanks to the second rear camera, Google was able to improve Portrait Mode on the Pixel 4 after relying on a single lens for two generations. The Google AI team today detailed how it’s leveraging both techniques for better depth estimation and bokeh, as well as how to take the best portrait images.

Previous Pixel phones relied on dual-pixel auto-focus to power the depth estimation needed for Portrait Mode. This technique results in a “virtual pair of cameras placed on either side of the main lens’ aperture.”

Dual-pixels work by splitting every pixel in half, such that each half pixel sees a different half of the main lens’ aperture. By reading out each of these half-pixel images separately, you get two slightly different views of the scene.

With this year’s phone, Google is using wide and telephoto cameras that are physically 13 mm apart. That said, Google is still leveraging “useful” dual pixels to capture as much information as possible for the Pixel 4’s Portrait Mode. Arranged perpendicularly, this is particularly helpful for estimating “depth for lines of any orientation.”

Having one input only allows us to predict depth accurately for lines in one direction (horizontal for dual-pixels and vertical for dual-cameras). With both signals, we can recover the depth on lines in all directions.

Google noted how the dual-cameras and dual-pixels approach for depth “only kicks in when the camera is at least 20 cm [7.8 in] from the subject.” That is the minimum focus distance of the telephoto lens.

Meanwhile, Google also improved bokeh to “more closely match that of a professional SLR camera.” A gallery posted today shows how blurred backgrounds are brighter with matching saturation. This is also available on the Pixel 3 and 3a’s rear camera with Google Camera 7.2.

The solution to this problem is to blur the merged raw image produced by HDR+ and then apply tone mapping. In addition to the brighter and more obvious bokeh disks, the background is saturated in the same way as the foreground.