Google Photos now allows complex edits that keep Ultra HDR on Pixel

Abner Li | Oct 16 2024 - 2:01 pm PT

Google today detailed how you can now make complex image edits, like with Magic Eraser, in Photos while maintaining Ultra HDR quality on the Pixel 8+.

Previously, anything more complex than crop or rotate would always create a new copy of the image in SDR quality. At issue is how the metadata associated with Ultra HDR (introduced with Android 14) increases the “complexity of image editing operations.”

Specifically, the issue is with the Gain Map (third image below), or “a log-encoded image stored alongside the SDR image that indicates how much to brighten each SDR pixel to produce the target HDR rendition.”

ML-powered editing tools like Magic Eraser, Photo Unblur, Magic Editor, and Portrait Blur were “developed for SDR images, with models expecting SDR images as inputs and producing SDR images as outputs.” Google explains:

“Using Magic Eraser, for example, you can erase a distracting element of an image, applying image inpainting to generate new pixels in the erased region. For an HDR image, if the erased portion was bright, then it’s likely that the matching Gain Map area was bright as well. As with the crop example, now you’d also have to inpaint the Gain Map. Otherwise, you would see an undesirable “ghosting” effect in the HDR rendition — the original Gain Map appearing as an overlay on the edited SDR image.

Google Photos solved this with a new ML model that reconstructs the missing Gain Map regions by predicting a new one and then blending with the original.

“We captured diverse photographs using the HDR+ burst photography pipeline, applying the dynamic range compression algorithms of the current Pixel camera to generate SDR and HDR image pairs. With this data and a dataset of random mask shapes, we trained a lightweight image-to-image translation model to predict the Gain Map given the edited SDR image and a masked version of the original Gain Map. It is even flexible enough to produce a Gain Map given only an SDR image as input — optimizing rendition for HDR displays.”

Core to our technique is predicting the HDR image metadata missing after editing, using an ML model trained on a large dataset of HDR images with complete metadata.

Google trained a reconstruction model on several thousand HDR images. It’s “under 1 MB and runs at interactive frame rates on mobile devices.” These Ultra HDR edits with Google Photos are now available on the Pixel 8 and newer devices.

Original Ultra HDR image vs. post-Magic Editor image that is still Ultra HDR