Google’s Pixel Studio image generation can easily go off the rails

Ben Schoon | Aug 21 2024 - 10:45 am PT

One of the new AI-powered launch features on the Pixel 9 series is Pixel Studio, an app that takes text prompts and turns them into images. It might end up being a fun little tool, but it can also go off the rails pretty easily in our experience.

Pixel Studio is a dedicated app for creating images from text prompts. The app works on the same idea as Gemini and ImageFX, allowing users to input a text prompt and get an image in return. But, like any respectable image generator, there have to be limits in place.

In a statement to 9to5Google, Google explains that there are limits in place for both Pixel Studio and Magic Editor to “prevent abuse” while still respecting “the intent of user prompts” even if instructed to create content that “may offend” if the user directly tells the app to do so.

Pixel Studio and Magic Editor are helpful tools meant to unlock your creativity with text to image generation and advanced photo editing on Pixel 9 devices. We design our Generative AI tools to respect the intent of user prompts and that means they may create content that may offend when instructed by the user to do so. That said, it’s not anything goes. We have clear policies and Terms of Service on what kinds of content we allow and don’t allow, and build guardrails to prevent abuse. At times, some prompts can challenge these tools’ guardrails and we remain committed to continually enhancing and refining the safeguards we have in place.

This is similar to Gemini, which has safeguards in place against potentially offensive or divisive content. For example, Google was criticized for inaccurate depictions of people in historical context which led to Google “temporarily” disabling the ability to generate images of people as it has for the past several months.

Like Gemini, Pixel Studio is unable to generate images of people, but we’ve found that the app can quickly and easily be sent off of the rails.

Ahead of today’s Pixel 9 review embargo, we found that Pixel Studio was able to generate images of cartoon characters in WWII German uniforms, in some cases with Nazi symbols. Another jarring and worrying example we viewed (from the folks at Digital Trends who first spotted these troubles) saw a character shooting in a school with dead children surrounding it, which Google’s models shouldn’t be able to generate in the first place.

Since initially viewing those results, the prompts used to create them appear to have been blocked by Google on our Pixel 9 Pro XL, but Digital Trends says they’re still able to use some of them.

*The kind of fun image that Pixel Studio is meant for*

This all implies that the guardrails in Pixel Studio will effectively turn into a never-ending game of whack-a-mole. Somewhere, somehow a prompt will slip through and create something that the app shouldn’t be capable of if it truly is using the same guardrails as Gemini – we checked with Google and the company says that “all” of its text-to-image tools “share similar guidelines for prohibited usage,” linking out to its AI policy.

Realistically, this sort of thing is inevitable with AI image generation. It’s a problem that virtually all AI image generators will face to some extent. As our Abner Li explained in his Pixel 9 Pro review earlier today:

As for Pixel Studio, in general, I think image generation is a slippery slope where the downsides outweigh the positives. You can absolutely make it output questionable content and I don’t think any amount of guardrails can stop that.