Report: Google preps ‘Jarvis’ AI agent that works in Chrome

At I/O 2024 in May, Google gave two examples of agentive experiences that you’d access through Gemini. Google might be ready to share more about agents that work in Chrome and are powered by Gemini 2.0 this December with Project Jarvis. 

“I think about [agents] as intelligent systems that show reasoning, planning, and memory. Are able to think multiple steps ahead, work across software and systems, all to get something done on your behalf, and, most importantly, under your supervision.”

—Sundar Pichai on AI agents 

According to The Information, Google is “developing artificial intelligence that takes over a person’s web browser to complete tasks such as gathering research, purchasing a product or booking a flight.” 

“Project Jarvis” — in a nod to J.A.R.V.I.S. in Iron Man — would operate in Google Chrome and is a consumer-facing (rather than enterprise) feature to “automate everyday, web-based tasks.” The article doesn’t specify whether this would be for mobile or desktop.

At I/O, Pichai showed off “Gemini and Chrome working together to help you do a number of things to get ready: Organizing, reasoning, synthesizing on your behalf.” That on-stage scenario was generically happening via gemini.google.com with no other UI shown off compared to the previous example happening through Gemini for Android.

Given a command/action, Jarvis works by taking “frequent screenshots of what’s on their computer screen, and interpreting the shots before taking actions like clicking on a button or typing into a text field.” Today’s report says Jarvis “operates relatively slowly because the model needs to think for a few seconds before taking each action.” As such, this is most likely not working on-device yet and still requires the cloud.

Jarvis is said to be powered by Gemini 2.0 and might be previewed “as early as December,” thus confirming another rumor yesterday. After that, Jarvis might be made available to early testers, so a launch does not seem imminent. It makes sense for Google to have a flagship example of something powered by Gemini 2.0. It has done that for past model launches, and Jarvis seems much more tangible.