Gemini Advanced now uses 1.5 Pro as Google details more Extensions, custom ‘Gems’ 

Google announced Gemini 1.5 Pro in February and is now launching it in the paid Gemini Advanced subscription today.

Gemini Advanced with Gemini 1.5 Pro

Gemini 1.5 Pro’s defining feature is a long context window “starting at 1 million tokens.” Since the February unveil, Google has enhanced “code generation, logical reasoning and planning, multi-turn conversation, and audio and image understanding through data and algorithmic improvements. “

Gemini Advanced can now process “multiple large documents, up to 1,500-pages total, or summarize 100 emails.” On the web, you will be able to “upload files via Google Drive or directly from your device” to “get answers and insights about dense documents. On the privacy front, Google says “Gemini keeps your files private to you, and they’re not used to train our models.”

…like figuring out the details of the pet policy in your rental agreement or comparing key arguments of multiple long research papers. 

An upcoming feature is uploading and understanding spreadsheets, as well as other data files, to analyze, find insights, and create custom visualizations and charts. Data analysis (Google Sheets, CSVs, and Excel files supported) will be available in the coming weeks. 

Meanwhile, Gemini 1.5 Pro is better at understanding images:

“…you can snap a photo of a dish at your favorite restaurant and ask for a recipe, or take a picture of a math problem and get step-by-step instructions on how to solve it — all from a single image.”

Another coming “soon” capability is the ability to “handle an hour of video content or codebases with more than 30,000 lines.”

Gemini Advanced using 1.5 Pro is launching today and available in 35+ languages and over 150+ countries/territories.

Gemini Extensions

Meanwhile, Gemini Extensions are expanding with Google Calendar, Tasks, Keep, and what Google calls “Utilities,” like the Clock app in the coming months. For example, you can take a picture of a printed various with multiple upcoming dates and Gemini will create Calendar events for you.

Launching today is the long-awaited YouTube Music extension that lets you search for songs by “mentioning a favorite verse or a featured artist.” 

These join the existing ones for Gmail, Drive and Docs, as well as Google Flights, Hotels, Maps, and YouTube. Extensions are available to free Gemini and Gemini Advanced users.

Gems

In the coming months, Gemini Advanced users (and business customers) will be able to create “Gems,” or “customized versions of Gemini.” Examples include a “gym buddy, sous chef, coding partner or creative writing guide.”

Simply describe what you want your Gem to do and how you want it to respond — like “you’re my running coach, give me a daily running plan and be positive, upbeat and motivating.” Gemini will take those instructions and, with one click, enhance them to create a Gem that meets your specific needs.

All Gemini users will have access to a number of pre-made Gems, like Learning Coach.

Gemini Advanced: Immersive planner

In the coming months, Gemini Advanced on the web is getting an “immersive planner” that can create a custom, timeline-based itinerary. Google says this “new planning experience will go beyond showing a list of suggested activities.”  

If you ask: “My family and I are going to Miami for Labor Day. My son loves art and my husband really wants fresh seafood. Can you pull my flight and hotel info from Gmail and help me plan the weekend?”

Gemini takes into account your flight timing, meal preferences and information about local museums, while also understanding where each stop is located and how long it will take to travel between each activity.

Gemini will take into account your flight information in Gmail, Google Maps recommendations for food and museums near your hotel, and Search for other activities, as well as travel times between stops. It will be presented in a “dynamic UI” with a side-by-side view that lets you edit visually or through chat.

Gemini 1.5 Flash, Gemma 2

On the developer front, Google is introducing 1.5 Flash today as its “fastest and most versatile multimodal AI model.” It has the same 1 million context window and is aimed at use cases where low latency and cost matters the most. It’s a lighter weight model than 1.5 Pro but retains multimodal reasoning capabilities:

This is because it’s been trained by 1.5 Pro through a process called “distillation,” where the most essential knowledge and skills from a larger model are transferred to a smaller, more efficient model.

Example use cases include summarization, chat applications, image/video captioning, data extraction from long documents and tables, and more. Flash joins the three other sizes that span from phones to data centers:

• Gemini Nano: Most efficient model for on-device tasks
• Gemini Pro: Best model for scaling across a wide range of tasks
• Gemini Ultra: Largest and most capable model for highly complex tasks

It’s available as a public preview through the Gemini API in Google AI Studio for 200+ countries and territories, including the EEA, UK, and Switzerland, with Gemini 1.5 Pro seeing similar access today.

Meanwhile, Google is now previewing (waitlist) a 2 million context window for Gemini 1.5 Pro.

Elsewhere, the Gemini API is getting the ability to “call multiple functions at the same time with parallel function calling” and “reason with video content with native video frame extraction.”

Coming soon is a Context Caching capability that can cache frequently used context, or files. 

This is ideal for scenarios like brainstorming content ideas based on your existing work, analyzing complex documents, or providing summaries of research papers and training materials. Context Caching is coming soon to the Gemini API. 

Meanwhile, Google teased Gemma 2 and a 27B parameter version that ” outperforms models twice its size and runs on a single TPUv5e.” It will join the existing 2B and 7B variants

Google also announced its 6th generation TPU called “Trillium.” As the “most performant and most energy-efficient TPU to date,” it touts 4.7X increase in peak compute performance per chip compared to TPU v5e.”