Experience Google AI in even more ways on Android (2024)

We’re at a once-in-a-generation moment where the latest advancements in AI are reinventing what phones can do. With Google AI at the core of Android’s operating system, the billions of people who use Android can now interact with their devices in entirely new ways.

Today, we’re sharing updates that let you experience Google AI on Android.

Circle to Search can now help students with homework

With Circle to Search built directly into the user experience, you can search anything you see on your phone using a simple gesture — without having to stop what you’re doing or switch to a different app. Since launching at Samsung Unpacked, we’ve added new capabilities to Circle to Search, like full-screen translation, and we’ve expanded availability to more Pixel and Samsung devices.

Starting today, Circle to Search can now help students with homework, giving them a deeper understanding, not just an answer — directly from their phones and tablets. When students circle a prompt they’re stuck on, they’ll get step-by-step instructions to solve a range of physics and math1 word problems without leaving their digital info sheet or syllabus. Later this year, Circle to Search will be able to help solve even more complex problems involving symbolic formulas, diagrams, graphs and more. This is all possible due to our LearnLM effort to enhance our models and products for learning.

Circle to Search is already available on more than 100 million devices today. With plans to bring the experience to more devices, we’re on track to double that by the end of the year.

Gemini will get even better at understanding context to assist you in getting things done

Gemini on Android is a new kind of assistant that uses generative AI to help you be more creative and productive. This experience, which is integrated into Android, is getting even better at understanding the context of what’s on your screen and what app you’re using.

Soon, you’ll be able to bring up Gemini's overlay on top of the app you're in to easily use Gemini in more ways. For example, you can drag and drop generated images into Gmail, Google Messages and other places, or tap “Ask this video” to find specific information in a YouTube video. If you have Gemini Advanced, you’ll also have the option to “Ask this PDF” to quickly get answers without having to scroll through multiple pages. This update will roll out to hundreds of millions of devices over the next few months.

And we’ll continue to improve Gemini to give you more dynamic suggestions related to what’s on your screen.

Full multimodal capabilities coming to Gemini Nano

Android is the first mobile operating system that includes a built-in, on-device foundation model. With Gemini Nano, we’re able to bring experiences to you quickly and keep your information completely private to you. Starting with Pixel later this year, we’ll be introducing our latest model, Gemini Nano with Multimodality. This means your phone will not just be able to process text input but also understand more information in context like sights, sounds and spoken language.

Clearer descriptions with TalkBack

Later this year, Gemini Nano’s multimodal capabilities are coming to TalkBack, helping people who experience blindness or low vision get richer and clearer descriptions of what’s happening in an image. On average, TalkBack users come across 90 unlabeled images per day. This update will help fill in missing information — whether it’s more details about what’s in a photo that family or friends sent or the style and cut of clothes when shopping online. Since Gemini Nano is on-device, these descriptions happen quickly and even work when there's no network connection.

Receive alerts for suspected scams during phone calls

According to a recent report, in a 12-month period, people lost more than $1 trillion to fraud. We’re testing a new feature that uses Gemini Nano to provide real-time alerts during a call if it detects conversation patterns commonly associated with scams. For example, you would receive an alert if a “bank representative” asks you to urgently transfer funds, make a payment with a gift card or requests personal information like card PINs or passwords, which are uncommon bank requests. This protection all happens on-device, so your conversation stays private to you. We’ll share more about this opt-in feature later this year.

More to come on Android

We’re just getting started with how on-device AI can change what your phone can do, and we’ll continue building Google AI into every part of the smartphone experience with Pixel, Samsung and more. If you’re a developer, check out the Android Developers blog to learn how you can build with our latest AI models and tools, like Gemini Nano and Gemini in Android Studio.

You can also subscribe with a .