Google has started rolling out new AI features to Gemini Live that let it “see” your screen or through your smartphone camera and answer questions about either in real-time, Google spokesperson Alex Joseph has confirmed in an email to The Verge. The features come nearly a year after Google first demonstrated the “Project Astra” work that powers them.
Google is rolling out Gemini’s real-time AI video features
Gemini can ‘see’ screens and camera feeds in real-time for some Google One AI Premium subscribers now.
A Reddit user said the feature showed up on their Xiaomi phone, as spotted by 9to5Google. Today, the same user posted the video below demonstrating Gemini’s new screen-reading ability. It’s one of the two features Google said in early March would “start rolling out to Gemini Advanced Subscribers as part of the Google One AI Premium plan” later in the month.
The other Astra capability rolling out now is live video, which lets Gemini interpret the feed from your smartphone camera in real-time and answer questions about it. In the demonstration video below that Google published this month, a person uses the feature to ask Gemini for help deciding on a paint color to use for their freshly-glazed pottery.
Google’s rollout of these features is a fresh example of the company’s big AI assistant lead as Amazon is preparing the limited early access debut of its Alexa Plus upgrade and Apple has delayed its upgraded Siri. Both are supposed to have capabilities similar to the ones Astra is starting to enable now. Meanwhile, Samsung still has Bixby, but Gemini is nevertheless the default assistant on its phones.