A Google team has suggested leveraging mobile phone data, including images and searches, to build a “bird’s-eye” perspective of users’ life through artificial intelligence technology.
Project Ellman is only one of many ideas Google has to use AI to develop or enhance its products. Google unveiled Gemini, its newest and “most capable” AI model to date, on Wednesday. In certain scenarios, Gemini fared better than OpenAI’s GPT-4. The business intends to licence Gemini to numerous clients via Google Cloud so they can utilise it in their own apps. Gemini’s multimodality—the ability to receive and comprehend information other than text, such as graphics, video, and audio—is one of its most notable qualities.
At a recent internal summit, a product manager for Google Photos introduced Project Ellman with Gemini teams.In order to characterise a user’s images more fully than “just pixels with labels and metadata,” Ellmann may incorporate context through biographies, earlier events, and later photos, according to the presentation. It suggests having the ability to pinpoint a range of experiences, such as college years, Bay Area years, and parent years.
A picture of a young child playing with a dog in the dirt is placed next to a caption that states, “We can’t answer tough questions or tell good stories without a bird’s-eye view of your life.”
According to a presentation slide, “We comb through your photos, looking at their tags and locations to identify a meaningful moment.” “Your overarching story becomes evident when we take a step back and comprehend your life as a whole.”
Large language models, according to the presentation, may infer events like the birth of a user’s child. “This LLM can deduce that Jack is James and Gemma’s first and only child, and that he was born here by using information from higher in the tree.”
A slide with an illustration of a user’s various life “moments” and “chapters” reads, “One of the reasons that an LLM is so powerful for this bird’s-eye approach, is that it’s able to take unstructured context from all different elevations across this tree, and use it to improve how it understands other regions of the tree.”
Another example of identifying a user who had just attended a class reunion was provided by the presenters. In their presentation, the team hinted that it was likely a reunion because it had been exactly ten years since he graduated and there were many faces they hadn’t seen in that time.
With the description, “Imagine opening ChatGPT but it already knows everything about your life,” the developers also showcased “Ellmann Chat.” What questions would you pose to it?
It showed an example of a chat where a user asks, “Do I have a pet?” In response, the user said that indeed, they had a dog that wore a red raincoat. They also provided the name of the dog and the names of the two family members they frequently see with the dog.
One such instance of a person asking when their siblings last visited was in the chat. Another asked for a list of towns that are comparable to their current residence because they are considering moving. Ellmann responded to both.
Other slides included an overview of the user’s eating habits provided by Ellmann. “It looks like you like eating Italian. Along with a picture of a pizza, there are other pictures of pasta dishes. Furthermore, it stated that the user appeared to love trying new foods, as seen by a shot in which it was shown a menu item it wasn’t familiar with. According to the presentation, the system also used the user’s screenshots to determine what things the user was thinking about buying, their interests, and their work and travel schedules. Additionally, it implied that it will be able to identify their preferred apps and websites, using Google Docs, Reddit, and Instagram as examples.
Google Photos has long employed AI to help users search through their images and videos, and we’re enthusiastic about the potential of LLMs to unlock even more beneficial experiences, a Google representative told CNBC. This was a preliminary internal investigation, and should we choose to release new features in the future, we would take the necessary time to make sure they were beneficial to consumers and made with their safety and privacy as our top priorities.
The rush by Big Tech to develop “memories” powered by AI
In the arms race amongst digital titans to produce more customised life memories, Google may benefit from the proposed Project Ellmann.
For years, albums have been created and “memories” have been served by Google Photos and Apple Photos based on photo trends.
Google said in November that it can now use AI to group together comparable photographs and arrange screenshots into easily accessible albums in Google photographs.
In June, Apple revealed that the most recent software update would enable its camera app to identify people, dogs, and cats in images. Users may already search for faces by name and have them sorted by face.
Apple also unveiled the Journal App, which would leverage AI on the iPhone to generate tailored recommendations based on recent images, locations, songs, and exercise routines to encourage users to write passages that recount their experiences and recollections.
However, the difficulties of correctly detecting and showing photos continue to be a challenge for Google, Apple, and other computer behemoths.
Apple and Google, for example, continue not to designate gorillas despite claims from 2015 that the business was mislabeling Black people as gorillas. This year, a New York Times investigation revealed that the Android operating system, developed by Apple and Google, which powers the majority of smartphones worldwide, disabled the option to visually search for primates out of concern that doing so would identify a person as an animal.
For More: News
Over time, companies such as Google, Facebook, and Apple have implemented controls to reduce the appearance of undesirable memories. However, consumers have claimed that these memories occasionally still appear and that minimising them requires navigating through multiple settings.