Google DeepMind has unveiled 'Genie 4,' a world model that generates virtual spaces and sounds in real time. Announced on October 27, Genie 4 creates environments where scenery and character behavior change based on user interaction. It also generates music, ambient noise, and character dialogue alongside the scenes.

Users can move around the space and control the camera using the keyboard. During navigation, they can also change the weather or add new people and objects using text. Gemini interprets input instructions based on the surrounding context and adjusts the progression to ensure the scene flows seamlessly. Google explained that it has also improved the ability to render object collisions, water flow, and human movement.

The public demonstration featured stained-glass landscapes and scenes of skiing down a snowy mountain. It employs a method where visual conditions, such as the texture of glass or documentary filming techniques, are specified in text, allowing the viewer to examine the space by shifting their viewpoint within that framework. A key feature of this production method is the simultaneous composition of on-screen visuals, user movement, and ambient sounds.

Sound is a new feature added to Genie 4. It generates not only sounds like wind blowing through trees or waves on the beach, but also dialogue where a character speaks in response to user input and surrounding events. Google is researching ways to utilize this for training environments for robots and AI agents, as well as for new interactive content.

Currently, a research preview is being provided to selected testers. We plan to expand the release to Project Genie's Google AI Ultra subscribers and select development partners later. A full release date for general users has not been announced.

See more and sources
https://deepmind.google/blog/genie-4-a-new-generation-of-world-simulation/
https://deepmind.google/models/genie/