Google Creates AI Agent That Follows Natural Language Gameplay Instructions
-
Google's DeepMind has created SIMA, an AI agent that can interact with and follow instructions in a variety of 3D game environments.
-
SIMA is trained on 9 different open-world games with a focus on cooperation over competition. The goal is an "obedient partner" rather than just a "superhuman opponent."
-
SIMA takes in visual input and provides control outputs to mimic human gameplay without access to internal game data. This makes it generalizable.
-
Training uses human gameplay footage annotated with natural language descriptions of what's happening. Integration with language/vision models helps avoid learning from scratch.
-
SIMA is early research but could eventually enable AI game characters that players can talk to and give instructions in cooperative gameplay.