Introducing SIMA: the first generalist AI agent to follow natural-language instructions in a broad range of 3D virtual environments and video games. 🕹️ It can complete tasks similar to a human, and outperforms an agent trained in just one setting. 🧵

We partnered with gaming studios to train SIMA (Scalable Instructable Multiworld Agent) on @NoMansSky, @Teardowngame, @ValheimGame and others. 🎮 These offer a wide range of distinct skills for it to learn, from flying a spaceship to crafting a helmet.

SIMA needs only the images provided by the 3D environment and natural-language instructions given by the user. 🖱️ With mouse and keyboard outputs, it is evaluated across 600 skills, spanning areas like navigation and object interaction - such as "turn left" or "chop down tree." ↓

We found SIMA agents trained on all of our domains significantly outperformed those trained on just one world. 📈 When it faced an unseen environment, it performed nearly as well as the specialized agent - highlighting its ability to generalize to new spaces. ↓

Unlike our previous work, SIMA isn't about achieving high game scores. 🕹️ It's about developing embodied AI agents that can translate abstract language into useful actions. And using video games as sandboxes offer a safe, accessible way of testing them. →

The SIMA research builds towards more general AI that can understand and safely carry out instructions in both virtual and physical settings. Such generalizable systems will make AI-powered technology more helpful and intuitive. Find out more. →

@apples_jimmy

Very cool!

Bro slow down a bit.

"First" you sure about that?