The future of embodied AI revolves around *collaborative* multi agent scenarios that need natural language communication, task delegation, resource sharing, and more ⛏️ Here are MINDcraft and MineCollab, a simulator and benchmark purpose built to enable research in this area!

MINDcraft is special as it has already existed as OSS on GitHub for a while and built up a great community of thousands of users, sparking many unique benchmarks. Today we bring it to the AI/RL/LLM research communities as a scientifically rigorous env

MINDcraft lets users and LLMs interact with embodied envs with a mix of high and low level actions. Tool use + acquisition, goal / open ended learning + more In particular in our benchmark MineCollab we test multi agent collaboration in cooking, crafting, and construction

All our tasks are designed so that you need at least two agents, in fact the more agents the faster you can theoretically finish a task! Despite this, we find that no existing agent right now is able to effectively collaborate, communication coordination is out of reach!

How we can overcome such bottlenecks via data curation, current failure modes, and more in our paper: 📃 🌐 With @isadorcw @KolbyNottingham Ayush Maniar @max_romana @Lianhuiq from @ucsd_cse and @LatitudeGamesAI

Announcing: Our most advanced speech-to-text model goes beyond accuracy to capture the real-world complexity of human conversation and deliver reliable, source-of-truth audio data. Explore Universal-2 updates 👇

@VoidAsuka cool multi-agent setting here!