We built an AI assistant that plays Minecraft with you. Start building a house—it figures out what you’re doing and jumps in to help. This assistant *wasn't* trained with RLHF. Instead, it's powered by *assistance games*, a better path forward for building AI assistants. 🧵

RLHF is great but it encourages short-term optimization: trying to solve the user's entire problem in a single response. For example, if you ask ChatGPT to "clean up some disk space," it will immediately give you a program to run without asking which files are okay to delete!

A better assistant would maintain *uncertainty* about its goal and ask clarification questions until it really understood, leading to a better solution. Assistance games can enable this.

Unlike RLHF, assistance games explicitly treat the user-assistant interaction as a two player game, where the user knows their goal but the assistant doesn't. AGs model *communication* about the goal from the user to the assistant and *collaboration* between them to achieve it.

In Minecraft, we use an assistance game formulation where a simulated human is given random houses to build, and an AI assistants learns via RL to help the human out. The assistant can't see the goal house, so it has to predict the goal and maintain uncertainty to be helpful.

Our new RL algorithm, AssistanceZero, trains an assistant that displays emergent helpful behaviors like *active learning* and *learning from corrections*.

Real human users rate our AssistanceZero assistant much higher than one trained via a pretraining+SFT pipeline! And, it enables people to build houses while placing fewer blocks than building alone.

We conclude our paper with a vision of how AssistanceZero could be applied to post-training of LLMs. We think that our approach could remove incentives for deception and other unsafe behavior in LLMs and make them more helpful. We may or may not be already working on this 😉

Check out the full paper for more details. And download our code to play with the assistant! Joint work with @eli_bronstein @timg51237 @dylanfeng_ @LukasBerglund2 @justinsvegliato @ancadianadragan

Stop wasting time following up with leads. Let our AI agents do it for you.

True AGI is gunna come from minecraft isn't it

Lets talk. We also have ai assistants in Minecraft as well. Perhaps we could share some ideas to further both of our projects.