1/ RL for improving task success has made LLMs... remarkably capable. But what happens when the model fails? Turns out, training for graceful failure can be surprisingly tricky! Very excited to share our new paper: Training Language Models to Fail Gracefully 🧵show more

Excited to share CrystalReasoner, a reasoning model for crystal... structure generation with LLMs and property-conditioned generation through RL: Website: Paper: Code:show more

Reinforcement learning should be able to improve upon behaviors... seen when training. In practice, RL agents often struggle to generalize to new long-horizon behaviors. Our new paper studies *horizon generalization*, the degree RL algorithms generalize to reaching distant goals. 1/show more

ChatGPT uses your data for training its Large Language... Model (LLM). But now you can ask ChatGPT to NOT use data for training its model. Go to "Settings." Click on "Show" in front of "Data Controls." Turn off "Chat History and Training."show more

My Melody wishes for the whole world to be... origami! 🩷 But when everything turns paper-perfect, she realizes real life might be even sweeter. Find out what happens on the #HelloKittyandFriends YouTube channel:show more

How can agents like LLMs become decision-making partners for... humans? 💬 Excited to share a new paper + suite of envs for 𝘥𝘦𝘤𝘪𝘴𝘪𝘰𝘯-𝘰𝘳𝘪𝘦𝘯𝘵𝘦𝘥 𝘥𝘪𝘢𝘭𝘰𝘨𝘶𝘦𝘴, where agents + humans collab to solve hard everyday problems. [1/n] Site:show more

Models love reward hacking. Maybe we should just let... them! In our new paper, we let models reward hack during RL but we dictate which parameters learn this behavior. When we disable these parameters for deployment, the model keeps its new capabilities but doesn't hack!show more

As a supporter of the open weights ecosystem, we're... proud to be a post-training partner for NVIDIA Nemotron. We post-train Nemotron models for customer use cases, de-risk mainline RL runs on our AC2 platform and training stack, and contribute aggregate workload statistics for inference benchmarking. This is how open models get better, and we're excited to keep working closely with NVIDIA AI.show more

We're very excited to introduce TAPNext: a model that... sets a new state-of-art for Tracking Any Point in videos, by formulating the task as Next Token Prediction. For more, see: 🧵show more

Diffusion models are great, but we can squeeze out... so much more from them. The only problem is that it usually requires extra training or manual representation editing. In our new paper, we show that with the current capabilities of LLMs, it is much simpler than we thought!show more

Excited to share GeoPT: Scaling Physics Simulation via Lifted... Geometric Pre-Training, which received the Best Paper Award at the ICLR 2026 Workshop on Foundation Models for Science. Can we scale neural physics simulation without scaling expensive solver-generated labels? (1/6) (The below results are all predicted by GeoPT.)show more

Most of what I actually need help with, I... never think to tell a model. But why is it on me to remember? Our new paper asks: what if AI could proactively specialize to individuals and the tasks they’re carrying out at this very moment? 🧵show more

Really excited to launch FrontierSWE v2! In one task,... we ask models to build an OpenGL engine that can render a flight simulator game. Excited to share more detailed analysis in the coming days but checkout the blog for more.show more

Wouldn’t it be nice if ChatGPT could find your... missing keys for you? Our latest research from Berkeley AI Research + Google AI suggests that robots can use large language models (LLMs) to find hidden objects faster. 🧵👇show more

🤔 How to fine-tune an Imitation Learning policy (e.g.,... Diffusion Policy, ACT) with RL? As an RL practitioner, I’ve been struggling with this problem for a while. Here’s why it’s tough: 1️⃣ Special designs (usually for multimodal action distributions) in modern IL models make them non-trivial to fine-tune by RL. 2️⃣ Large policy models + RL's poor sample efficiency = a nightmare But finally, we figured out a simple solution that works for any model architecture! 🌟 Check out our #ICLR2025 paper: “Policy Decorator: Model-Agnostic Online Refinement for Large Policy Models”, led by my amazing mentee Xiu Yuan. 🔗 🧵 Read more below!show more

🚀Thrilled to share what we’ve been building at TRI... over the past several months: our first Large Behavior Models (LBMs) are here! I’m proud to have been a core contributor to the multi-task policy learning and post-training efforts. At TRI, we’ve been researching how LBMs can help robots learn faster, better, and more efficiently. The key takeaways: ✅ We built an evaluation pipeline to benchmark LBM performance with real 𝐬𝐭𝐚𝐭𝐢𝐬𝐭𝐢𝐜𝐚𝐥 𝐜𝐨𝐧𝐟𝐢𝐝𝐞𝐧𝐜𝐞 ✅ Pre-training on hundreds of tasks makes models more robust—plus, we can teach new, complex tasks with 80% 𝐥𝐞𝐬𝐬 𝐝𝐚𝐭𝐚 ✅ The bigger and more diverse the pre-training, the better the results Check out our overview video, webpage and paper for more details: ✨ 🌎 📄 We hope this work helps move the field of robotics forward!show more

Why am I working on RL for LLMs, when... I can go work on RL for Olaf... #humanoidssummitshow more

What happens when AI agents put thousands of GPUs... to work improving AI itself? Meet OpenRSI Index. 👇 The goal: measure whether AI research agents can improve on human-designed training recipes. Real model-development tasks: 🔹 Pre-training 🔹 Post-training 🔹 Vision generation ⚡ Up to 20,864 H100-hours per run in the featured examples. Can agents discover better methods? Let’s put them to the test. Great work by OpenRSI Foundation and the research community! 🚀show more

What if you kept asking an LLM to "make... it better"? In some recent work at FAIR, we investigate how we can efficiently use RL to fine-tune LLMs to iteratively self-improve on their previous solutions at inference-time. Training for iterated self-improvement can be costly. The naive approach to training for K self-improvement steps leads to K times the number of rollout steps per episode. We introduce Exploratory Iteration (ExIt), an RL-based automatic curriculum method that bootstraps diverse training distributions of self-improvement tasks by upcycling the LLM's own responses at previous turns as the starting points for both self-improvement and *self-divergence.* In order to decide what task to train on next, the curriculum prioritizes sampling of partial turn histories that led to higher return variance in its GRPO group (a learnability score that comes for free). This automatic curriculum over the bootstrapped task space teaches the model how to perform iterated self-improvement while only ever training the model on single-step self-improvement tasks. We look at ExIt's impact in both single-turn (contest math problems) and multi-turn (BFCLv3 multi-turn tasks), as well as MLE-bench, where the LLM is run in a search scaffold to produce solutions to real Kaggle competitions. Across these eval settings, we find ExIt produces models with greater capacity for inference-time self-improvement compared to GRPO. Notably, ExIt models can self-improve on test tasks for many more steps than the typical solution depth encountered during training, including a 22% improvement in MLE-bench performance compared to GRPO.show more

Ever wondered how training dynamics differ between LLMs 🖋️... and Vision 👁️ models? We explore this and close the gap between VMs and LLMs in our #NeurIPS2024 paper "TrAct: Making First-layer Pre-Activations Trainable". Paper📜 Video🎥show more