A science of post-training can be built on interpretability.... Models know when they reward hack, and we see it, yet training ignores it. We propose Mechanistic Shaping: using the model’s own concepts to write training rules. In vision, DPO & RL, and its effects on monitoring 🧵show more

We cut through the mystery around interpretability in training.... A single, mathematically rigorous framework transforms model internals into training feedback and works across domains, problems, and modalities. We call this *mechanistic shaping*, and we’ve used it to mitigate reward hacking, remove shortcuts in chest x-rays, and reduce sycophancy and harmfulness. We believe this is a big step forward for technical alignment. But it carries a known risk: training against a model’s internals can make it harder to monitor. We explore both the technique and this concern below. 🧵show more

New work on Training Foundation Models with a Single... Algorithm: Paper: Code, Models: This work introduces Unified Training, a framework that unifies pretraining, midtraining, supervised fine-tuning and RL under a single maximum-likelihood objective. Building on MaxRL, it enables static data and RL-generated data to be trained together within the same system. Empirically, Unified Training improves accuracy and compute efficiency, generalizes better to harder reasoning problems, and reduces forgetting. Interestingly, reusing data across training stages can improve performance even when conventional staged training fails. It also produces models that adapt more effectively to new tasks, leading to stronger continual learning. See an excellent thread by Yuda Song.show more

Harness adapters training is now live! We are training... LoRA adapters on LFM2.5 / LFM3 from Liquid AI and the recent Twil-9B model from webAI using RL to make them more efficient and accurate on agentic tasks! You can follow the training going live here:show more

As a supporter of the open weights ecosystem, we're... proud to be a post-training partner for NVIDIA Nemotron. We post-train Nemotron models for customer use cases, de-risk mainline RL runs on our AC2 platform and training stack, and contribute aggregate workload statistics for inference benchmarking. This is how open models get better, and we're excited to keep working closely with NVIDIA AI.show more

Training large models with RL typically runs on a... fixed homogeneous cluster of the latest accelerators for numerical stability and faster throughput. This throws away inference optimization work done for other vendors and generations, limits the capacity you can utilize, and is unnecessarily expensive. So we trained a model on agentic coding tasks using fully asynchronous RL and mixed-precision, mixed-accelerator samplers. With off-policy correction, training stayed stable through convergence, on par with homogeneous samplers, at a lower cost.show more

1/ RL for improving task success has made LLMs... remarkably capable. But what happens when the model fails? Turns out, training for graceful failure can be surprisingly tricky! Very excited to share our new paper: Training Language Models to Fail Gracefully 🧵show more

Models love reward hacking. Maybe we should just let... them! In our new paper, we let models reward hack during RL but we dictate which parameters learn this behavior. When we disable these parameters for deployment, the model keeps its new capabilities but doesn't hack!show more

I don’t know who needs to hear this but….... If you’re using mostly free weights and very few well-built machines in your training, you are FOR SURE costing yourself gains If you’re using a blend of free weights and well-built machines in your training, you’re PROBABLY doing a pretty good job with exercise selection but you may be leaving SOME gains on the table If you’re using mostly well-built machines and very few free weights in your training, you are very likely putting yourself in a position to MAXIMIZE gains I DO NOT MAKE THE RULES!show more

New work: The Value Axis 🎯 How do LLMs... choose which path to take mid-task? We find they internally track the chance of reaching their goal along a linear axis, akin to a value function in RL. We show it modulates confidence in math & coding and can be reshaped with DPO and SFT.show more

We barely made it to training. We had heavy... hail and rain, but don't be sad.show more

Introducing autoresearch for arXiv papers: replicate and experiment on... any arXiv paper using an army of Claude/Codex agents For post-training, your agents can use Tinker to launch concurrent RL runs while you watch your experiment tree grow Try it out now on OpenResearch:show more

We found a new way to get language models... to reason. 🤯 No RL, no training, no verifiers, no prompting. ❌ With better sampling, base models can achieve single-shot reasoning on par with (or better than!) GRPO while avoiding its characteristic loss in generation diversity.show more

What happens when we lose an engine after decision... speed? We go. We’re committed. We can’t stop on the runway. One of the toughest maneuvers in aviation and one we practice ad nauseum in training. I can do it in my sleep. Which is a good thing. Rest assured your crew can do it too. 🔥show more

Models usually know when they’re doing something wrong. Probes... let us “read their minds” to detect offensive cyber intent, reward hacking, strategic deception, and more. If you build or serve models, they’re your first line of defense. Our latest post explains how they work & how to use them - the first of a series of educational posts on applied AI interpretability. 🧵show more

📖 The Ultimate MoE Training Handbook 📖 We spent... 1/2M GPUh to optimize MoE training and wrote the most comprehensive blog about it. With 30k words and 50+ interactive figures it covers everything from profiling to large cluster training.show more

We built an AI model to simulate how a... fruit fly walks, flies and behaves – in partnership with HHMI | Janelia. 🪰 Our computerized insect replicates realistic motion, and can even use its eyes to control its actions. Here’s how we developed it – and what it means for science. 🧵show more

Monfils : « Im training hard on clay to... finish it well in Paris » The training session :show more

ChatGPT uses your data for training its Large Language... Model (LLM). But now you can ask ChatGPT to NOT use data for training its model. Go to "Settings." Click on "Show" in front of "Data Controls." Turn off "Chat History and Training."show more

Can “chicken” make a base model reason? 🐔 Yes!... Why? With the right first tokens, base models can match RL in reasoning. We trace this to learned training data associations and use a simple data edit to make “chicken” elicit reasoning too. blog:show more