Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

THE BEST visual explainer of how information propagates through a transformer. If you want to have more than intuition about how the Transformer architecture is ruling the LLM world - open-source project explains everything about LLM Transformer Models! - A great resource for anyone looking to gain a deeper... show more

Rohan Paul

150,798 subscribers

106,897 views • 2 years ago •via X (Twitter)

Science & Technology News & Politics Education

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

Wow. This is one of the best interactive sites I’ve seen for learning how LLMs work! 🔥 It starts w/ a clear intro and guides you through every core component: from Embedding, Layer Norm, and Self-Attention to MLPs, Transformer blocks, Softmax, and Output layers. link in 🧵↓

Wow. This is one of the best interactive sites I’ve seen for learning how LLMs work! 🔥 It starts w/ a clear intro and guides you through every core component: from Embedding, Layer Norm, and Self-Attention to MLPs, Transformer blocks, Softmax, and Output layers. link in 🧵↓

Charly Wargnier

52,092 views • 1 year ago

Transformer by hand ✍️ ~ 6 steps walkthrough below Open the hood of a transformer and the parts list is overwhelming: embeddings, positional encoding, attention weighting, self-attention, cross-attention, multi-head attention, layer norm, skip connections, softmax, linear, Nx, shifted right, query, key, value, masking. Which of those actually make the car run? Two of them. Attention weighting and the feed-forward network. Everything else is an enhancement to make it run faster and longer, which is how we got from a car to a truck, and to the word "large" in large language model. So I drew and calculated those two parts entirely by hand. Goal: push five features through one transformer block, filling in every cell yourself. 1. Given Five positions of input features, arriving from the previous block. 2. Attention matrix Let us feed all five features to a query-key module (QK) and read back an attention weight matrix, A. The details of that module are a post of their own. 3. Attention weighting We multiply the input features by A to get the attention weighted features, Z. Still five positions. The effect is to combine features *across positions*, horizontally: X1 becomes X1 + X2, X2 becomes X2 + X3, and so on. 4. First layer Let us feed all five weighted features into the first layer of the FFN. Multiply by the weights and biases. This time the combining happens *across feature dimensions*, vertically, and each feature grows from 3 numbers to 4. Note that every position goes through the same weight matrix. That is what "position-wise" means. 5. ReLU We cross out the negatives. They become zeros. 6. Second layer Let us bring it back down: 4 dimensions to 3. The output feeds the next block, which has a completely separate set of parameters, and the whole thing runs again. You have just calculated a transformer block by hand. ✍️ The takeaway: the two parts are doing two different jobs, and neither one alone is enough. Attention mixes *across positions*, so a feature can see its neighbours. The FFN mixes *across feature dimensions*, so each position can think about itself. Horizontal, then vertical. Then that pattern repeats N times, each block with its own separate set of weights. That is the Nx from the list up top, and that is what makes the transformer run. 💾 Save this post! #AIbyHand #Transformers #DeepLearning

Transformer by hand ✍️ ~ 6 steps walkthrough below Open the hood of a transformer and the parts list is overwhelming: embeddings, positional encoding, attention weighting, self-attention, cross-attention, multi-head attention, layer norm, skip connections, softmax, linear, Nx, shifted right, query, key, value, masking. Which of those actually make the car run? Two of them. Attention weighting and the feed-forward network. Everything else is an enhancement to make it run faster and longer, which is how we got from a car to a truck, and to the word "large" in large language model. So I drew and calculated those two parts entirely by hand. Goal: push five features through one transformer block, filling in every cell yourself. 1. Given Five positions of input features, arriving from the previous block. 2. Attention matrix Let us feed all five features to a query-key module (QK) and read back an attention weight matrix, A. The details of that module are a post of their own. 3. Attention weighting We multiply the input features by A to get the attention weighted features, Z. Still five positions. The effect is to combine features across positions, horizontally: X1 becomes X1 + X2, X2 becomes X2 + X3, and so on. 4. First layer Let us feed all five weighted features into the first layer of the FFN. Multiply by the weights and biases. This time the combining happens across feature dimensions, vertically, and each feature grows from 3 numbers to 4. Note that every position goes through the same weight matrix. That is what "position-wise" means. 5. ReLU We cross out the negatives. They become zeros. 6. Second layer Let us bring it back down: 4 dimensions to 3. The output feeds the next block, which has a completely separate set of parameters, and the whole thing runs again. You have just calculated a transformer block by hand. ✍️ The takeaway: the two parts are doing two different jobs, and neither one alone is enough. Attention mixes across positions, so a feature can see its neighbours. The FFN mixes across feature dimensions, so each position can think about itself. Horizontal, then vertical. Then that pattern repeats N times, each block with its own separate set of weights. That is the Nx from the list up top, and that is what makes the transformer run. 💾 Save this post! #AIbyHand #Transformers #DeepLearning

Tom Yeh

26,089 views • 2 months ago

Part 2/30 of the LLM Series: RoPE (Rotary Position Embedding) How does a transformer know the difference between - "the dog bit the man" and "the man bit the dog"? The words are almost identical, but the meaning changes completely. RoPE encodes position as rotation, allowing transformers to understand relative order through geometry. Read more:

Part 2/30 of the LLM Series: RoPE (Rotary Position Embedding) How does a transformer know the difference between - "the dog bit the man" and "the man bit the dog"? The words are almost identical, but the meaning changes completely. RoPE encodes position as rotation, allowing transformers to understand relative order through geometry. Read more:

TensorTonic

11,766 views • 4 months ago

$Transformer and Mixture of Experts, explained visually! Mixture of Experts (MoE) is a popular architecture that uses different experts to improve Transformer models. Transformer and MoE differ in the decoder block: - Transformer uses a feed-forward network. - MoE uses experts, which are feed-forward networks but smaller compared to those Transformer. During inference, a subset of experts are selected. This makes inference faster in MoE. Also, since the network has multiple decoder layers: - The text passes through different experts across layers. - The chosen experts also differ between tokens. But how does the model decide which experts should be ideal? The router does that. It is a multi-class classifier that produces softmax scores over experts to select the top K experts. The router is trained with the network, and it learns to select the best experts. But it isn't straightforward. There are challenges! Challenge 1) Notice this pattern at the start of training: - Say, the model selects "Expert 2" - This expert gets a bit better - It may get selected again since it's the "best" - It learns more - It gets selected again in the next iteration - It learns more, and so on! This means many experts can go under-trained due to the overselection of a few experts! We solve this in two steps: - Add noise to the feed-forward output of the router so that other experts can get higher logits. - Set all but the top K logits to -infinity. After softmax, these scores become zero. This way, other experts also get the opportunity to train. Challenge 2) Some experts may get exposed to more tokens than others, leading to under-trained experts. We prevent this by limiting the number of tokens an expert can process. If an expert reaches the limit, the token is passed to the next best expert. Overall, MoEs have more parameters to load. But a fraction of them are activated during inference. This leads to faster inference. Mixtral 8x7B and Llama 4 are two popular MoE-based LLMs. Have you used MoEs in production yet? To dive deeper into how MoE inference works in production, we wrote a full article covering token dispatch, grouped expert computation, model-weight memory, multi-GPU communication, expert placement, load imbalance, and performance diagnosis. Read it below.$

Transformer and Mixture of Experts, explained visually! Mixture of Experts (MoE) is a popular architecture that uses different experts to improve Transformer models. Transformer and MoE differ in the decoder block: - Transformer uses a feed-forward network. - MoE uses experts, which are feed-forward networks but smaller compared to those Transformer. During inference, a subset of experts are selected. This makes inference faster in MoE. Also, since the network has multiple decoder layers: - The text passes through different experts across layers. - The chosen experts also differ between tokens. But how does the model decide which experts should be ideal? The router does that. It is a multi-class classifier that produces softmax scores over experts to select the top K experts. The router is trained with the network, and it learns to select the best experts. But it isn't straightforward. There are challenges! Challenge 1) Notice this pattern at the start of training: - Say, the model selects "Expert 2" - This expert gets a bit better - It may get selected again since it's the "best" - It learns more - It gets selected again in the next iteration - It learns more, and so on! This means many experts can go under-trained due to the overselection of a few experts! We solve this in two steps: - Add noise to the feed-forward output of the router so that other experts can get higher logits. - Set all but the top K logits to -infinity. After softmax, these scores become zero. This way, other experts also get the opportunity to train. Challenge 2) Some experts may get exposed to more tokens than others, leading to under-trained experts. We prevent this by limiting the number of tokens an expert can process. If an expert reaches the limit, the token is passed to the next best expert. Overall, MoEs have more parameters to load. But a fraction of them are activated during inference. This leads to faster inference. Mixtral 8x7B and Llama 4 are two popular MoE-based LLMs. Have you used MoEs in production yet? To dive deeper into how MoE inference works in production, we wrote a full article covering token dispatch, grouped expert computation, model-weight memory, multi-GPU communication, expert placement, load imbalance, and performance diagnosis. Read it below.

Daily Dose of Data Science

53,763 views • 3 days ago

🚨 YOUR GPU IS PROBABLY WASTING MORE THAN YOU THINK. vLLM is built to squeeze far more useful work out of your GPU when serving LLMs. Running an LLM at scale isn’t just about having a powerful GPU. The real problem is how efficiently you use its memory and compute. That’s where vLLM comes in. → High-throughput LLM inference and serving → PagedAttention for smarter KV-cache memory management → Continuous batching to keep GPUs busy → Prefix caching + chunked prefill → OpenAI-compatible API out of the box → Supports a huge range of modern LLM architectures → Quantization support for running models more efficiently And the crazy part? You can start an OpenAI-compatible inference server with: `vllm serve ` So your application can talk to your own model almost like it’s talking to OpenAI. The bigger idea: Don’t just buy more GPUs. Make the GPUs you already have work harder. That’s why vLLM has become such a major project in LLM inference. 🔥 #vLLM #AI #LLM #Inference #GPU #MachineLearning #AIInfrastructure #OpenSource #AIAgents #Developers

🚨 YOUR GPU IS PROBABLY WASTING MORE THAN YOU THINK. vLLM is built to squeeze far more useful work out of your GPU when serving LLMs. Running an LLM at scale isn’t just about having a powerful GPU. The real problem is how efficiently you use its memory and compute. That’s where vLLM comes in. → High-throughput LLM inference and serving → PagedAttention for smarter KV-cache memory management → Continuous batching to keep GPUs busy → Prefix caching + chunked prefill → OpenAI-compatible API out of the box → Supports a huge range of modern LLM architectures → Quantization support for running models more efficiently And the crazy part? You can start an OpenAI-compatible inference server with: `vllm serve ` So your application can talk to your own model almost like it’s talking to OpenAI. The bigger idea: Don’t just buy more GPUs. Make the GPUs you already have work harder. That’s why vLLM has become such a major project in LLM inference. 🔥 #vLLM #AI #LLM #Inference #GPU #MachineLearning #AIInfrastructure #OpenSource #AIAgents #Developers

Vikas gupta

14,138 views • 10 days ago

A major question in multimodal modeling is how to leverage strong pre-trained models, such as industry-scale LLMs and VLMs, during training to avoid starting from scratch. This is important as the demand for rich and domain-specific multimodal models continues to increase, while training them from scratch is obviously impractical due to limited data and compute. The so-called “any-to-any” multimodal models can model a large diverse dictionary of modalities. That’s good. Their downside is that their architectures are often not decoder-only, which has limited their performance in practice and prevents them from leveraging strong pre-trained decoder-only models as priors. We are releasing MODUS (ICML Conference '26) a decoder-only any-to-any multimodal model to address some of these questions. A single transformer decoder predicts any modality from any others with no modality-specific heads, losses, or task pipelines. We show efficient adaptation of established models, e.g., BAGEL, to rich any-to-any multimodal modeling. We are releasing 14B to 77B parameter models. All materials are open-source. The download links and demos here 🧵

A major question in multimodal modeling is how to leverage strong pre-trained models, such as industry-scale LLMs and VLMs, during training to avoid starting from scratch. This is important as the demand for rich and domain-specific multimodal models continues to increase, while training them from scratch is obviously impractical due to limited data and compute. The so-called “any-to-any” multimodal models can model a large diverse dictionary of modalities. That’s good. Their downside is that their architectures are often not decoder-only, which has limited their performance in practice and prevents them from leveraging strong pre-trained decoder-only models as priors. We are releasing MODUS (ICML Conference '26) a decoder-only any-to-any multimodal model to address some of these questions. A single transformer decoder predicts any modality from any others with no modality-specific heads, losses, or task pipelines. We show efficient adaptation of established models, e.g., BAGEL, to rich any-to-any multimodal modeling. We are releasing 14B to 77B parameter models. All materials are open-source. The download links and demos here 🧵

Amir Zamir

24,028 views • 2 months ago

🇯🇵 A JAPANESE LAB SAYS IT MATCHED FABLE 5 LEVELS. While the world braced for China to crack the frontier, a Tokyo lab did something stranger. Sakana AI, co-founded by one of the authors of the original Transformer paper, launched Fugu today. It's not a bigger model. It's a conductor. One API that quietly assembles a team of other models, splits the work, verifies it, and hands you one answer. Sakana claims its top tier matches Anthropic's Fable 5 and Mythos.

🇯🇵 A JAPANESE LAB SAYS IT MATCHED FABLE 5 LEVELS. While the world braced for China to crack the frontier, a Tokyo lab did something stranger. Sakana AI, co-founded by one of the authors of the original Transformer paper, launched Fugu today. It's not a bigger model. It's a conductor. One API that quietly assembles a team of other models, splits the work, verifies it, and hands you one answer. Sakana claims its top tier matches Anthropic's Fable 5 and Mythos.

CryptoGoos

27,467 views • 3 months ago

What if aircraft were built like open-source software? Project Caribou is an ambitious open-source effort to build a 200 kg-class heavy-lift hexacopter that anyone can study, improve, and contribute to. The long-term goal is to accelerate the development of larger open-source aircraft and eVTOL platforms. The project targets a ~200 kg maximum takeoff weight with an ~100 kg payload, uses an 18S power architecture, and is built around an ArduPilot-based flight stack. The repository includes CAD models, PCB designs, documentation, and engineering workflows that are evolving in the open. If you are interested in drones, aerospace, robotics, or embedded systems, this is one of the most exciting open-source engineering projects to follow.

What if aircraft were built like open-source software? Project Caribou is an ambitious open-source effort to build a 200 kg-class heavy-lift hexacopter that anyone can study, improve, and contribute to. The long-term goal is to accelerate the development of larger open-source aircraft and eVTOL platforms. The project targets a ~200 kg maximum takeoff weight with an ~100 kg payload, uses an 18S power architecture, and is built around an ArduPilot-based flight stack. The repository includes CAD models, PCB designs, documentation, and engineering workflows that are evolving in the open. If you are interested in drones, aerospace, robotics, or embedded systems, this is one of the most exciting open-source engineering projects to follow.

Praveen Kumar Verma

37,152 views • 1 month ago

Models usually know when they’re doing something wrong. Probes let us “read their minds” to detect offensive cyber intent, reward hacking, strategic deception, and more. If you build or serve models, they’re your first line of defense. Our latest post explains how they work & how to use them - the first of a series of educational posts on applied AI interpretability. 🧵

Models usually know when they’re doing something wrong. Probes let us “read their minds” to detect offensive cyber intent, reward hacking, strategic deception, and more. If you build or serve models, they’re your first line of defense. Our latest post explains how they work & how to use them - the first of a series of educational posts on applied AI interpretability. 🧵

Goodfire

28,034 views • 12 days ago

I’m launching my first open-source project. If you are building an app like sage by buildspace It’s a generative UI framework for AI applications. This is the demo we created at the ai tinkerers hackathon this Saturday. I am looking for users & contributors. Drop a comment below with how you want to use it.

I’m launching my first open-source project. If you are building an app like sage by buildspace It’s a generative UI framework for AI applications. This is the demo we created at the ai tinkerers hackathon this Saturday. I am looking for users & contributors. Drop a comment below with how you want to use it.

Michael Magán

14,634 views • 2 years ago

i just ran Google's brand new Unsloth Gemma4 12B dense GGUF on my RTX 4060 using llama.cpp + CUDA 13.2 21 tokens per second. on a budget consumer GPU. locally. no API. no cloud. no subscription. and the benchmarks are absolutely cooked # first let's talk architecture because this is genuinely different every multimodal model you've used has a frozen vision encoder + frozen audio encoder + LLM backbone glued together Gemma 4 12B is different it's a single decoder only transformer. that's it. vision? raw 48×48 pixel patches → one matmul → projected directly into the LLM audio? raw 16kHz signal sliced into 40ms frames → linear projection → same LLM input space no encoder tax. no latency penalty. no fragmented memory to put the encoder savings in perspective: old Gemma 4 26B approach: - 550M param vision encoder (frozen) - 300M param audio encoder (frozen) - LLM backbone Gemma 4 12B: - 35M param vision embedder (a single matmul) - no audio encoder at all - LLM backbone handles EVERYTHING 550M → 35M for vision alone. that's a 15x reduction this is why the gemma-4-12b-it-Q4_K_M.gguf is just 6.6 GBs!!! and it has 256K native context context # Benchmarks: AIME 2026 (math olympiad): 77.5% GPQA Diamond (expert science): 78.8% LiveCodeBench v6 (real code): 72% Codeforces ELO: 1659 MMLU Pro: 77.2% MATH-Vision: 79.7% BigBench Extra Hard: 53% inference → llama.cpp, LM Studio, vLLM, SGLang llamacpp flags: -m "gemma-4-12b-it-Q4_K_M.gguf" -ngl 99 -c 8000 -v --port 8080 Available on huggingface now! Link below

Alok

281,007 views • 3 months ago

Can industrial AI do more than just observe? Nicolas Cerisier, Vice President of 3DEXPERIENCE Platform R&D at Dassault Systèmes explains how the shift from SaaS to "Agent-as-a-Service" is introducing "Industry World Models" that understand the scientific laws of physics and engineering to power autonomous virtual companions.

Can industrial AI do more than just observe? Nicolas Cerisier, Vice President of 3DEXPERIENCE Platform R&D at Dassault Systèmes explains how the shift from SaaS to "Agent-as-a-Service" is introducing "Industry World Models" that understand the scientific laws of physics and engineering to power autonomous virtual companions.

NVIDIA

25,785 views • 4 months ago

Sharing some more UI previews for Superlogical, this time the "tab peek." If you three-finger swipe down, you can "peek" at the other session tabs. These are live-updating, metal-rendered previews including splits. If you keep dragging, you enter a "mission control" style mode. This is one of those UI patterns that once you get used to, you mindlessly find yourself doing it ALL THE TIME. It's a great way to quickly take a look at everything thats going on. You aren't reading the mini text of course, but the shape of it tells you a lot about where things are. The peek pushes the terminal down off screen, it doesn't resize it, so it doesn't force any weird reflow or anything either. There's a lot more to talk about, especially about how this interacts with how we expose what needs attention so you can easily find the terminals/sessions/etc. that require your attention. Of course, you'll be able to drive this with the keyboard too. :) As always, a disclaimer: this is all early work subject to change, just trying to share some of what we're building in public. A lot of the UI experiential design is almonk so follow him for more updates too!

Sharing some more UI previews for Superlogical, this time the "tab peek." If you three-finger swipe down, you can "peek" at the other session tabs. These are live-updating, metal-rendered previews including splits. If you keep dragging, you enter a "mission control" style mode. This is one of those UI patterns that once you get used to, you mindlessly find yourself doing it ALL THE TIME. It's a great way to quickly take a look at everything thats going on. You aren't reading the mini text of course, but the shape of it tells you a lot about where things are. The peek pushes the terminal down off screen, it doesn't resize it, so it doesn't force any weird reflow or anything either. There's a lot more to talk about, especially about how this interacts with how we expose what needs attention so you can easily find the terminals/sessions/etc. that require your attention. Of course, you'll be able to drive this with the keyboard too. :) As always, a disclaimer: this is all early work subject to change, just trying to share some of what we're building in public. A lot of the UI experiential design is almonk so follow him for more updates too!

Mitchell Hashimoto

351,264 views • 1 month ago

If you want to understand how far AI-assisted game development has moved in less than a year, IsoCity is one of the best examples. The original version took roughly 40–50 hours with Cursor Composer + cloud agents. Its assets were AI-generated, while the game itself grew into a surprisingly deep city simulation with traffic, pedestrians, trains, planes, zoning, resource management, saving, and mobile support. Under the hood: → Next.js 16 → TypeScript → Native HTML5 Canvas rendering → No external game engine → Custom isometric renderer and simulation systems And the best part: the whole project is open source under MIT, so you can study it, fork it, modify it, or use parts of the architecture in your own experiments. The visual jump from that 2025 build to what agents are producing in 2026 is pretty ridiculous. Built by Andrew Milich and contributors. Repo:

If you want to understand how far AI-assisted game development has moved in less than a year, IsoCity is one of the best examples. The original version took roughly 40–50 hours with Cursor Composer + cloud agents. Its assets were AI-generated, while the game itself grew into a surprisingly deep city simulation with traffic, pedestrians, trains, planes, zoning, resource management, saving, and mobile support. Under the hood: → Next.js 16 → TypeScript → Native HTML5 Canvas rendering → No external game engine → Custom isometric renderer and simulation systems And the best part: the whole project is open source under MIT, so you can study it, fork it, modify it, or use parts of the architecture in your own experiments. The visual jump from that 2025 build to what agents are producing in 2026 is pretty ridiculous. Built by Andrew Milich and contributors. Repo:

Token Gremlin

28,231 views • 1 month ago

Republicans like Nancy Mace insist that Liam should be using the female restroom. Really? A few questions: - Why? - How does this make sense to anyone beyond a kindergarten-level understanding of gender? - Have you actually thought this through, or is this just about scoring cheap political points? The point of posts like this is simple: Complex, multi-faceted issues can’t be reduced to oversimplified, black-and-white answers. But hey, I guess nuance isn’t for everyone.

Republicans like Nancy Mace insist that Liam should be using the female restroom. Really? A few questions: - Why? - How does this make sense to anyone beyond a kindergarten-level understanding of gender? - Have you actually thought this through, or is this just about scoring cheap political points? The point of posts like this is simple: Complex, multi-faceted issues can’t be reduced to oversimplified, black-and-white answers. But hey, I guess nuance isn’t for everyone.

Brian Krassenstein

621,330 views • 1 year ago

Really enjoyed this Go Wayback! guide from KranousPlays. If you want to know how to survive the endless maps of prologue: Go Wayback!, or just learn more about the game, this is a great place to start! o7 Full video here:

Really enjoyed this Go Wayback! guide from KranousPlays. If you want to know how to survive the endless maps of prologue: Go Wayback!, or just learn more about the game, this is a great place to start! o7 Full video here:

PLAYERUNKNOWN

24,259 views • 6 months ago

Tired of manual prompt tweaking? Watch the latest Llama tutorial on how to optimize your existing GPT or other LLM prompts for Llama with `llama-prompt-ops`, the open-source Python library! In this video, Partner Engineer Justin Lee demonstrates installation, project setup, migrating your first prompt, and analyzing performance gains. Watch now to discover: ✨ Why systematic prompt optimization is crucial for migrating from GPT to Llama. 💻 A live code walkthrough of `llama-prompt-ops` in action for a customer service classification task. Watch the full tutorial here:

Tired of manual prompt tweaking? Watch the latest Llama tutorial on how to optimize your existing GPT or other LLM prompts for Llama with `llama-prompt-ops`, the open-source Python library! In this video, Partner Engineer Justin Lee demonstrates installation, project setup, migrating your first prompt, and analyzing performance gains. Watch now to discover: ✨ Why systematic prompt optimization is crucial for migrating from GPT to Llama. 💻 A live code walkthrough of `llama-prompt-ops` in action for a customer service classification task. Watch the full tutorial here:

Meta for Developers

38,166 views • 1 year ago

What if you kept asking an LLM to "make it better"? In some recent work at FAIR, we investigate how we can efficiently use RL to fine-tune LLMs to iteratively self-improve on their previous solutions at inference-time. Training for iterated self-improvement can be costly. The naive approach to training for K self-improvement steps leads to K times the number of rollout steps per episode. We introduce Exploratory Iteration (ExIt), an RL-based automatic curriculum method that bootstraps diverse training distributions of self-improvement tasks by upcycling the LLM's own responses at previous turns as the starting points for both self-improvement and *self-divergence.* In order to decide what task to train on next, the curriculum prioritizes sampling of partial turn histories that led to higher return variance in its GRPO group (a learnability score that comes for free). This automatic curriculum over the bootstrapped task space teaches the model how to perform iterated self-improvement while only ever training the model on single-step self-improvement tasks. We look at ExIt's impact in both single-turn (contest math problems) and multi-turn (BFCLv3 multi-turn tasks), as well as MLE-bench, where the LLM is run in a search scaffold to produce solutions to real Kaggle competitions. Across these eval settings, we find ExIt produces models with greater capacity for inference-time self-improvement compared to GRPO. Notably, ExIt models can self-improve on test tasks for many more steps than the typical solution depth encountered during training, including a 22% improvement in MLE-bench performance compared to GRPO.

What if you kept asking an LLM to "make it better"? In some recent work at FAIR, we investigate how we can efficiently use RL to fine-tune LLMs to iteratively self-improve on their previous solutions at inference-time. Training for iterated self-improvement can be costly. The naive approach to training for K self-improvement steps leads to K times the number of rollout steps per episode. We introduce Exploratory Iteration (ExIt), an RL-based automatic curriculum method that bootstraps diverse training distributions of self-improvement tasks by upcycling the LLM's own responses at previous turns as the starting points for both self-improvement and self-divergence. In order to decide what task to train on next, the curriculum prioritizes sampling of partial turn histories that led to higher return variance in its GRPO group (a learnability score that comes for free). This automatic curriculum over the bootstrapped task space teaches the model how to perform iterated self-improvement while only ever training the model on single-step self-improvement tasks. We look at ExIt's impact in both single-turn (contest math problems) and multi-turn (BFCLv3 multi-turn tasks), as well as MLE-bench, where the LLM is run in a search scaffold to produce solutions to real Kaggle competitions. Across these eval settings, we find ExIt produces models with greater capacity for inference-time self-improvement compared to GRPO. Notably, ExIt models can self-improve on test tasks for many more steps than the typical solution depth encountered during training, including a 22% improvement in MLE-bench performance compared to GRPO.

Minqi Jiang

41,147 views • 1 year ago

Rare-event discovery for robots training is what we’ve been working on lately How would a robot react to an unusual event? What if a firefighter drone won’t be able to choke a fire (like on the video below)? Robo-doctors use cases? How to mass-produce these situations to train robots to see & react the best way? I think that open-sourced, world models like LTX can become the solution for such training process. And they might become the differentiator for the future of robotics. Of course, I’m not an expert in robotics and ML, but this topic makes me curious - and I’m curious about your thoughts

Rare-event discovery for robots training is what we’ve been working on lately How would a robot react to an unusual event? What if a firefighter drone won’t be able to choke a fire (like on the video below)? Robo-doctors use cases? How to mass-produce these situations to train robots to see & react the best way? I think that open-sourced, world models like LTX can become the solution for such training process. And they might become the differentiator for the future of robotics. Of course, I’m not an expert in robotics and ML, but this topic makes me curious - and I’m curious about your thoughts

AmirMušić

41,839 views • 1 month ago