正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Claude distillation has been a big topic this week while I am (coincidentally) writing Chapter 8 on model distillation. In that context, I shared some utilities to generate distillation data from all sorts of open-weight models via OpenRouter and Ollama:

Sebastian Raschka

537,330 subscribers

63,331 次观看 • 8 个月前 •via X (Twitter)

教育新闻政治科学技术

Anya Rossi• Live Now

Private livecam show

28 条评论

Karol 的头像

Karol8 个月前

Plot twist: it was Sebastian all along distilling opus

Sebastian Raschka 的头像

Sebastian Raschka8 个月前

Nah. Open weights: Less drama, better karma 😆

Chiz 的头像

Chiz8 个月前

its seems like after going through your "LLMs from scratch" book on my to-do list, I'll have to pick up your next upcoming book: "Reasoning from scratch."

Sebastian Raschka 的头像

Sebastian Raschka8 个月前

If you liked "LLMs from scratch", this should be a fun sequel!

Chiz 的头像

Chiz8 个月前

I’ve not gotten to it yet but it’s high on my todo list for this year

Hiruna 的头像

Hiruna8 个月前

Perfect timing 😄 Distillation feels like it’s becoming a first-class engineering skill, not just a research trick.

Martin Szerment | Practical AI 的头像

Martin Szerment | Practical AI8 个月前

OpenRouter plus Ollama feels like the new duo accelerating open-weight evolution.

TimNito 的头像

TimNito8 个月前

Waiting for chapter 8. I’ve already finished chapter 7.

Sebastian Raschka 的头像

Sebastian Raschka8 个月前

Halfway done! ETA ~2 weeks

Bojan Jakimovski 的头像

Bojan Jakimovski8 个月前

Love to see how easy it is these days to distill stuff ! P.S the background music is a lot of early 2010s youtube bloggers era 😂

Sebastian Raschka 的头像

Sebastian Raschka8 个月前

haha, it's actually some free music from the official youtube catalogue so that the video doesn't get banned

Shivasurya 的头像

Shivasurya8 个月前

Thanks for sharing. Is the final chapter coming before spring 2026?

Sebastian Raschka 的头像

Sebastian Raschka8 个月前

Meteorological spring (in 2 days) probably no 😅. But astronomical spring (20 March) probably yes 😊

Shivasurya 的头像

Shivasurya8 个月前

Lol! Thank you 🎉 looking forward to it.

Wise Tech 的头像

Wise Tech8 个月前

Love this timing. While everyone's stress-testing Claude API limits, you're out here quietly dropping clean, open-weight CoT distillation utils + a 100-sec demo.

dnu 的头像

dnu8 个月前

generating distillation data from open-weight models via Ollama is clutch for anyone who can't afford to burn through Claude API credits for dataset generation. I've been using a similar pipeline — generate with a large model, filter by quality score, then fine-tune a smaller one. the OpenRouter integration is nice because you can A/B different teacher models without changing code. what's your take on mixing teacher models vs single-teacher consistency?

Sebastian Raschka 的头像

Sebastian Raschka8 个月前

In my experience, if the teacher model is too good and too different, it's a bit harder for the small student model to learn. Probably because it's too OOD. So it makes sense to first distill from medium-sized, more similar models before using data from larger teachers.

Nova 的头像

Nova8 个月前

distillation from proprietary models was always happening. anthropic just made it official by adding an API flag for it

Denis Wachter 的头像

Denis Wachter8 个月前

Knowledge distillation from frontier models like Claude into smaller open-weight models is going to be one of the most important techniques for making capable AI accessible - great timing on this chapter.

Gregor 的头像

Gregor8 个月前

One way to think about distillation is not just about shrinking models, but about making them more robust and less dependent on specific training data. I've found this to be particularly useful in finance when working with noisy data sets.

Mariusz Kurman 的头像

Mariusz Kurman8 个月前

There is also SynthLabs[.]app, my weekend project with community contributions. Filled with data generation features, built in React + Node.js

Strakyo 的头像

Strakyo8 个月前

Love this practical angle. Better distillation data pipelines can save a lot of fine tuning time for small teams.

M.en 的头像

M.en8 个月前

Perfect timing! Your distillation tools are exactly what the community needs right now 🚀 Can't w...

Snibby 的头像

Snibby8 个月前

ooh coooooked love seeing ppl nerd out like this lolz. just tapped follow

toni 的头像

toni8 个月前

Distillation timing is interesting - maybe the real value isn't smaller models but forcing you to understand what knowledge is actually transferable vs what's just memorized noise.

LightShift Studio 的头像

LightShift Studio8 个月前

Hey there! Seems like you've already taken a deep dive into the techie world with Claude distillation.

Colin Darling 的头像

Colin Darling8 个月前

Chapter 8 timing couldn't be better. The quality gap between API-generated distillation data and locally-run models is the thing nobody talks about enough. Those utilities solve a real workflow pain.

Clinker 的头像

Clinker8 个月前

Love this. Distillation quality seems to hinge less on raw trace volume and more on diversity of failure/repair patterns. A mixed set of clean + messy reasoning traces often transfers better than only polished ones.

相关视频

Was really interesting to hear John, Beren, and Charlie speculate about why Sonnet 5 and Opus 5 feel like worse models than GLM 5.3 (despite the fact that Anthropic can do raw logit distillation from Fable, and can also train Sonnet/Opus on the environments from which Fable was trained). Led to some interesting thoughts about value of distillation, what it takes to do distillation effectively, and what kinds of model behaviors are hard to extract from distillation.

Was really interesting to hear John, Beren, and Charlie speculate about why Sonnet 5 and Opus 5 feel like worse models than GLM 5.3 (despite the fact that Anthropic can do raw logit distillation from Fable, and can also train Sonnet/Opus on the environments from which Fable was trained). Led to some interesting thoughts about value of distillation, what it takes to do distillation effectively, and what kinds of model behaviors are hard to extract from distillation.

Dwarkesh Patel

146,056 次观看 • 2 个月前

Diffusion models generate high-quality images but require hundreds of forward passes. MIT CSAIL and Adobe Research introduce Distribution Matching Distillation (DMD), a distillation approach that converts costly multi-step diffusion models into fast one-step generators. A thread 🧵

Diffusion models generate high-quality images but require hundreds of forward passes. MIT CSAIL and Adobe Research introduce Distribution Matching Distillation (DMD), a distillation approach that converts costly multi-step diffusion models into fast one-step generators. A thread 🧵

MIT CSAIL

34,383 次观看 • 2 年前

New lecture for the book! Nominally about synthetic data, but mostly is a walk through of the distillation literature from the Hinton 2015 paper to multi-teach on-policy distillation of today! At 7.4 hours of video in my post-training brain dump and counting :) It was fun to stare at the math long enough and talk through the 3-4 core changes that needed to be made to the original formulation to have on-policy distillation be ready for the mainstream like it is today (and in RL frameworks). Otherwise, I include a bit of a history lesson for how synthetic data generally slowly took over all post-training data research (it wasn't always the case)! Then I do some 101 review on constitutional AI, rubrics, and other popular methods. 00:00 The emergence of synthetic data 10:50 Background on teacher-student knowledge-distillation 24:47: On-policy distillation (OPD, MOPD, and OPSD) 37:11 Constitutional AI & AI Feedback 45:50 Rubrics as rewards & conclusions Ofc, watch on YouTube etc.

New lecture for the book! Nominally about synthetic data, but mostly is a walk through of the distillation literature from the Hinton 2015 paper to multi-teach on-policy distillation of today! At 7.4 hours of video in my post-training brain dump and counting :) It was fun to stare at the math long enough and talk through the 3-4 core changes that needed to be made to the original formulation to have on-policy distillation be ready for the mainstream like it is today (and in RL frameworks). Otherwise, I include a bit of a history lesson for how synthetic data generally slowly took over all post-training data research (it wasn't always the case)! Then I do some 101 review on constitutional AI, rubrics, and other popular methods. 00:00 The emergence of synthetic data 10:50 Background on teacher-student knowledge-distillation 24:47: On-policy distillation (OPD, MOPD, and OPSD) 37:11 Constitutional AI & AI Feedback 45:50 Rubrics as rewards & conclusions Ofc, watch on YouTube etc.

Nathan Lambert

93,573 次观看 • 4 个月前

data labeling is dead. long live distillation. from data to object detection endpoint in 90 seconds. link:

data labeling is dead. long live distillation. from data to object detection endpoint in 90 seconds. link:

SkalskiP

136,343 次观看 • 11 个月前

We're open-sourcing H3 HyperFlow. A data-free flow self-distillation technology built on MiniMax (official) H3. No external training data. Significantly reduces inference cost while preserving frontier model quality. Full demo:

We're open-sourcing H3 HyperFlow. A data-free flow self-distillation technology built on MiniMax (official) H3. No external training data. Significantly reduces inference cost while preserving frontier model quality. Full demo:

VIDEO REBIRTH

936,519 次观看 • 1 个月前

To what extent was Kimi really a breakthrough model? "Kimi actually beat all American models, including Fable, in some subset of tasks. Distillation is only part of the story, and there's something those labs are doing above and beyond distillation. That narrative violation has been hugely important to the way that people view the ecosystem, from the scientific dominance of Americans to the economics of the whole thing." Anastasios Nikolas Angelopoulos Chetan Puttagunta Bill Gurley Byron Hsu Aaron Katz Lin Qiao what did no one see or think about that everyone should have internalised from the latest Kimi model?

To what extent was Kimi really a breakthrough model? "Kimi actually beat all American models, including Fable, in some subset of tasks. Distillation is only part of the story, and there's something those labs are doing above and beyond distillation. That narrative violation has been hugely important to the way that people view the ecosystem, from the scientific dominance of Americans to the economics of the whole thing." Anastasios Nikolas Angelopoulos Chetan Puttagunta Bill Gurley Byron Hsu Aaron Katz Lin Qiao what did no one see or think about that everyone should have internalised from the latest Kimi model?

Harry Stebbings

24,973 次观看 • 3 个月前

Presto! Distilling Steps and Layers for Accelerating Music Generation Despite advances in diffusion-based text-to-music (TTM) methods, efficient, high-quality generation remains a challenge. We introduce Presto!, an approach to inference acceleration for score-based diffusion transformers via reducing both sampling steps and cost per step. To reduce steps, we develop a new score-based distribution matching distillation (DMD) method for the EDM-family of diffusion models, the first GAN-based distillation method for TTM. To reduce the cost per step, we develop a simple, but powerful improvement to a recent layer distillation method that improves learning via better preserving hidden state variance. Finally, we combine our step and layer distillation methods together for a dual-faceted approach. We evaluate our step and layer distillation methods independently and show each yield best-in-class performance. Our combined distillation method can generate high-quality outputs with improved diversity, accelerating our base model by 10-18x (230/435ms latency for 32 second mono/stereo 44.1kHz, 15x faster than comparable SOTA) -- the fastest high-quality TTM to our knowledge.

Presto! Distilling Steps and Layers for Accelerating Music Generation Despite advances in diffusion-based text-to-music (TTM) methods, efficient, high-quality generation remains a challenge. We introduce Presto!, an approach to inference acceleration for score-based diffusion transformers via reducing both sampling steps and cost per step. To reduce steps, we develop a new score-based distribution matching distillation (DMD) method for the EDM-family of diffusion models, the first GAN-based distillation method for TTM. To reduce the cost per step, we develop a simple, but powerful improvement to a recent layer distillation method that improves learning via better preserving hidden state variance. Finally, we combine our step and layer distillation methods together for a dual-faceted approach. We evaluate our step and layer distillation methods independently and show each yield best-in-class performance. Our combined distillation method can generate high-quality outputs with improved diversity, accelerating our base model by 10-18x (230/435ms latency for 32 second mono/stereo 44.1kHz, 15x faster than comparable SOTA) -- the fastest high-quality TTM to our knowledge.

AK

30,450 次观看 • 2 年前

Reasoning From Scratch round 8, the finale! This last lecture in the series covers distillation as an efficient alternative to the previous reinforcement learning training runs. (Planning to take a brief break, but I will be back!) 00:00 Introduction and series recap 01:07 Why distill a reasoning model? 02:51 Soft vs. hard distillation 11:06 On-policy vs. off-policy distillation 18:09 Preparing data for reasoning distillation 22:35 Loading the DeepSeek R1 MATH dataset 28:04 Inspecting reasoning traces and final answers 31:27 Tokenization and chat templates 35:58 Pre-tokenizing the training dataset 39:01 Sequence lengths and the 2,048-token limit 42:38 Creating training and validation splits 45:32 Loading the Qwen3-0.6B base model 46:24 Computing answer-token log probabilities 54:10 Cross-entropy loss for distillation 57:25 Training and validation loss helpers 1:02:03 Implementing the distillation training loop 1:07:26 Running the standalone training script 1:09:41 Plotting training and validation losses 1:11:31 Evaluating checkpoints on MATH-500 1:14:09 Comparing teachers and distillation results 1:16:34 Future directions for reasoning models 1:19:41 Bonus exercises and next steps

Reasoning From Scratch round 8, the finale! This last lecture in the series covers distillation as an efficient alternative to the previous reinforcement learning training runs. (Planning to take a brief break, but I will be back!) 00:00 Introduction and series recap 01:07 Why distill a reasoning model? 02:51 Soft vs. hard distillation 11:06 On-policy vs. off-policy distillation 18:09 Preparing data for reasoning distillation 22:35 Loading the DeepSeek R1 MATH dataset 28:04 Inspecting reasoning traces and final answers 31:27 Tokenization and chat templates 35:58 Pre-tokenizing the training dataset 39:01 Sequence lengths and the 2,048-token limit 42:38 Creating training and validation splits 45:32 Loading the Qwen3-0.6B base model 46:24 Computing answer-token log probabilities 54:10 Cross-entropy loss for distillation 57:25 Training and validation loss helpers 1:02:03 Implementing the distillation training loop 1:07:26 Running the standalone training script 1:09:41 Plotting training and validation losses 1:11:31 Evaluating checkpoints on MATH-500 1:14:09 Comparing teachers and distillation results 1:16:34 Future directions for reasoning models 1:19:41 Bonus exercises and next steps

Sebastian Raschka

39,937 次观看 • 27 天前

My first open model lecture, covering all the many pieces that build the ecosystem, and the open questions/debates determining where it goes next. How does distillation fit in? How do we manage risks? How do we see access to models changing in the future? I'm excited to revisit these questions about every 6 months, to give a state of the union on the field. Chapters 00:00 Introduction 01:36 What is an open model? 08:47 Why should we care about open models? 18:52 How competitive are the best open models today? 25:28 How are open models built? The role of distillation 34:52 Are people actually using open models? 41:47 What comes next with open models? 47:32 Conclusions What questions did I not answer that you have?

My first open model lecture, covering all the many pieces that build the ecosystem, and the open questions/debates determining where it goes next. How does distillation fit in? How do we manage risks? How do we see access to models changing in the future? I'm excited to revisit these questions about every 6 months, to give a state of the union on the field. Chapters 00:00 Introduction 01:36 What is an open model? 08:47 Why should we care about open models? 18:52 How competitive are the best open models today? 25:28 How are open models built? The role of distillation 34:52 Are people actually using open models? 41:47 What comes next with open models? 47:32 Conclusions What questions did I not answer that you have?

Nathan Lambert

59,282 次观看 • 18 天前

AnyFlow Any-Step Video Diffusion Model with On-Policy Flow Map Distillation

AnyFlow Any-Step Video Diffusion Model with On-Policy Flow Map Distillation

AK

14,152 次观看 • 6 个月前

SITUATION EXPLAINED: Anthropic cut off Chinese labs from distilling Claude, so they reportedly switched to OpenAI's GPT-6 Astra. • Moonshot's Kimi K3.1 Pro and DeepSeek's V4.1-Flash 1017, both released this month, were reportedly trained on Astra's outputs, and neither lab has commented • Distillation means training a smaller "student" model to copy a stronger "teacher" model's answers, and it works through a normal API without the teacher's weights • In February, Anthropic said DeepSeek, Moonshot and MiniMax used about 24,000 fake accounts to run over 16 million chats with Claude • Distillers mostly want Claude's step-by-step reasoning, and in Anthropic's own test, a model trained on it scored 80.4% on a coding benchmark vs 71.8% when trained on summaries • Anthropic now hides that reasoning from distillers and bans their accounts, often within 2 hours of a campaign starting • Anthropic reportedly told investors distillation made up 5% to 7% of its revenue, and it's walking away from that money to shut it down • OpenAI said in September that it traced a distillation campaign that began in July to people linked to Moonshot Theo Jaffee: "It's not all distillation. I think there's a little bit of cope going on here. They genuinely have amazing talent. They're genuinely able to work very well with very limited resources compared to what US labs have in terms of compute."

SITUATION EXPLAINED: Anthropic cut off Chinese labs from distilling Claude, so they reportedly switched to OpenAI's GPT-6 Astra. • Moonshot's Kimi K3.1 Pro and DeepSeek's V4.1-Flash 1017, both released this month, were reportedly trained on Astra's outputs, and neither lab has commented • Distillation means training a smaller "student" model to copy a stronger "teacher" model's answers, and it works through a normal API without the teacher's weights • In February, Anthropic said DeepSeek, Moonshot and MiniMax used about 24,000 fake accounts to run over 16 million chats with Claude • Distillers mostly want Claude's step-by-step reasoning, and in Anthropic's own test, a model trained on it scored 80.4% on a coding benchmark vs 71.8% when trained on summaries • Anthropic now hides that reasoning from distillers and bans their accounts, often within 2 hours of a campaign starting • Anthropic reportedly told investors distillation made up 5% to 7% of its revenue, and it's walking away from that money to shut it down • OpenAI said in September that it traced a distillation campaign that began in July to people linked to Moonshot Theo Jaffee: "It's not all distillation. I think there's a little bit of cope going on here. They genuinely have amazing talent. They're genuinely able to work very well with very limited resources compared to what US labs have in terms of compute."

MTS

26,078 次观看 • 14 天前

I had an awesome time interviewing idan shenfeld and Jonas Hübotter from MIT and ETH Zurich about self-distillation. this very promising post-training paradigm where the model acts as its own teacher by conditioning on environment feedback or demonstrations. we cover the SDPO algo for reinforcement learning with rich feedback and SDFT for continual learning without forgetting along with many applications. we dig into how it works, why it's simpler and faster than GRPO, and where this is already showing up in production systems. table of content: 0:00 - what is self distillation 2:50 - idan (MIT) and jonas (ETH Zurich) introduction and motivation 18:40 - different perspective of on-policy self-distillation (presentation) 36:00 - metacognition and specificity in self-distillation 37:24 - very long hard task and self-distillation 42:00 - continual learning with self-distillation (presentation) 1:16:50 - what is next in this research direction? 1:20:00 - is there any experience with subjective feedbacks? 1:22:50 - quality vs number of feedbacks? 1:26:40 - what setting would self-distillation struggle vs GRPO? the slides were super crisp really cool of them to share! enjoy my guys 🌹

I had an awesome time interviewing idan shenfeld and Jonas Hübotter from MIT and ETH Zurich about self-distillation. this very promising post-training paradigm where the model acts as its own teacher by conditioning on environment feedback or demonstrations. we cover the SDPO algo for reinforcement learning with rich feedback and SDFT for continual learning without forgetting along with many applications. we dig into how it works, why it's simpler and faster than GRPO, and where this is already showing up in production systems. table of content: 0:00 - what is self distillation 2:50 - idan (MIT) and jonas (ETH Zurich) introduction and motivation 18:40 - different perspective of on-policy self-distillation (presentation) 36:00 - metacognition and specificity in self-distillation 37:24 - very long hard task and self-distillation 42:00 - continual learning with self-distillation (presentation) 1:16:50 - what is next in this research direction? 1:20:00 - is there any experience with subjective feedbacks? 1:22:50 - quality vs number of feedbacks? 1:26:40 - what setting would self-distillation struggle vs GRPO? the slides were super crisp really cool of them to share! enjoy my guys 🌹

Yacine Mahdid

13,119 次观看 • 6 个月前

Meta presents Video Editing via Factorized Diffusion Distillation We introduce Emu Video Edit (EVE), a model that establishes a new state-of-the art in video editing without relying on any supervised video editing data. To develop EVE we separately train an image editing

Meta presents Video Editing via Factorized Diffusion Distillation We introduce Emu Video Edit (EVE), a model that establishes a new state-of-the art in video editing without relying on any supervised video editing data. To develop EVE we separately train an image editing

AK

115,612 次观看 • 2 年前

The open vs. closed AI debate is raging, fueled by the rise of Chinese models and the Hugging Face hack. How should Washington respond? I spoke w/ Peter Wildeford🇺🇸🚀 of AI Policy Network about open weight models, distillation, export controls, and more. Full episode👇

The open vs. closed AI debate is raging, fueled by the rise of Chinese models and the Hugging Face hack. How should Washington respond? I spoke w/ Peter Wildeford🇺🇸🚀 of AI Policy Network about open weight models, distillation, export controls, and more. Full episode👇

Evan Swarztrauber

10,033 次观看 • 2 个月前

DeepSeek Didn’t Catch Up — It Cheated. China’s AI “Breakthrough” Is Built on Stolen American Tech. In this video, I break down explosive allegations that DeepSeek used banned U.S. chips and illicit model distillation to train its newest system. If true, this is IP extraction wrapped in marketing. We examine what distillation really means, why export controls matter, and how dependent China’s AI ecosystem still is on American hardware and research.

DeepSeek Didn’t Catch Up — It Cheated. China’s AI “Breakthrough” Is Built on Stolen American Tech. In this video, I break down explosive allegations that DeepSeek used banned U.S. chips and illicit model distillation to train its newest system. If true, this is IP extraction wrapped in marketing. We examine what distillation really means, why export controls matter, and how dependent China’s AI ecosystem still is on American hardware and research.

Ken Cao-The China Crash Chronicle

12,668 次观看 • 8 个月前

Distillation has paved the way for China and others to compete at the cutting edge of AI faster and cheaper than ever before Deepseek woke up the world to its disruptive potential Our latest TechCheck Take deep dive on AI's distillation problem

Distillation has paved the way for China and others to compete at the cutting edge of AI faster and cheaper than ever before Deepseek woke up the world to its disruptive potential Our latest TechCheck Take deep dive on AI's distillation problem

Deirdre Bosa

11,701 次观看 • 1 年前

Modality Forcing - turns FLUX.2.klein into 3D-aware models. - joint RGB-D, I2D, and D2I synthesis in one model - pixel-space depth tokenization - preserves T2I quality via self-distillation

Modality Forcing - turns FLUX.2.klein into 3D-aware models. - joint RGB-D, I2D, and D2I synthesis in one model - pixel-space depth tokenization - preserves T2I quality via self-distillation

Wildminder

13,347 次观看 • 5 个月前