Loading video...

Video Failed to Load

Go Home

Claude distillation has been a big topic this week while I am (coincidentally) writing Chapter 8 on model distillation. In that context, I shared some utilities to generate distillation data from all sorts of open-weight models via OpenRouter and Ollama:

rasbt's profile picture

Sebastian Raschka

416,500 subscribers

62,458 views • 3 months ago •via X (Twitter)

EducationNews & PoliticsScience & Technology
LIVE

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

Diffusion models generate high-quality images but require hundreds of forward passes. MIT CSAIL and Adobe Research introduce Distribution Matching Distillation (DMD), a distillation approach that converts costly multi-step diffusion models into fast one-step generators. A thread 🧵
0:24

Diffusion models generate high-quality images but require hundreds of forward passes. MIT CSAIL and Adobe Research introduce Distribution Matching Distillation (DMD), a distillation approach that converts costly multi-step diffusion models into fast one-step generators. A thread 🧵

MIT CSAIL

34,383 views • 2 years ago

Nvidia presents Articulated Kinematics Distillation from Video Diffusion Models
4:11

Nvidia presents Articulated Kinematics Distillation from Video Diffusion Models

AK

39,189 views • 1 year ago

data labeling is dead. long live distillation. from data to object detection endpoint in 90 seconds. link:
1:23

data labeling is dead. long live distillation. from data to object detection endpoint in 90 seconds. link:

SkalskiP

135,852 views • 6 months ago

Presto! Distilling Steps and Layers for Accelerating Music Generation Despite advances in diffusion-based text-to-music (TTM) methods, efficient, high-quality generation remains a challenge. We introduce Presto!, an approach to inference acceleration for score-based diffusion transformers via reducing both sampling steps and cost per step. To reduce steps, we develop a new score-based distribution matching distillation (DMD) method for the EDM-family of diffusion models, the first GAN-based distillation method for TTM. To reduce the cost per step, we develop a simple, but powerful improvement to a recent layer distillation method that improves learning via better preserving hidden state variance. Finally, we combine our step and layer distillation methods together for a dual-faceted approach. We evaluate our step and layer distillation methods independently and show each yield best-in-class performance. Our combined distillation method can generate high-quality outputs with improved diversity, accelerating our base model by 10-18x (230/435ms latency for 32 second mono/stereo 44.1kHz, 15x faster than comparable SOTA) -- the fastest high-quality TTM to our knowledge.
1:20

Presto! Distilling Steps and Layers for Accelerating Music Generation Despite advances in diffusion-based text-to-music (TTM) methods, efficient, high-quality generation remains a challenge. We introduce Presto!, an approach to inference acceleration for score-based diffusion transformers via reducing both sampling steps and cost per step. To reduce steps, we develop a new score-based distribution matching distillation (DMD) method for the EDM-family of diffusion models, the first GAN-based distillation method for TTM. To reduce the cost per step, we develop a simple, but powerful improvement to a recent layer distillation method that improves learning via better preserving hidden state variance. Finally, we combine our step and layer distillation methods together for a dual-faceted approach. We evaluate our step and layer distillation methods independently and show each yield best-in-class performance. Our combined distillation method can generate high-quality outputs with improved diversity, accelerating our base model by 10-18x (230/435ms latency for 32 second mono/stereo 44.1kHz, 15x faster than comparable SOTA) -- the fastest high-quality TTM to our knowledge.

AK

30,430 views • 1 year ago

FastWan Generating a 5-Second Video in 5 Seconds via Sparse Distillation
0:30

FastWan Generating a 5-Second Video in 5 Seconds via Sparse Distillation

AK

34,589 views • 10 months ago

I had an awesome time interviewing idan shenfeld and Jonas Hübotter from MIT and ETH Zurich about self-distillation. this very promising post-training paradigm where the model acts as its own teacher by conditioning on environment feedback or demonstrations. we cover the SDPO algo for reinforcement learning with rich feedback and SDFT for continual learning without forgetting along with many applications. we dig into how it works, why it's simpler and faster than GRPO, and where this is already showing up in production systems. table of content: 0:00 - what is self distillation 2:50 - idan (MIT) and jonas (ETH Zurich) introduction and motivation 18:40 - different perspective of on-policy self-distillation (presentation) 36:00 - metacognition and specificity in self-distillation 37:24 - very long hard task and self-distillation 42:00 - continual learning with self-distillation (presentation) 1:16:50 - what is next in this research direction? 1:20:00 - is there any experience with subjective feedbacks? 1:22:50 - quality vs number of feedbacks? 1:26:40 - what setting would self-distillation struggle vs GRPO? the slides were super crisp really cool of them to share! enjoy my guys 🌹
1:31:19

I had an awesome time interviewing idan shenfeld and Jonas Hübotter from MIT and ETH Zurich about self-distillation. this very promising post-training paradigm where the model acts as its own teacher by conditioning on environment feedback or demonstrations. we cover the SDPO algo for reinforcement learning with rich feedback and SDFT for continual learning without forgetting along with many applications. we dig into how it works, why it's simpler and faster than GRPO, and where this is already showing up in production systems. table of content: 0:00 - what is self distillation 2:50 - idan (MIT) and jonas (ETH Zurich) introduction and motivation 18:40 - different perspective of on-policy self-distillation (presentation) 36:00 - metacognition and specificity in self-distillation 37:24 - very long hard task and self-distillation 42:00 - continual learning with self-distillation (presentation) 1:16:50 - what is next in this research direction? 1:20:00 - is there any experience with subjective feedbacks? 1:22:50 - quality vs number of feedbacks? 1:26:40 - what setting would self-distillation struggle vs GRPO? the slides were super crisp really cool of them to share! enjoy my guys 🌹

Yacine Mahdid

12,945 views • 1 month ago

Modality Forcing - turns FLUX.2.klein into 3D-aware models. - joint RGB-D, I2D, and D2I synthesis in one model - pixel-space depth tokenization - preserves T2I quality via self-distillation
0:34

Modality Forcing - turns FLUX.2.klein into 3D-aware models. - joint RGB-D, I2D, and D2I synthesis in one model - pixel-space depth tokenization - preserves T2I quality via self-distillation

Wildminder

12,739 views • 8 days ago

Distillation has paved the way for China and others to compete at the cutting edge of AI faster and cheaper than ever before Deepseek woke up the world to its disruptive potential Our latest TechCheck Take deep dive on AI's distillation problem
1:00

Distillation has paved the way for China and others to compete at the cutting edge of AI faster and cheaper than ever before Deepseek woke up the world to its disruptive potential Our latest TechCheck Take deep dive on AI's distillation problem

Deirdre Bosa

11,694 views • 1 year ago

Meta presents Video Editing via Factorized Diffusion Distillation We introduce Emu Video Edit (EVE), a model that establishes a new state-of-the art in video editing without relying on any supervised video editing data. To develop EVE we separately train an image editing
0:24

Meta presents Video Editing via Factorized Diffusion Distillation We introduce Emu Video Edit (EVE), a model that establishes a new state-of-the art in video editing without relying on any supervised video editing data. To develop EVE we separately train an image editing

AK

115,598 views • 2 years ago

DeepSeek Didn’t Catch Up — It Cheated. China’s AI “Breakthrough” Is Built on Stolen American Tech. In this video, I break down explosive allegations that DeepSeek used banned U.S. chips and illicit model distillation to train its newest system. If true, this is IP extraction wrapped in marketing. We examine what distillation really means, why export controls matter, and how dependent China’s AI ecosystem still is on American hardware and research.
11:22

DeepSeek Didn’t Catch Up — It Cheated. China’s AI “Breakthrough” Is Built on Stolen American Tech. In this video, I break down explosive allegations that DeepSeek used banned U.S. chips and illicit model distillation to train its newest system. If true, this is IP extraction wrapped in marketing. We examine what distillation really means, why export controls matter, and how dependent China’s AI ecosystem still is on American hardware and research.

Ken Cao-The China Crash Chronicle

12,668 views • 3 months ago

ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation propose to model the 3D parameter as a random variable instead of a constant as in SDS and present variational score distillation (VSD), a principled particle-based variational framework to explain and address the aforementioned issues in text-to-3D generation. We show that SDS is a special case of VSD and leads to poor samples with both small and large CFG weights. In comparison, VSD works well with various CFG weights as ancestral sampling from diffusion models and simultaneously improves the diversity and sample quality with a common CFG weight (i.e., 7.5). We further present various improvements in the design space for text-to-3D such as distillation time schedule and density initialization, which are orthogonal to the distillation algorithm yet not well explored. Our overall approach, dubbed ProlificDreamer, can generate high rendering resolution (i.e., 512times512) and high-fidelity NeRF with rich structure and complex effects (e.g., smoke and drops). Further, initialized from NeRF, meshes fine-tuned by VSD are meticulously detailed and photo-realistic paper page:
0:43

ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation propose to model the 3D parameter as a random variable instead of a constant as in SDS and present variational score distillation (VSD), a principled particle-based variational framework to explain and address the aforementioned issues in text-to-3D generation. We show that SDS is a special case of VSD and leads to poor samples with both small and large CFG weights. In comparison, VSD works well with various CFG weights as ancestral sampling from diffusion models and simultaneously improves the diversity and sample quality with a common CFG weight (i.e., 7.5). We further present various improvements in the design space for text-to-3D such as distillation time schedule and density initialization, which are orthogonal to the distillation algorithm yet not well explored. Our overall approach, dubbed ProlificDreamer, can generate high rendering resolution (i.e., 512times512) and high-fidelity NeRF with rich structure and complex effects (e.g., smoke and drops). Further, initialized from NeRF, meshes fine-tuned by VSD are meticulously detailed and photo-realistic paper page:

AK

46,150 views • 3 years ago

📢 Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation Got only one or a few images and wondering if recovering the 3D environment is a reconstruction or generation problem? Why not do it with a generative reconstruction model! We show that a camera-conditioned video diffusion model can be transformed into a generative reconstruction model that directly outputs a high-quality 3D Gaussian Splatting representation through self-distillation, without requiring real-world training data. Check out our results in the video (wait for dynamic scenes in the second half!) : Project Page: Code and Models: Paper:
0:50

📢 Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation Got only one or a few images and wondering if recovering the 3D environment is a reconstruction or generation problem? Why not do it with a generative reconstruction model! We show that a camera-conditioned video diffusion model can be transformed into a generative reconstruction model that directly outputs a high-quality 3D Gaussian Splatting representation through self-distillation, without requiring real-world training data. Check out our results in the video (wait for dynamic scenes in the second half!) : Project Page: Code and Models: Paper:

Sherwin Bahmani

66,417 views • 9 months ago

This is some serious driving! A 420-ton distillation tower, made by Iran, is being transferred
1:05

This is some serious driving! A 420-ton distillation tower, made by Iran, is being transferred

Soureh 🟩☫🟥

426,524 views • 8 months ago

Progressive Rendering Distillation released on Hugging Face Adapting Stable Diffusion for Instant Text-to-Mesh Generation without 3D Data
0:20

Progressive Rendering Distillation released on Hugging Face Adapting Stable Diffusion for Instant Text-to-Mesh Generation without 3D Data

AK

12,586 views • 1 year ago

Gemma and the power of post-training. 💪 Google DeepMind’s Léonard Hussenot details: 🔍 How Gemma facilitates post-training research 🚀 Best-of-N Distillation (BOND) for model efficiency 🏆 Improving reward models with weight averaging 👀 →
0:57

Gemma and the power of post-training. 💪 Google DeepMind’s Léonard Hussenot details: 🔍 How Gemma facilitates post-training research 🚀 Best-of-N Distillation (BOND) for model efficiency 🏆 Improving reward models with weight averaging 👀 →

Google for Developers

18,616 views • 1 year ago

Both Google and OpenAI are now publicly saying Chinese models are running large-scale distillation attacks against them. Distillation means using outputs from a frontier model to train a cheaper competitor. Systematically. At scale. GLM-5 is already benchmarking near Claude and GPT on real engineering tasks. The export controls were meant to slow capability transfer through chips. Nobody put controls on API access. The capability gap the West assumed was a moat is being closed through the front door. Legally. Quietly. Quickly.
1:13

Both Google and OpenAI are now publicly saying Chinese models are running large-scale distillation attacks against them. Distillation means using outputs from a frontier model to train a cheaper competitor. Systematically. At scale. GLM-5 is already benchmarking near Claude and GPT on real engineering tasks. The export controls were meant to slow capability transfer through chips. Nobody put controls on API access. The capability gap the West assumed was a moat is being closed through the front door. Legally. Quietly. Quickly.

Shawn Chauhan

44,437 views • 4 months ago

👀Big one: Discussing A.I model distillation & the U.S vs China rivalry today is our CEO TravisGood along with Ben Fielding from gensyn and Bayley from PrismaX. What a treat in the studio!
20:54

👀Big one: Discussing A.I model distillation & the U.S vs China rivalry today is our CEO TravisGood along with Ben Fielding from gensyn and Bayley from PrismaX. What a treat in the studio!

ambient.xyz

25,359 views • 2 months ago

AccVideo just dropped on Hugging Face Accelerating Video Diffusion Model with Synthetic Dataset present a efficient distillation method to accelerate video diffusion models with synthetic dataset method is 8.5x faster than HunyuanVideo
1:04

AccVideo just dropped on Hugging Face Accelerating Video Diffusion Model with Synthetic Dataset present a efficient distillation method to accelerate video diffusion models with synthetic dataset method is 8.5x faster than HunyuanVideo

AK

20,633 views • 1 year ago