正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Claude distillation has been a big topic this week while I am (coincidentally) writing Chapter 8 on model distillation. In that context, I shared some utilities to generate distillation data from all sorts of open-weight models via OpenRouter and Ollama:

Sebastian Raschka

474,171 subscribers

62,838 次观看 • 5 个月前 •via X (Twitter)

教育新闻政治科学技术

Anya Rossi• Live Now

Private livecam show

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

New lecture for the book! Nominally about synthetic data, but mostly is a walk through of the distillation literature from the Hinton 2015 paper to multi-teach on-policy distillation of today! At 7.4 hours of video in my post-training brain dump and counting :) It was fun to stare at the math long enough and talk through the 3-4 core changes that needed to be made to the original formulation to have on-policy distillation be ready for the mainstream like it is today (and in RL frameworks). Otherwise, I include a bit of a history lesson for how synthetic data generally slowly took over all post-training data research (it wasn't always the case)! Then I do some 101 review on constitutional AI, rubrics, and other popular methods. 00:00 The emergence of synthetic data 10:50 Background on teacher-student knowledge-distillation 24:47: On-policy distillation (OPD, MOPD, and OPSD) 37:11 Constitutional AI & AI Feedback 45:50 Rubrics as rewards & conclusions Ofc, watch on YouTube etc.

New lecture for the book! Nominally about synthetic data, but mostly is a walk through of the distillation literature from the Hinton 2015 paper to multi-teach on-policy distillation of today! At 7.4 hours of video in my post-training brain dump and counting :) It was fun to stare at the math long enough and talk through the 3-4 core changes that needed to be made to the original formulation to have on-policy distillation be ready for the mainstream like it is today (and in RL frameworks). Otherwise, I include a bit of a history lesson for how synthetic data generally slowly took over all post-training data research (it wasn't always the case)! Then I do some 101 review on constitutional AI, rubrics, and other popular methods. 00:00 The emergence of synthetic data 10:50 Background on teacher-student knowledge-distillation 24:47: On-policy distillation (OPD, MOPD, and OPSD) 37:11 Constitutional AI & AI Feedback 45:50 Rubrics as rewards & conclusions Ofc, watch on YouTube etc.

Nathan Lambert

92,497 次观看 • 1 个月前

Presto! Distilling Steps and Layers for Accelerating Music Generation Despite advances in diffusion-based text-to-music (TTM) methods, efficient, high-quality generation remains a challenge. We introduce Presto!, an approach to inference acceleration for score-based diffusion transformers via reducing both sampling steps and cost per step. To reduce steps, we develop a new score-based distribution matching distillation (DMD) method for the EDM-family of diffusion models, the first GAN-based distillation method for TTM. To reduce the cost per step, we develop a simple, but powerful improvement to a recent layer distillation method that improves learning via better preserving hidden state variance. Finally, we combine our step and layer distillation methods together for a dual-faceted approach. We evaluate our step and layer distillation methods independently and show each yield best-in-class performance. Our combined distillation method can generate high-quality outputs with improved diversity, accelerating our base model by 10-18x (230/435ms latency for 32 second mono/stereo 44.1kHz, 15x faster than comparable SOTA) -- the fastest high-quality TTM to our knowledge.

Presto! Distilling Steps and Layers for Accelerating Music Generation Despite advances in diffusion-based text-to-music (TTM) methods, efficient, high-quality generation remains a challenge. We introduce Presto!, an approach to inference acceleration for score-based diffusion transformers via reducing both sampling steps and cost per step. To reduce steps, we develop a new score-based distribution matching distillation (DMD) method for the EDM-family of diffusion models, the first GAN-based distillation method for TTM. To reduce the cost per step, we develop a simple, but powerful improvement to a recent layer distillation method that improves learning via better preserving hidden state variance. Finally, we combine our step and layer distillation methods together for a dual-faceted approach. We evaluate our step and layer distillation methods independently and show each yield best-in-class performance. Our combined distillation method can generate high-quality outputs with improved diversity, accelerating our base model by 10-18x (230/435ms latency for 32 second mono/stereo 44.1kHz, 15x faster than comparable SOTA) -- the fastest high-quality TTM to our knowledge.

AK

30,430 次观看 • 1 年前

Meta presents Video Editing via Factorized Diffusion Distillation We introduce Emu Video Edit (EVE), a model that establishes a new state-of-the art in video editing without relying on any supervised video editing data. To develop EVE we separately train an image editing

Meta presents Video Editing via Factorized Diffusion Distillation We introduce Emu Video Edit (EVE), a model that establishes a new state-of-the art in video editing without relying on any supervised video editing data. To develop EVE we separately train an image editing

AK

115,612 次观看 • 2 年前

I had an awesome time interviewing idan shenfeld and Jonas Hübotter from MIT and ETH Zurich about self-distillation. this very promising post-training paradigm where the model acts as its own teacher by conditioning on environment feedback or demonstrations. we cover the SDPO algo for reinforcement learning with rich feedback and SDFT for continual learning without forgetting along with many applications. we dig into how it works, why it's simpler and faster than GRPO, and where this is already showing up in production systems. table of content: 0:00 - what is self distillation 2:50 - idan (MIT) and jonas (ETH Zurich) introduction and motivation 18:40 - different perspective of on-policy self-distillation (presentation) 36:00 - metacognition and specificity in self-distillation 37:24 - very long hard task and self-distillation 42:00 - continual learning with self-distillation (presentation) 1:16:50 - what is next in this research direction? 1:20:00 - is there any experience with subjective feedbacks? 1:22:50 - quality vs number of feedbacks? 1:26:40 - what setting would self-distillation struggle vs GRPO? the slides were super crisp really cool of them to share! enjoy my guys 🌹

I had an awesome time interviewing idan shenfeld and Jonas Hübotter from MIT and ETH Zurich about self-distillation. this very promising post-training paradigm where the model acts as its own teacher by conditioning on environment feedback or demonstrations. we cover the SDPO algo for reinforcement learning with rich feedback and SDFT for continual learning without forgetting along with many applications. we dig into how it works, why it's simpler and faster than GRPO, and where this is already showing up in production systems. table of content: 0:00 - what is self distillation 2:50 - idan (MIT) and jonas (ETH Zurich) introduction and motivation 18:40 - different perspective of on-policy self-distillation (presentation) 36:00 - metacognition and specificity in self-distillation 37:24 - very long hard task and self-distillation 42:00 - continual learning with self-distillation (presentation) 1:16:50 - what is next in this research direction? 1:20:00 - is there any experience with subjective feedbacks? 1:22:50 - quality vs number of feedbacks? 1:26:40 - what setting would self-distillation struggle vs GRPO? the slides were super crisp really cool of them to share! enjoy my guys 🌹

Yacine Mahdid

13,060 次观看 • 3 个月前

ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation propose to model the 3D parameter as a random variable instead of a constant as in SDS and present variational score distillation (VSD), a principled particle-based variational framework to explain and address the aforementioned issues in text-to-3D generation. We show that SDS is a special case of VSD and leads to poor samples with both small and large CFG weights. In comparison, VSD works well with various CFG weights as ancestral sampling from diffusion models and simultaneously improves the diversity and sample quality with a common CFG weight (i.e., 7.5). We further present various improvements in the design space for text-to-3D such as distillation time schedule and density initialization, which are orthogonal to the distillation algorithm yet not well explored. Our overall approach, dubbed ProlificDreamer, can generate high rendering resolution (i.e., 512times512) and high-fidelity NeRF with rich structure and complex effects (e.g., smoke and drops). Further, initialized from NeRF, meshes fine-tuned by VSD are meticulously detailed and photo-realistic paper page:

ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation propose to model the 3D parameter as a random variable instead of a constant as in SDS and present variational score distillation (VSD), a principled particle-based variational framework to explain and address the aforementioned issues in text-to-3D generation. We show that SDS is a special case of VSD and leads to poor samples with both small and large CFG weights. In comparison, VSD works well with various CFG weights as ancestral sampling from diffusion models and simultaneously improves the diversity and sample quality with a common CFG weight (i.e., 7.5). We further present various improvements in the design space for text-to-3D such as distillation time schedule and density initialization, which are orthogonal to the distillation algorithm yet not well explored. Our overall approach, dubbed ProlificDreamer, can generate high rendering resolution (i.e., 512times512) and high-fidelity NeRF with rich structure and complex effects (e.g., smoke and drops). Further, initialized from NeRF, meshes fine-tuned by VSD are meticulously detailed and photo-realistic paper page:

AK

46,151 次观看 • 3 年前

DeepSeek Didn’t Catch Up — It Cheated. China’s AI “Breakthrough” Is Built on Stolen American Tech. In this video, I break down explosive allegations that DeepSeek used banned U.S. chips and illicit model distillation to train its newest system. If true, this is IP extraction wrapped in marketing. We examine what distillation really means, why export controls matter, and how dependent China’s AI ecosystem still is on American hardware and research.

DeepSeek Didn’t Catch Up — It Cheated. China’s AI “Breakthrough” Is Built on Stolen American Tech. In this video, I break down explosive allegations that DeepSeek used banned U.S. chips and illicit model distillation to train its newest system. If true, this is IP extraction wrapped in marketing. We examine what distillation really means, why export controls matter, and how dependent China’s AI ecosystem still is on American hardware and research.

Ken Cao-The China Crash Chronicle

12,668 次观看 • 5 个月前

David Sacks says Anthropic could stop Chinese distillation if they wanted to, but they won't because it slows growth "The way that you know that this whole distillation thing is fake, is because if stopping distillation was their primary objective, Anthropic would push to ban Chinese access to American models, not American access to Chinese models." "They’re the ones in the best position to block it. If industrial scale distillation is a national security threat, they’re the ones who need to stop it, because that is the place where distillation occurs. You have to stop it at the source." "They know that if they KYC their customers, it’ll slow their growth. So instead, what they’re saying is, “Hey, ban our competitors.” If they really think it’s that big a threat, they should use a few points of their 90% gross margins to do that." "It seems to me that this debate is all backwards. The question should be on Anthropic to explain why it’s doing such a bad job, not on the whole American open source ecosystem to be punished for Anthropic’s failure."

David Sacks says Anthropic could stop Chinese distillation if they wanted to, but they won't because it slows growth "The way that you know that this whole distillation thing is fake, is because if stopping distillation was their primary objective, Anthropic would push to ban Chinese access to American models, not American access to Chinese models." "They’re the ones in the best position to block it. If industrial scale distillation is a national security threat, they’re the ones who need to stop it, because that is the place where distillation occurs. You have to stop it at the source." "They know that if they KYC their customers, it’ll slow their growth. So instead, what they’re saying is, “Hey, ban our competitors.” If they really think it’s that big a threat, they should use a few points of their 90% gross margins to do that." "It seems to me that this debate is all backwards. The question should be on Anthropic to explain why it’s doing such a bad job, not on the whole American open source ecosystem to be punished for Anthropic’s failure."

dnap

191,137 次观看 • 22 天前

Sequoia's Dean Meyer: "If distillation is real, the question is, how do we address the asymmetry?" "Western companies aren't going to set up 35,000 proxy accounts to steal from Anthropic." "There *is* a structural asymmetry, and it's something we have to think about." "The gap between closed and open source has closed a lot." "And the key conversation online, over the past week, was distillation the reason for this? Or is there some other, mysterious reason?" "The truth is, from the conversations we've had, is that it's somewhere in the middle." "Whether it's 20% or 40%, it doesn't really matter as much. But it's meaningful that we know it's happening." "We know from the labs that there's 10s of millions of queries through these large-scale proxy distillation accounts." "We also know, if you speak to anyone who's leading in synthetic data gen on pre- or post-training at the labs, they will tell you that even if you do hardcore KYC, and then just get access to the output (let's assume you just get the output from the prompt), it is still extremely valuable." "That is 100% true. We know distillation is real."

Sequoia's Dean Meyer: "If distillation is real, the question is, how do we address the asymmetry?" "Western companies aren't going to set up 35,000 proxy accounts to steal from Anthropic." "There is a structural asymmetry, and it's something we have to think about." "The gap between closed and open source has closed a lot." "And the key conversation online, over the past week, was distillation the reason for this? Or is there some other, mysterious reason?" "The truth is, from the conversations we've had, is that it's somewhere in the middle." "Whether it's 20% or 40%, it doesn't really matter as much. But it's meaningful that we know it's happening." "We know from the labs that there's 10s of millions of queries through these large-scale proxy distillation accounts." "We also know, if you speak to anyone who's leading in synthetic data gen on pre- or post-training at the labs, they will tell you that even if you do hardcore KYC, and then just get access to the output (let's assume you just get the output from the prompt), it is still extremely valuable." "That is 100% true. We know distillation is real."

TBPN

43,027 次观看 • 20 天前

Jensen Huang on "distillation" On his new interview with axios, he was asked this question "Should open source model companies be allowed to distill closed models" "Distillation—learning from AI, learning from other people, and learning from other sources of knowledge, is fundamental to intelligence. We are constantly learning from other people. I am learning from you through the questions you are asking, and you are learning from me. All day long, we are learning from one another. AI also has to learn from something. The original AI models, whether they were open or closed, were trained on previously created knowledge from the internet. Now, AI is generating more content than humans. In a few more years, the internet could be 99% AI-generated content, and that content will have been created by some form of AI. As a result, AI systems will constantly be distilling knowledge and intelligence from other AI systems. The fact that AI can learn is a good thing. We want AI systems to be intelligent because a smarter AI can also be a safer AI." ---- From "Axios" YouTube channel, (full video link in comment)

Jensen Huang on "distillation" On his new interview with axios, he was asked this question "Should open source model companies be allowed to distill closed models" "Distillation—learning from AI, learning from other people, and learning from other sources of knowledge, is fundamental to intelligence. We are constantly learning from other people. I am learning from you through the questions you are asking, and you are learning from me. All day long, we are learning from one another. AI also has to learn from something. The original AI models, whether they were open or closed, were trained on previously created knowledge from the internet. Now, AI is generating more content than humans. In a few more years, the internet could be 99% AI-generated content, and that content will have been created by some form of AI. As a result, AI systems will constantly be distilling knowledge and intelligence from other AI systems. The fact that AI can learn is a good thing. We want AI systems to be intelligent because a smarter AI can also be a safer AI." ---- From "Axios" YouTube channel, (full video link in comment)

Rohan Paul

441,746 次观看 • 23 天前

Palantir CTO Shyam Sankar says Chinese open source models pose a "threat to American Prosperity": "These Chinese open source models are really the result of distillation attacks." "Most of this is just stolen American IP from frontier labs, and I think the frontier labs should do more to protect that IP, and that's in their own economic interest." "The gravest threat to us is not homicide, it's suicide." "It's an inability to turn these tokens into economic value and generate American prosperity for it." "You see that with the proposed moratorium in New York on data centers. Turning our back on AI would be as consequential a mistake as turning our back on the atom in the '70s." "It's just the beginning, but we're starting to see a burgeoning US open model ecosystem forming." "NVIDIA's Nemotron models are very good." "I think you're going to see in the next month a slate of announcements of other American companies, non-frontier labs, neo labs, who are putting out open weight models that companies are going to be able to cheaply fine-tune and capture their own alpha in weights they control." Via Bloomberg

Palantir CTO Shyam Sankar says Chinese open source models pose a "threat to American Prosperity": "These Chinese open source models are really the result of distillation attacks." "Most of this is just stolen American IP from frontier labs, and I think the frontier labs should do more to protect that IP, and that's in their own economic interest." "The gravest threat to us is not homicide, it's suicide." "It's an inability to turn these tokens into economic value and generate American prosperity for it." "You see that with the proposed moratorium in New York on data centers. Turning our back on AI would be as consequential a mistake as turning our back on the atom in the '70s." "It's just the beginning, but we're starting to see a burgeoning US open model ecosystem forming." "NVIDIA's Nemotron models are very good." "I think you're going to see in the next month a slate of announcements of other American companies, non-frontier labs, neo labs, who are putting out open weight models that companies are going to be able to cheaply fine-tune and capture their own alpha in weights they control." Via Bloomberg

Jawwwn

180,421 次观看 • 29 天前

What happens to Chinese AI once American companies stop releasing frontier models internationally? Gavin Baker says that China's made a "terrible mistake" by relying on its own chips and distilling American models: "It feels like the administration has been willing to let them buy H200s or B30s. And I think [the Chinese] have this crazy belief that, 'Oh, our own internal chips are good enough.' They're not." "And what's making them think that is that Chinese labs are very, very good at distillation...we've all seen those iPhone farms in China. You know they've got the same thing for distillation. They've got 100,000 endpoints distilling these models across every API available." "They've gotten really, really good at that. But all that goes away if people stop releasing these models at the frontier. And I think Mythos is a sign of things to come there."

What happens to Chinese AI once American companies stop releasing frontier models internationally? Gavin Baker says that China's made a "terrible mistake" by relying on its own chips and distilling American models: "It feels like the administration has been willing to let them buy H200s or B30s. And I think [the Chinese] have this crazy belief that, 'Oh, our own internal chips are good enough.' They're not." "And what's making them think that is that Chinese labs are very, very good at distillation...we've all seen those iPhone farms in China. You know they've got the same thing for distillation. They've got 100,000 endpoints distilling these models across every API available." "They've gotten really, really good at that. But all that goes away if people stop releasing these models at the frontier. And I think Mythos is a sign of things to come there."

TBPN

122,701 次观看 • 2 个月前

What happens when robot world models learn from human experience at scale? 🤔 DreamDojo from NVIDIA Research is a generalist robot world model pretrained on 44K hours of egocentric human videos and then post-trained on robot data to generalize across new objects and environments. After distillation, it runs at 10 FPS for live teleoperation, policy evaluation, and model-based planning. Read the ICML paper to learn more 📄

What happens when robot world models learn from human experience at scale? 🤔 DreamDojo from NVIDIA Research is a generalist robot world model pretrained on 44K hours of egocentric human videos and then post-trained on robot data to generalize across new objects and environments. After distillation, it runs at 10 FPS for live teleoperation, policy evaluation, and model-based planning. Read the ICML paper to learn more 📄

NVIDIA Robotics

22,882 次观看 • 1 个月前

Sam on Kimi, distillation, and open source: “I have always assumed that there are going to be great cheap models in the world, and we better be the greatest and the cheapest. You get a better deal today, at least at a particular latency, using OpenAI's models than Kimi. We distill our own models. That’s how we make smaller, cheaper models. That’s a very good thing to do. There will clearly be an important place for open source models in the world and people that will want their own weights. We have so much usage of our models that we do not need to be a gigantically high-margin business to be able to afford model training. I would rather people not distill from us, for sure. But this is not in my top ten list of worries.”

Sam on Kimi, distillation, and open source: “I have always assumed that there are going to be great cheap models in the world, and we better be the greatest and the cheapest. You get a better deal today, at least at a particular latency, using OpenAI's models than Kimi. We distill our own models. That’s how we make smaller, cheaper models. That’s a very good thing to do. There will clearly be an important place for open source models in the world and people that will want their own weights. We have so much usage of our models that we do not need to be a gigantically high-margin business to be able to afford model training. I would rather people not distill from us, for sure. But this is not in my top ten list of worries.”

Patrick OShaughnessy

269,240 次观看 • 19 天前

Inferact Co-founder Simon Mo on the trajectory of open and closed models: "What really differentiates open-weight models from closed-weight models, right? In the end, there's not much differentiation. It's more about the distribution strategy and go-to-market strategy... Capability-wise, I don't really see a big gap, not even today." "It's not about just source data. It's not about where to get the data from. Rather, it's who can build the best environment and who can make the most optimization and algorithmic choices to leverage all this learning from this environment." "So the next year is all gonna be about that. It's about how open-weight model labs are differentiating and really getting the model to meet the real world." "Project out in a year, there's not gonna be any difference." Simon Mo

Inferact Co-founder Simon Mo on the trajectory of open and closed models: "What really differentiates open-weight models from closed-weight models, right? In the end, there's not much differentiation. It's more about the distribution strategy and go-to-market strategy... Capability-wise, I don't really see a big gap, not even today." "It's not about just source data. It's not about where to get the data from. Rather, it's who can build the best environment and who can make the most optimization and algorithmic choices to leverage all this learning from this environment." "So the next year is all gonna be about that. It's about how open-weight model labs are differentiating and really getting the model to meet the real world." "Project out in a year, there's not gonna be any difference." Simon Mo

a16z

33,782 次观看 • 10 天前

Senior Fellow at CFR Chris McGuire, who crafted the Biden administration's chip export controls, on why he considers Chinese distillation of US models a form of IP theft, and how to stop it "There's a difference between distillation a company does within itself, which is standard accepted practice, and attacking another model by engaging in distillation that is explicitly against the terms of service. That's what we're talking about here." "It's pretty clear this is providing material uplift to China's AI capabilities, significantly shortening the US lead. This is effectively stealing US IP. The leading labs are spending hundreds of billions of dollars, and the Chinese are able to just back out a significant part of the training process." "You can't distill your way all the way from zero to a fully trained model. It's only part of the process. The pre-training, the other parts, you still need to do yourself, and you still need a lot of computing power." "They're not gonna be able to distill their way all the way up the ladder, so if they're gonna get partway up, let's make that as hard as possible, and then let's take out the rungs on the whole rest of the ladder." Chris McGuire Glinert 🇺🇸 🏭

Senior Fellow at CFR Chris McGuire, who crafted the Biden administration's chip export controls, on why he considers Chinese distillation of US models a form of IP theft, and how to stop it "There's a difference between distillation a company does within itself, which is standard accepted practice, and attacking another model by engaging in distillation that is explicitly against the terms of service. That's what we're talking about here." "It's pretty clear this is providing material uplift to China's AI capabilities, significantly shortening the US lead. This is effectively stealing US IP. The leading labs are spending hundreds of billions of dollars, and the Chinese are able to just back out a significant part of the training process." "You can't distill your way all the way from zero to a fully trained model. It's only part of the process. The pre-training, the other parts, you still need to do yourself, and you still need a lot of computing power." "They're not gonna be able to distill their way all the way up the ladder, so if they're gonna get partway up, let's make that as hard as possible, and then let's take out the rungs on the whole rest of the ladder." Chris McGuire Glinert 🇺🇸 🏭

MTS

10,901 次观看 • 25 天前

🚨David Sacks: There is no White House decision to ban open source models. Trump is listening to a chorus of voices — and Sacks says it would be a tragic mistake to take action against the open source ecosystem. His argument: Anthropic is the fastest growing tech company at scale in history. $10B ARR at the start of the year. $70B now. This is not a company that needs government protection. And yet they've spent enormous energy trying to panic Washington into banning American developers from using Chinese open source models. If distillation by Chinese companies was really their primary concern — a genuine national security threat — they would push to ban Chinese access to American models. Instead they're pushing to ban American access to Chinese models. Because this helps Anthropic's competitive position. Sacks: "That's the way you know the whole distillation thing is fake." He went further. If distillation is happening at industrial scale, it's obvious to see. Anthropic has 90% gross margins. Why aren't they stopping it at the source? Answer: because KYC'ing their customers would slow their growth. So instead of fixing their own house, they're asking the government to kneecap their competitors. "The question should be on Anthropic to explain why it's doing such a bad job — not on the whole American open source ecosystem to be punished for Anthropic's failure." Trump's AI Czar on The All-In Podcast.

🚨David Sacks: There is no White House decision to ban open source models. Trump is listening to a chorus of voices — and Sacks says it would be a tragic mistake to take action against the open source ecosystem. His argument: Anthropic is the fastest growing tech company at scale in history. $10B ARR at the start of the year. $70B now. This is not a company that needs government protection. And yet they've spent enormous energy trying to panic Washington into banning American developers from using Chinese open source models. If distillation by Chinese companies was really their primary concern — a genuine national security threat — they would push to ban Chinese access to American models. Instead they're pushing to ban American access to Chinese models. Because this helps Anthropic's competitive position. Sacks: "That's the way you know the whole distillation thing is fake." He went further. If distillation is happening at industrial scale, it's obvious to see. Anthropic has 90% gross margins. Why aren't they stopping it at the source? Answer: because KYC'ing their customers would slow their growth. So instead of fixing their own house, they're asking the government to kneecap their competitors. "The question should be on Anthropic to explain why it's doing such a bad job — not on the whole American open source ecosystem to be punished for Anthropic's failure." Trump's AI Czar on The All-In Podcast.

KanekoaTheGreat

48,358 次观看 • 23 天前

“AI MODELS COULD BECOME 10× SMALLER WITHOUT LOSING MOST OF THEIR INTELLIGENCE.” Mark Zuckerberg says one of the most powerful developments in AI is model distillation: taking the capabilities of a much larger model and transferring them into a much smaller one. - Meta's huge frontier models can serve as "teachers" for smaller models. - Zuckerberg says a model could potentially retain around 90–95% of the larger model's intelligence while being only ~10% of its size. - That means dramatically lower inference costs, memory requirements and computing power. - Developers can distill and customize these capabilities for their own applications. And that's where open-source models come into the picture.

“AI MODELS COULD BECOME 10× SMALLER WITHOUT LOSING MOST OF THEIR INTELLIGENCE.” Mark Zuckerberg says one of the most powerful developments in AI is model distillation: taking the capabilities of a much larger model and transferring them into a much smaller one. - Meta's huge frontier models can serve as "teachers" for smaller models. - Zuckerberg says a model could potentially retain around 90–95% of the larger model's intelligence while being only ~10% of its size. - That means dramatically lower inference costs, memory requirements and computing power. - Developers can distill and customize these capabilities for their own applications. And that's where open-source models come into the picture.

SciTech Era

76,479 次观看 • 7 天前

Announcing the Open Model Series: a new series on open source AI models, with merve of Hugging Face. Hugging Face is the open model hub: it hosts model weights, datasets, evals, & demo apps, and maintains the open-source libraries (like transformers) used to run and train them. I think of it like GitHub, but for AI models. You can download open models and run them on your own machines. In this series we'll cover what open source and open weight models are, how they work, how to run them yourself, and why they're important. By the way - this video is edited entirely with Claude! I can make a tutorial on how I did it. If there's a question you want answered about open models, leave it in the comments. We will read them all. Episode 1 is out soon.

Announcing the Open Model Series: a new series on open source AI models, with merve of Hugging Face. Hugging Face is the open model hub: it hosts model weights, datasets, evals, & demo apps, and maintains the open-source libraries (like transformers) used to run and train them. I think of it like GitHub, but for AI models. You can download open models and run them on your own machines. In this series we'll cover what open source and open weight models are, how they work, how to run them yourself, and why they're important. By the way - this video is edited entirely with Claude! I can make a tutorial on how I did it. If there's a question you want answered about open models, leave it in the comments. We will read them all. Episode 1 is out soon.

Harper Carroll

30,392 次观看 • 13 天前