Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

Andrej Karpathy: Internet training data is terrible, so big models end up compressing "memory" instead of doing cognitive work Use intelligent models to filter to the cognitive core With cleaner data, smaller models, likely distilled from a stronger one, are enough

Haider.

68,657 subscribers

384,349 Aufrufe • vor 11 Monaten •via X (Twitter)

Wissenschaft & Technologie Nachrichten & Politik Bildung

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

Andrej Karpathy just made one of the most interesting arguments about AI model design that most people are completely missing. His take is that frontier AI models are not too big because the technology is complex and too big because the training data is garbage. When you or I think of the internet, we picture Wall Street Journal articles, Wikipedia entries, serious writing. That is not what a pretraining dataset looks like. When researchers at frontier labs look at random documents from the actual training corpus, it is stock ticker symbols, broken HTML, spam, gibberish. One estimate puts Llama 3's information compression at just 0.07 bits per token meaning the model has only a hazy recollection of most of what it trained on. So we build trillion parameter models not because we need a trillion parameter brain but because we need a trillion-parameter compression engine to squeeze some intelligence out of a firehose of noise. Most of those parameters are doing memory work, not cognitive work. Karpathy's prediction is separate the two entirely. Build a cognitive core, a model that contains only the algorithms for reasoning and problem-solving, stripped of encyclopedic memorization and pair it with external memory that it can query when it needs facts. He thinks a cognitive core trained on high-quality data could hit genuine intelligence at around one billion parameters. For reference, today's flagship models run between 200 billion and 1.8 trillion parameters with most of that weight dedicated to remembering the internet's slop. The trend is already moving his direction. GPT-4o operates at roughly 200 billion parameters and outperforms the original 1.8 trillion-parameter GPT-4. Inference costs for GPT-3.5-level performance dropped 280-fold between 2022 and 2024 driven almost entirely by smaller, cleaner, better-architected models. The real bottleneck in AI right now is not compute but rather data quality.

Andrej Karpathy just made one of the most interesting arguments about AI model design that most people are completely missing. His take is that frontier AI models are not too big because the technology is complex and too big because the training data is garbage. When you or I think of the internet, we picture Wall Street Journal articles, Wikipedia entries, serious writing. That is not what a pretraining dataset looks like. When researchers at frontier labs look at random documents from the actual training corpus, it is stock ticker symbols, broken HTML, spam, gibberish. One estimate puts Llama 3's information compression at just 0.07 bits per token meaning the model has only a hazy recollection of most of what it trained on. So we build trillion parameter models not because we need a trillion parameter brain but because we need a trillion-parameter compression engine to squeeze some intelligence out of a firehose of noise. Most of those parameters are doing memory work, not cognitive work. Karpathy's prediction is separate the two entirely. Build a cognitive core, a model that contains only the algorithms for reasoning and problem-solving, stripped of encyclopedic memorization and pair it with external memory that it can query when it needs facts. He thinks a cognitive core trained on high-quality data could hit genuine intelligence at around one billion parameters. For reference, today's flagship models run between 200 billion and 1.8 trillion parameters with most of that weight dedicated to remembering the internet's slop. The trend is already moving his direction. GPT-4o operates at roughly 200 billion parameters and outperforms the original 1.8 trillion-parameter GPT-4. Inference costs for GPT-3.5-level performance dropped 280-fold between 2022 and 2024 driven almost entirely by smaller, cleaner, better-architected models. The real bottleneck in AI right now is not compute but rather data quality.

Milk Road AI

200,719 Aufrufe • vor 5 Monaten

The First 100 begins now. Verify Quality is live on PrismaX. For the first time, anyone can score the robot training data that models learn from, earn points, and compete to become one of The First 100. Better data. Better models. The standard starts with you.

The First 100 begins now. Verify Quality is live on PrismaX. For the first time, anyone can score the robot training data that models learn from, earn points, and compete to become one of The First 100. Better data. Better models. The standard starts with you.

PrismaX

232,379 Aufrufe • vor 3 Monaten

💡Why Vana? “Ultimately, AI models are only as good as their training data. So if you want to build the best AI, you need the best data.” shares the vision behind Vana’s mission, centered on DataDAOs that aggregate specific datasets, reward data owners for their contributions and level up AI models with high-quality data🏆

💡Why Vana? “Ultimately, AI models are only as good as their training data. So if you want to build the best AI, you need the best data.” shares the vision behind Vana’s mission, centered on DataDAOs that aggregate specific datasets, reward data owners for their contributions and level up AI models with high-quality data🏆

vana

108,377 Aufrufe • vor 1 Jahr

Yoshua Bengio says AI does not create enough new jobs to balance the ones it replaces A handful of engineers earn huge salaries, while a vast number of workers face displacement as models master cognitive tasks "if we automate most of the cognitive work, what's gonna be left?"

Yoshua Bengio says AI does not create enough new jobs to balance the ones it replaces A handful of engineers earn huge salaries, while a vast number of workers face displacement as models master cognitive tasks "if we automate most of the cognitive work, what's gonna be left?"

Haider.

151,063 Aufrufe • vor 9 Monaten

This is the best way to understand how ML models actually work! Use Drawdata to draw a 2D dataset in Jupyter. Use it to actively pick data from the widget and update the model as the data is being drawn! Fully interactive, real-time, and open-source!

This is the best way to understand how ML models actually work! Use Drawdata to draw a 2D dataset in Jupyter. Use it to actively pick data from the widget and update the model as the data is being drawn! Fully interactive, real-time, and open-source!

Daily Dose of Data Science

52,103 Aufrufe • vor 10 Monaten

Hot tip for anyone doing AI dev: Use Ollama to easily run models like Deepseek-r1 or Gemma locally on your machine. It downloads them and spins up a server with an OpenAI SDK compatible API The smaller models are fast and good enough to work on new features or debug streaming without having to pay for API requests

Hot tip for anyone doing AI dev: Use Ollama to easily run models like Deepseek-r1 or Gemma locally on your machine. It downloads them and spins up a server with an OpenAI SDK compatible API The smaller models are fast and good enough to work on new features or debug streaming without having to pay for API requests

Wes Bos

153,499 Aufrufe • vor 1 Jahr

Data ownership & privacy are key to building an internet that benefits & protects its end-users AI is no different — Users’ data must be protected while interacting with AI Oasis' privacy-preserving smart contracts enable users to contribute data to AI models on their own terms

Data ownership & privacy are key to building an internet that benefits & protects its end-users AI is no different — Users’ data must be protected while interacting with AI Oasis' privacy-preserving smart contracts enable users to contribute data to AI models on their own terms

Oasis Labs

32,852 Aufrufe • vor 3 Jahren

A large portion of animal intelligence doesn't require any learning, claims Andrej Karpathy: it's baked into DNA. AI models, by contrast, start from random weights. They have to learn their intelligence, mostly by imitating the internet. This is so different that Andrej thinks it's a fundamentally different kind of intelligence: LLMs are more like ghosts than animals.

A large portion of animal intelligence doesn't require any learning, claims Andrej Karpathy: it's baked into DNA. AI models, by contrast, start from random weights. They have to learn their intelligence, mostly by imitating the internet. This is so different that Andrej thinks it's a fundamentally different kind of intelligence: LLMs are more like ghosts than animals.

Dwarkesh Patel

82,232 Aufrufe • vor 4 Monaten

Over the last few months, we’ve been thinking about how to learn from “off-domain” data - data from non-robot sources like video or simulation. These data sources are not quite good enough to learn policies (even monolithic VLA models) directly, but they still contain lots of information that can be useful for generalizable robot control. How can we develop robot learning models that are able to make use of this type of data for generalizable control? In new work, that we call HAMSTER, we show that VLMs can be useful for enabling robotic learning from off-domain data, but specifically when used through hierarchical VLA architectures. We show that this class of models can learn generalizable robot policies for the real world from large-scale, off-domain data. A 🧵 (1/10)

Over the last few months, we’ve been thinking about how to learn from “off-domain” data - data from non-robot sources like video or simulation. These data sources are not quite good enough to learn policies (even monolithic VLA models) directly, but they still contain lots of information that can be useful for generalizable robot control. How can we develop robot learning models that are able to make use of this type of data for generalizable control? In new work, that we call HAMSTER, we show that VLMs can be useful for enabling robotic learning from off-domain data, but specifically when used through hierarchical VLA architectures. We show that this class of models can learn generalizable robot policies for the real world from large-scale, off-domain data. A 🧵 (1/10)

Abhishek Gupta

11,994 Aufrufe • vor 1 Jahr

The AV industry is hitting a “Data Wall.” Brute-force training on petabytes of data is reaching a dead end. As models improve, further gains get harder. We bypassed this wall with Factored Embodied AI—achieving zero-shot autonomous steering with just 1,000 hours of driving data.

The AV industry is hitting a “Data Wall.” Brute-force training on petabytes of data is reaching a dead end. As models improve, further gains get harder. We bypassed this wall with Factored Embodied AI—achieving zero-shot autonomous steering with just 1,000 hours of driving data.

Helm.ai

8,650,862 Aufrufe • vor 9 Monaten

Curious whether video generation models (like #SORA) qualify as world models? We conduct a systematic study to answer this question by investigating whether a video gen model is able to learn physical laws. Three are three key messages to take home: 1⃣The model generalises perfectly for in-distribution data, but fails to do out-of-distribution generalization. For combinatorial scenarios, scaling law is observed. 2⃣The models fail to abstract general rules and instead tries to mimic the closest training example. 3⃣The model prioritizes different attributes when referencing training data: color > size > velocity > shape. This work is a joint effort with our outstanding intern Yang Yue. Paper: Webpage:

Curious whether video generation models (like #SORA) qualify as world models? We conduct a systematic study to answer this question by investigating whether a video gen model is able to learn physical laws. Three are three key messages to take home: 1⃣The model generalises perfectly for in-distribution data, but fails to do out-of-distribution generalization. For combinatorial scenarios, scaling law is observed. 2⃣The models fail to abstract general rules and instead tries to mimic the closest training example. 3⃣The model prioritizes different attributes when referencing training data: color > size > velocity > shape. This work is a joint effort with our outstanding intern Yang Yue. Paper: Webpage:

Bingyi Kang

606,981 Aufrufe • vor 1 Jahr

BREAKING: AI has eaten the Internet. Data labeling is so over. & $30 trillion of human work is on the verge of automation. Inside The $2.2B AI Research Accelerator, Turing Founder & CEO, Jonathan Siddharth (), joins Sourcery to break down the severe power shift in AI training: from commodity data labeling → expert research Positioning Turing apart from AI data providers like Scale AI, Mercor, & Surge. (00:00) AI Ate The Internet (00:49) Training Superintelligence: the race to AGI (02:31) Viral tweet (03:24) What Turing actually does (04:43) The internet data is “used up” — where will new data come from? (05:34) Four pillars of superintelligence: multimodality, reasoning, tool use, coding (06:07) Automating $30T of global knowledge work (09:18) The $1B revenue opportunity (10:59) Why Turing is a research-first accelerator, not a data labeler (13:45) Jonathan’s Stanford AI Lab roots & founding DNA (17:57) How models are built: pre-training vs. post-training (20:14) RLHF, reinforcement learning, & “breaking the models” (25:19) GPT-5 and the myth of rapid takeoff (30:46) Safety debates and human-in-the-loop systems (34:53) Closing Enterprise Gap: finance, insurance, & pharma (39:23) Why proprietary enterprise data is the next moat in AI

BREAKING: AI has eaten the Internet. Data labeling is so over. & $30 trillion of human work is on the verge of automation. Inside The $2.2B AI Research Accelerator, Turing Founder & CEO, Jonathan Siddharth (), joins Sourcery to break down the severe power shift in AI training: from commodity data labeling → expert research Positioning Turing apart from AI data providers like Scale AI, Mercor, & Surge. (00:00) AI Ate The Internet (00:49) Training Superintelligence: the race to AGI (02:31) Viral tweet (03:24) What Turing actually does (04:43) The internet data is “used up” — where will new data come from? (05:34) Four pillars of superintelligence: multimodality, reasoning, tool use, coding (06:07) Automating $30T of global knowledge work (09:18) The $1B revenue opportunity (10:59) Why Turing is a research-first accelerator, not a data labeler (13:45) Jonathan’s Stanford AI Lab roots & founding DNA (17:57) How models are built: pre-training vs. post-training (20:14) RLHF, reinforcement learning, & “breaking the models” (25:19) GPT-5 and the myth of rapid takeoff (30:46) Safety debates and human-in-the-loop systems (34:53) Closing Enterprise Gap: finance, insurance, & pharma (39:23) Why proprietary enterprise data is the next moat in AI

Molly O’Shea

70,094 Aufrufe • vor 11 Monaten

⚛️ TxGemma is a collection of open models designed to accelerate therapeutic data analysis with the power of @GoogleDeepmind’s Gemma.

⚛️ TxGemma is a collection of open models designed to accelerate therapeutic data analysis with the power of @GoogleDeepmind’s Gemma.

Google AI Developers

127,027 Aufrufe • vor 1 Jahr

Geoffrey Hinton says the current path of scaling is hitting a limit Most high-value data is locked inside companies, and the "free internet" is largely exhausted The solution is for models to generate their own training data through reasoning "that's how AlphaGo beat humans"

Geoffrey Hinton says the current path of scaling is hitting a limit Most high-value data is locked inside companies, and the "free internet" is largely exhausted The solution is for models to generate their own training data through reasoning "that's how AlphaGo beat humans"

Haider.

149,442 Aufrufe • vor 9 Monaten

$WDC CEO says storage demand and requirement compound and grow whether AI runs on US frontier models or cheaper open-source ones. "Whether it's the traditional frontier models that you've seen in the U.S. that are premium-based, or some of the new open-source, more economical models, the underlying requirement is still that they're going to require a lot of data to support the training of these models." "These models are going to generate a lot more data that requires storage for us." "So we view it as very positive, because even as compute maybe drives greater efficiency in terms of compute and memory resources, the requirement for storage is just going to compound and grow." "So we view it very positively for demand going forward."

$WDC CEO says storage demand and requirement compound and grow whether AI runs on US frontier models or cheaper open-source ones. "Whether it's the traditional frontier models that you've seen in the U.S. that are premium-based, or some of the new open-source, more economical models, the underlying requirement is still that they're going to require a lot of data to support the training of these models." "These models are going to generate a lot more data that requires storage for us." "So we view it as very positive, because even as compute maybe drives greater efficiency in terms of compute and memory resources, the requirement for storage is just going to compound and grow." "So we view it very positively for demand going forward."

Fireside Alpha

14,114 Aufrufe • vor 1 Monat

OpenAI researcher Lukasz Kaiser explains why he's optimistic AI can remain truly open: intelligence may ultimately work better as many distributed models than one giant model "If you look at the world, humans are the most amazing computers in the world. Our brains are still unmatched by the models, and we are not generalists, and there is not one big brain of humanity. It's a lot of separate people." "Given a fixed amount of data, the best way to learn from it is to have a lot of distributed models that are strong on its own, but even stronger when they're, If you look at some basic research, there are actually reasons to believe it, like ensembles and things like that." "Transformers are great when you feed them all of the internet, but we know there is an algorithm that's even better and it can learn from much smaller data and be amazing at the things it's learning. We are the proof. We have not found it yet, but we will." "All these big models, they're all the same, they're kind of boring. I think they'll be much more fun." OpenAI

OpenAI researcher Lukasz Kaiser explains why he's optimistic AI can remain truly open: intelligence may ultimately work better as many distributed models than one giant model "If you look at the world, humans are the most amazing computers in the world. Our brains are still unmatched by the models, and we are not generalists, and there is not one big brain of humanity. It's a lot of separate people." "Given a fixed amount of data, the best way to learn from it is to have a lot of distributed models that are strong on its own, but even stronger when they're, If you look at some basic research, there are actually reasons to believe it, like ensembles and things like that." "Transformers are great when you feed them all of the internet, but we know there is an algorithm that's even better and it can learn from much smaller data and be amazing at the things it's learning. We are the proof. We have not found it yet, but we will." "All these big models, they're all the same, they're kind of boring. I think they'll be much more fun." OpenAI

MTS

15,860 Aufrufe • vor 24 Tagen

World Labs CEO Dr. Fei-Fei Li & SceniX Co-Founder Yunzhu Li on the data bottleneck in robotics and how world models help: "The lack of data in training, the lack of data in evaluation, this is very, very different from language models, where data is abundant on the internet." "And we know that in order for robotics to work, we have to somehow unlock the power of scaling law. But where does that come from?" "It's a profound problem that everybody's battling with in robotics." "We see a lot of unlock in being able to do this whole process through the modeling of the environments." "Being able to create these digital worlds... that's just going to unlock so much more potential for being able to replace all the costly and the unsafe data in the real environments with the data generated from the worlds for robots to be able to do scalable learning and evaluations." Fei-Fei Li Yunzhu Li martin_casado

World Labs CEO Dr. Fei-Fei Li & SceniX Co-Founder Yunzhu Li on the data bottleneck in robotics and how world models help: "The lack of data in training, the lack of data in evaluation, this is very, very different from language models, where data is abundant on the internet." "And we know that in order for robotics to work, we have to somehow unlock the power of scaling law. But where does that come from?" "It's a profound problem that everybody's battling with in robotics." "We see a lot of unlock in being able to do this whole process through the modeling of the environments." "Being able to create these digital worlds... that's just going to unlock so much more potential for being able to replace all the costly and the unsafe data in the real environments with the data generated from the worlds for robots to be able to do scalable learning and evaluations." Fei-Fei Li Yunzhu Li martin_casado

a16z

77,065 Aufrufe • vor 2 Monaten

Larry Ellison basically explained why OpenAI is positioned to win the AI race. Today’s models all trained on the same thing: public data. The internet. Books. Papers. General knowledge. Impressive, but capped. Ellison’s point: peak value only comes when AI is trained on private data. That’s where OpenAI is moving: • partnerships with universities to work with unpublished research • experts teaching models real job tasks, not just theory • deep access to how science, medicine, law, and finance actually work Public data makes AI smart. Private data makes it decisive. The next AI leap won’t come from reading more of the internet. It’ll come from being invited behind the curtain. That’s the real moat.

Larry Ellison basically explained why OpenAI is positioned to win the AI race. Today’s models all trained on the same thing: public data. The internet. Books. Papers. General knowledge. Impressive, but capped. Ellison’s point: peak value only comes when AI is trained on private data. That’s where OpenAI is moving: • partnerships with universities to work with unpublished research • experts teaching models real job tasks, not just theory • deep access to how science, medicine, law, and finance actually work Public data makes AI smart. Private data makes it decisive. The next AI leap won’t come from reading more of the internet. It’ll come from being invited behind the curtain. That’s the real moat.

VraserX e/acc

167,094 Aufrufe • vor 9 Monaten

.Trenton Bricken explains how we know LLMs are actually generalizing - aka they're not just stochastic parrots: - Training models on code makes them better at reasoning in language. - Models fine tuned on math problems become better at entity detection. - We can just straightforwardly read the world-models developed by smaller NNs which are easier to interpret (Othello). Transfer learning shows models are developing a deeper understanding of their data. Full episode out Thursday.

.Trenton Bricken explains how we know LLMs are actually generalizing - aka they're not just stochastic parrots: - Training models on code makes them better at reasoning in language. - Models fine tuned on math problems become better at entity detection. - We can just straightforwardly read the world-models developed by smaller NNs which are easier to interpret (Othello). Transfer learning shows models are developing a deeper understanding of their data. Full episode out Thursday.

Dwarkesh Patel

237,812 Aufrufe • vor 2 Jahren