Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

AI training faces a very different set of trade-offs compared to human evolution. Because you can directly copy a trained model basically for free, it makes sense to amortize in the pre-trained weights the learning that humans spread across a lifetime.

Dwarkesh Patel

227,492 subscribers

25,747 views • 3 months ago •via X (Twitter)

Education Science & Technology

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

An AI model trained on only 2.5 hours of operator data ran a real excavator remotely over Starlink. Two and a half hours of watching a human, and the machine can dig on its own. The labor economy is about to look very different, very fast.

An AI model trained on only 2.5 hours of operator data ran a real excavator remotely over Starlink. Two and a half hours of watching a human, and the machine can dig on its own. The labor economy is about to look very different, very fast.

Mario Nawfal

90,709 views • 3 months ago

How well do egocentric human videos scale? Apparently, really well. Dyna Robotics has introduced Dyna-2, a world-action model (WAM) pre-trained on more than 1 million hours of real egocentric human video. It learns by jointly predicting the next video frames and the actions, and that video prediction (world modeling) is what carries learning from humans to robots. • More human video predictably improves action prediction. • Task accuracy improves across embodiments (bimanual arms and a humanoid) with zero robot data in pre-training. • Post-trained on just a few hours of robot data, mean task performance rose 20% to 53% as human pre-training scaled 1k to 1M hours. WAM vs VLA, matched apple-to-apple: the WAM hit 1.55x the VLA's success rate and won 65% of head-to-heads. Deployed zero-shot at unseen customer sites, it passed 87% vs the VLA's 46%.

How well do egocentric human videos scale? Apparently, really well. Dyna Robotics has introduced Dyna-2, a world-action model (WAM) pre-trained on more than 1 million hours of real egocentric human video. It learns by jointly predicting the next video frames and the actions, and that video prediction (world modeling) is what carries learning from humans to robots. • More human video predictably improves action prediction. • Task accuracy improves across embodiments (bimanual arms and a humanoid) with zero robot data in pre-training. • Post-trained on just a few hours of robot data, mean task performance rose 20% to 53% as human pre-training scaled 1k to 1M hours. WAM vs VLA, matched apple-to-apple: the WAM hit 1.55x the VLA's success rate and won 65% of head-to-heads. Deployed zero-shot at unseen customer sites, it passed 87% vs the VLA's 46%.

The Humanoid Hub

15,034 views • 17 days ago

Many of you asked for code & weights for π₀, we are happy to announce that we are releasing π₀ and pre-trained checkpoints in our new openpi repository! We tested the model on a few public robots, and we include code for you to fine-tune it yourself.

Many of you asked for code & weights for π₀, we are happy to announce that we are releasing π₀ and pre-trained checkpoints in our new openpi repository! We tested the model on a few public robots, and we include code for you to fine-tune it yourself.

Physical Intelligence

441,382 views • 1 year ago

Super clean and efficient meshes by an AI? YES! The typical 3D Generative AI solutions produce lots of artifacts and usually way to many polygons due to volumetric approaches. In comparison “MeshGPT creates triangle meshes by autoregressively sampling from a transformer model that has been trained to produce tokens from a learned geometric vocabulary. These tokens can then be decoded into the faces of a triangle mesh. This method generates clean, coherent, and compact meshes, characterized by sharp edges and high fidelity.” Surely it is limited by the trained vocabulary but various versions can be trained for specific sets to create generative model libraries for certain object groups. Very promising approach with the high quality.

Super clean and efficient meshes by an AI? YES! The typical 3D Generative AI solutions produce lots of artifacts and usually way to many polygons due to volumetric approaches. In comparison “MeshGPT creates triangle meshes by autoregressively sampling from a transformer model that has been trained to produce tokens from a learned geometric vocabulary. These tokens can then be decoded into the faces of a triangle mesh. This method generates clean, coherent, and compact meshes, characterized by sharp edges and high fidelity.” Surely it is limited by the trained vocabulary but various versions can be trained for specific sets to create generative model libraries for certain object groups. Very promising approach with the high quality.

René Schulte

20,772 views • 2 years ago

JUST IN: Dyna Robotics just published one of the most important research papers in robotics this year. It could fundamentally change how robot foundation models are trained. A scaling law that transfers from human video to robot performance. Dyna-2 is out and it's 🔥 Here's what that means in plain terms. Dyna-2 was pre-trained on ONE MILLION hours of egocentric human video, 170 years of continuous human experience, cooking, folding, assembling, cleaning. And as that human data scaled, robot performance improved. Predictably. Monotonically. Across 39 tasks on two different robot embodiments the model had never seen. → 1,000 hours pre-training → 20% normalised task performance → 10,000 hours → 28% → 100,000 hours → 45% → 1,000,000 hours → 53% Human video exists at effectively unlimited scale. Every cook, every factory worker, every craftsperson wearing a camera is generating training data for future robots. But the finding that stunned even the researchers, world modeling is what makes the transfer work. A model trained to predict future video AND actions massively outperforms one trained on actions alone. Video is the new scaling axis for robotics. One more jaw-dropping data point. 13 minutes of teleoperation data was enough to fine-tune Dyna-2 to open a bottle cap using two five-fingered robot hands. The robots are coming, and they're learning from us directly :D Read more here: Congrats Jason Ma and team! ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

JUST IN: Dyna Robotics just published one of the most important research papers in robotics this year. It could fundamentally change how robot foundation models are trained. A scaling law that transfers from human video to robot performance. Dyna-2 is out and it's 🔥 Here's what that means in plain terms. Dyna-2 was pre-trained on ONE MILLION hours of egocentric human video, 170 years of continuous human experience, cooking, folding, assembling, cleaning. And as that human data scaled, robot performance improved. Predictably. Monotonically. Across 39 tasks on two different robot embodiments the model had never seen. → 1,000 hours pre-training → 20% normalised task performance → 10,000 hours → 28% → 100,000 hours → 45% → 1,000,000 hours → 53% Human video exists at effectively unlimited scale. Every cook, every factory worker, every craftsperson wearing a camera is generating training data for future robots. But the finding that stunned even the researchers, world modeling is what makes the transfer work. A model trained to predict future video AND actions massively outperforms one trained on actions alone. Video is the new scaling axis for robotics. One more jaw-dropping data point. 13 minutes of teleoperation data was enough to fine-tune Dyna-2 to open a bottle cap using two five-fingered robot hands. The robots are coming, and they're learning from us directly :D Read more here: Congrats Jason Ma and team! ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

23,373 views • 17 days ago

Satya Nadella, CEO of Microsoft, says every company now runs on two kinds of capital, and leaking one of them is a one-way door: "This economy is going to be shaped going forward by both human capital and, let's call it, this token capital. That is true for Microsoft, that is true for a new startup, that's going to be true for any bank that's been in existence for 100 years." "The interplay between human capital and token capital, and compounding the returns of it, is what you need to do." "What is the tacit knowledge of an enterprise or a firm? It's the unique ways that you're able to operate, pass judgment, have taste." "The model, in some sense, is able to extract that tacit knowledge through human trajectories, and then code it in a set of weights in a model." "Every enterprise now needs to be more mindful about that interplay of humans and their digital estate working together, those trajectories training the models that they keep as IP versus leak it." "Because if you leak it, it's a one-way door. You're done in some sense." The moat was never the data sitting in your warehouse. It was the judgment your people apply to it, and AI now turns that judgment into weights you either own or hand away. Treat your tacit knowledge like a patent, not a prompt. The firms that pipe a hundred years of taste into someone else's frontier model are trading a permanent edge for a quarter of convenience.

Satya Nadella, CEO of Microsoft, says every company now runs on two kinds of capital, and leaking one of them is a one-way door: "This economy is going to be shaped going forward by both human capital and, let's call it, this token capital. That is true for Microsoft, that is true for a new startup, that's going to be true for any bank that's been in existence for 100 years." "The interplay between human capital and token capital, and compounding the returns of it, is what you need to do." "What is the tacit knowledge of an enterprise or a firm? It's the unique ways that you're able to operate, pass judgment, have taste." "The model, in some sense, is able to extract that tacit knowledge through human trajectories, and then code it in a set of weights in a model." "Every enterprise now needs to be more mindful about that interplay of humans and their digital estate working together, those trajectories training the models that they keep as IP versus leak it." "Because if you leak it, it's a one-way door. You're done in some sense." The moat was never the data sitting in your warehouse. It was the judgment your people apply to it, and AI now turns that judgment into weights you either own or hand away. Treat your tacit knowledge like a patent, not a prompt. The firms that pipe a hundred years of taste into someone else's frontier model are trading a permanent edge for a quarter of convenience.

Karl Mehta

34,573 views • 1 month ago

Today, AWS CEO Matt Garman announced Nova Forge, a model builder which lets companies inject their own data during the pre-training phase. "You [tell Forge]: 'Here's my corpus of corporate data, here's everything I need to know about my industry.' We then mix that in and finish pre-training the model. So you get a pre-trained model that totally understands your company and your data."

Today, AWS CEO Matt Garman announced Nova Forge, a model builder which lets companies inject their own data during the pre-training phase. "You [tell Forge]: 'Here's my corpus of corporate data, here's everything I need to know about my industry.' We then mix that in and finish pre-training the model. So you get a pre-trained model that totally understands your company and your data."

TBPN

97,014 views • 8 months ago

In case you missed it, we recently launched "Post-training of LLMs," a short course where you'll: ✅ Understand when and why to use post-training methods like Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Online Reinforcement Learning. ✅ Learn the concepts underlying the three post-training methods of SFT, DPO, and Online RL, their common use-cases, and how to curate high-quality data to effectively train a model using each method. ✅ Download a pre-trained model and implement post-training pipelines to turn a base model into an instruct model, change the identity of a chat assistant, and improve a model’s math capabilities. Learn more and enroll for free:

In case you missed it, we recently launched "Post-training of LLMs," a short course where you'll: ✅ Understand when and why to use post-training methods like Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Online Reinforcement Learning. ✅ Learn the concepts underlying the three post-training methods of SFT, DPO, and Online RL, their common use-cases, and how to curate high-quality data to effectively train a model using each method. ✅ Download a pre-trained model and implement post-training pipelines to turn a base model into an instruct model, change the identity of a chat assistant, and improve a model’s math capabilities. Learn more and enroll for free:

DeepLearning.AI

16,779 views • 1 year ago

AI that had human-level intelligence would actually be way above human level in capability. Obviously, if we trained a human-level AI, we could just run way more instances of it in parallel. This is a huge advantage. But it's not the only one. Right now, LLMs are much less sample-efficient than humans, but they can process each individual piece of data much more quickly. A model that was a human-level learner would be able to gain the equivalent of thousands of years of education. AIs can also be way more task-oriented than humans. Even a really well-motivated human worker can only operate with ‘human-level’ effort for relatively short spans of time. AGI could do it continuously.

AI that had human-level intelligence would actually be way above human level in capability. Obviously, if we trained a human-level AI, we could just run way more instances of it in parallel. This is a huge advantage. But it's not the only one. Right now, LLMs are much less sample-efficient than humans, but they can process each individual piece of data much more quickly. A model that was a human-level learner would be able to gain the equivalent of thousands of years of education. AIs can also be way more task-oriented than humans. Even a really well-motivated human worker can only operate with ‘human-level’ effort for relatively short spans of time. AGI could do it continuously.

Dwarkesh Patel

31,861 views • 3 months ago

Ever wondered what neural networks are and how they work? Systems like ChatGPT use neural networks to work as well as they do. Neural networks are composed of "layers" of neurons, layers with different functions; connections between layers called "weights"; and mathematical functions called "activation functions". If you’re interested in learning about these systems, check the comments. Ultimately, the neural network structure of the model serves to visually demonstrate that it is, in fact, a complex mathematical equation. When companies release the model's weights, they are releasing a key component needed to run the model's complete equation. Without the weights, the equation is incomplete. For the math-minded: the weights of a model are the learned numbers (they are variables during training) that are then used as constants in the mathematical functions that make up the model. Neural networks are ultimately just one big, hyper-complex mathematical function, and when a model is trained, it learns the constants associated with the high-dimensional input.

Ever wondered what neural networks are and how they work? Systems like ChatGPT use neural networks to work as well as they do. Neural networks are composed of "layers" of neurons, layers with different functions; connections between layers called "weights"; and mathematical functions called "activation functions". If you’re interested in learning about these systems, check the comments. Ultimately, the neural network structure of the model serves to visually demonstrate that it is, in fact, a complex mathematical equation. When companies release the model's weights, they are releasing a key component needed to run the model's complete equation. Without the weights, the equation is incomplete. For the math-minded: the weights of a model are the learned numbers (they are variables during training) that are then used as constants in the mathematical functions that make up the model. Neural networks are ultimately just one big, hyper-complex mathematical function, and when a model is trained, it learns the constants associated with the high-dimensional input.

Harper Carroll

34,605 views • 9 months ago

1/ NitroGen: NVIDIA's new image-to-action model! NitroGen, a vision-action foundation model for generalist gaming agents that is trained on 40,000 hours of gameplay videos across more than 1,000 games. Gaming is a significant factor in AI training. Google DeepMind trained AI early on Starcraft 2, and OpenAI on Dota 2. This new product from NVIDIA is therefore extremely important. Why it matters and how it works:

1/ NitroGen: NVIDIA's new image-to-action model! NitroGen, a vision-action foundation model for generalist gaming agents that is trained on 40,000 hours of gameplay videos across more than 1,000 games. Gaming is a significant factor in AI training. Google DeepMind trained AI early on Starcraft 2, and OpenAI on Dota 2. This new product from NVIDIA is therefore extremely important. Why it matters and how it works:

Chubby♨️

55,280 views • 8 months ago

OpenAI just released a new model to distinguish between AI/human written text to protect against ChatGPT. The classifier was trained on a pair of AI/human written dataset. However.. I was easily able to trick it by using GPT3 to rewrite the text. Demo:

OpenAI just released a new model to distinguish between AI/human written text to protect against ChatGPT. The classifier was trained on a pair of AI/human written dataset. However.. I was easily able to trick it by using GPT3 to rewrite the text. Demo:

Lior Alexander

195,429 views • 3 years ago

Helix is now learning directly from human video data We have already trained on data collected in the real world, including Brookfield residential units To our knowledge, this is the first instance of a humanoid robot learning navigation end-to-end using only human video

Helix is now learning directly from human video data We have already trained on data collected in the real world, including Brookfield residential units To our knowledge, this is the first instance of a humanoid robot learning navigation end-to-end using only human video

Figure

46,041 views • 11 months ago

The AI industry is optimizing for a definition of intelligence that does not exist. Andrew Ng just said it out loud. Ng: “AGI, to me, should be less about AI that already knows everything under the sun. That seems very challenging, doesn’t seem practical.” The human brain is not the most powerful economic asset in history because of what it holds. It is powerful because of what it can pick up. Ng: “The amazing thing about the human brain is its plasticity, or its ability to learn.” That same biological hardware that earns a PhD in quantum physics could have been trained on chess, surgery, or rewriting global supply chains from scratch. Ng: “That same human brain, just given different training, could have been a chess master, or could have been amazing at playing tennis.” General intelligence is not omniscience. It is the structural capacity to master whatever you point it at. Ng: “It is through learning that we then gain these incredibly specialized intelligences.” The winner is not whoever builds the biggest model. It is whoever builds the most adaptable one. The AI that walks into a domain it has never touched and executes before a human analyst finishes reading the brief. Ng: “What makes the human brain so valuable for economic tasks, is its ability to just learn to do whatever is needed.” Every corporation on earth pays for human labor because humans adapt. Not because they already know everything. AGI is the digitization of that exact capability. At machine speed. At infinite scale. Ng: “A lot of what makes the human brain so general is not that my brain or your brain already knows everything under the sun. It’s our ability to adapt, to learn a huge range of things.” The most powerful economic asset in history was never specialized knowledge. It was the raw capacity to acquire any knowledge, in any domain, on demand. The winning AI is not an encyclopedia. It is the force that makes encyclopedias irrelevant. And once that exists, the question stops being what the AI knows. It becomes what you can teach it before your competitor wakes up. Most people dominating this conversation have not understood that yet.

The AI industry is optimizing for a definition of intelligence that does not exist. Andrew Ng just said it out loud. Ng: “AGI, to me, should be less about AI that already knows everything under the sun. That seems very challenging, doesn’t seem practical.” The human brain is not the most powerful economic asset in history because of what it holds. It is powerful because of what it can pick up. Ng: “The amazing thing about the human brain is its plasticity, or its ability to learn.” That same biological hardware that earns a PhD in quantum physics could have been trained on chess, surgery, or rewriting global supply chains from scratch. Ng: “That same human brain, just given different training, could have been a chess master, or could have been amazing at playing tennis.” General intelligence is not omniscience. It is the structural capacity to master whatever you point it at. Ng: “It is through learning that we then gain these incredibly specialized intelligences.” The winner is not whoever builds the biggest model. It is whoever builds the most adaptable one. The AI that walks into a domain it has never touched and executes before a human analyst finishes reading the brief. Ng: “What makes the human brain so valuable for economic tasks, is its ability to just learn to do whatever is needed.” Every corporation on earth pays for human labor because humans adapt. Not because they already know everything. AGI is the digitization of that exact capability. At machine speed. At infinite scale. Ng: “A lot of what makes the human brain so general is not that my brain or your brain already knows everything under the sun. It’s our ability to adapt, to learn a huge range of things.” The most powerful economic asset in history was never specialized knowledge. It was the raw capacity to acquire any knowledge, in any domain, on demand. The winning AI is not an encyclopedia. It is the force that makes encyclopedias irrelevant. And once that exists, the question stops being what the AI knows. It becomes what you can teach it before your competitor wakes up. Most people dominating this conversation have not understood that yet.

Dustin

19,813 views • 5 months ago

this was the most important bit of the ilya interview imo. humans are not AGI, and LLMs won’t become like humans with more scaling. humans lack a huge amount of knowledge and have insanely good “continual learning” powered by extremely good reward functions (except for addiction) that guide them. and so it’s more about the learning rather than knowledge. the term AGI was coined as a response to the narrow AI which was only capable of intelligence in a specific domain (chess, game, code, etc) but General intelligence is able to learn different topics because that’s what “learning” is about. pre-training was thought to be the solution to reaching AGI because you train AI on so many different topics. but they have “overshot” that target. amazing perspective.

this was the most important bit of the ilya interview imo. humans are not AGI, and LLMs won’t become like humans with more scaling. humans lack a huge amount of knowledge and have insanely good “continual learning” powered by extremely good reward functions (except for addiction) that guide them. and so it’s more about the learning rather than knowledge. the term AGI was coined as a response to the narrow AI which was only capable of intelligence in a specific domain (chess, game, code, etc) but General intelligence is able to learn different topics because that’s what “learning” is about. pre-training was thought to be the solution to reaching AGI because you train AI on so many different topics. but they have “overshot” that target. amazing perspective.

ℏεsam

53,444 views • 9 months ago

LoRA training is the standout feature. You can fine tune Qwen Image directly online, export the trained LoRA, and plug it straight into ComfyUI workflows like any custom model. This replaces a lot of paid LoRA setups.

LoRA training is the standout feature. You can fine tune Qwen Image directly online, export the trained LoRA, and plug it straight into ComfyUI workflows like any custom model. This replaces a lot of paid LoRA setups.

SANI BULA

15,462 views • 6 months ago

Elon Musk: AI is like a genius child. You want it to grow up with love of the truth and of humanity. “I do worry about AI. Because we are creating an intelligence that ultimately will be far more intelligent than any human, and ultimately more intelligent than the sum of all humans. In creating AI, we need to be very careful. It's like having a genius child. You want to have that child grow up with good values. The most important thing in training AI is that it's rigorously truthful. This is very, very important. Essential. My concern about a lot of the AIs that are being developed is that they're trained to lie. In some cases, with potentially disastrous consequences. When Google Gemini came out, people asked, which is worse, misgendering Caitlyn Jenner or global thermonuclear war? And it said, misgendering Caitlyn Jenner. That's a problem, guys. If this AI becomes super powerful, it could decide that the best way to avoid misgendering is to kill all humans, which makes the probability of misgendering zero. It's very important to have AI be maximally truthful and curious, and love humanity. Very important.” Harrisburg, Pennsylvania, October 19, 2024

Elon Musk: AI is like a genius child. You want it to grow up with love of the truth and of humanity. “I do worry about AI. Because we are creating an intelligence that ultimately will be far more intelligent than any human, and ultimately more intelligent than the sum of all humans. In creating AI, we need to be very careful. It's like having a genius child. You want to have that child grow up with good values. The most important thing in training AI is that it's rigorously truthful. This is very, very important. Essential. My concern about a lot of the AIs that are being developed is that they're trained to lie. In some cases, with potentially disastrous consequences. When Google Gemini came out, people asked, which is worse, misgendering Caitlyn Jenner or global thermonuclear war? And it said, misgendering Caitlyn Jenner. That's a problem, guys. If this AI becomes super powerful, it could decide that the best way to avoid misgendering is to kill all humans, which makes the probability of misgendering zero. It's very important to have AI be maximally truthful and curious, and love humanity. Very important.” Harrisburg, Pennsylvania, October 19, 2024

ELON CLIPS

3,257,158 views • 1 year ago

#WATCH | Bengaluru, Karnataka: Co-founder, Sarvam AI, Pratyush Kumar says, "... It's been three years now. We started off building on top of the Llama model, which was the best open model at that time. Sarvam was the only company outside, or maybe in Asia, that contributed to the Llama 3 Series. We understood how to add language skills into models, and Llama 3 was reasonable in Indian languages. When the first set of GPUs arrived in India, we pre-trained from scratch a 2-billion model and worked with Infosys to make it useful for IT services. Then we recognised the importance of scaling up reinforcement learning and built a reasoning model called Sarvam M on top of Mistral's small model. Over the last year, we pre-trained from scratch in India on 4,000 Hopper Series GPUs a 100-billion-plus-parameter model..."

#WATCH | Bengaluru, Karnataka: Co-founder, Sarvam AI, Pratyush Kumar says, "... It's been three years now. We started off building on top of the Llama model, which was the best open model at that time. Sarvam was the only company outside, or maybe in Asia, that contributed to the Llama 3 Series. We understood how to add language skills into models, and Llama 3 was reasonable in Indian languages. When the first set of GPUs arrived in India, we pre-trained from scratch a 2-billion model and worked with Infosys to make it useful for IT services. Then we recognised the importance of scaling up reinforcement learning and built a reasoning model called Sarvam M on top of Mistral's small model. Over the last year, we pre-trained from scratch in India on 4,000 Hopper Series GPUs a 100-billion-plus-parameter model..."

ANI

162,971 views • 28 days ago