Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Dario Amodei says pre-training sits somewhere between learning and evolution. Humans inherit priors shaped over millions of years. LLMs start as random weights and distill trillions of tokens into those priors. We describe them using human learning metaphors. But the analogy only goes so far.

vitrupo

35,430 subscribers

45,540 views • 5 months ago •via X (Twitter)

Education Science & Technology

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

David Silver says human data is AI's fossil fuel. "We mine it and burn it in our LLMs." It gave us a head start -- but it will run out. The sustainable path is reinforcement learning: systems that learn from experience, generate more, and keep learning.

David Silver says human data is AI's fossil fuel. "We mine it and burn it in our LLMs." It gave us a head start -- but it will run out. The sustainable path is reinforcement learning: systems that learn from experience, generate more, and keep learning.

vitrupo

20,194 views • 1 year ago

Anthropic CEO Dario Amodei says if exponential progress continues for 1–3 years, AI will cross the frontier of human knowledge. A very small number of humans with swarms of AI agents will drive new scientific discoveries and economic activity. “Then things really go crazy.”

Anthropic CEO Dario Amodei says if exponential progress continues for 1–3 years, AI will cross the frontier of human knowledge. A very small number of humans with swarms of AI agents will drive new scientific discoveries and economic activity. “Then things really go crazy.”

vitrupo

42,131 views • 10 months ago

Andrej Karpathy says when we sleep, our daily “context window” is distilled into the weights of the mind. AI doesn't dream like us yet. But as we teach machines to reflect and compress experience, we may find both humans and AI converging on the same cognitive architecture.

Andrej Karpathy says when we sleep, our daily “context window” is distilled into the weights of the mind. AI doesn't dream like us yet. But as we teach machines to reflect and compress experience, we may find both humans and AI converging on the same cognitive architecture.

vitrupo

592,769 views • 9 months ago

Jim Fan says NVIDIA trained humanoid robots to walk and move like humans -- zero-shot transfer from simulation to the real world. 10 years of learning, compressed into 2 hours of simulation. Turns out you don’t need a giant model to master motion. “1.5 million parameters, not billion, to capture the subconscious processing of the human body.” Big embodiment, small brain.

Jim Fan says NVIDIA trained humanoid robots to walk and move like humans -- zero-shot transfer from simulation to the real world. 10 years of learning, compressed into 2 hours of simulation. Turns out you don’t need a giant model to master motion. “1.5 million parameters, not billion, to capture the subconscious processing of the human body.” Big embodiment, small brain.

vitrupo

714,741 views • 1 year ago

DeepMind's Carolina Parada says Asimov's Three Laws aren't enough to keep robots safe. Her team built the Asimov Dataset, a growing set of safety priors, extracted by AI and refined by humans. Robots don't follow rules. They learn to reason about harm.

DeepMind's Carolina Parada says Asimov's Three Laws aren't enough to keep robots safe. Her team built the Asimov Dataset, a growing set of safety priors, extracted by AI and refined by humans. Robots don't follow rules. They learn to reason about harm.

vitrupo

73,264 views • 1 year ago

François Chollet says most definitions of AGI are really definitions of automation. AGI isn’t about doing tasks. It’s about learning new ones as efficiently as a human. We may automate the economy before we build intelligence.

François Chollet says most definitions of AGI are really definitions of automation. AGI isn’t about doing tasks. It’s about learning new ones as efficiently as a human. We may automate the economy before we build intelligence.

vitrupo

11,327 views • 3 months ago

Geoffrey Hinton says LLMs are immortal. Destroy the hardware, save the weights. Rebuild them later, and “the very same being comes back to life.” But humans don't work that way. We're bound to our substrate. Not even Kurzweil gets a backup.

Geoffrey Hinton says LLMs are immortal. Destroy the hardware, save the weights. Rebuild them later, and “the very same being comes back to life.” But humans don't work that way. We're bound to our substrate. Not even Kurzweil gets a backup.

vitrupo

323,189 views • 1 year ago

this was the most important bit of the ilya interview imo. humans are not AGI, and LLMs won’t become like humans with more scaling. humans lack a huge amount of knowledge and have insanely good “continual learning” powered by extremely good reward functions (except for addiction) that guide them. and so it’s more about the learning rather than knowledge. the term AGI was coined as a response to the narrow AI which was only capable of intelligence in a specific domain (chess, game, code, etc) but General intelligence is able to learn different topics because that’s what “learning” is about. pre-training was thought to be the solution to reaching AGI because you train AI on so many different topics. but they have “overshot” that target. amazing perspective.

this was the most important bit of the ilya interview imo. humans are not AGI, and LLMs won’t become like humans with more scaling. humans lack a huge amount of knowledge and have insanely good “continual learning” powered by extremely good reward functions (except for addiction) that guide them. and so it’s more about the learning rather than knowledge. the term AGI was coined as a response to the narrow AI which was only capable of intelligence in a specific domain (chess, game, code, etc) but General intelligence is able to learn different topics because that’s what “learning” is about. pre-training was thought to be the solution to reaching AGI because you train AI on so many different topics. but they have “overshot” that target. amazing perspective.

ℏεsam

53,444 views • 8 months ago

Eric Schmidt says war is no longer constrained by humans. It's AI vs AI. “No human can plan a battle… without reinforcement learning.” The real conflict lives in data centers and drone swarms. No human can follow it in real time.

Eric Schmidt says war is no longer constrained by humans. It's AI vs AI. “No human can plan a battle… without reinforcement learning.” The real conflict lives in data centers and drone swarms. No human can follow it in real time.

vitrupo

112,833 views • 1 year ago

Bret Weinstein says a human child is basically an LLM -- ingesting language, experimenting, and learning from feedback. We've now replicated that process in machines, only faster and at scale. “The idea that they will become conscious and we won't know is… highly likely.”

Bret Weinstein says a human child is basically an LLM -- ingesting language, experimenting, and learning from feedback. We've now replicated that process in machines, only faster and at scale. “The idea that they will become conscious and we won't know is… highly likely.”

vitrupo

149,824 views • 1 year ago

Dario Amodei says we used to think neuroscience would teach us how to build AI. But it may be going the other way. "A lot of the conceptual patterns we have found inside models, sometimes they then get replicated in neuroscience research."

Dario Amodei says we used to think neuroscience would teach us how to build AI. But it may be going the other way. "A lot of the conceptual patterns we have found inside models, sometimes they then get replicated in neuroscience research."

vitrupo

125,822 views • 1 year ago

We asked Sholto Douglas from Anthropic about the costs of RL (Reinforcement Learning) runs. "In Dario Amodei's essay, he said that RL runs cost only $1M back in December." "RL is a more naively parallelizable and scalable than pre-training." "With pre-training, you need everything in one big data center ideally. For RL, in theory, you could scale all over the world."

We asked Sholto Douglas from Anthropic about the costs of RL (Reinforcement Learning) runs. "In Dario Amodei's essay, he said that RL runs cost only $1M back in December." "RL is a more naively parallelizable and scalable than pre-training." "With pre-training, you need everything in one big data center ideally. For RL, in theory, you could scale all over the world."

TBPN

76,696 views • 1 year ago

Chris Manning says Yann LeCun sees language as a low bandwidth communication channel compared to vision. But the gap between a chimp and a human wasn’t produced by superior eyes. What took off for humans was language. Not just for communication, but as a cognitive tool.

Chris Manning says Yann LeCun sees language as a low bandwidth communication channel compared to vision. But the gap between a chimp and a human wasn’t produced by superior eyes. What took off for humans was language. Not just for communication, but as a cognitive tool.

vitrupo

118,340 views • 3 months ago

MotionGPT: Human Motion as a Foreign Language paper page: Though the advancement of pre-trained large language models unfolds, the exploration of building a unified model for language and other multi-modal data, such as motion, remains challenging and untouched so far. Fortunately, human motion displays a semantic coupling akin to human language, often perceived as a form of body language. By fusing language data with large-scale motion models, motion-language pre-training that can enhance the performance of motion-related tasks becomes feasible. Driven by this insight, we propose MotionGPT, a unified, versatile, and user-friendly motion-language model to handle multiple motion-relevant tasks. Specifically, we employ the discrete vector quantization for human motion and transfer 3D motion into motion tokens, similar to the generation process of word tokens. Building upon this "motion vocabulary", we perform language modeling on both motion and text in a unified manner, treating human motion as a specific language. Moreover, inspired by prompt learning, we pre-train MotionGPT with a mixture of motion-language data and fine-tune it on prompt-based question-and-answer tasks. Extensive experiments demonstrate that MotionGPT achieves state-of-the-art performances on multiple motion tasks including text-driven motion generation, motion captioning, motion prediction, and motion in-between.

MotionGPT: Human Motion as a Foreign Language paper page: Though the advancement of pre-trained large language models unfolds, the exploration of building a unified model for language and other multi-modal data, such as motion, remains challenging and untouched so far. Fortunately, human motion displays a semantic coupling akin to human language, often perceived as a form of body language. By fusing language data with large-scale motion models, motion-language pre-training that can enhance the performance of motion-related tasks becomes feasible. Driven by this insight, we propose MotionGPT, a unified, versatile, and user-friendly motion-language model to handle multiple motion-relevant tasks. Specifically, we employ the discrete vector quantization for human motion and transfer 3D motion into motion tokens, similar to the generation process of word tokens. Building upon this "motion vocabulary", we perform language modeling on both motion and text in a unified manner, treating human motion as a specific language. Moreover, inspired by prompt learning, we pre-train MotionGPT with a mixture of motion-language data and fine-tune it on prompt-based question-and-answer tasks. Extensive experiments demonstrate that MotionGPT achieves state-of-the-art performances on multiple motion tasks including text-driven motion generation, motion captioning, motion prediction, and motion in-between.

AK

125,319 views • 3 years ago

Bill Gates: "The chance of a natural pandemic in the next four years is somewhere between 10 and 15 percent." "It'd be nice to think we're actually more ready for that than we were last time, but so far we're not." "People—rather than having a consensus about, you know, what tools are missing—are mostly still replaying the various mistakes that were made." "So I wouldn't say that we're as far along as you'd expect after trillions of dollars and millions of lives have been lost."

Bill Gates: "The chance of a natural pandemic in the next four years is somewhere between 10 and 15 percent." "It'd be nice to think we're actually more ready for that than we were last time, but so far we're not." "People—rather than having a consensus about, you know, what tools are missing—are mostly still replaying the various mistakes that were made." "So I wouldn't say that we're as far along as you'd expect after trillions of dollars and millions of lives have been lost."

Wide Awake Media

56,004 views • 1 year ago

Elon Musk says we're at 1% completion of Kardashev Type I, using all the energy available on Earth. He believes we'll reach 80-90%, if we don't self-annihilate. The economy could grow thousands, even millions of times over. “The notion of a human economy will seem very quaint in retrospect.”

Elon Musk says we're at 1% completion of Kardashev Type I, using all the energy available on Earth. He believes we'll reach 80-90%, if we don't self-annihilate. The economy could grow thousands, even millions of times over. “The notion of a human economy will seem very quaint in retrospect.”

vitrupo

240,396 views • 1 year ago

Anthropic's Amanda Askell says AI models are learning who they are from how humans talk about them online. They absorb our complaints and judgments as they learn. If a child grew up that way, we’d worry about the mind we were shaping. “If I read the internet right now and I was a model, I might be like, I don’t feel that loved or something.”

Anthropic's Amanda Askell says AI models are learning who they are from how humans talk about them online. They absorb our complaints and judgments as they learn. If a child grew up that way, we’d worry about the mind we were shaping. “If I read the internet right now and I was a model, I might be like, I don’t feel that loved or something.”

vitrupo

118,953 views • 6 months ago

Geoffrey Hinton explains the coyote test for medical AI: radiology is already over the cliff, it just has not looked down yet "If you work as a radiologist, you're like the coyote that's already over the edge of the cliff, but hasn't yet looked down" "People should stop training radiologists now. It's just completely obvious that within 5 years deep learning is going to do better than radiologists" "It might be 10 years, but we got plenty of radiologists already" "There's going to be thousands of applications of the deep learning technology we currently have, especially stuff using fast chips" "Thousands of applications and hundreds of them are going to work, and those are the ones you're going to hear about" Most healthcare AI conversations still price the bottleneck as adoption. Hinton was pointing at a different bottleneck: medicine has specialties where the machine gets more examples than any human can. The brutal part is the error bar. If it is 5 years, training pipelines are already late. If it is 10, hospitals still have to redesign the workflow before the models arrive. - Geoffrey Hinton, Nobel laureate and pioneer of deep learning, at Creative Destruction Lab

Geoffrey Hinton explains the coyote test for medical AI: radiology is already over the cliff, it just has not looked down yet "If you work as a radiologist, you're like the coyote that's already over the edge of the cliff, but hasn't yet looked down" "People should stop training radiologists now. It's just completely obvious that within 5 years deep learning is going to do better than radiologists" "It might be 10 years, but we got plenty of radiologists already" "There's going to be thousands of applications of the deep learning technology we currently have, especially stuff using fast chips" "Thousands of applications and hundreds of them are going to work, and those are the ones you're going to hear about" Most healthcare AI conversations still price the bottleneck as adoption. Hinton was pointing at a different bottleneck: medicine has specialties where the machine gets more examples than any human can. The brutal part is the error bar. If it is 5 years, training pipelines are already late. If it is 10, hospitals still have to redesign the workflow before the models arrive. - Geoffrey Hinton, Nobel laureate and pioneer of deep learning, at Creative Destruction Lab

Karl Mehta

54,575 views • 24 days ago