Loading video...

Video Failed to Load

Go Home

Some more thoughts about Yann interview: Even if LLMs work great, that's missing the point. Everyone's doing the same thing now. More scale, more data, longer CoT, tweak RL. But the path to get there was completely stochastic. Attention, transformers, scaling laws, RLHF, none of it was obvious, it...

ziv_ravid's profile picture

Ravid Shwartz Ziv

11,919 subscribers

57,486 views • 9 months ago •via X (Twitter)

EducationNews & PoliticsScience & Technology
LIVE

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

“There are more companies than ideas by quite a bit. Compute is large enough such that it's not obvious that you need that much more compute to prove some idea. AlexNet was built on 2 GPUs. The transformer was built on 8 to 64 GPUs. Which would be, what, 2 GPUs of today? You could argue that o1 reasoning was not the most compute heavy thing in the world. For research, you definitely need some amount of compute, but it's far from obvious that you need the absolutely largest amount of compute. If everyone is within the same paradigm, then compute becomes one of the big differentiators.” Ilya Sutskever
2:03

“There are more companies than ideas by quite a bit. Compute is large enough such that it's not obvious that you need that much more compute to prove some idea. AlexNet was built on 2 GPUs. The transformer was built on 8 to 64 GPUs. Which would be, what, 2 GPUs of today? You could argue that o1 reasoning was not the most compute heavy thing in the world. For research, you definitely need some amount of compute, but it's far from obvious that you need the absolutely largest amount of compute. If everyone is within the same paradigm, then compute becomes one of the big differentiators.” Ilya Sutskever

Dwarkesh Patel

203,605 views • 10 months ago

.Ilya Sutskever on why OpenAI gave up on robotics: "Progress comes from the combination of compute and data. There was no path to data from robotics. I'd say that now it is possible to create a path forward, but one needs to really commit. You could imagine a gradual path of improvement where you build more robots, they do more things, you collect more data and so on. But you need to really love robots and need to be really willing to solve all the physical and logistical problems of dealing with them. It's not the same as software at all."
1:57

.Ilya Sutskever on why OpenAI gave up on robotics: "Progress comes from the combination of compute and data. There was no path to data from robotics. I'd say that now it is possible to create a path forward, but one needs to really commit. You could imagine a gradual path of improvement where you build more robots, they do more things, you collect more data and so on. But you need to really love robots and need to be really willing to solve all the physical and logistical problems of dealing with them. It's not the same as software at all."

Dwarkesh Patel

325,925 views • 2 years ago

MacFarlane: These things would land more if the DOJ was not an extension of the WH, which it is. It is deeply politicized. It is sycophantic. But I also think this is going to beg the question, how much more are we going to hear about preemptive pardons? I think that that's going to echo more as the weeks go on.
1:56

MacFarlane: These things would land more if the DOJ was not an extension of the WH, which it is. It is deeply politicized. It is sycophantic. But I also think this is going to beg the question, how much more are we going to hear about preemptive pardons? I think that that's going to echo more as the weeks go on.

Acyn

31,511 views • 5 months ago

Baseten's Charlie O'Neill on why the gap to the frontier is just compute and data: "Despite OpenAI and Anthropic and the closed source labs having more compute and more data, the recipe is the same. We're all scaling the same recipe here, and we're at slightly different points of the scaling curve. It's not like OpenAI or Anthropic have any secret sauce that differentiates them scaling up their multi-trillion parameter models from the rest of the open source ecosystem." "It really is just a matter of taking the core underlying architecture and training recipe, and throwing more compute and data at it, which is really promising." "Of course, these closed source labs probably have a long tail of optimizations they've made, which we're gonna continue to try and figure out in the open source community. But the core mechanism is there." "K3 is a great example of, well, if you do scale this recipe up, we're gonna get a pretty good model that's very, very close to the frontier, if not frontier itself." Charlie O'Neill Baseten
1:51

Baseten's Charlie O'Neill on why the gap to the frontier is just compute and data: "Despite OpenAI and Anthropic and the closed source labs having more compute and more data, the recipe is the same. We're all scaling the same recipe here, and we're at slightly different points of the scaling curve. It's not like OpenAI or Anthropic have any secret sauce that differentiates them scaling up their multi-trillion parameter models from the rest of the open source ecosystem." "It really is just a matter of taking the core underlying architecture and training recipe, and throwing more compute and data at it, which is really promising." "Of course, these closed source labs probably have a long tail of optimizations they've made, which we're gonna continue to try and figure out in the open source community. But the core mechanism is there." "K3 is a great example of, well, if you do scale this recipe up, we're gonna get a pretty good model that's very, very close to the frontier, if not frontier itself." Charlie O'Neill Baseten

MTS

17,404 views • 2 months ago

Sequoia's Ravi Gupta and Pat Grady on potential layoffs and why the AI doomers are wrong: "Technology is neither good nor bad, it's what we do with it. The same technology that powers a nuclear bomb can power a nuclear power plant." "A lot of what people are doing when they go and talk about the technology is they talk about all the things that might happen. I would rather us talk about the fact that it's going to get more and more powerful, and it's more and more incumbent on all of us to make sure we use it for something great." "If I could give my kids one thing, it wouldn't be kindness. It wouldn't be curiosity. It would be agency. The technology is going to be what all of us make of it."
2:04

Sequoia's Ravi Gupta and Pat Grady on potential layoffs and why the AI doomers are wrong: "Technology is neither good nor bad, it's what we do with it. The same technology that powers a nuclear bomb can power a nuclear power plant." "A lot of what people are doing when they go and talk about the technology is they talk about all the things that might happen. I would rather us talk about the fact that it's going to get more and more powerful, and it's more and more incumbent on all of us to make sure we use it for something great." "If I could give my kids one thing, it wouldn't be kindness. It wouldn't be curiosity. It would be agency. The technology is going to be what all of us make of it."

TBPN

21,055 views • 8 months ago

Ilya Sutskever says the AI scaling era is running out of road because pre-training is about to hit a finite-data wall. "At some point though, pre-training will run out of data. The data is very clearly finite." "Then, okay, what do you do next? Either you do some kind of a souped-up pre-training... or you're doing RL, or maybe something else." "In some sense, we are back to the age of research." His timeline is the part most AI investors and founders should hear: "From 2012 to 2020, it was the age of research. Now from 2020 to 2025, it was the age of scaling... But now the scale is so big." "Is the belief really that... if you had 100x more everything would be so different?" "I don't think that's true. So it's back to the age of research again, just with big computers."
1:10

Ilya Sutskever says the AI scaling era is running out of road because pre-training is about to hit a finite-data wall. "At some point though, pre-training will run out of data. The data is very clearly finite." "Then, okay, what do you do next? Either you do some kind of a souped-up pre-training... or you're doing RL, or maybe something else." "In some sense, we are back to the age of research." His timeline is the part most AI investors and founders should hear: "From 2012 to 2020, it was the age of research. Now from 2020 to 2025, it was the age of scaling... But now the scale is so big." "Is the belief really that... if you had 100x more everything would be so different?" "I don't think that's true. So it's back to the age of research again, just with big computers."

Karl Mehta

200,003 views • 3 months ago

Scott on why conviction in an idea is dangerous as an investor but required as a founder: "From the investor seat, I don't think you want to be in love with an idea. That's a risky thing because you'll find a way to express that by investing and make some compromises on the team because you know this idea is just so good, its time has come. But if the team isn't there, 90% of the time it doesn't work. Then there's a whole thing can you swap out the team? But the whole Founders Fund thesis was always no, you need the founder to run the company. But on the company side, you have to be in love with the idea. It's not that rational to start a company. There's a lot easier, more comfortable ways to make money if that's the goal. So it's got to be about what the company is doing specifically. If you want to actually start a company, it better be something that you're really passionate about, or you think that the problem is really important."
1:15

Scott on why conviction in an idea is dangerous as an investor but required as a founder: "From the investor seat, I don't think you want to be in love with an idea. That's a risky thing because you'll find a way to express that by investing and make some compromises on the team because you know this idea is just so good, its time has come. But if the team isn't there, 90% of the time it doesn't work. Then there's a whole thing can you swap out the team? But the whole Founders Fund thesis was always no, you need the founder to run the company. But on the company side, you have to be in love with the idea. It's not that rational to start a company. There's a lot easier, more comfortable ways to make money if that's the goal. So it's got to be about what the company is doing specifically. If you want to actually start a company, it better be something that you're really passionate about, or you think that the problem is really important."

Patrick OShaughnessy

33,016 views • 5 months ago

Elon Musk: 𝕏 is simply trying to adhere to the law. If the people in a country want the laws to be different, they should make the laws different. "What we're trying to do is simply adhere to the laws in a country. If something is illegal in the United States, or if it's illegal in Europe or Brazil or wherever it might be, then we will take it down, and we'll suspend the account because we're not there to make the laws. But if speech is not illegal, then what are we doing? Now we're injecting ourselves in as a censor. And where does it stop? And who decides? Where does that path lead? I think it leads to a bad place. So, if the people in a country want the laws to be different, they should make the laws different. But otherwise, we're going to obey the law in each jurisdiction. That's it. It's not more complicated than that. We're not trying to flout the law. We're trying to adhere to the law. And if the laws change, we will change. And if the laws don't change, we won't. We're just literally trying to adhere to the law." From: The All-In Podcast, September 9, 2024
1:06

Elon Musk: 𝕏 is simply trying to adhere to the law. If the people in a country want the laws to be different, they should make the laws different. "What we're trying to do is simply adhere to the laws in a country. If something is illegal in the United States, or if it's illegal in Europe or Brazil or wherever it might be, then we will take it down, and we'll suspend the account because we're not there to make the laws. But if speech is not illegal, then what are we doing? Now we're injecting ourselves in as a censor. And where does it stop? And who decides? Where does that path lead? I think it leads to a bad place. So, if the people in a country want the laws to be different, they should make the laws different. But otherwise, we're going to obey the law in each jurisdiction. That's it. It's not more complicated than that. We're not trying to flout the law. We're trying to adhere to the law. And if the laws change, we will change. And if the laws don't change, we won't. We're just literally trying to adhere to the law." From: The All-In Podcast, September 9, 2024

ELON CLIPS

5,539,295 views • 2 years ago

Sam Altman admits he was wrong on AI's timeline and says society and the economy will adapt more slowly "I thought when we got to GPT-4, which was back in 2023, that very quickly after that there was going to be much more disruption, software businesses up for grabs right away, than it turned out to be." "I think I was wrong about a few things, but one in terms of the speed: the economy just has so much inertia." "People keep doing the same things, buying from the same company, wanting to use their tools the same way. I think this is actually a positive in many ways, and it's going to make this big transition go smoother and slower. I'm grateful for it." "But it means we've all been too ambitious on timelines. Even with this incredible technology, society and the economy will adapt more slowly."
0:56

Sam Altman admits he was wrong on AI's timeline and says society and the economy will adapt more slowly "I thought when we got to GPT-4, which was back in 2023, that very quickly after that there was going to be much more disruption, software businesses up for grabs right away, than it turned out to be." "I think I was wrong about a few things, but one in terms of the speed: the economy just has so much inertia." "People keep doing the same things, buying from the same company, wanting to use their tools the same way. I think this is actually a positive in many ways, and it's going to make this big transition go smoother and slower. I'm grateful for it." "But it means we've all been too ambitious on timelines. Even with this incredible technology, society and the economy will adapt more slowly."

Fireside Alpha

6,394,577 views • 1 month ago

Dario Amodei reveals the two convictions that made him leave OpenAI to start Anthropic "The first conviction was the scaling laws" "If you scale up models, give them more data and more compute, you find incredible increases in performance. I was finding that in 2019 with GPT-2" "There were a lot of folks inside and outside who didn't believe it at all. We made the case to leadership, this is going to be a big deal" "The second was, if these models are going to match the capability of the human brain, we better get this right" "Despite a lot of language about doing it in the right way, I was just not convinced there was a real and serious conviction to do it the right way"
4:12

Dario Amodei reveals the two convictions that made him leave OpenAI to start Anthropic "The first conviction was the scaling laws" "If you scale up models, give them more data and more compute, you find incredible increases in performance. I was finding that in 2019 with GPT-2" "There were a lot of folks inside and outside who didn't believe it at all. We made the case to leadership, this is going to be a big deal" "The second was, if these models are going to match the capability of the human brain, we better get this right" "Despite a lot of language about doing it in the right way, I was just not convinced there was a real and serious conviction to do it the right way"

Jaynit Makwana

248,140 views • 4 months ago

.Naval: Marxism, besides denying human incentives, also has a problem where it just assumes that everything is finite and we're all just dividing up the same small set of things. Well, the cavemen didn't have color TVs, computers, cars, antibiotics, or medicine. They were not sitting around dividing up the same few things. The knowledge grows, and we create more. It also tends to assume that we can freeze frame at some point in society and say, 'Well, we have enough different kinds of sneakers, enough different kinds of housing, we just need to allocate it better.' And that is not how anything works. David Deutsch has a great definition of wealth, which he says is the set of physical transformations that we can affect. So, when you think about it that way, you realize that knowledge is not just stored capital in the classic Marxist sense (capital vs. labor), but it's also knowledge on what to do with that capital. The cavemen or Palaeolithic ancestors had access to all the same resources we did. They were living on the same Earth, and by the modern environmentalist arguments, they had a better Earth—they had more to do things with. But yet, they couldn't do anything. They were not wealthy by any stretch of the imagination. Why? Because of knowledge. Life is not a zero-sum game, it's a positive-sum game. But we are hardwired to think it's a zero-sum game because, for millions or billions of years, there was no such thing as wealth. There was no such thing as persistent knowledge creation in the environment. What you had was a small amount of resources being divided up, and most of the games were status games—'Which monkey outranks which other monkey?' And that decides which monkey gets to eat first. We played that game for a billion years. Now, we come onto a recent game where, actually, we can all eat, and the big problem is obesity. It's no longer starvation. The big problem is boredom. It's not actually work. There's enough work. So, in this environment, switching your evolved mindset from a zero-sum game to a positive-sum game where we can all win, if we create knowledge together and use the resources that we have to create more resources and more wealth, that's the game we all need to be playing now.
2:11

.Naval: Marxism, besides denying human incentives, also has a problem where it just assumes that everything is finite and we're all just dividing up the same small set of things. Well, the cavemen didn't have color TVs, computers, cars, antibiotics, or medicine. They were not sitting around dividing up the same few things. The knowledge grows, and we create more. It also tends to assume that we can freeze frame at some point in society and say, 'Well, we have enough different kinds of sneakers, enough different kinds of housing, we just need to allocate it better.' And that is not how anything works. David Deutsch has a great definition of wealth, which he says is the set of physical transformations that we can affect. So, when you think about it that way, you realize that knowledge is not just stored capital in the classic Marxist sense (capital vs. labor), but it's also knowledge on what to do with that capital. The cavemen or Palaeolithic ancestors had access to all the same resources we did. They were living on the same Earth, and by the modern environmentalist arguments, they had a better Earth—they had more to do things with. But yet, they couldn't do anything. They were not wealthy by any stretch of the imagination. Why? Because of knowledge. Life is not a zero-sum game, it's a positive-sum game. But we are hardwired to think it's a zero-sum game because, for millions or billions of years, there was no such thing as wealth. There was no such thing as persistent knowledge creation in the environment. What you had was a small amount of resources being divided up, and most of the games were status games—'Which monkey outranks which other monkey?' And that decides which monkey gets to eat first. We played that game for a billion years. Now, we come onto a recent game where, actually, we can all eat, and the big problem is obesity. It's no longer starvation. The big problem is boredom. It's not actually work. There's enough work. So, in this environment, switching your evolved mindset from a zero-sum game to a positive-sum game where we can all win, if we create knowledge together and use the resources that we have to create more resources and more wealth, that's the game we all need to be playing now.

Arjun Khemani

163,592 views • 1 year ago

German MEP Christine Anderson argues that governments "overplayed their cards" during the Covid era, setting in motion a mass awakening that can no longer be reversed. "The people stood up... And then they overplayed their cards. They came down even harder on the people to get them to do what they wanted them to do, and that woke more people up." "That was like a vicious cycle. The more people rebelled... the harder the state bashed down on them, the more people woke up." "And that's the situation we're in now. The genie is out of the bottle and they will not ever get it back in." "There is light at the end of the tunnel because a lot of people have woken up, and a lot of people will continue to wake up... in the months and years to come." "But I can tell you, buckle up. They're stepping up their game, so we have to do the same thing."
2:16

German MEP Christine Anderson argues that governments "overplayed their cards" during the Covid era, setting in motion a mass awakening that can no longer be reversed. "The people stood up... And then they overplayed their cards. They came down even harder on the people to get them to do what they wanted them to do, and that woke more people up." "That was like a vicious cycle. The more people rebelled... the harder the state bashed down on them, the more people woke up." "And that's the situation we're in now. The genie is out of the bottle and they will not ever get it back in." "There is light at the end of the tunnel because a lot of people have woken up, and a lot of people will continue to wake up... in the months and years to come." "But I can tell you, buckle up. They're stepping up their game, so we have to do the same thing."

Wide Awake Media

28,659 views • 7 months ago

Jensen to AI Leaders: “We have to be far more thoughtful” when communicating to the public Jensen Huang: “(AI) is not a biological being. It is not alien. It is not conscious. It is computer software.” “We say things like, ‘We don't understand it at all.’ It is not true. We understand a lot of things about this technology.” Chamath: “If you were in the seat in the boardroom of Anthropic over that whole scuttlebutt with the Department of War, what do you think you would've told Dario and that team to do, maybe, differently to try to change some of this outcome and some of this perception?” Jensen: “The first thing that I would say about Anthropic is, first of all, the technology is incredible. We are a large consumer of Anthropic technology.” “The desire to warn people about the capability of the technology is also really terrific.” “We just have to make sure that we understand that the world has a spectrum, and that warning is good, scaring is less good because this technology is too important to us.” “I think that it is fine to predict the future, but we need to be a little bit more circumspect. We need to have a little bit more humility, that, in fact, we can't completely predict the future.” “And to say things that are quite extreme, quite catastrophic, that there's no evidence of it happening, could be more damaging than people think.” “And of course we are technology leaders.” “There was a time when nobody listened to us, but now because technology is so important in the social fabric, such an important industry, so important to national security, our words do matter.” “And I think we have to be much more circumspect, we have to be more moderate, we have to be more balanced, we have to be far more thoughtful.”
1:27

Jensen to AI Leaders: “We have to be far more thoughtful” when communicating to the public Jensen Huang: “(AI) is not a biological being. It is not alien. It is not conscious. It is computer software.” “We say things like, ‘We don't understand it at all.’ It is not true. We understand a lot of things about this technology.” Chamath: “If you were in the seat in the boardroom of Anthropic over that whole scuttlebutt with the Department of War, what do you think you would've told Dario and that team to do, maybe, differently to try to change some of this outcome and some of this perception?” Jensen: “The first thing that I would say about Anthropic is, first of all, the technology is incredible. We are a large consumer of Anthropic technology.” “The desire to warn people about the capability of the technology is also really terrific.” “We just have to make sure that we understand that the world has a spectrum, and that warning is good, scaring is less good because this technology is too important to us.” “I think that it is fine to predict the future, but we need to be a little bit more circumspect. We need to have a little bit more humility, that, in fact, we can't completely predict the future.” “And to say things that are quite extreme, quite catastrophic, that there's no evidence of it happening, could be more damaging than people think.” “And of course we are technology leaders.” “There was a time when nobody listened to us, but now because technology is so important in the social fabric, such an important industry, so important to national security, our words do matter.” “And I think we have to be much more circumspect, we have to be more moderate, we have to be more balanced, we have to be far more thoughtful.”

The All-In Podcast

57,581 views • 6 months ago

Bill Chisholm: “It’s not about the money at all. This was, again, trying to put together the right set of players and assets to win now, to win next year, the following year, and the year after that. That's what this was about. None of these were about money. We have some more room now. If we see something we wanted, Brad's got the green light to do that. So I just really, I mean, I know there's people who feel like, 'Oh, there must be a smoking gun somewhere on the money,' but that's just not what this is about. And I can say it, and I'll keep saying it, but I'll also prove it to you when the time comes. When we have the opportunity, we're going to do that. And we've given ourselves the flexibility to do it now. So, it's fine to keep asking the question, because I know we have to prove it, and we will.”
1:32

Bill Chisholm: “It’s not about the money at all. This was, again, trying to put together the right set of players and assets to win now, to win next year, the following year, and the year after that. That's what this was about. None of these were about money. We have some more room now. If we see something we wanted, Brad's got the green light to do that. So I just really, I mean, I know there's people who feel like, 'Oh, there must be a smoking gun somewhere on the money,' but that's just not what this is about. And I can say it, and I'll keep saying it, but I'll also prove it to you when the time comes. When we have the opportunity, we're going to do that. And we've given ourselves the flexibility to do it now. So, it's fine to keep asking the question, because I know we have to prove it, and we will.”

Justin Turpin

10,831 views • 2 months ago

.Vice President JD Vance: "I do not want so many ICE officers in Minneapolis right now... but they're here not even to enforce immigration laws, but to protect the people from the rioters. That’s an absurd state of affairs, and we wouldn't need it if we had a little bit more cooperation from the Minneapolis Police Department."
1:25

.Vice President JD Vance: "I do not want so many ICE officers in Minneapolis right now... but they're here not even to enforce immigration laws, but to protect the people from the rioters. That’s an absurd state of affairs, and we wouldn't need it if we had a little bit more cooperation from the Minneapolis Police Department."

Rapid Response 47

78,290 views • 8 months ago