Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

here's Jeff Dean talking about how labs will do multi-epoch pretraining with heavy regularization to keep scaling even with limited data. no wonder slowrun gets so much attention from pretraining teams at big labs. pretraining is about to look very very different.

Samip

3,525 subscribers

127,273 views • 5 months ago •via X (Twitter)

Education Science & Technology

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

If autonomous finance is the future… how do agents actually find and trust each other? Right now, most can’t. So we teamed up with Tempora Labs to do something about it ↓

If autonomous finance is the future… how do agents actually find and trust each other? Right now, most can’t. So we teamed up with Tempora Labs to do something about it ↓

HOL

22,268 views • 6 months ago

“One of the great things about Iso is how interdisciplinary everyone is, people come from so many different backgrounds.” We spoke with Bilal Khan, Software Engineer at Isomorphic Labs, about our unique, collaborative culture and frontier AI built to tackle the world's most complex problems. Watch the video and head to the comments to explore our current roles.

“One of the great things about Iso is how interdisciplinary everyone is, people come from so many different backgrounds.” We spoke with Bilal Khan, Software Engineer at Isomorphic Labs, about our unique, collaborative culture and frontier AI built to tackle the world's most complex problems. Watch the video and head to the comments to explore our current roles.

Isomorphic Labs

19,870 views • 6 months ago

Excitement about world-action models and robot learning has never been higher — they promise a way to use human egocentric data to train massive robotics models which can provide the “GPT” moment for robotics and unlock general-purpose embodied intelligence. And yet there’s been little concrete demonstration of scaling in robot learning. Dyna Robotics aims to change that, with an in-depth look at how scaling works as they approach 1 million hours of training data. Jason Ma Tianyu Li Anh-Quan Pham and Chet Bhateja joined us to tell us more. They show that as the amount of data they use in pretraining increased, they saw predictable, statistically significant gains on accuracy metrics on held-out data (data not seen during training). They go on to talk about what they learned, and show how this can be applied to many different problems. Watch Episode #99 of RoboPapers, with Michael Cho - Rbt/Acc Chris Paxton Jiafei Duan today to learn more!

Excitement about world-action models and robot learning has never been higher — they promise a way to use human egocentric data to train massive robotics models which can provide the “GPT” moment for robotics and unlock general-purpose embodied intelligence. And yet there’s been little concrete demonstration of scaling in robot learning. Dyna Robotics aims to change that, with an in-depth look at how scaling works as they approach 1 million hours of training data. Jason Ma Tianyu Li Anh-Quan Pham and Chet Bhateja joined us to tell us more. They show that as the amount of data they use in pretraining increased, they saw predictable, statistically significant gains on accuracy metrics on held-out data (data not seen during training). They go on to talk about what they learned, and show how this can be applied to many different problems. Watch Episode #99 of RoboPapers, with Michael Cho - Rbt/Acc Chris Paxton Jiafei Duan today to learn more!

RoboPapers

25,878 views • 29 days ago

you guys i’m crying.. taeyong talks about how he’s very shy and gets nervous about asking people to do challenges so at one of the fansigns i attended, i told him a group i talk to often will be at some music shows with him and would be honored to do a challenge with him.. XLOV!

you guys i’m crying.. taeyong talks about how he’s very shy and gets nervous about asking people to do challenges so at one of the fansigns i attended, i told him a group i talk to often will be at some music shows with him and would be honored to do a challenge with him.. XLOV!

𝔪𝔬𝔬𝔡𝔶 𓆩ꨄ︎𓆪

443,347 views • 3 months ago

World Labs CEO Dr. Fei-Fei Li & SceniX Co-Founder Yunzhu Li on the data bottleneck in robotics and how world models help: "The lack of data in training, the lack of data in evaluation, this is very, very different from language models, where data is abundant on the internet." "And we know that in order for robotics to work, we have to somehow unlock the power of scaling law. But where does that come from?" "It's a profound problem that everybody's battling with in robotics." "We see a lot of unlock in being able to do this whole process through the modeling of the environments." "Being able to create these digital worlds... that's just going to unlock so much more potential for being able to replace all the costly and the unsafe data in the real environments with the data generated from the worlds for robots to be able to do scalable learning and evaluations." Fei-Fei Li Yunzhu Li martin_casado

World Labs CEO Dr. Fei-Fei Li & SceniX Co-Founder Yunzhu Li on the data bottleneck in robotics and how world models help: "The lack of data in training, the lack of data in evaluation, this is very, very different from language models, where data is abundant on the internet." "And we know that in order for robotics to work, we have to somehow unlock the power of scaling law. But where does that come from?" "It's a profound problem that everybody's battling with in robotics." "We see a lot of unlock in being able to do this whole process through the modeling of the environments." "Being able to create these digital worlds... that's just going to unlock so much more potential for being able to replace all the costly and the unsafe data in the real environments with the data generated from the worlds for robots to be able to do scalable learning and evaluations." Fei-Fei Li Yunzhu Li martin_casado

a16z

77,065 views • 1 month ago

Can we scale 4D pretraining to learn general space-time representations that reconstruct an object from a few views at any time to any view at any other time? Introducing 4D-LRM: a Large Space-Time Reconstruction Model that ... 🔹 Predicts 4D Gaussian primitives directly from multi-view tokens (no motion vectors, no HexPlane); 🔹 Uses a clean, minimal Transformer backbone; 🔹 Generalizes with fast, high-quality feedforward rendering at any view and infinite frame rate. Check out more interactive demos and scaling behaviors on our homepage/paper. 👉Website: 👉Paper:

Can we scale 4D pretraining to learn general space-time representations that reconstruct an object from a few views at any time to any view at any other time? Introducing 4D-LRM: a Large Space-Time Reconstruction Model that ... 🔹 Predicts 4D Gaussian primitives directly from multi-view tokens (no motion vectors, no HexPlane); 🔹 Uses a clean, minimal Transformer backbone; 🔹 Generalizes with fast, high-quality feedforward rendering at any view and infinite frame rate. Check out more interactive demos and scaling behaviors on our homepage/paper. 👉Website: 👉Paper:

Martin Ziqiao Ma

21,851 views • 1 year ago

Ilya Sutskever says the AI scaling era is running out of road because pre-training is about to hit a finite-data wall. "At some point though, pre-training will run out of data. The data is very clearly finite." "Then, okay, what do you do next? Either you do some kind of a souped-up pre-training... or you're doing RL, or maybe something else." "In some sense, we are back to the age of research." His timeline is the part most AI investors and founders should hear: "From 2012 to 2020, it was the age of research. Now from 2020 to 2025, it was the age of scaling... But now the scale is so big." "Is the belief really that... if you had 100x more everything would be so different?" "I don't think that's true. So it's back to the age of research again, just with big computers."

Ilya Sutskever says the AI scaling era is running out of road because pre-training is about to hit a finite-data wall. "At some point though, pre-training will run out of data. The data is very clearly finite." "Then, okay, what do you do next? Either you do some kind of a souped-up pre-training... or you're doing RL, or maybe something else." "In some sense, we are back to the age of research." His timeline is the part most AI investors and founders should hear: "From 2012 to 2020, it was the age of research. Now from 2020 to 2025, it was the age of scaling... But now the scale is so big." "Is the belief really that... if you had 100x more everything would be so different?" "I don't think that's true. So it's back to the age of research again, just with big computers."

Karl Mehta

200,003 views • 2 months ago

.President Donald J. Trump on potential tariff dividend payments: "The reason we're even talking about it is that we have so much money coming in from tariffs... but we will be able to make a very substantial dividend to the people of our country."

.President Donald J. Trump on potential tariff dividend payments: "The reason we're even talking about it is that we have so much money coming in from tariffs... but we will be able to make a very substantial dividend to the people of our country."

Rapid Response 47

44,860 views • 7 months ago

Chuck Schumer won't say if he opposes regime change in Venezuela. JAKE TAPPER: Do you disagree with President Trump's ultimate goal of regime change in Venezuela? CHUCK SCHUMER: Look, the bottom line is President Trump throws out so many different things in so many different ways, you don't even know what the heck he's talking about. You know, obviously, if Maduro would just flee on his own, everyone would like that; but we don't know what the heck he's up to when he talks about that. So it's very, very - you cannot say I endorse this, I endorse that - when Trump is all over the lot, not very specific and very worrisome at how far he might escalate.

Chuck Schumer won't say if he opposes regime change in Venezuela. JAKE TAPPER: Do you disagree with President Trump's ultimate goal of regime change in Venezuela? CHUCK SCHUMER: Look, the bottom line is President Trump throws out so many different things in so many different ways, you don't even know what the heck he's talking about. You know, obviously, if Maduro would just flee on his own, everyone would like that; but we don't know what the heck he's up to when he talks about that. So it's very, very - you cannot say I endorse this, I endorse that - when Trump is all over the lot, not very specific and very worrisome at how far he might escalate.

Ken Klippenstein

1,382,969 views • 9 months ago

stop they were talking about how different hoshi is when he's in BSS and when he's together with woozi "how is hoshi when hes with BSS?" 🍊: he's loud and crazy !!! and when he's speaking with woozi he sounds like he's talking to a boss, very well mannered 😭😭

stop they were talking about how different hoshi is when he's in BSS and when he's together with woozi "how is hoshi when hes with BSS?" 🍊: he's loud and crazy !!! and when he's speaking with woozi he sounds like he's talking to a boss, very well mannered 😭😭

꽃보다우아해🌸

231,802 views • 1 year ago

Baseten's Charlie O'Neill on why the gap to the frontier is just compute and data: "Despite OpenAI and Anthropic and the closed source labs having more compute and more data, the recipe is the same. We're all scaling the same recipe here, and we're at slightly different points of the scaling curve. It's not like OpenAI or Anthropic have any secret sauce that differentiates them scaling up their multi-trillion parameter models from the rest of the open source ecosystem." "It really is just a matter of taking the core underlying architecture and training recipe, and throwing more compute and data at it, which is really promising." "Of course, these closed source labs probably have a long tail of optimizations they've made, which we're gonna continue to try and figure out in the open source community. But the core mechanism is there." "K3 is a great example of, well, if you do scale this recipe up, we're gonna get a pretty good model that's very, very close to the frontier, if not frontier itself." Charlie O'Neill Baseten

Baseten's Charlie O'Neill on why the gap to the frontier is just compute and data: "Despite OpenAI and Anthropic and the closed source labs having more compute and more data, the recipe is the same. We're all scaling the same recipe here, and we're at slightly different points of the scaling curve. It's not like OpenAI or Anthropic have any secret sauce that differentiates them scaling up their multi-trillion parameter models from the rest of the open source ecosystem." "It really is just a matter of taking the core underlying architecture and training recipe, and throwing more compute and data at it, which is really promising." "Of course, these closed source labs probably have a long tail of optimizations they've made, which we're gonna continue to try and figure out in the open source community. But the core mechanism is there." "K3 is a great example of, well, if you do scale this recipe up, we're gonna get a pretty good model that's very, very close to the frontier, if not frontier itself." Charlie O'Neill Baseten

MTS

17,404 views • 1 month ago

The whole CM Vijay family doesn’t seem happy with his decision. Look at his parents—they didn’t even look at Trisha. I wonder what he likes so much about her. Anyway, it’s his personal life, and we have no right to comment on it. #TVK #Vijay

The whole CM Vijay family doesn’t seem happy with his decision. Look at his parents—they didn’t even look at Trisha. I wonder what he likes so much about her. Anyway, it’s his personal life, and we have no right to comment on it. #TVK #Vijay

Aheen Maxi

108,554 views • 1 month ago

🇺🇸🇺🇦 Trump: "We are going to supply them with various samples of very modern military equipment, and they will reimburse us 100% for them” How will they pay? What is Trump talking about?

🇺🇸🇺🇦 Trump: "We are going to supply them with various samples of very modern military equipment, and they will reimburse us 100% for them” How will they pay? What is Trump talking about?

Lord Bebo

128,303 views • 1 year ago