Video yükleniyor...

Video Yüklenemedi

Bu video yüklenirken bir sorun oluştu. Bu geçici bir ağ sorunundan kaynaklanıyor olabilir veya video kullanılamıyor olabilir.

Ana Sayfaya Dön

here's Jeff Dean talking about how labs will do multi-epoch pretraining with heavy regularization to keep scaling even with limited data. no wonder slowrun gets so much attention from pretraining teams at big labs. pretraining is about to look very very different.

Samip

2,774 subscribers

127,068 görüntüleme • 3 ay önce •via X (Twitter)

Eğitim Bilim & Teknoloji

Anya Rossi• Live Now

Private livecam show

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

D2E Scaling Vision-Action Pretraining on Desktop Data for Transfer to Embodied AI

D2E Scaling Vision-Action Pretraining on Desktop Data for Transfer to Embodied AI

AK

25,095 görüntüleme • 9 ay önce

What if you could induce models to be more concise via pretraining data curation?

What if you could induce models to be more concise via pretraining data curation?

Matthew Leavitt

7,135,743 görüntüleme • 16 gün önce

New short course on Pretraining LLMs! Developed with Upstage and taught by their CEO Sung Kim and CSO Lucy Park. While prompting or fine-tuning existing models works well for many general language tasks, pretraining is valuable for specialized domains or languages with limited representation in current models. This course walks you through the LLM pretraining pipeline: 1. Data preparation: Learn to source, clean, and prepare training data using HuggingFace. 2. Model architecture: Configure transformer networks, including modifying existing models. 3. Training: Set up and run training using open-source libraries. 4. Evaluation: Benchmark performance using popular evaluation strategies. As an example use case, you'll also compare the output of a base model with its fine-tuned and further pretrained variants, to see the impact of pretraining on a model's ability to write Python. The course also explores an innovative technique called depth up-scaling, which Upstage used to train their Solar model family, reducing pretraining compute costs by up to 70%. This technique works by first duplicating layers of a smaller pretrained model to form a larger model, and then further pretraining the result. Sign up here!

New short course on Pretraining LLMs! Developed with Upstage and taught by their CEO Sung Kim and CSO Lucy Park. While prompting or fine-tuning existing models works well for many general language tasks, pretraining is valuable for specialized domains or languages with limited representation in current models. This course walks you through the LLM pretraining pipeline: 1. Data preparation: Learn to source, clean, and prepare training data using HuggingFace. 2. Model architecture: Configure transformer networks, including modifying existing models. 3. Training: Set up and run training using open-source libraries. 4. Evaluation: Benchmark performance using popular evaluation strategies. As an example use case, you'll also compare the output of a base model with its fine-tuned and further pretrained variants, to see the impact of pretraining on a model's ability to write Python. The course also explores an innovative technique called depth up-scaling, which Upstage used to train their Solar model family, reducing pretraining compute costs by up to 70%. This technique works by first duplicating layers of a smaller pretrained model to form a larger model, and then further pretraining the result. Sign up here!

Andrew Ng

85,657 görüntüleme • 2 yıl önce

Excited to introduce 𝐋𝐀𝐏𝐀: the first unsupervised pretraining method for Vision-Language-Action models. Outperforms SOTA models trained with ground-truth actions 30x more efficient than conventional VLA pretraining 📝: 🧵 1/9

Excited to introduce 𝐋𝐀𝐏𝐀: the first unsupervised pretraining method for Vision-Language-Action models. Outperforms SOTA models trained with ground-truth actions 30x more efficient than conventional VLA pretraining 📝: 🧵 1/9

Joel Jang

46,018 görüntüleme • 1 yıl önce

“Pretraining as we know it will end” Robotics: Hold my 🍺 , new data coming Source: Agibot

“Pretraining as we know it will end” Robotics: Hold my 🍺 , new data coming Source: Agibot

Xuxin Cheng

74,897 görüntüleme • 1 yıl önce

We bust some common myths in visuo-motor pretraining! Our data-centric experiments show: 📷 ImageNet > Ego4D for *self-supervised* robotic pretraining 🏆 Diversity matters more than content: training on a broader image distribution beats SOTA by 30% 📊Sim evals can mislead! 1/n

We bust some common myths in visuo-motor pretraining! Our data-centric experiments show: 📷 ImageNet > Ego4D for self-supervised robotic pretraining 🏆 Diversity matters more than content: training on a broader image distribution beats SOTA by 30% 📊Sim evals can mislead! 1/n

Sudeep Dasari

54,329 görüntüleme • 2 yıl önce

Mike Novogratz of Galaxy talks about how $GLXY has raised so much to keep a big $BTC reserves They have been very clear about the directive of the association to be friendly with crypto

Mike Novogratz of Galaxy talks about how $GLXY has raised so much to keep a big $BTC reserves They have been very clear about the directive of the association to be friendly with crypto

CryptoDaddy

10,193 görüntüleme • 8 ay önce

I'll let Noam Brown (OpenAI research) answer Yann's “told you so” again. There is nothing to add. There is no wall, there is no end in sight! "Now, we are in a world, like I said earlier, where the amount of compute that's going into pretraining for things large language models is very, very high. But the inference costs are very low. And there was a reasonable concern among various people that we were going to start seeing diminishing returns from AI progress because the costs and the amount of data that you need for pretraining would become so astronomical. And I think the really important takeaway from o1 is that that wall doesn't actually exist, that we can actually push this a lot further. Because, now, we can scale up inference compute. And there's so much room to scale up inference compute."

I'll let Noam Brown (OpenAI research) answer Yann's “told you so” again. There is nothing to add. There is no wall, there is no end in sight! "Now, we are in a world, like I said earlier, where the amount of compute that's going into pretraining for things large language models is very, very high. But the inference costs are very low. And there was a reasonable concern among various people that we were going to start seeing diminishing returns from AI progress because the costs and the amount of data that you need for pretraining would become so astronomical. And I think the really important takeaway from o1 is that that wall doesn't actually exist, that we can actually push this a lot further. Because, now, we can scale up inference compute. And there's so much room to scale up inference compute."

Chubby♨️

132,808 görüntüleme • 1 yıl önce

Physical Intelligence's π₀, a general-purpose robot foundation model that combines Internet-scale vision-language pretraining with robot interaction data to execute tasks. They aim "to develop foundation models that can control any robot to perform any task" Autonomous demos:

Physical Intelligence's π₀, a general-purpose robot foundation model that combines Internet-scale vision-language pretraining with robot interaction data to execute tasks. They aim "to develop foundation models that can control any robot to perform any task" Autonomous demos:

The Humanoid Hub

196,055 görüntüleme • 1 yıl önce

LingBot-Video is out on Hugging Face MoE-based video foundation model built for embodied intelligence 30B params, only 3B active at inference Augmented with 70K hours of embodied data on top of large-scale internet video pretraining

LingBot-Video is out on Hugging Face MoE-based video foundation model built for embodied intelligence 30B params, only 3B active at inference Augmented with 70K hours of embodied data on top of large-scale internet video pretraining

AK

37,178 görüntüleme • 3 gün önce

NVIDIA just announced EgoScale 🤖🧠 NVIDIA Research has uncovered a log-linear scaling law for robot dexterity by pretraining VLA models on over 20,000 hours of egocentric human video This massive dataset is 20 times larger than previous efforts and proves that robot intelligence follows a predictable path: the more human data, the lower the loss The secret is a simple recipe combining large-scale human pretraining with a small amount of aligned human-robot mid-training to bridge the gap In testing, this method boosted the average success rate by 54% on a 22-DoF robotic hand compared to policies built without pretraining EgoScale also enables one-shot task adaptation and works across different hardware, suggesting that human motion is a universal motor prior for robots Website: Paper: Source: NVIDIA Research #Robot #Humanoid #Robotics #AI #EmbodiedAI #PhysicalAI #NVIDIA #EgoScale #GR00T

NVIDIA just announced EgoScale 🤖🧠 NVIDIA Research has uncovered a log-linear scaling law for robot dexterity by pretraining VLA models on over 20,000 hours of egocentric human video This massive dataset is 20 times larger than previous efforts and proves that robot intelligence follows a predictable path: the more human data, the lower the loss The secret is a simple recipe combining large-scale human pretraining with a small amount of aligned human-robot mid-training to bridge the gap In testing, this method boosted the average success rate by 54% on a 22-DoF robotic hand compared to policies built without pretraining EgoScale also enables one-shot task adaptation and works across different hardware, suggesting that human motion is a universal motor prior for robots Website: Paper: Source: NVIDIA Research #Robot #Humanoid #Robotics #AI #EmbodiedAI #PhysicalAI #NVIDIA #EgoScale #GR00T

RoboHub🤖

43,752 görüntüleme • 4 ay önce

If autonomous finance is the future… how do agents actually find and trust each other? Right now, most can’t. So we teamed up with Tempora Labs to do something about it ↓

If autonomous finance is the future… how do agents actually find and trust each other? Right now, most can’t. So we teamed up with Tempora Labs to do something about it ↓

HOL

22,257 görüntüleme • 4 ay önce

“One of the great things about Iso is how interdisciplinary everyone is, people come from so many different backgrounds.” We spoke with Bilal Khan, Software Engineer at Isomorphic Labs, about our unique, collaborative culture and frontier AI built to tackle the world's most complex problems. Watch the video and head to the comments to explore our current roles.

“One of the great things about Iso is how interdisciplinary everyone is, people come from so many different backgrounds.” We spoke with Bilal Khan, Software Engineer at Isomorphic Labs, about our unique, collaborative culture and frontier AI built to tackle the world's most complex problems. Watch the video and head to the comments to explore our current roles.

Isomorphic Labs

19,614 görüntüleme • 4 ay önce

“he listens to everything, even just things that people wouldn’t pick up on. he pays attention and that’s very sweet, and i love that about him. [..] and one thing i admire about dean is that when you need his attention, he will be there 100%.” stephen about dean for quinn

“he listens to everything, even just things that people wouldn’t pick up on. he pays attention and that’s very sweet, and i love that about him. [..] and one thing i admire about dean is that when you need his attention, he will be there 100%.” stephen about dean for quinn

deanallie lovebot 💭

10,524 görüntüleme • 1 ay önce

you guys i’m crying.. taeyong talks about how he’s very shy and gets nervous about asking people to do challenges so at one of the fansigns i attended, i told him a group i talk to often will be at some music shows with him and would be honored to do a challenge with him.. XLOV!

you guys i’m crying.. taeyong talks about how he’s very shy and gets nervous about asking people to do challenges so at one of the fansigns i attended, i told him a group i talk to often will be at some music shows with him and would be honored to do a challenge with him.. XLOV!

𝔪𝔬𝔬𝔡𝔶 𓆩ꨄ︎𓆪

438,810 görüntüleme • 1 ay önce

She’s talking about how much grace she would give to someone who is speeding and how much she lets them get away with, she’s sounds very responsible to me! While other Leo’s are much stricter. 🙃

She’s talking about how much grace she would give to someone who is speeding and how much she lets them get away with, she’s sounds very responsible to me! While other Leo’s are much stricter. 🙃

Kathleen Winchell ❤️🤍💙🇺🇸🇺🇸

37,223 görüntüleme • 8 ay önce

Punchline: distill world models from simulation to enable fast, stable real-world robot adaptation. Simulation is nearly always wrong. But in Simulation Distillation, we ask a simple question: How do we perform simulation pretraining such that real-world adaptation becomes trivially easy? Let's take a closer look (1/n)

Punchline: distill world models from simulation to enable fast, stable real-world robot adaptation. Simulation is nearly always wrong. But in Simulation Distillation, we ask a simple question: How do we perform simulation pretraining such that real-world adaptation becomes trivially easy? Let's take a closer look (1/n)

Abhishek Gupta

32,411 görüntüleme • 1 ay önce

Why is no one talking about how grok is starting to generate very disturbing and illegal photos

Why is no one talking about how grok is starting to generate very disturbing and illegal photos

✮

135,710 görüntüleme • 6 ay önce

Kana Labs is now Xyra Labs ✨ Change isn’t just about a new name, it’s about growth. Kana Labs is evolving into Xyra Labs, carrying forward the same mission with a fresh vision. The journey continues - stronger, bolder, as Xyra Labs. Read more:

Kana Labs is now Xyra Labs ✨ Change isn’t just about a new name, it’s about growth. Kana Labs is evolving into Xyra Labs, carrying forward the same mission with a fresh vision. The journey continues - stronger, bolder, as Xyra Labs. Read more:

Xyra Labs

25,355 görüntüleme • 9 ay önce

What is it about Black labs? They are built different.

What is it about Black labs? They are built different.

Dead End King

52,837 görüntüleme • 3 ay önce