Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

We used RL to train models that create curated context from long documents for downstream use by agents. The models sometimes learn to invent their own abbreviations and shorthand. Optimizing with RL for downstream use produces very different artifacts from ordinary summaries: shorter, denser, creatively concise. We call these... show more

Applied Compute

5,020 subscribers

36,155 views • 17 days ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

We developed an RL method for fine-tuning our models for precise tasks in just a few hours or even minutes. Instead of training the whole model, we add an “RL token” output to π-0.6, our latest model, which is used by a tiny actor and critic to learn quickly with RL.

We developed an RL method for fine-tuning our models for precise tasks in just a few hours or even minutes. Instead of training the whole model, we add an “RL token” output to π-0.6, our latest model, which is used by a tiny actor and critic to learn quickly with RL.

Physical Intelligence

434,203 views • 4 months ago

For 3D pose some use different keypoints, others SMPL and other models. It's a mess! With Neural Localizer Fields, we can choose the output at test time! allowing to train using any. Results are real time and SOTA across the board. István Sárándi

For 3D pose some use different keypoints, others SMPL and other models. It's a mess! With Neural Localizer Fields, we can choose the output at test time! allowing to train using any. Results are real time and SOTA across the board. István Sárándi

Gerard Pons-Moll

43,472 views • 2 years ago

Over the last few months, we’ve been thinking about how to learn from “off-domain” data - data from non-robot sources like video or simulation. These data sources are not quite good enough to learn policies (even monolithic VLA models) directly, but they still contain lots of information that can be useful for generalizable robot control. How can we develop robot learning models that are able to make use of this type of data for generalizable control? In new work, that we call HAMSTER, we show that VLMs can be useful for enabling robotic learning from off-domain data, but specifically when used through hierarchical VLA architectures. We show that this class of models can learn generalizable robot policies for the real world from large-scale, off-domain data. A 🧵 (1/10)

Over the last few months, we’ve been thinking about how to learn from “off-domain” data - data from non-robot sources like video or simulation. These data sources are not quite good enough to learn policies (even monolithic VLA models) directly, but they still contain lots of information that can be useful for generalizable robot control. How can we develop robot learning models that are able to make use of this type of data for generalizable control? In new work, that we call HAMSTER, we show that VLMs can be useful for enabling robotic learning from off-domain data, but specifically when used through hierarchical VLA architectures. We show that this class of models can learn generalizable robot policies for the real world from large-scale, off-domain data. A 🧵 (1/10)

Abhishek Gupta

11,994 views • 1 year ago

Exa CEO Will Bryk explains why retrieval can help solve the tokenpocalypse: "We should not be using gigantic models for every task." "You should use a family of models of different sizes. The big model decides what to do, and it dishes out commands to the small models, and those small models can be way more accurate and reliable if they're using retrieval." "Retrieval helps small models act like big models... We do save our customers a huge amount of tokens because they can use smaller models and use retrieval." "We could save 20x on cost for customers compared to other providers by being very efficient in what information from the web the agent actually sees." Will Bryk with Sarah Wang

Exa CEO Will Bryk explains why retrieval can help solve the tokenpocalypse: "We should not be using gigantic models for every task." "You should use a family of models of different sizes. The big model decides what to do, and it dishes out commands to the small models, and those small models can be way more accurate and reliable if they're using retrieval." "Retrieval helps small models act like big models... We do save our customers a huge amount of tokens because they can use smaller models and use retrieval." "We could save 20x on cost for customers compared to other providers by being very efficient in what information from the web the agent actually sees." Will Bryk with Sarah Wang

a16z

25,144 views • 1 month ago

Today's Training Data episode takes us BTS on the infrastructure challenges required to do large RL runs at scale, featuring Federico Cassano (Composer Lead at Cursor) and Dmytro Dzhulgakov (Co-Founder at Fireworks AI). The Cursor team trained Composer 2 on Fireworks by starting with a strong base model (Kimi 2.5) and performing large-scale mid-training on code tokens and web data to learn common patterns and libraries, followed by a large-scale Reinforcement Learning run to learn how to navigate the Cursor harness, call tools, and write correct code. Today's episode dives into the systems and infrastructure challenges of making that large RL run happening, and there were many (!!), from numerical mismatch to global distribution to synchronizing rollouts across asynchronous pipelines to keeping track of expert activation across runs and more. Extremely nerdy in-the-weeds challenges that Federico and Dima were delighted to nerd out on together :) Beyond RL infra, we also discussed Online vs Simulated rollouts, self-summarization for long-horizon agents, environment design ("the most powerful RL environment is the product itself"), and other technical nuggets. PS: We filmed this episode before the SpaceX news, while the Cursor team was still compute-constrained. While Cursor now has *all* the flops, the takeaways and hurdles crossed ring true for any serious application-level company that is racing to post-train their own models. I believe that more serious application companies will go the way of Cursor and post-train their own models. 00:00 Introduction 00:53 Why Cursor Trained Composer 2 04:55 Specialization vs Bitter Lesson 06:16 Composer 2 Training Recipe 16:32 Scaling RL Infrastructure Globally 23:32 Floating Point Drift 25:11 MoE Sensitivity Explained 26:25 Router Replay Fix 27:19 Real Time RL Loop 31:49 Long Horizon Agents 34:29 Why RL Everywhere 37:34 LLM as Judge Rewards 39:14 RL in Hard Domains 40:13 Build Your Own Environments 44:34 Closing Thoughts

Today's Training Data episode takes us BTS on the infrastructure challenges required to do large RL runs at scale, featuring Federico Cassano (Composer Lead at Cursor) and Dmytro Dzhulgakov (Co-Founder at Fireworks AI). The Cursor team trained Composer 2 on Fireworks by starting with a strong base model (Kimi 2.5) and performing large-scale mid-training on code tokens and web data to learn common patterns and libraries, followed by a large-scale Reinforcement Learning run to learn how to navigate the Cursor harness, call tools, and write correct code. Today's episode dives into the systems and infrastructure challenges of making that large RL run happening, and there were many (!!), from numerical mismatch to global distribution to synchronizing rollouts across asynchronous pipelines to keeping track of expert activation across runs and more. Extremely nerdy in-the-weeds challenges that Federico and Dima were delighted to nerd out on together :) Beyond RL infra, we also discussed Online vs Simulated rollouts, self-summarization for long-horizon agents, environment design ("the most powerful RL environment is the product itself"), and other technical nuggets. PS: We filmed this episode before the SpaceX news, while the Cursor team was still compute-constrained. While Cursor now has all the flops, the takeaways and hurdles crossed ring true for any serious application-level company that is racing to post-train their own models. I believe that more serious application companies will go the way of Cursor and post-train their own models. 00:00 Introduction 00:53 Why Cursor Trained Composer 2 04:55 Specialization vs Bitter Lesson 06:16 Composer 2 Training Recipe 16:32 Scaling RL Infrastructure Globally 23:32 Floating Point Drift 25:11 MoE Sensitivity Explained 26:25 Router Replay Fix 27:19 Real Time RL Loop 31:49 Long Horizon Agents 34:29 Why RL Everywhere 37:34 LLM as Judge Rewards 39:14 RL in Hard Domains 40:13 Build Your Own Environments 44:34 Closing Thoughts

Sonya Huang 🐥

78,706 views • 1 month ago

New short course: Build Long-Context AI Apps with Jamba. Learn about state space models (SSMs), which have emerged as an alternative to transformers! Specifically, Jamba is a hybrid transformer-Mamba architecture that combines strengths of the transformer with ideas from SSMs. This course is built with AI21 Labs and taught by Chen Wang and Chen Almagor. The transformer architecture is computationally expensive when handling very long input contexts. But there's an alternative called Mamba, a selective state space model that can process very long contexts with a much lower computational cost. However, researchers found that the pure Mamba architecture underperforms in understanding the context, and gives lower-quality responses. To overcome this, AI21 developed the Jamba model, which combines Mamba's computational efficiency with the transformer's attention mechanism to help with the output quality. In this course, you’ll learn about how state space models, and Jamba, work. You’ll also learn how to prompt Jamba, use it to process long documents, and build long-context RAG apps. - Learn how Jamba combines transformer and state space model architectures to achieve high performance and quality - Use the AI21 SDK, with an example of prompting over a large 200k-token annual financial report of Nvidia - Use Jamba for tool-calling, with hands-on examples from calling simple arithmetic calculations to a function that returns quarterly company financial reports. - Learn how training for long context is done, and the metrics used for its evaluation - Create a RAG app using the AI21 Conversational RAG tool and build your own RAG pipeline that uses Jamba and LangChain. By the end of this course, you'll learn how to build applications that can handle context as long as an entire book. Please sign up here:

New short course: Build Long-Context AI Apps with Jamba. Learn about state space models (SSMs), which have emerged as an alternative to transformers! Specifically, Jamba is a hybrid transformer-Mamba architecture that combines strengths of the transformer with ideas from SSMs. This course is built with AI21 Labs and taught by Chen Wang and Chen Almagor. The transformer architecture is computationally expensive when handling very long input contexts. But there's an alternative called Mamba, a selective state space model that can process very long contexts with a much lower computational cost. However, researchers found that the pure Mamba architecture underperforms in understanding the context, and gives lower-quality responses. To overcome this, AI21 developed the Jamba model, which combines Mamba's computational efficiency with the transformer's attention mechanism to help with the output quality. In this course, you’ll learn about how state space models, and Jamba, work. You’ll also learn how to prompt Jamba, use it to process long documents, and build long-context RAG apps. - Learn how Jamba combines transformer and state space model architectures to achieve high performance and quality - Use the AI21 SDK, with an example of prompting over a large 200k-token annual financial report of Nvidia - Use Jamba for tool-calling, with hands-on examples from calling simple arithmetic calculations to a function that returns quarterly company financial reports. - Learn how training for long context is done, and the metrics used for its evaluation - Create a RAG app using the AI21 Conversational RAG tool and build your own RAG pipeline that uses Jamba and LangChain. By the end of this course, you'll learn how to build applications that can handle context as long as an entire book. Please sign up here:

Andrew Ng

77,792 views • 1 year ago

So we did a bunch of projects with real world reinforcement learning - but it was often too inefficient to be practical to train tabula rasa. This suggests we need better priors, but acquiring these from on-robot data can often be expensive as well. In our recent work, we show that despite being fundamentally inaccurate, simulation can guide provide a cheap way to guide real-world RL finetuning to be super efficient! We propose Simulation-Guided Fine-Tuning (SGFT) - a simple paradigm for sim2real finetuning that uses simulation to provide reward shaping that accelerates real world RL finetuning *beyond* just providing an initialization. TLDR: Use value functions from sim to shape rewards for real-world RL, see large sample efficiency improvements 🧵(1/6)

So we did a bunch of projects with real world reinforcement learning - but it was often too inefficient to be practical to train tabula rasa. This suggests we need better priors, but acquiring these from on-robot data can often be expensive as well. In our recent work, we show that despite being fundamentally inaccurate, simulation can guide provide a cheap way to guide real-world RL finetuning to be super efficient! We propose Simulation-Guided Fine-Tuning (SGFT) - a simple paradigm for sim2real finetuning that uses simulation to provide reward shaping that accelerates real world RL finetuning beyond just providing an initialization. TLDR: Use value functions from sim to shape rewards for real-world RL, see large sample efficiency improvements 🧵(1/6)

Abhishek Gupta

13,637 views • 1 year ago

Today I’m excited to announce Spreadsheet Agents 📊🤖 - a brand-new LlamaIndex feature that allows users to both do data transformation and QA over unnormalized Excel sheets. A lot of knowledge work happens in Excel (or Sheets, Numbers, etc.) - from insurance to tax to corporate finance. It is an extremely nonstandard format that is unsolved by LLMs - neither text-to-CSV nor RAG nor code interpreter work well over these use cases. We built an agent that has really advanced technical capabilities: 1. We do RL-based semantic structure parsing to understand the structure of the Excel sheet 2. We expose these data models through specialized tools to allow an Excel agent to interact with these sheets Huge shoutout to Daniel B. and Zhaoqi Li for working on this 🙌 Currently available in private preview, check out our blog: Come talk to us if you’re interested: Come talk to us if you’re interested:

Jerry Liu

37,901 views • 1 year ago

Learn how to build a Gemini 3 Deep Agents with this video from LangChain. These agents can reason through complex, long horizon tasks by: > Breaking goals into actionable steps > Delegating specific work for specialized models > Leveraging file systems and code execution

Learn how to build a Gemini 3 Deep Agents with this video from LangChain. These agents can reason through complex, long horizon tasks by: > Breaking goals into actionable steps > Delegating specific work for specialized models > Leveraging file systems and code execution

Google AI Developers

19,484 views • 7 months ago

Earlier this week at GTC, we announced our partnership with Nvidia. We will work with Nvidia to build strong, American open-source models that are at the frontier of scientific reasoning. These models will be essential for the US to compete with China on science in the coming decades. Jensen is committing to spend tens of billions of dollars developing open-source models, and we are excited to be a partner with them in figuring out how to benchmark, train and use those agents to accelerate scientific research. We have already open-sourced some of the work we have done with them, and are looking forward to open-sourcing more. There are few things today that are more important. See our blog post below, and watch the video to learn more, narrated by the man himself.

Earlier this week at GTC, we announced our partnership with Nvidia. We will work with Nvidia to build strong, American open-source models that are at the frontier of scientific reasoning. These models will be essential for the US to compete with China on science in the coming decades. Jensen is committing to spend tens of billions of dollars developing open-source models, and we are excited to be a partner with them in figuring out how to benchmark, train and use those agents to accelerate scientific research. We have already open-sourced some of the work we have done with them, and are looking forward to open-sourcing more. There are few things today that are more important. See our blog post below, and watch the video to learn more, narrated by the man himself.

Sam Rodriques

23,301 views • 4 months ago

If we're going to have AIs that fully automate math, they not only need to solve existing problems. We also need to teach them how to recognize what problem to solve next. Human mathematicians have heuristic models that they use to decide what to work on, like, “There’s something important going on with the randomness of primes - let’s codify some patterns we see into specific conjectures.” But these aren’t currently precise enough for RL.

If we're going to have AIs that fully automate math, they not only need to solve existing problems. We also need to teach them how to recognize what problem to solve next. Human mathematicians have heuristic models that they use to decide what to work on, like, “There’s something important going on with the randomness of primes - let’s codify some patterns we see into specific conjectures.” But these aren’t currently precise enough for RL.

Dwarkesh Patel

54,583 views • 3 months ago

3. Agent factory: Foundry is the complete app platform for building apps and agents. We are adding support for more models from Grok, Hugging Face, Meta, Mistral, and more. Plus: Agentic retrieval in Azure AI Search, Foundry Agent Service, integration with Copilot Studio, and more. And we are ensuring the tools you already use for identity, management, and security will now all extend to agents too. The NFL has already used Foundry, along with our data stack, to run its recent Combine.

3. Agent factory: Foundry is the complete app platform for building apps and agents. We are adding support for more models from Grok, Hugging Face, Meta, Mistral, and more. Plus: Agentic retrieval in Azure AI Search, Foundry Agent Service, integration with Copilot Studio, and more. And we are ensuring the tools you already use for identity, management, and security will now all extend to agents too. The NFL has already used Foundry, along with our data stack, to run its recent Combine.

Satya Nadella

180,396 views • 1 year ago

Humans learn and improve from failures. Similarly, foundation models adapt based on human feedback. Can we leverage this failure understanding to enhance robotics systems that use foundation models? Introducing AHA—a vision-language model for detecting and reasoning over failures in robotic manipulation. Project page: 🧵Thread👇 Aha!

Humans learn and improve from failures. Similarly, foundation models adapt based on human feedback. Can we leverage this failure understanding to enhance robotics systems that use foundation models? Introducing AHA—a vision-language model for detecting and reasoning over failures in robotic manipulation. Project page: 🧵Thread👇 Aha!

Jiafei Duan

48,773 views • 1 year ago

.Polymath is training world generation models to automate the creation of RL environments. Traditionally, RL environment generation has been bottlenecked by human data. Superintelligence will never be achieved by human data alone. Polymath is building the core technology to enable automated environment generation using far less human effort than traditionally required, and eventually none. This allows for more complex and realistic worlds, and higher quality, scale, and diversity of tasks. This will be essential to unlock RL scaling. The end goal is to create large-scale, long-horizon environments from a text description alone. This will enable the creation of worlds of arbitrary complexity and scale, which is foundational for training & evaluating autonomous, superintelligent AI agents. Congrats on the launch, Dylan Ma and Naren Yenuganti!

.Polymath is training world generation models to automate the creation of RL environments. Traditionally, RL environment generation has been bottlenecked by human data. Superintelligence will never be achieved by human data alone. Polymath is building the core technology to enable automated environment generation using far less human effort than traditionally required, and eventually none. This allows for more complex and realistic worlds, and higher quality, scale, and diversity of tasks. This will be essential to unlock RL scaling. The end goal is to create large-scale, long-horizon environments from a text description alone. This will enable the creation of worlds of arbitrary complexity and scale, which is foundational for training & evaluating autonomous, superintelligent AI agents. Congrats on the launch, Dylan Ma and Naren Yenuganti!

Y Combinator

44,192 views • 4 months ago

Every home is different. That means that to build a useful home robot, we must be able to perform zero-shot generalization on a wide range of tasks. Humanoid company 1X has a solution: world models. 1X Director of Evaluations Daniel Ho joins us on RoboPapers to talk about: - why world models are the future for scaling robot learning - how to use world models for robot control - what world models unlock for evaluating robot model performance - how we can hill-climb from here to general purpose robots Watch Episode #61 of RoboPapers, with Michael Cho - Rbt/Acc and Chris Paxton, now!

Every home is different. That means that to build a useful home robot, we must be able to perform zero-shot generalization on a wide range of tasks. Humanoid company 1X has a solution: world models. 1X Director of Evaluations Daniel Ho joins us on RoboPapers to talk about: - why world models are the future for scaling robot learning - how to use world models for robot control - what world models unlock for evaluating robot model performance - how we can hill-climb from here to general purpose robots Watch Episode #61 of RoboPapers, with Michael Cho - Rbt/Acc and Chris Paxton, now!

RoboPapers

27,567 views • 5 months ago

"One of the very confusing things about the models right now: how to reconcile the fact that they are doing so well on evals. And you look at the evals and you go, 'Those are pretty hard evals.' But the economic impact seems to be dramatically behind. There is [a possible] explanation. Back when people were doing pre-training, the question of what data to train on was answered, because that answer was everything. So you don't have to think if it's going to be this data or that data. When people do RL training, they say, 'Okay, we want to have this kind of RL training for this thing and that kind of RL training for that thing.' You say, 'Hey, I would love our model to do really well when we release it. I want the evals to look great. What would be RL training that could help on this task?' If you combine this with generalization of the models actually being inadequate, that has the potential to explain a lot of what we are seeing, this disconnect between eval performance and actual real-world performance"

"One of the very confusing things about the models right now: how to reconcile the fact that they are doing so well on evals. And you look at the evals and you go, 'Those are pretty hard evals.' But the economic impact seems to be dramatically behind. There is [a possible] explanation. Back when people were doing pre-training, the question of what data to train on was answered, because that answer was everything. So you don't have to think if it's going to be this data or that data. When people do RL training, they say, 'Okay, we want to have this kind of RL training for this thing and that kind of RL training for that thing.' You say, 'Hey, I would love our model to do really well when we release it. I want the evals to look great. What would be RL training that could help on this task?' If you combine this with generalization of the models actually being inadequate, that has the potential to explain a lot of what we are seeing, this disconnect between eval performance and actual real-world performance"

Dwarkesh Patel

502,060 views • 7 months ago

The network for machine intelligence Two years ago, we laid out our vision for a machine learning compute protocol. One that connects every device in the world into an open network for machine intelligence, with no gatekeepers or artificial boundaries. This week, we’ll be sharing some of our early progress, beginning with RL Swarm, a peer-to-peer system for collaborative reinforcement learning over the internet. Next month, we’ll open our Testnet, allowing anyone to contribute to the frontier of open machine intelligence. Introducing RL Swarm RL Swarm is a fully open source system for collaborative reinforcement learning over the internet. It is a live demo of our research findings, which show that models training with RL learn faster when they train as a collective swarm than they do on their own. Join our swarm now to see this in practice. You can participate with consumer hardware at home or a powerful GPU in the cloud. You can follow along with the swarm’s progress by following the links below.

The network for machine intelligence Two years ago, we laid out our vision for a machine learning compute protocol. One that connects every device in the world into an open network for machine intelligence, with no gatekeepers or artificial boundaries. This week, we’ll be sharing some of our early progress, beginning with RL Swarm, a peer-to-peer system for collaborative reinforcement learning over the internet. Next month, we’ll open our Testnet, allowing anyone to contribute to the frontier of open machine intelligence. Introducing RL Swarm RL Swarm is a fully open source system for collaborative reinforcement learning over the internet. It is a live demo of our research findings, which show that models training with RL learn faster when they train as a collective swarm than they do on their own. Join our swarm now to see this in practice. You can participate with consumer hardware at home or a powerful GPU in the cloud. You can follow along with the swarm’s progress by following the links below.

gensyn

228,703 views • 1 year ago

Many thanks to all those who train our soldiers, who share their experience with our heroes and who themselves learn from Ukrainians. Learn 🇺🇦 resilience, courage and an unbreakable desire to fight for the life for own country and Ukrainian children. It will definitely be – we will fight for this! We will prevail.

Many thanks to all those who train our soldiers, who share their experience with our heroes and who themselves learn from Ukrainians. Learn 🇺🇦 resilience, courage and an unbreakable desire to fight for the life for own country and Ukrainian children. It will definitely be – we will fight for this! We will prevail.

Volodymyr Zelenskyy / Володимир Зеленський

904,267 views • 3 years ago

Percy Liang teaches Stanford students how AI actually thinks now we get to learn it too. same lectures Stanford undergrads pay nearly $90,000 a year for. it teaches the math underneath every AI tool you use: / search and planning -> why agents loop the way they do / MDPs and RL -> why models reward-hack and how to spot it / probability and Bayes nets -> why outputs are calibrated (or aren't) / machine learning fundamentals -> what fine-tuning actually changes

Percy Liang teaches Stanford students how AI actually thinks now we get to learn it too. same lectures Stanford undergrads pay nearly $90,000 a year for. it teaches the math underneath every AI tool you use: / search and planning -> why agents loop the way they do / MDPs and RL -> why models reward-hack and how to spot it / probability and Bayes nets -> why outputs are calibrated (or aren't) / machine learning fundamentals -> what fine-tuning actually changes

dunik

18,446 views • 1 month ago