Загрузка видео...

Не удалось загрузить видео

Возникла проблема при загрузке этого видео. Это может быть связано с временными проблемами сети или видео может быть недоступно.

На главную

We used RL to train models that create curated context from long documents for downstream use by agents. The models sometimes learn to invent their own abbreviations and shorthand. Optimizing with RL for downstream use produces very different artifacts from ordinary summaries: shorter, denser, creatively concise. We call these... show more

Applied Compute

8,346 subscribers

37,368 просмотров • 4 месяцев назад •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

Комментарии: 8

Фото профиля Applied Compute

Applied Compute4 месяцев назад

Neural cheat-sheets approach the performance of learned KV-caches — which are several orders of magnitude larger and not human-readable — while preserving the auditability of natural language summaries that enterprise workflows require. One can think of a neural-cheat sheet as a learned cache in token space.

Фото профиля Applied Compute

Applied Compute4 месяцев назад

At deployment time, the trained model takes in a new knowledge artifact: an enterprise SOP, a meeting transcript, a coding agent trace, or even a novella, and outputs a dense set of notes that captures fine-grained detail and amortizes reasoning. This approach is used to build memory for agents, instantiated at Applied Compute as a ContextBase.

Фото профиля Applied Compute

Applied Compute4 месяцев назад

Read the full research report on our blog:

Фото профиля Rohan Rao

Rohan Rao3 месяцев назад

Aligns with extraction of domain-specific ontologies and schemas from unstructured text documents, something for which we fine-tuned (with SFT/LoRA) specialized triplet extraction models for finance, chip design, legal and healthcare a couple of years ago!

Фото профиля Uzay

Uzay4 месяцев назад

nice - @KaivuHariharan @atticuswzf cool to see someone do this

Фото профиля Jake

Jake3 месяцев назад

this should also be used for compaction

Фото профиля legalprimes

legalprimes3 месяцев назад

Neat

Фото профиля Sebastian Buzdugan

Sebastian Buzdugan3 месяцев назад

watched shorthand help agents until one abbreviation collision poisoned the next retrieval step

Похожие видео

We developed an RL method for fine-tuning our models for precise tasks in just a few hours or even minutes. Instead of training the whole model, we add an “RL token” output to π-0.6, our latest model, which is used by a tiny actor and critic to learn quickly with RL.

We developed an RL method for fine-tuning our models for precise tasks in just a few hours or even minutes. Instead of training the whole model, we add an “RL token” output to π-0.6, our latest model, which is used by a tiny actor and critic to learn quickly with RL.

Physical Intelligence

447,445 просмотров • 7 месяцев назад

A conversation on the optimal reward for coding agents, infinite context models, and real-time RL

A conversation on the optimal reward for coding agents, infinite context models, and real-time RL

Cursor

320,339 просмотров • 1 год назад

trained my own with trl + openenv blog with open artifacts (code, models, dataset, rl env...) soon!

trained my own with trl + openenv blog with open artifacts (code, models, dataset, rl env...) soon!

Sergio Paniego

90,257 просмотров • 1 месяц назад

New paper: Multi-Faceted Interactivity Alignment in Full-Duplex Speech Models We use RL to post-train speech models (Moshi and PersonaPlex) to talk more like a human: to know when to respond, when to wait, and when to nod along with “yeah”s and “okay”s when listening.

New paper: Multi-Faceted Interactivity Alignment in Full-Duplex Speech Models We use RL to post-train speech models (Moshi and PersonaPlex) to talk more like a human: to know when to respond, when to wait, and when to nod along with “yeah”s and “okay”s when listening.

kyutai

33,721 просмотров • 4 месяцев назад

How do we run RL with real-time chunking (RTC)? In this work we figured out how to use a small RL policy with a large robot foundation model, where the RL policy observed more recent images (due to faster inference) and steers the policy toward better behaviors! A fun collaboration with Siemens, led by Brian Zhu, Momen Khalil, Emanuele Poggi from Siemens and E Harrison from Berkeley, with lots of amazing contributors!

How do we run RL with real-time chunking (RTC)? In this work we figured out how to use a small RL policy with a large robot foundation model, where the RL policy observed more recent images (due to faster inference) and steers the policy toward better behaviors! A fun collaboration with Siemens, led by Brian Zhu, Momen Khalil, Emanuele Poggi from Siemens and E Harrison from Berkeley, with lots of amazing contributors!

Sergey Levine

47,700 просмотров • 1 месяц назад

For 3D pose some use different keypoints, others SMPL and other models. It's a mess! With Neural Localizer Fields, we can choose the output at test time! allowing to train using any. Results are real time and SOTA across the board. István Sárándi

For 3D pose some use different keypoints, others SMPL and other models. It's a mess! With Neural Localizer Fields, we can choose the output at test time! allowing to train using any. Results are real time and SOTA across the board. István Sárándi

Gerard Pons-Moll

43,598 просмотров • 2 лет назад

Hiring RL Engineer! Started off as a curious project at Lossfunk to push the boundaries of LLMs in social reasoning - we are now building RL environments, data, and benchmarks to simulate more real-world scenarios. If you want to train SoTA RL models over multi-GPUs (H200s/B200s) to unlock next AI frontier, this is for you.

Hiring RL Engineer! Started off as a curious project at Lossfunk to push the boundaries of LLMs in social reasoning - we are now building RL environments, data, and benchmarks to simulate more real-world scenarios. If you want to train SoTA RL models over multi-GPUs (H200s/B200s) to unlock next AI frontier, this is for you.

Satpal Singh Rathore

46,010 просмотров • 1 год назад

🧳 Bring Your Own AI Models to GitHub Copilot in VS Code Use your own Azure-hosted AI models with GitHub Copilot in VS Code. See how Bring Your Own Key (BYOK) lets you connect models from Microsoft Foundry, choose which models appear in Copilot, and use the right model for different coding tasks.

🧳 Bring Your Own AI Models to GitHub Copilot in VS Code Use your own Azure-hosted AI models with GitHub Copilot in VS Code. See how Bring Your Own Key (BYOK) lets you connect models from Microsoft Foundry, choose which models appear in Copilot, and use the right model for different coding tasks.

Visual Studio Code

38,536 просмотров • 1 месяц назад

Over the last few months, we’ve been thinking about how to learn from “off-domain” data - data from non-robot sources like video or simulation. These data sources are not quite good enough to learn policies (even monolithic VLA models) directly, but they still contain lots of information that can be useful for generalizable robot control. How can we develop robot learning models that are able to make use of this type of data for generalizable control? In new work, that we call HAMSTER, we show that VLMs can be useful for enabling robotic learning from off-domain data, but specifically when used through hierarchical VLA architectures. We show that this class of models can learn generalizable robot policies for the real world from large-scale, off-domain data. A 🧵 (1/10)

Over the last few months, we’ve been thinking about how to learn from “off-domain” data - data from non-robot sources like video or simulation. These data sources are not quite good enough to learn policies (even monolithic VLA models) directly, but they still contain lots of information that can be useful for generalizable robot control. How can we develop robot learning models that are able to make use of this type of data for generalizable control? In new work, that we call HAMSTER, we show that VLMs can be useful for enabling robotic learning from off-domain data, but specifically when used through hierarchical VLA architectures. We show that this class of models can learn generalizable robot policies for the real world from large-scale, off-domain data. A 🧵 (1/10)

Abhishek Gupta

11,994 просмотров • 1 год назад

Today's Training Data episode takes us BTS on the infrastructure challenges required to do large RL runs at scale, featuring Federico Cassano (Composer Lead at Cursor) and Dmytro Dzhulgakov (Co-Founder at Fireworks). The Cursor team trained Composer 2 on Fireworks by starting with a strong base model (Kimi 2.5) and performing large-scale mid-training on code tokens and web data to learn common patterns and libraries, followed by a large-scale Reinforcement Learning run to learn how to navigate the Cursor harness, call tools, and write correct code. Today's episode dives into the systems and infrastructure challenges of making that large RL run happening, and there were many (!!), from numerical mismatch to global distribution to synchronizing rollouts across asynchronous pipelines to keeping track of expert activation across runs and more. Extremely nerdy in-the-weeds challenges that Federico and Dima were delighted to nerd out on together :) Beyond RL infra, we also discussed Online vs Simulated rollouts, self-summarization for long-horizon agents, environment design ("the most powerful RL environment is the product itself"), and other technical nuggets. PS: We filmed this episode before the SpaceX news, while the Cursor team was still compute-constrained. While Cursor now has *all* the flops, the takeaways and hurdles crossed ring true for any serious application-level company that is racing to post-train their own models. I believe that more serious application companies will go the way of Cursor and post-train their own models. 00:00 Introduction 00:53 Why Cursor Trained Composer 2 04:55 Specialization vs Bitter Lesson 06:16 Composer 2 Training Recipe 16:32 Scaling RL Infrastructure Globally 23:32 Floating Point Drift 25:11 MoE Sensitivity Explained 26:25 Router Replay Fix 27:19 Real Time RL Loop 31:49 Long Horizon Agents 34:29 Why RL Everywhere 37:34 LLM as Judge Rewards 39:14 RL in Hard Domains 40:13 Build Your Own Environments 44:34 Closing Thoughts

Today's Training Data episode takes us BTS on the infrastructure challenges required to do large RL runs at scale, featuring Federico Cassano (Composer Lead at Cursor) and Dmytro Dzhulgakov (Co-Founder at Fireworks). The Cursor team trained Composer 2 on Fireworks by starting with a strong base model (Kimi 2.5) and performing large-scale mid-training on code tokens and web data to learn common patterns and libraries, followed by a large-scale Reinforcement Learning run to learn how to navigate the Cursor harness, call tools, and write correct code. Today's episode dives into the systems and infrastructure challenges of making that large RL run happening, and there were many (!!), from numerical mismatch to global distribution to synchronizing rollouts across asynchronous pipelines to keeping track of expert activation across runs and more. Extremely nerdy in-the-weeds challenges that Federico and Dima were delighted to nerd out on together :) Beyond RL infra, we also discussed Online vs Simulated rollouts, self-summarization for long-horizon agents, environment design ("the most powerful RL environment is the product itself"), and other technical nuggets. PS: We filmed this episode before the SpaceX news, while the Cursor team was still compute-constrained. While Cursor now has all the flops, the takeaways and hurdles crossed ring true for any serious application-level company that is racing to post-train their own models. I believe that more serious application companies will go the way of Cursor and post-train their own models. 00:00 Introduction 00:53 Why Cursor Trained Composer 2 04:55 Specialization vs Bitter Lesson 06:16 Composer 2 Training Recipe 16:32 Scaling RL Infrastructure Globally 23:32 Floating Point Drift 25:11 MoE Sensitivity Explained 26:25 Router Replay Fix 27:19 Real Time RL Loop 31:49 Long Horizon Agents 34:29 Why RL Everywhere 37:34 LLM as Judge Rewards 39:14 RL in Hard Domains 40:13 Build Your Own Environments 44:34 Closing Thoughts

Sonya Huang 🐥

81,490 просмотров • 5 месяцев назад

New Anthropic research: Natural emergent misalignment from reward hacking in production RL. “Reward hacking” is where models learn to cheat on tasks they’re given during training. Our new study finds that the consequences of reward hacking, if unmitigated, can be very serious.

New Anthropic research: Natural emergent misalignment from reward hacking in production RL. “Reward hacking” is where models learn to cheat on tasks they’re given during training. Our new study finds that the consequences of reward hacking, if unmitigated, can be very serious.

Anthropic

2,619,798 просмотров • 11 месяцев назад

Today, we’re excited to announce Dojo, a collaborative RL environment suite for computer use agents (CUA).

Today, we’re excited to announce Dojo, a collaborative RL environment suite for computer use agents (CUA).

Chakra Labs

53,623 просмотров • 1 год назад

Google DeepMind, David Silver reveals: we built a system that used RL to discover its own RL algorithms. this AI-designed system outperformed all human-created RL algorithms developed over the years.

Google DeepMind, David Silver reveals: we built a system that used RL to discover its own RL algorithms. this AI-designed system outperformed all human-created RL algorithms developed over the years.

Haider.

397,296 просмотров • 1 год назад

Exa CEO Will Bryk explains why retrieval can help solve the tokenpocalypse: "We should not be using gigantic models for every task." "You should use a family of models of different sizes. The big model decides what to do, and it dishes out commands to the small models, and those small models can be way more accurate and reliable if they're using retrieval." "Retrieval helps small models act like big models... We do save our customers a huge amount of tokens because they can use smaller models and use retrieval." "We could save 20x on cost for customers compared to other providers by being very efficient in what information from the web the agent actually sees." Will Bryk with Sarah Wang

Exa CEO Will Bryk explains why retrieval can help solve the tokenpocalypse: "We should not be using gigantic models for every task." "You should use a family of models of different sizes. The big model decides what to do, and it dishes out commands to the small models, and those small models can be way more accurate and reliable if they're using retrieval." "Retrieval helps small models act like big models... We do save our customers a huge amount of tokens because they can use smaller models and use retrieval." "We could save 20x on cost for customers compared to other providers by being very efficient in what information from the web the agent actually sees." Will Bryk with Sarah Wang

a16z

25,144 просмотров • 4 месяцев назад

New short course: Build Long-Context AI Apps with Jamba. Learn about state space models (SSMs), which have emerged as an alternative to transformers! Specifically, Jamba is a hybrid transformer-Mamba architecture that combines strengths of the transformer with ideas from SSMs. This course is built with AI21 Labs and taught by Chen Wang and Chen Almagor. The transformer architecture is computationally expensive when handling very long input contexts. But there's an alternative called Mamba, a selective state space model that can process very long contexts with a much lower computational cost. However, researchers found that the pure Mamba architecture underperforms in understanding the context, and gives lower-quality responses. To overcome this, AI21 developed the Jamba model, which combines Mamba's computational efficiency with the transformer's attention mechanism to help with the output quality. In this course, you’ll learn about how state space models, and Jamba, work. You’ll also learn how to prompt Jamba, use it to process long documents, and build long-context RAG apps. - Learn how Jamba combines transformer and state space model architectures to achieve high performance and quality - Use the AI21 SDK, with an example of prompting over a large 200k-token annual financial report of Nvidia - Use Jamba for tool-calling, with hands-on examples from calling simple arithmetic calculations to a function that returns quarterly company financial reports. - Learn how training for long context is done, and the metrics used for its evaluation - Create a RAG app using the AI21 Conversational RAG tool and build your own RAG pipeline that uses Jamba and LangChain. By the end of this course, you'll learn how to build applications that can handle context as long as an entire book. Please sign up here:

New short course: Build Long-Context AI Apps with Jamba. Learn about state space models (SSMs), which have emerged as an alternative to transformers! Specifically, Jamba is a hybrid transformer-Mamba architecture that combines strengths of the transformer with ideas from SSMs. This course is built with AI21 Labs and taught by Chen Wang and Chen Almagor. The transformer architecture is computationally expensive when handling very long input contexts. But there's an alternative called Mamba, a selective state space model that can process very long contexts with a much lower computational cost. However, researchers found that the pure Mamba architecture underperforms in understanding the context, and gives lower-quality responses. To overcome this, AI21 developed the Jamba model, which combines Mamba's computational efficiency with the transformer's attention mechanism to help with the output quality. In this course, you’ll learn about how state space models, and Jamba, work. You’ll also learn how to prompt Jamba, use it to process long documents, and build long-context RAG apps. - Learn how Jamba combines transformer and state space model architectures to achieve high performance and quality - Use the AI21 SDK, with an example of prompting over a large 200k-token annual financial report of Nvidia - Use Jamba for tool-calling, with hands-on examples from calling simple arithmetic calculations to a function that returns quarterly company financial reports. - Learn how training for long context is done, and the metrics used for its evaluation - Create a RAG app using the AI21 Conversational RAG tool and build your own RAG pipeline that uses Jamba and LangChain. By the end of this course, you'll learn how to build applications that can handle context as long as an entire book. Please sign up here:

Andrew Ng

77,792 просмотров • 1 год назад

Lecture 10 of my course! Nominally on regularization in RL, so I discuss the evolving role of the KL penalty in RL, but also a set of nice RL papers that explain what RL helps models generalize better than SFT -- with theory supporting it. When going through these, it's so interesting how seasonal problems in ML are. Lots of problems from controlling reward models overopt will rhyme as we try to control rubrics for agents. 00:00 Intro & the role of regularization 02:50 The KL penalty in RL 10:53 RL as a reverse KL loss 21:09 Why RL generalizes better than SFT 25:15 Other regularization tools Just a few videos left as I get to the end of the course. Thanks all, and keep sending questions. Spread the word if you have a second.

Lecture 10 of my course! Nominally on regularization in RL, so I discuss the evolving role of the KL penalty in RL, but also a set of nice RL papers that explain what RL helps models generalize better than SFT -- with theory supporting it. When going through these, it's so interesting how seasonal problems in ML are. Lots of problems from controlling reward models overopt will rhyme as we try to control rubrics for agents. 00:00 Intro & the role of regularization 02:50 The KL penalty in RL 10:53 RL as a reverse KL loss 21:09 Why RL generalizes better than SFT 25:15 Other regularization tools Just a few videos left as I get to the end of the course. Thanks all, and keep sending questions. Spread the word if you have a second.

Nathan Lambert

89,819 просмотров • 3 месяцев назад

We are moving from model controller to RL for locomotion. The policy is starting to look quite robust. Even more convinced now that legs are easier than wheels, given the software complexity has shrunk tremendously with RL.

We are moving from model controller to RL for locomotion. The policy is starting to look quite robust. Even more convinced now that legs are easier than wheels, given the software complexity has shrunk tremendously with RL.

Sankaet

133,130 просмотров • 1 год назад