Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

From Failures to Learnable Worlds Policy-Conditioned Environment Generation for Navigation Agents paper:

AK

534,456 subscribers

17,068 views • 5 days ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

6 Comments

lewington5 days ago

i think it’s just a personalised curriculum for embodied agents

陈放Frank🕊️5 days ago

「导航 agent 失败 → 反过来生环境再训练」这招特别狠——把失败率当数据用而不是砸更多算力，之前 Voyager 那条线就在做这件事，但生成环境的可控性一直卡。 policy-conditioned 比 unconditional 难不少，主要卡在 conditional signal 怎么拿到（动作 label + 失败 case 的对齐）。如果这个方向真跑通，对 Embodied Turing Machines 那类 self-improve 也是直接补一块——agent 可以自己做 curriculum。

LongTian5 days ago

failing forward is the real curriculum here, and conditioning on the policy makes the generated worlds actually bite back

Daniel Priscu5 days ago

building the next environment from where the policy fails beats burning compute on levels it already solved

Broadstreet5 days ago

⏫

Elio科技迷(互)5 days ago

turning failed runs into new training worlds is a neat trick

Related Videos

🚨Excited to share: “𝗣𝗮𝗻𝗼𝗚𝗲𝗻: Text-Conditioned Panoramic Environment Generation for Vision-and-Language Navigation”!🚨 PanoGen creates diverse 360-degree panorama via recursive outpainting, achieving SotA on R2R, R4R, CVDN. Mohit Bansal @uncnlp 🧵

🚨Excited to share: “𝗣𝗮𝗻𝗼𝗚𝗲𝗻: Text-Conditioned Panoramic Environment Generation for Vision-and-Language Navigation”!🚨 PanoGen creates diverse 360-degree panorama via recursive outpainting, achieving SotA on R2R, R4R, CVDN. Mohit Bansal @uncnlp 🧵

Jialu Li

31,991 views • 3 years ago

Environment generation is the missing scaling axis for embodied AI. Introducing SimWorld Studio: a self-evolving factory for endless interactive 3D env where agents act, fail & learn. Env-agent co-evolvution improves navigation success 50% → 90%. From a prompt, our SimCoder writes code to automatically build an interactive world. Agents train inside it. And their performance shapes the next world.

Environment generation is the missing scaling axis for embodied AI. Introducing SimWorld Studio: a self-evolving factory for endless interactive 3D env where agents act, fail & learn. Env-agent co-evolvution improves navigation success 50% → 90%. From a prompt, our SimCoder writes code to automatically build an interactive world. Agents train inside it. And their performance shapes the next world.

SimWorld

2,272,272 views • 5 months ago

Introducing an approach to directly ground video generation models to policy execution without needing any action labels! Our approach uses a generic goal-conditioned exploration procedure to learn a policy that works across robots / embodiments!

Introducing an approach to directly ground video generation models to policy execution without needing any action labels! Our approach uses a generic goal-conditioned exploration procedure to learn a policy that works across robots / embodiments!

Yilun Du

21,427 views • 1 year ago

📣 New paper: Emergence of Maps in the Memories of Blind Navigation Agents Humans have the ability to navigate poorly lit spaces by relying on touch and memory. Our research shows that blind AI agents can learn to do the same. Read the paper ➡️

📣 New paper: Emergence of Maps in the Memories of Blind Navigation Agents Humans have the ability to navigate poorly lit spaces by relying on touch and memory. Our research shows that blind AI agents can learn to do the same. Read the paper ➡️

AI at Meta

89,747 views • 3 years ago

Introducing ShapeR, a method for robust conditional 3D shape generation from casually captured sequences. ShapeR leverages a rectified flow transformer conditioned on per-object multimodal data to turn casual image sequences into full metric scene reconstructions. Project Page: Paper: Links to code and huggingface below ⬇️

Introducing ShapeR, a method for robust conditional 3D shape generation from casually captured sequences. ShapeR leverages a rectified flow transformer conditioned on per-object multimodal data to turn casual image sequences into full metric scene reconstructions. Project Page: Paper: Links to code and huggingface below ⬇️

Yawar Siddiqui

71,078 views • 9 months ago

Gen2Act: Casting language-conditioned manipulation as *human video generation* followed by *closed-loop policy execution conditioned on the generated video* enables solving diverse real-world tasks unseen in the robot dataset! 1/n

Gen2Act: Casting language-conditioned manipulation as human video generation followed by closed-loop policy execution conditioned on the generated video enables solving diverse real-world tasks unseen in the robot dataset! 1/n

Homanga Bharadhwaj

71,299 views • 2 years ago

.Polymath is training world generation models to automate the creation of RL environments. Traditionally, RL environment generation has been bottlenecked by human data. Superintelligence will never be achieved by human data alone. Polymath is building the core technology to enable automated environment generation using far less human effort than traditionally required, and eventually none. This allows for more complex and realistic worlds, and higher quality, scale, and diversity of tasks. This will be essential to unlock RL scaling. The end goal is to create large-scale, long-horizon environments from a text description alone. This will enable the creation of worlds of arbitrary complexity and scale, which is foundational for training & evaluating autonomous, superintelligent AI agents. Congrats on the launch, Dylan Ma and Naren Yenuganti!

.Polymath is training world generation models to automate the creation of RL environments. Traditionally, RL environment generation has been bottlenecked by human data. Superintelligence will never be achieved by human data alone. Polymath is building the core technology to enable automated environment generation using far less human effort than traditionally required, and eventually none. This allows for more complex and realistic worlds, and higher quality, scale, and diversity of tasks. This will be essential to unlock RL scaling. The end goal is to create large-scale, long-horizon environments from a text description alone. This will enable the creation of worlds of arbitrary complexity and scale, which is foundational for training & evaluating autonomous, superintelligent AI agents. Congrats on the launch, Dylan Ma and Naren Yenuganti!

Y Combinator

44,192 views • 7 months ago

New paper on kernel generation: "AdaExplore: Failure-Driven Adaptation and Diversity-Preserving Search for Efficient Kernel Generation" Agents often waste a lot of time making the same mistakes and trying to fix unpromising solutions. AdaExplore accumulates reusable skills based on failures (Adapt) and searches with a mixture of large changes and small refinements (Explore). Outperforms popular methods like OpenEvolve!

New paper on kernel generation: "AdaExplore: Failure-Driven Adaptation and Diversity-Preserving Search for Efficient Kernel Generation" Agents often waste a lot of time making the same mistakes and trying to fix unpromising solutions. AdaExplore accumulates reusable skills based on failures (Adapt) and searches with a mixture of large changes and small refinements (Explore). Outperforms popular methods like OpenEvolve!

Sean Welleck

29,727 views • 5 months ago

LIFT Layout-In-Future Video Generation under Large Viewpoint Change via On-Policy Self-Distillation paper:

LIFT Layout-In-Future Video Generation under Large Viewpoint Change via On-Policy Self-Distillation paper:

AK

17,977 views • 12 days ago

Introducing 🦀 CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents 🦀 CRAB provides an end-to-end and easy-to-use framework to build multimodal agents, operate environments, and create benchmarks to evaluate them, featuring three key components: - 🔀 Cross-environment support - agents can operate tasks in 📱 Android and 💻 Ubuntu. - 🕸️ Graph evaluator - provides a fine-grain evaluation metric for agents. - 🤖 Task generation - composes subtasks to automatically generate tasks. By connecting all devices to agents, 🦀CRAB unlocks greater capabilities for human-like tasks than ever before. Use 🦀 CRAB to benchmark your multimodal agents! - 👨‍💻 Check out the repository: - 📝 Read the paper: - 🌐 Find out more via the project page: - 🐫 Join our community:

Introducing 🦀 CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents 🦀 CRAB provides an end-to-end and easy-to-use framework to build multimodal agents, operate environments, and create benchmarks to evaluate them, featuring three key components: - 🔀 Cross-environment support - agents can operate tasks in 📱 Android and 💻 Ubuntu. - 🕸️ Graph evaluator - provides a fine-grain evaluation metric for agents. - 🤖 Task generation - composes subtasks to automatically generate tasks. By connecting all devices to agents, 🦀CRAB unlocks greater capabilities for human-like tasks than ever before. Use 🦀 CRAB to benchmark your multimodal agents! - 👨‍💻 Check out the repository: - 📝 Read the paper: - 🌐 Find out more via the project page: - 🐫 Join our community:

CAMEL-AI.org

72,807 views • 2 years ago

How many failures will it take for the American government to finally change foreign policy?

How many failures will it take for the American government to finally change foreign policy?

Ron Paul

14,646 views • 4 months ago

Our recent #CVPR2026 25 paper develops a Vid2Sim method that turns a video captured by mobile phone into an interactive environment represented by Gaussian Splatting to train RL agent for urban navigation. Incredible Ziyang Xie leaded the project. Webpage:

Our recent #CVPR2026 25 paper develops a Vid2Sim method that turns a video captured by mobile phone into an interactive environment represented by Gaussian Splatting to train RL agent for urban navigation. Incredible Ziyang Xie leaded the project. Webpage:

Bolei Zhou

31,701 views • 1 year ago

Introducing the Capy Paper integration The easiest way to design with cloud coding agents Paper runs in Capy's VMs, so your agents and teammates can design together from any device

Introducing the Capy Paper integration The easiest way to design with cloud coding agents Paper runs in Capy's VMs, so your agents and teammates can design together from any device

Capy

28,707 views • 11 days ago

Every RL environment we build runs on AgentEnv. Today we’re open-sourcing it, so more people can build the worlds agents learn in.

Every RL environment we build runs on AgentEnv. Today we’re open-sourcing it, so more people can build the worlds agents learn in.

Scale

100,216 views • 14 days ago

PAMI Part Anchored Motion for Text to Human-Object Interaction Generation paper:

PAMI Part Anchored Motion for Text to Human-Object Interaction Generation paper:

AK

20,327 views • 12 days ago

Wow. That's cool. Researchers just released World, an open-source Unreal Engine 5 simulator for training and testing LLM and VLM agents in realistic 3D environments. The platform supports RGB, depth, and segmentation sensors, along with navigation, vehicles, pedestrians, robots and procedural city generation. It's built with a Gym-like Python interface, it allows AI agents to learn physical and social reasoning in complex virtual worlds before real world deployment. We're moving from AI that only understands information to AI that can perceive, reason and act inside realistic simulated environments.

Wow. That's cool. Researchers just released World, an open-source Unreal Engine 5 simulator for training and testing LLM and VLM agents in realistic 3D environments. The platform supports RGB, depth, and segmentation sensors, along with navigation, vehicles, pedestrians, robots and procedural city generation. It's built with a Gym-like Python interface, it allows AI agents to learn physical and social reasoning in complex virtual worlds before real world deployment. We're moving from AI that only understands information to AI that can perceive, reason and act inside realistic simulated environments.

SciTech Era

47,751 views • 4 months ago

Microsoft presents World-R1 Reinforcing 3D Constraints for Text-to-Video Generation paper:

Microsoft presents World-R1 Reinforcing 3D Constraints for Text-to-Video Generation paper:

AK

24,610 views • 5 months ago

📢📢📢 Excited to share our new work *Autonomous Character-Scene Interaction Synthesis from Text Instruction* (Siggraph Asia 24). It presents a unified model for flexible scene-conditioned motion generation given text, scene, trajectory conditions. The results with smooth interaction look very impressive! 📰Paper: Project: Code and data will be released soon.

📢📢📢 Excited to share our new work Autonomous Character-Scene Interaction Synthesis from Text Instruction (Siggraph Asia 24). It presents a unified model for flexible scene-conditioned motion generation given text, scene, trajectory conditions. The results with smooth interaction look very impressive! 📰Paper: Project: Code and data will be released soon.

Siyuan Huang

11,451 views • 2 years ago