正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Scalable, reproducible, and reliable robotic evaluation remains an open challenge, especially in the age of generalist robot foundation models. Can simulation effectively predict real-world robot policy performance & behavior? Presenting SIMPLER!👇

Xuanlin Li (Simon)

1,183 subscribers

86,431 次观看 • 2 年前 •via X (Twitter)

科学技术新闻政治教育

Anya Rossi• Live Now

Private livecam show

3 条评论

Xuanlin Li (Simon) 的头像

Xuanlin Li (Simon)2 年前

SIMPLER contributes to open-source robotics by enabling the evaluation of real-world robot policies without extensive hardware setups, continuous human supervision, or manual environment resets. See @kylehkhsu’s thread for a deep dive!

Yunhao Fang 的头像

Yunhao Fang2 年前

Nice work Xuanlin.👏

Mustafa 的头像

Mustafa2 年前

Let’s chat. Unfortunately can’t dm you

相关视频

📢 Announcing one of the most exciting works from us this year on **scalable robot policy evaluation through real-to-sim transfer**, moving toward a scalable evaluation engine with structured world models that capture the appearance, geometry, and dynamics of environments involving deformable objects. 🤖 Evaluation remains one of the biggest bottlenecks in building general-purpose robots. Today, robots are still evaluated only in the real world, which is **orders of magnitude slower** than the development of language agents. We propose a new framework where simulation performance **strongly correlates** with the real world (r > 0.9), even for deformable objects. The key difference from existing work lies in the correlation between simulation and reality: if a robot model performs better in the digital world, does it also perform better in the real world? This question has long made people hesitant about simulation-based evaluation — especially for deformable objects. We are changing that. Our pipeline achieves effective real-to-sim transfer, establishing **state-of-the-art correlation** between simulation and reality for deformable object manipulation. It provides a **scalable and reproducible evaluation engine** for robot learning. 🌐

📢 Announcing one of the most exciting works from us this year on scalable robot policy evaluation through real-to-sim transfer, moving toward a scalable evaluation engine with structured world models that capture the appearance, geometry, and dynamics of environments involving deformable objects. 🤖 Evaluation remains one of the biggest bottlenecks in building general-purpose robots. Today, robots are still evaluated only in the real world, which is orders of magnitude slower than the development of language agents. We propose a new framework where simulation performance strongly correlates with the real world (r > 0.9), even for deformable objects. The key difference from existing work lies in the correlation between simulation and reality: if a robot model performs better in the digital world, does it also perform better in the real world? This question has long made people hesitant about simulation-based evaluation — especially for deformable objects. We are changing that. Our pipeline achieves effective real-to-sim transfer, establishing state-of-the-art correlation between simulation and reality for deformable object manipulation. It provides a scalable and reproducible evaluation engine for robot learning. 🌐

Yunzhu Li

39,960 次观看 • 9 个月前

Thrilled to announce Octo 🐙, an open-source robot foundation model! Octo is a sota generalist robot policy based on transformer+diffusion. Most importantly, you can finetune Octo *today* with flexible observation and action spaces on your robot setup!

Thrilled to announce Octo 🐙, an open-source robot foundation model! Octo is a sota generalist robot policy based on transformer+diffusion. Most importantly, you can finetune Octo today with flexible observation and action spaces on your robot setup!

Oier Mees

44,944 次观看 • 2 年前

Given the strong community adoption and real-world deployment of MolmoAct2 on YAM, we're introducing zero-shot evaluation of MolmoAct2 Bimanual YAM in simulation. Now you can test out our models without a real-world robot and build on them! Code: Simulation built on Maniskill!

Given the strong community adoption and real-world deployment of MolmoAct2 on YAM, we're introducing zero-shot evaluation of MolmoAct2 Bimanual YAM in simulation. Now you can test out our models without a real-world robot and build on them! Code: Simulation built on Maniskill!

Jiafei Duan

36,869 次观看 • 1 个月前

How do we make robot policy evaluation reproducible across labs? Announcing OpenArm 02: a fully open-source dual arm platform, with a unified setup for reproducible evaluation: controlled tasks, backgrounds, lighting, and camera angles. ✅ Fully open-source, so anyone can replicate the setup anywhere ✅ AutoEval: 24/7 real-world evaluation with minimal human intervention (building on work by Paul Zhou) ✅ Fair side-by-side policy comparison OpenArm is the open foundation to turn isolated results into shared progress. Releasing soon. Stay tuned.

How do we make robot policy evaluation reproducible across labs? Announcing OpenArm 02: a fully open-source dual arm platform, with a unified setup for reproducible evaluation: controlled tasks, backgrounds, lighting, and camera angles. ✅ Fully open-source, so anyone can replicate the setup anywhere ✅ AutoEval: 24/7 real-world evaluation with minimal human intervention (building on work by Paul Zhou) ✅ Fair side-by-side policy comparison OpenArm is the open foundation to turn isolated results into shared progress. Releasing soon. Stay tuned.

Enactic

64,759 次观看 • 5 个月前

Evaluation is a critical bottleneck in building robot foundation models. Check out our latest work RoboLab, led by Xuning Yang, which addresses this exact challenge. Its a high-fidelity simulation environment for testing these models. A truly generalist policy should be able to complete these tasks zero-shot, and this benchmark highlights exactly how far we still have to go. More info 👇

Evaluation is a critical bottleneck in building robot foundation models. Check out our latest work RoboLab, led by Xuning Yang, which addresses this exact challenge. Its a high-fidelity simulation environment for testing these models. A truly generalist policy should be able to complete these tasks zero-shot, and this benchmark highlights exactly how far we still have to go. More info 👇

Ankit Goyal

29,910 次观看 • 3 个月前

We’re organizing the RoboArena Challenge at CoRL this year! Show the performance of your best generalist policy, in a fair, open benchmark for the robotics community! 🤖 Sign up, even if you don’t have a robot! More details in 🧵👇

We’re organizing the RoboArena Challenge at CoRL this year! Show the performance of your best generalist policy, in a fair, open benchmark for the robotics community! 🤖 Sign up, even if you don’t have a robot! More details in 🧵👇

Karl Pertsch

26,775 次观看 • 1 年前

The future of robot butlers starts with mobile manipulation. We’re announcing the NeurIPS 2023 Open-Vocabulary Mobile Manipulation Challenge! - Full robot stack ✅ - Parallel sim and real evaluation ✅ - No robot required ✅👀

The future of robot butlers starts with mobile manipulation. We’re announcing the NeurIPS 2023 Open-Vocabulary Mobile Manipulation Challenge! - Full robot stack ✅ - Parallel sim and real evaluation ✅ - No robot required ✅👀

Chris Paxton

178,996 次观看 • 3 年前

Qwen-VLA feels like one of the first real robotics foundation models. A single system trained across robot manipulation, navigation, egocentric human video, simulation, and vision-language reasoning instead of isolated robot policies.

Qwen-VLA feels like one of the first real robotics foundation models. A single system trained across robot manipulation, navigation, egocentric human video, simulation, and vision-language reasoning instead of isolated robot policies.

Robots Digest 🤖

14,722 次观看 • 2 个月前

#AGIBOTAIWeek Day4: Introducing GE-Sim 2.0 (Genie Envisioner World Simulator 2.0) , a comprehensive embodied world simulator for robotic manipulation. By integrating future video generation, proprioceptive state estimation, and reward-based policy assessment, GE-Sim 2.0 moves beyond visual simulation toward an embodied world simulator with built-in evaluation capability, laying the foundation for a world-model-centric platform for closed-loop and interactive policy learning. It provides a practical framework for scalable evaluation and training of robotic manipulation policies. Project page: #AGIBOT #AGIBOTAIWeek #GenieEnvisioner #Robotics #visualsimulation

#AGIBOTAIWeek Day4: Introducing GE-Sim 2.0 (Genie Envisioner World Simulator 2.0) , a comprehensive embodied world simulator for robotic manipulation. By integrating future video generation, proprioceptive state estimation, and reward-based policy assessment, GE-Sim 2.0 moves beyond visual simulation toward an embodied world simulator with built-in evaluation capability, laying the foundation for a world-model-centric platform for closed-loop and interactive policy learning. It provides a practical framework for scalable evaluation and training of robotic manipulation policies. Project page: #AGIBOT #AGIBOTAIWeek #GenieEnvisioner #Robotics #visualsimulation

AGIBOT

28,219 次观看 • 3 个月前

Excited to release τ0-WM: an open-source unified video-action world model for robotic manipulation. It's a 5B-parameter robotic foundation model trained on 27.3K hours of real-robot teleoperation, UMI-style demonstrations, and egocentric interaction videos.

Excited to release τ0-WM: an open-source unified video-action world model for robotic manipulation. It's a 5B-parameter robotic foundation model trained on 27.3K hours of real-robot teleoperation, UMI-style demonstrations, and egocentric interaction videos.

Jianlan Luo

63,831 次观看 • 2 个月前

Why do generalist robotic models fail when a cup is moved just two inches to the left? It’s not a lack of motor skill, it’s an alignment problem. Today, we introduce VLS: Vision-Language Steering of Pretrained Robot Policies, a training-free framework that guides robot behavior in real time. Check out the project: 👇🧵 (Watch till the end: VLS runs uncut, steering pretrained policies across long-horizon tasks.)

Why do generalist robotic models fail when a cup is moved just two inches to the left? It’s not a lack of motor skill, it’s an alignment problem. Today, we introduce VLS: Vision-Language Steering of Pretrained Robot Policies, a training-free framework that guides robot behavior in real time. Check out the project: 👇🧵 (Watch till the end: VLS runs uncut, steering pretrained policies across long-horizon tasks.)

Jiafei Duan

72,465 次观看 • 6 个月前

NVIDIA Isaac GR00T N1 is an open generalist foundation model for #humanoidrobots. 🤖 Discover how the model can easily generalize common tasks, boosting robot reasoning and skills. ▶️

NVIDIA Isaac GR00T N1 is an open generalist foundation model for #humanoidrobots. 🤖 Discover how the model can easily generalize common tasks, boosting robot reasoning and skills. ▶️

NVIDIA Robotics

53,497 次观看 • 1 年前

Testing robot policies on hardware is slow, expensive and hard to scale. World models offer a promising path to accelerating robot policy development. We're sharing new research from the Runway Robotics team, in which we simulated 8 robot policies inside our General World Model and found 0.95 correlation with real-world results. Those early results point to world model simulation as a practical substitute for hardware evaluation, comparing favorably to existing real-to-sim approaches. Learn more at the link below.

Testing robot policies on hardware is slow, expensive and hard to scale. World models offer a promising path to accelerating robot policy development. We're sharing new research from the Runway Robotics team, in which we simulated 8 robot policies inside our General World Model and found 0.95 correlation with real-world results. Those early results point to world model simulation as a practical substitute for hardware evaluation, comparing favorably to existing real-to-sim approaches. Learn more at the link below.

Runway

13,464 次观看 • 5 个月前

What happens when robot world models learn from human experience at scale? 🤔 DreamDojo from NVIDIA Research is a generalist robot world model pretrained on 44K hours of egocentric human videos and then post-trained on robot data to generalize across new objects and environments. After distillation, it runs at 10 FPS for live teleoperation, policy evaluation, and model-based planning. Read the ICML paper to learn more 📄

What happens when robot world models learn from human experience at scale? 🤔 DreamDojo from NVIDIA Research is a generalist robot world model pretrained on 44K hours of egocentric human videos and then post-trained on robot data to generalize across new objects and environments. After distillation, it runs at 10 FPS for live teleoperation, policy evaluation, and model-based planning. Read the ICML paper to learn more 📄

NVIDIA Robotics

22,413 次观看 • 29 天前

We open-source Action Images — a new representation that translates 7-DoF robot actions into interpretable images. Video models are emerging as powerful robotic foundation models, but a key challenge remains: how can we seamlessly integrate robot policies into video models? Instead of representing actions as low-dimensional control tokens, Action Images provide a pixel-grounded action representation, reframing policy learning as a visual tracking problem! By unifying observations and actions in the same video space, Action Images enable a unified robotics world model that supports video-action joint generation, action-conditioned video generation, and action labeling! Code: Paper:

We open-source Action Images — a new representation that translates 7-DoF robot actions into interpretable images. Video models are emerging as powerful robotic foundation models, but a key challenge remains: how can we seamlessly integrate robot policies into video models? Instead of representing actions as low-dimensional control tokens, Action Images provide a pixel-grounded action representation, reframing policy learning as a visual tracking problem! By unifying observations and actions in the same video space, Action Images enable a unified robotics world model that supports video-action joint generation, action-conditioned video generation, and action labeling! Code: Paper:

Chuang Gan

19,585 次观看 • 1 个月前

How to harness foundation models for *generalization in the wild* in robot manipulation? Introducing VoxPoser: use LLM+VLM to label affordances and constraints directly in 3D perceptual space for zero-shot robot manipulation in the real world! 🌐 🧵👇

How to harness foundation models for generalization in the wild in robot manipulation? Introducing VoxPoser: use LLM+VLM to label affordances and constraints directly in 3D perceptual space for zero-shot robot manipulation in the real world! 🌐 🧵👇

Wenlong Huang

293,910 次观看 • 3 年前

World Model meets robot policy! Robbyant's LingBot-VA: unifies video world modeling and robotic policy learning. - A single model generates both future video and the actions to make it real. - Long-term memory enables long-horizon tasks. - Claims significant outperformance over π₀.₅ in real-world tasks. - It's open-source

World Model meets robot policy! Robbyant's LingBot-VA: unifies video world modeling and robotic policy learning. - A single model generates both future video and the actions to make it real. - Long-term memory enables long-horizon tasks. - Claims significant outperformance over π₀.₅ in real-world tasks. - It's open-source

The Humanoid Hub

17,721 次观看 • 6 个月前

Tired of teleoperation? One human video → 1,000s of robot demos. (📍GitHub ) Scaling Robot Data Without Dynamics Simulation or Robot Hardware Real2Render2Real (R2R2R) is a new way to scale robot data without physics simulation or hardware. You take a phone scan + a single monocular human demo. It tracks the motion, renders photorealistic scenes, and generates diverse, robot-agnostic trajectories ready for training. > No teleop, no sim, no robot, just a phone and a video > Train VLA models and diffusion policies directly on the output > Supports multiple robot embodiments with kinematic consistency > 1000s of demos in 1/27 the time of real-world collection Thank you, Max Fu, for sharing!! Project: Paper: Code coming soon: It shows that with the right pipeline, you can scale robot learning data without touching a robot. One of the most interesting directions in scalable robotics today. —— Weekly robotics and AI insights. Subscribe free:

Tired of teleoperation? One human video → 1,000s of robot demos. (📍GitHub ) Scaling Robot Data Without Dynamics Simulation or Robot Hardware Real2Render2Real (R2R2R) is a new way to scale robot data without physics simulation or hardware. You take a phone scan + a single monocular human demo. It tracks the motion, renders photorealistic scenes, and generates diverse, robot-agnostic trajectories ready for training. > No teleop, no sim, no robot, just a phone and a video > Train VLA models and diffusion policies directly on the output > Supports multiple robot embodiments with kinematic consistency > 1000s of demos in 1/27 the time of real-world collection Thank you, Max Fu, for sharing!! Project: Paper: Code coming soon: It shows that with the right pipeline, you can scale robot learning data without touching a robot. One of the most interesting directions in scalable robotics today. —— Weekly robotics and AI insights. Subscribe free:

Ilir Aliu

42,864 次观看 • 6 个月前

I placed 🥈 2nd in the LeHome Challenge (at IEEE ICRA 2026) earlier this month, and before that I was 🥇 1st of 62 teams in the simulation round. Now I am sharing my solution — with a detailed logic walkthrough and open-source code. The task was to teach a cheap two-armed robot to fold different garments in simulation and on a real robot. I trained a VLA policy with an RL loop to make it work. Let's break it down 👇

I placed 🥈 2nd in the LeHome Challenge (at IEEE ICRA 2026) earlier this month, and before that I was 🥇 1st of 62 teams in the simulation round. Now I am sharing my solution — with a detailed logic walkthrough and open-source code. The task was to teach a cheap two-armed robot to fold different garments in simulation and on a real robot. I trained a VLA policy with an RL loop to make it work. Let's break it down 👇

Ilia

21,948 次观看 • 1 个月前

An interactive world model developed by NVIDIA in collaboration with academic partners. - DreamDojo turns egocentric human video data into physical intelligence. - Human data is more scalable than robotics data but lacks action labels. - To solve this, a dedicated action model extracts latent actions by identifying physics and motion deltas between frames. Training - A massive 44k hours of video data are used for pre-training. - Post-training on small-scale robot datasets maps human physics to specific robot embodiments. - An additional distillation stage converts the model into an autoregressive, few-step diffusion model, enabling real-time, action-controllable simulation. Primary Use Cases - Live Teleoperation: Controlling a robot inside a world simulation in real-time. - Model-based Planning: Previewing and curating the best actions for improved success. - Policy Evaluation: Testing robot policies in realistic, out-of-distribution scenarios. Everything that's open-sourced: weights, code, post-training dataset, eval set, and details to reproduce.

An interactive world model developed by NVIDIA in collaboration with academic partners. - DreamDojo turns egocentric human video data into physical intelligence. - Human data is more scalable than robotics data but lacks action labels. - To solve this, a dedicated action model extracts latent actions by identifying physics and motion deltas between frames. Training - A massive 44k hours of video data are used for pre-training. - Post-training on small-scale robot datasets maps human physics to specific robot embodiments. - An additional distillation stage converts the model into an autoregressive, few-step diffusion model, enabling real-time, action-controllable simulation. Primary Use Cases - Live Teleoperation: Controlling a robot inside a world simulation in real-time. - Model-based Planning: Previewing and curating the best actions for improved success. - Policy Evaluation: Testing robot policies in realistic, out-of-distribution scenarios. Everything that's open-sourced: weights, code, post-training dataset, eval set, and details to reproduce.

The Humanoid Hub

11,575 次观看 • 5 个月前