Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Scalable, reproducible, and reliable robotic evaluation remains an open challenge, especially in the age of generalist robot foundation models. Can *simulation* effectively predict *real-world* robot policy performance & behavior? Presenting SIMPLER!👇

XuanlinLi2's profile picture

Xuanlin Li (Simon)

1,180 subscribers

86,460 Aufrufe • vor 2 Jahren •via X (Twitter)

Wissenschaft & TechnologieNachrichten & PolitikBildung
LIVE

Anya Rossi• Live Now

Private livecam show

3 Kommentare

Profilbild von Xuanlin Li (Simon)
Xuanlin Li (Simon)vor 2 Jahren

SIMPLER contributes to open-source robotics by enabling the evaluation of real-world robot policies without extensive hardware setups, continuous human supervision, or manual environment resets. See @kylehkhsu’s thread for a deep dive!

Profilbild von Yunhao Fang
Yunhao Fangvor 2 Jahren

Nice work Xuanlin.👏

Profilbild von Mustafa
Mustafavor 2 Jahren

Let’s chat. Unfortunately can’t dm you

Ähnliche Videos

📢 Announcing one of the most exciting works from us this year on **scalable robot policy evaluation through real-to-sim transfer**, moving toward a scalable evaluation engine with structured world models that capture the appearance, geometry, and dynamics of environments involving deformable objects. 🤖 Evaluation remains one of the biggest bottlenecks in building general-purpose robots. Today, robots are still evaluated only in the real world, which is **orders of magnitude slower** than the development of language agents. We propose a new framework where simulation performance **strongly correlates** with the real world (r > 0.9), even for deformable objects. The key difference from existing work lies in the correlation between simulation and reality: if a robot model performs better in the digital world, does it also perform better in the real world? This question has long made people hesitant about simulation-based evaluation — especially for deformable objects. We are changing that. Our pipeline achieves effective real-to-sim transfer, establishing **state-of-the-art correlation** between simulation and reality for deformable object manipulation. It provides a **scalable and reproducible evaluation engine** for robot learning. 🌐
3:04

📢 Announcing one of the most exciting works from us this year on **scalable robot policy evaluation through real-to-sim transfer**, moving toward a scalable evaluation engine with structured world models that capture the appearance, geometry, and dynamics of environments involving deformable objects. 🤖 Evaluation remains one of the biggest bottlenecks in building general-purpose robots. Today, robots are still evaluated only in the real world, which is **orders of magnitude slower** than the development of language agents. We propose a new framework where simulation performance **strongly correlates** with the real world (r > 0.9), even for deformable objects. The key difference from existing work lies in the correlation between simulation and reality: if a robot model performs better in the digital world, does it also perform better in the real world? This question has long made people hesitant about simulation-based evaluation — especially for deformable objects. We are changing that. Our pipeline achieves effective real-to-sim transfer, establishing **state-of-the-art correlation** between simulation and reality for deformable object manipulation. It provides a **scalable and reproducible evaluation engine** for robot learning. 🌐

Yunzhu Li

39,960 Aufrufe • vor 9 Monaten

Given the strong community adoption and real-world deployment of MolmoAct2 on YAM, we're introducing zero-shot evaluation of MolmoAct2 Bimanual YAM in simulation. Now you can test out our models without a real-world robot and build on them! Code: Simulation built on Maniskill!
0:20

Given the strong community adoption and real-world deployment of MolmoAct2 on YAM, we're introducing zero-shot evaluation of MolmoAct2 Bimanual YAM in simulation. Now you can test out our models without a real-world robot and build on them! Code: Simulation built on Maniskill!

Jiafei Duan

37,104 Aufrufe • vor 2 Monaten

How do we make robot policy evaluation reproducible across labs? Announcing OpenArm 02: a fully open-source dual arm platform, with a unified setup for reproducible evaluation: controlled tasks, backgrounds, lighting, and camera angles. ✅ Fully open-source, so anyone can replicate the setup anywhere ✅ AutoEval: 24/7 real-world evaluation with minimal human intervention (building on work by Paul Zhou) ✅ Fair side-by-side policy comparison OpenArm is the open foundation to turn isolated results into shared progress. Releasing soon. Stay tuned.
1:10

How do we make robot policy evaluation reproducible across labs? Announcing OpenArm 02: a fully open-source dual arm platform, with a unified setup for reproducible evaluation: controlled tasks, backgrounds, lighting, and camera angles. ✅ Fully open-source, so anyone can replicate the setup anywhere ✅ AutoEval: 24/7 real-world evaluation with minimal human intervention (building on work by Paul Zhou) ✅ Fair side-by-side policy comparison OpenArm is the open foundation to turn isolated results into shared progress. Releasing soon. Stay tuned.

Enactic

64,759 Aufrufe • vor 5 Monaten

Evaluation is a critical bottleneck in building robot foundation models. Check out our latest work RoboLab, led by Xuning Yang, which addresses this exact challenge. Its a high-fidelity simulation environment for testing these models. A truly generalist policy should be able to complete these tasks zero-shot, and this benchmark highlights exactly how far we still have to go. More info 👇
0:26

Evaluation is a critical bottleneck in building robot foundation models. Check out our latest work RoboLab, led by Xuning Yang, which addresses this exact challenge. Its a high-fidelity simulation environment for testing these models. A truly generalist policy should be able to complete these tasks zero-shot, and this benchmark highlights exactly how far we still have to go. More info 👇

Ankit Goyal

29,910 Aufrufe • vor 4 Monaten

We’re organizing the RoboArena Challenge at CoRL this year! Show the performance of your best generalist policy, in a fair, open benchmark for the robotics community! 🤖 Sign up, even if you don’t have a robot! More details in 🧵👇
0:48

We’re organizing the RoboArena Challenge at CoRL this year! Show the performance of your best generalist policy, in a fair, open benchmark for the robotics community! 🤖 Sign up, even if you don’t have a robot! More details in 🧵👇

Karl Pertsch

26,775 Aufrufe • vor 1 Jahr

The future of robot butlers starts with mobile manipulation. We’re announcing the NeurIPS 2023 Open-Vocabulary Mobile Manipulation Challenge! - Full robot stack ✅ - Parallel sim and real evaluation ✅ - No robot required ✅👀
1:24

The future of robot butlers starts with mobile manipulation. We’re announcing the NeurIPS 2023 Open-Vocabulary Mobile Manipulation Challenge! - Full robot stack ✅ - Parallel sim and real evaluation ✅ - No robot required ✅👀

Chris Paxton

179,032 Aufrufe • vor 3 Jahren

Qwen-VLA feels like one of the first real robotics foundation models. A single system trained across robot manipulation, navigation, egocentric human video, simulation, and vision-language reasoning instead of isolated robot policies.
2:52

Qwen-VLA feels like one of the first real robotics foundation models. A single system trained across robot manipulation, navigation, egocentric human video, simulation, and vision-language reasoning instead of isolated robot policies.

Robots Digest 🤖

14,722 Aufrufe • vor 2 Monaten

#AGIBOTAIWeek Day4: Introducing GE-Sim 2.0 (Genie Envisioner World Simulator 2.0) , a comprehensive embodied world simulator for robotic manipulation. By integrating future video generation, proprioceptive state estimation, and reward-based policy assessment, GE-Sim 2.0 moves beyond visual simulation toward an embodied world simulator with built-in evaluation capability, laying the foundation for a world-model-centric platform for closed-loop and interactive policy learning. It provides a practical framework for scalable evaluation and training of robotic manipulation policies. Project page: #AGIBOT #AGIBOTAIWeek #GenieEnvisioner #Robotics #visualsimulation
2:52

#AGIBOTAIWeek Day4: Introducing GE-Sim 2.0 (Genie Envisioner World Simulator 2.0) , a comprehensive embodied world simulator for robotic manipulation. By integrating future video generation, proprioceptive state estimation, and reward-based policy assessment, GE-Sim 2.0 moves beyond visual simulation toward an embodied world simulator with built-in evaluation capability, laying the foundation for a world-model-centric platform for closed-loop and interactive policy learning. It provides a practical framework for scalable evaluation and training of robotic manipulation policies. Project page: #AGIBOT #AGIBOTAIWeek #GenieEnvisioner #Robotics #visualsimulation

AGIBOT

28,219 Aufrufe • vor 4 Monaten

Excited to release τ0-WM: an open-source unified video-action world model for robotic manipulation. It's a 5B-parameter robotic foundation model trained on 27.3K hours of real-robot teleoperation, UMI-style demonstrations, and egocentric interaction videos.
3:26

Excited to release τ0-WM: an open-source unified video-action world model for robotic manipulation. It's a 5B-parameter robotic foundation model trained on 27.3K hours of real-robot teleoperation, UMI-style demonstrations, and egocentric interaction videos.

Jianlan Luo

63,831 Aufrufe • vor 2 Monaten

NVIDIA Isaac GR00T N1 is an open generalist foundation model for #humanoidrobots. 🤖 Discover how the model can easily generalize common tasks, boosting robot reasoning and skills. ▶️
0:30

NVIDIA Isaac GR00T N1 is an open generalist foundation model for #humanoidrobots. 🤖 Discover how the model can easily generalize common tasks, boosting robot reasoning and skills. ▶️

NVIDIA Robotics

53,497 Aufrufe • vor 1 Jahr

Why do generalist robotic models fail when a cup is moved just two inches to the left? It’s not a lack of motor skill, it’s an alignment problem. Today, we introduce VLS: Vision-Language Steering of Pretrained Robot Policies, a training-free framework that guides robot behavior in real time. Check out the project: 👇🧵 (Watch till the end: VLS runs uncut, steering pretrained policies across long-horizon tasks.)
1:47

Why do generalist robotic models fail when a cup is moved just two inches to the left? It’s not a lack of motor skill, it’s an alignment problem. Today, we introduce VLS: Vision-Language Steering of Pretrained Robot Policies, a training-free framework that guides robot behavior in real time. Check out the project: 👇🧵 (Watch till the end: VLS runs uncut, steering pretrained policies across long-horizon tasks.)

Jiafei Duan

72,465 Aufrufe • vor 6 Monaten

Testing robot policies on hardware is slow, expensive and hard to scale. World models offer a promising path to accelerating robot policy development. We're sharing new research from the Runway Robotics team, in which we simulated 8 robot policies inside our General World Model and found 0.95 correlation with real-world results. Those early results point to world model simulation as a practical substitute for hardware evaluation, comparing favorably to existing real-to-sim approaches. Learn more at the link below.
0:40

Testing robot policies on hardware is slow, expensive and hard to scale. World models offer a promising path to accelerating robot policy development. We're sharing new research from the Runway Robotics team, in which we simulated 8 robot policies inside our General World Model and found 0.95 correlation with real-world results. Those early results point to world model simulation as a practical substitute for hardware evaluation, comparing favorably to existing real-to-sim approaches. Learn more at the link below.

Runway

13,464 Aufrufe • vor 5 Monaten

What happens when robot world models learn from human experience at scale? 🤔 DreamDojo from NVIDIA Research is a generalist robot world model pretrained on 44K hours of egocentric human videos and then post-trained on robot data to generalize across new objects and environments. After distillation, it runs at 10 FPS for live teleoperation, policy evaluation, and model-based planning. Read the ICML paper to learn more 📄
0:32

What happens when robot world models learn from human experience at scale? 🤔 DreamDojo from NVIDIA Research is a generalist robot world model pretrained on 44K hours of egocentric human videos and then post-trained on robot data to generalize across new objects and environments. After distillation, it runs at 10 FPS for live teleoperation, policy evaluation, and model-based planning. Read the ICML paper to learn more 📄

NVIDIA Robotics

22,882 Aufrufe • vor 1 Monat

We open-source Action Images — a new representation that translates 7-DoF robot actions into interpretable images. Video models are emerging as powerful robotic foundation models, but a key challenge remains: how can we seamlessly integrate robot policies into video models? Instead of representing actions as low-dimensional control tokens, Action Images provide a pixel-grounded action representation, reframing policy learning as a visual tracking problem! By unifying observations and actions in the same video space, Action Images enable a unified robotics world model that supports video-action joint generation, action-conditioned video generation, and action labeling! Code: Paper:
0:51

We open-source Action Images — a new representation that translates 7-DoF robot actions into interpretable images. Video models are emerging as powerful robotic foundation models, but a key challenge remains: how can we seamlessly integrate robot policies into video models? Instead of representing actions as low-dimensional control tokens, Action Images provide a pixel-grounded action representation, reframing policy learning as a visual tracking problem! By unifying observations and actions in the same video space, Action Images enable a unified robotics world model that supports video-action joint generation, action-conditioned video generation, and action labeling! Code: Paper:

Chuang Gan

19,963 Aufrufe • vor 1 Monat

How to harness foundation models for *generalization in the wild* in robot manipulation? Introducing VoxPoser: use LLM+VLM to label affordances and constraints directly in 3D perceptual space for zero-shot robot manipulation in the real world! 🌐 🧵👇
0:50

How to harness foundation models for *generalization in the wild* in robot manipulation? Introducing VoxPoser: use LLM+VLM to label affordances and constraints directly in 3D perceptual space for zero-shot robot manipulation in the real world! 🌐 🧵👇

Wenlong Huang

293,910 Aufrufe • vor 3 Jahren

World Model meets robot policy! Robbyant's LingBot-VA: unifies video world modeling and robotic policy learning. - A single model generates both future video and the actions to make it real. - Long-term memory enables long-horizon tasks. - Claims significant outperformance over π₀.₅ in real-world tasks. - It's open-source
1:33

World Model meets robot policy! Robbyant's LingBot-VA: unifies video world modeling and robotic policy learning. - A single model generates both future video and the actions to make it real. - Long-term memory enables long-horizon tasks. - Claims significant outperformance over π₀.₅ in real-world tasks. - It's open-source

The Humanoid Hub

17,721 Aufrufe • vor 6 Monaten

Tired of teleoperation? One human video → 1,000s of robot demos. (📍GitHub ) Scaling Robot Data Without Dynamics Simulation or Robot Hardware Real2Render2Real (R2R2R) is a new way to scale robot data without physics simulation or hardware. You take a phone scan + a single monocular human demo. It tracks the motion, renders photorealistic scenes, and generates diverse, robot-agnostic trajectories ready for training. > No teleop, no sim, no robot, just a phone and a video > Train VLA models and diffusion policies directly on the output > Supports multiple robot embodiments with kinematic consistency > 1000s of demos in 1/27 the time of real-world collection Thank you, Max Fu, for sharing!! Project: Paper: Code coming soon: It shows that with the right pipeline, you can scale robot learning data without touching a robot. One of the most interesting directions in scalable robotics today. —— Weekly robotics and AI insights. Subscribe free:
0:29

Tired of teleoperation? One human video → 1,000s of robot demos. (📍GitHub ) Scaling Robot Data Without Dynamics Simulation or Robot Hardware Real2Render2Real (R2R2R) is a new way to scale robot data without physics simulation or hardware. You take a phone scan + a single monocular human demo. It tracks the motion, renders photorealistic scenes, and generates diverse, robot-agnostic trajectories ready for training. > No teleop, no sim, no robot, just a phone and a video > Train VLA models and diffusion policies directly on the output > Supports multiple robot embodiments with kinematic consistency > 1000s of demos in 1/27 the time of real-world collection Thank you, Max Fu, for sharing!! Project: Paper: Code coming soon: It shows that with the right pipeline, you can scale robot learning data without touching a robot. One of the most interesting directions in scalable robotics today. —— Weekly robotics and AI insights. Subscribe free:

Ilir Aliu

42,864 Aufrufe • vor 6 Monaten

I placed 🥈 2nd in the LeHome Challenge (at IEEE ICRA 2026) earlier this month, and before that I was 🥇 1st of 62 teams in the simulation round. Now I am sharing my solution — with a detailed logic walkthrough and open-source code. The task was to teach a cheap two-armed robot to fold different garments in simulation and on a real robot. I trained a VLA policy with an RL loop to make it work. Let's break it down 👇
0:33

I placed 🥈 2nd in the LeHome Challenge (at IEEE ICRA 2026) earlier this month, and before that I was 🥇 1st of 62 teams in the simulation round. Now I am sharing my solution — with a detailed logic walkthrough and open-source code. The task was to teach a cheap two-armed robot to fold different garments in simulation and on a real robot. I trained a VLA policy with an RL loop to make it work. Let's break it down 👇

Ilia

21,948 Aufrufe • vor 1 Monat

An interactive world model developed by NVIDIA in collaboration with academic partners. - DreamDojo turns egocentric human video data into physical intelligence. - Human data is more scalable than robotics data but lacks action labels. - To solve this, a dedicated action model extracts latent actions by identifying physics and motion deltas between frames. Training - A massive 44k hours of video data are used for pre-training. - Post-training on small-scale robot datasets maps human physics to specific robot embodiments. - An additional distillation stage converts the model into an autoregressive, few-step diffusion model, enabling real-time, action-controllable simulation. Primary Use Cases - Live Teleoperation: Controlling a robot inside a world simulation in real-time. - Model-based Planning: Previewing and curating the best actions for improved success. - Policy Evaluation: Testing robot policies in realistic, out-of-distribution scenarios. Everything that's open-sourced: weights, code, post-training dataset, eval set, and details to reproduce.
1:14

An interactive world model developed by NVIDIA in collaboration with academic partners. - DreamDojo turns egocentric human video data into physical intelligence. - Human data is more scalable than robotics data but lacks action labels. - To solve this, a dedicated action model extracts latent actions by identifying physics and motion deltas between frames. Training - A massive 44k hours of video data are used for pre-training. - Post-training on small-scale robot datasets maps human physics to specific robot embodiments. - An additional distillation stage converts the model into an autoregressive, few-step diffusion model, enabling real-time, action-controllable simulation. Primary Use Cases - Live Teleoperation: Controlling a robot inside a world simulation in real-time. - Model-based Planning: Previewing and curating the best actions for improved success. - Policy Evaluation: Testing robot policies in realistic, out-of-distribution scenarios. Everything that's open-sourced: weights, code, post-training dataset, eval set, and details to reproduce.

The Humanoid Hub

11,575 Aufrufe • vor 6 Monaten