Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

For large-scale robotic deployment🤖 in the real-world 🌏, robots must adapt to changes in environment and objects. Ever questioned the generalizability of your robot's manipulation policy? Put it to the test with The Colosseum 🏛️. Check out our project:

Jiafei Duan

6,059 subscribers

36,617 Aufrufe • vor 2 Jahren •via X (Twitter)

Wissenschaft & Technologie Bildung

Anya Rossi• Live Now

Private livecam show

14 Kommentare

Profilbild von Jiafei Duan

Jiafei Duanvor 2 Jahren

1/12 🧵: Introducing THE COLOSSEUM: A benchmark for evaluating robot manipulation against environmental changes. With perturbations in 20 tasks from RLBench (@stepjamUK), it spans 12 dimensions of challenges

Profilbild von Jiafei Duan

Jiafei Duanvor 2 Jahren

2/12 🧵: To set the foundation for this new robotics benchmark, we assessed 4 state-of-the-art (SotA) behavior cloning models - R3M, MVP, PerAct @mohito1905 , and RVT @imankitgoyal - across 12 perturbation factors and 20 robotics manipulation tasks. 📊

Profilbild von Jiafei Duan

Jiafei Duanvor 2 Jahren

3/12 🧵: Exploring BC models through THE COLOSSEUM, we delve into key research questions on generalization for BC policies. Are 2D-based BC models more generalizable than 3D? What factors most disrupt BC models' performance? Does the real world align with simulations in terms of model generalization? 🤔

Profilbild von Jiafei Duan

Jiafei Duanvor 2 Jahren

4/12 🧵: Fresh insights from our evaluation results! 🔥We've found that 3D baselines generally outperform 2D ones, showing significantly greater robustness to environmental perturbations.

Profilbild von Jiafei Duan

Jiafei Duanvor 2 Jahren

5/12 🧵: Zeroing in on 3D models (RVT and PerAct), we've identified that color nuances —from objects, tables, to lighting—and distractors 🚨 play pivotal roles in performance. In contrast, 2D models (R3M-MLP and MVP-MLP) show heightened sensitivity towards object and light color, texture, and the angle of the camera 📷.

Profilbild von Jiafei Duan

Jiafei Duanvor 2 Jahren

6/12 🧵: We've brought our simulations to life! We're testing real-world perturbations on a trained BC model by replicating 4 simulation tasks and 3D printing all assets. This enhances our benchmark's reproducibility and bridges the gap between simulation and real-world tasks.

Profilbild von Jiafei Duan

Jiafei Duanvor 2 Jahren

7/12 🧵: Real-world tests align with The Colosseum's simulation evaluation results, showing you can trust BC model sims to reflect similar performance drops under the same perturbation factors⚖️.

Profilbild von Jiafei Duan

Jiafei Duanvor 2 Jahren

8/12 🧵: Also, check out these cool prior works: Decomposing visual imitation learning– – great to have independent works converge on similar ideas and initiatives!

Profilbild von Jiafei Duan

Jiafei Duanvor 2 Jahren

9/12 🧵: The Colosseum will continue to serve as a challenge and provide a unified platform to develop, evaluate, and compare future robotic manipulation methods that stand the test of robustness and generalization.

Profilbild von Jiafei Duan

Jiafei Duanvor 2 Jahren

10/12 🧵: We're dedicated to keeping The Colosseum's leaderboard up-to-date ✨. We will add in more model evals (ACT3D, GNFactor, ACT, and many more). Excited to see new robotic manipulation policies step up and test their generalizability with us!

Profilbild von Jiafei Duan

Jiafei Duanvor 2 Jahren

11/12 🧵: This project could not have been possible without the team @wpumacay7567 @Ishika_S_ , and advisors @RanjayKrishna @_jessethomason_ Dieter Fox.

Profilbild von Jiafei Duan

Jiafei Duanvor 2 Jahren

12/12🧵: Stay tuned for the release of our code and complete benchmark. However, if you would like to evaluate your BC model with us before our release, feel free to contact us at: [email protected]

Profilbild von Jiafei Duan

Jiafei Duanvor 2 Jahren

Exciting benchmarks like ImageNet-A, WILDS, and GLUE have driven advancements in vision and language models by testing for robustness and generalization. It's time for robotics to get a piece of the action and push the field forward with this tailored benchmark!

Profilbild von Jiafei Duan

Jiafei Duanvor 2 Jahren

@allen_ai @nvidia @CSatUSC

Ähnliche Videos

🤖 How can robot policies zero-shot generalize to any new environment and any new object? Introducing our new project: 🚀Data Scaling Laws in Imitation Learning for Robotic Manipulation🚀—bringing us closer to the dream of having robots work as waiters in hot pot restaurants! 🍲

🤖 How can robot policies zero-shot generalize to any new environment and any new object? Introducing our new project: 🚀Data Scaling Laws in Imitation Learning for Robotic Manipulation🚀—bringing us closer to the dream of having robots work as waiters in hot pot restaurants! 🍲

Yang Gao

125,057 Aufrufe • vor 1 Jahr

Put simply, the real world web onboards robots to new physical spaces instantly. Simplify robot deployment → scale robot deployment. Accelerate.

Put simply, the real world web onboards robots to new physical spaces instantly. Simplify robot deployment → scale robot deployment. Accelerate.

Auki

534,978 Aufrufe • vor 10 Monaten

📢📢📢 Excited to release ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learning (CVPR25). 🤏🤙✌️With ManipTrans, we can transfer dexterous manipulation skills into robotic hands in simulation and deploy them on a real robot, using a residual policy learned for dex manipulation. 🤖🤖🤖The video below illustrates how the MoCap data can be transferred to Inspire, Shadow, Xhand, Allegro, and Mano. With ManipTrans, we can scale up dex manip data greatly with minimal effort. For more details, please check our -webpage: -code: -huggingface:

📢📢📢 Excited to release ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learning (CVPR25). 🤏🤙✌️With ManipTrans, we can transfer dexterous manipulation skills into robotic hands in simulation and deploy them on a real robot, using a residual policy learned for dex manipulation. 🤖🤖🤖The video below illustrates how the MoCap data can be transferred to Inspire, Shadow, Xhand, Allegro, and Mano. With ManipTrans, we can scale up dex manip data greatly with minimal effort. For more details, please check our -webpage: -code: -huggingface:

Siyuan Huang

20,983 Aufrufe • vor 1 Jahr

🦾 Our robot hand can rotate objects over 6+ axes in the real-world! Introducing RotateIt (CoRL 2023), a Sim-to-Real policy that can rotate many objects over many axes, using vision and touch! Check it out: Paper: #CoRL2023

🦾 Our robot hand can rotate objects over 6+ axes in the real-world! Introducing RotateIt (CoRL 2023), a Sim-to-Real policy that can rotate many objects over many axes, using vision and touch! Check it out: Paper: #CoRL2023

Haozhi Qi

51,041 Aufrufe • vor 2 Jahren

Today Mecka is announcing $60M in funding to become the data and deployment layer for physical AI This raise will allow us to scale our data infrastructure, invest into new verticals, and deploy robots into the real world

Today Mecka is announcing $60M in funding to become the data and deployment layer for physical AI This raise will allow us to scale our data infrastructure, invest into new verticals, and deploy robots into the real world

Josh

332,759 Aufrufe • vor 3 Monaten

Our current dual-arm setup for autonomous electronics assembly. The teleoperation controllers in front are used to teach the robots precise manipulation of small objects.

Our current dual-arm setup for autonomous electronics assembly. The teleoperation controllers in front are used to teach the robots precise manipulation of small objects.

Viktor Petrenko

866,871 Aufrufe • vor 1 Monat

📢 Announcing one of the most exciting works from us this year on **scalable robot policy evaluation through real-to-sim transfer**, moving toward a scalable evaluation engine with structured world models that capture the appearance, geometry, and dynamics of environments involving deformable objects. 🤖 Evaluation remains one of the biggest bottlenecks in building general-purpose robots. Today, robots are still evaluated only in the real world, which is **orders of magnitude slower** than the development of language agents. We propose a new framework where simulation performance **strongly correlates** with the real world (r > 0.9), even for deformable objects. The key difference from existing work lies in the correlation between simulation and reality: if a robot model performs better in the digital world, does it also perform better in the real world? This question has long made people hesitant about simulation-based evaluation — especially for deformable objects. We are changing that. Our pipeline achieves effective real-to-sim transfer, establishing **state-of-the-art correlation** between simulation and reality for deformable object manipulation. It provides a **scalable and reproducible evaluation engine** for robot learning. 🌐

📢 Announcing one of the most exciting works from us this year on scalable robot policy evaluation through real-to-sim transfer, moving toward a scalable evaluation engine with structured world models that capture the appearance, geometry, and dynamics of environments involving deformable objects. 🤖 Evaluation remains one of the biggest bottlenecks in building general-purpose robots. Today, robots are still evaluated only in the real world, which is orders of magnitude slower than the development of language agents. We propose a new framework where simulation performance strongly correlates with the real world (r > 0.9), even for deformable objects. The key difference from existing work lies in the correlation between simulation and reality: if a robot model performs better in the digital world, does it also perform better in the real world? This question has long made people hesitant about simulation-based evaluation — especially for deformable objects. We are changing that. Our pipeline achieves effective real-to-sim transfer, establishing state-of-the-art correlation between simulation and reality for deformable object manipulation. It provides a scalable and reproducible evaluation engine for robot learning. 🌐

Yunzhu Li

39,960 Aufrufe • vor 10 Monaten

The Real-to-Sim part of R2S2R transforms physical robots, sensors, environments, objects, and interactions into simulations that preserve task-relevant observations and dynamics - not only how the world looks, but how it acts when the robot interacts with it. These results set a new bar for sim-real alignment in robot manipulation.

The Real-to-Sim part of R2S2R transforms physical robots, sensors, environments, objects, and interactions into simulations that preserve task-relevant observations and dynamics - not only how the world looks, but how it acts when the robot interacts with it. These results set a new bar for sim-real alignment in robot manipulation.

Fei-Fei Li

25,720 Aufrufe • vor 1 Monat

How do you explain world models to your sibling? 🤖 NVIDIA Cosmos helps physical AI reason about real-world interactions and simulate possible futures before acting in the real world. Or, as they might put it, it teaches robots to dream. 💭

How do you explain world models to your sibling? 🤖 NVIDIA Cosmos helps physical AI reason about real-world interactions and simulate possible futures before acting in the real world. Or, as they might put it, it teaches robots to dream. 💭

NVIDIA

89,292 Aufrufe • vor 29 Tagen

Robots Using Their Feet to Manipulate Objects ETH Zurich researchers have advanced "pedipulation," where legged robots use their feet for manipulation instead of robotic arms. Their new approach trains robots to navigate and manipulate objects while avoiding static and dynamic obstacles. Tested on the ANYmal quadruped, the method enables precise foot control in complex, unseen environments after minimal training in simulation. Will this research open up new possibilities for efficient, obstacle-aware robot manipulation?

Robots Using Their Feet to Manipulate Objects ETH Zurich researchers have advanced "pedipulation," where legged robots use their feet for manipulation instead of robotic arms. Their new approach trains robots to navigate and manipulate objects while avoiding static and dynamic obstacles. Tested on the ANYmal quadruped, the method enables precise foot control in complex, unseen environments after minimal training in simulation. Will this research open up new possibilities for efficient, obstacle-aware robot manipulation?

Circuit

11,802 Aufrufe • vor 1 Jahr

Robots are gaining spatial awareness with Over the Reality. 🤖 Powered by the world’s largest 3D dataset, we enable robots to see, sense, and collaborate in real-world environments. This early test shows how a Unitree Go2 perceives its surroundings through OVER’s VPS — on the right, the real life robot; on the left, its real-time perception of the position inside one of OVER’s 3D maps. Robotic spatial awareness is here, living in 155,000+ mapped locations and expanding every day. One step closer to Physical AI. Stay tuned for our upcoming ROS 2 integration, unlocking a universal framework for robotic perception leveraging OVER's infrastructure!

Robots are gaining spatial awareness with Over the Reality. 🤖 Powered by the world’s largest 3D dataset, we enable robots to see, sense, and collaborate in real-world environments. This early test shows how a Unitree Go2 perceives its surroundings through OVER’s VPS — on the right, the real life robot; on the left, its real-time perception of the position inside one of OVER’s 3D maps. Robotic spatial awareness is here, living in 155,000+ mapped locations and expanding every day. One step closer to Physical AI. Stay tuned for our upcoming ROS 2 integration, unlocking a universal framework for robotic perception leveraging OVER's infrastructure!

Over the Reality 🌐

263,827 Aufrufe • vor 11 Monaten

For robots to be useful, they must be able to interact with a wide variety of environments; and yet, scaling interaction data is difficult, expensive, and time consuming. Instead, much research revolves around sim-to-real manipulation — but mostly this has not been mobile manipulation. Recently, though, this has begun to change. Two recent papers from Tairan He and Haoru Xue show us how to unlock the potential of this technique, building policies which, without any real data at all, can move objects around in the world and open doors in the real world with a humanoid robot. Watch Episode #60 of RoboPapers now to learn more, hosted by Chris Paxton and Jiafei Duan. In this episode, we cover two papers:. First is VIRAL: Visual Sim-to-Real at Scale for Humanoid Loco-Manipulation; and second is DoorMan: Opening the Sim-to-Real Door for Humanoid Pixel-to-Action Policy Transfer.

For robots to be useful, they must be able to interact with a wide variety of environments; and yet, scaling interaction data is difficult, expensive, and time consuming. Instead, much research revolves around sim-to-real manipulation — but mostly this has not been mobile manipulation. Recently, though, this has begun to change. Two recent papers from Tairan He and Haoru Xue show us how to unlock the potential of this technique, building policies which, without any real data at all, can move objects around in the world and open doors in the real world with a humanoid robot. Watch Episode #60 of RoboPapers now to learn more, hosted by Chris Paxton and Jiafei Duan. In this episode, we cover two papers:. First is VIRAL: Visual Sim-to-Real at Scale for Humanoid Loco-Manipulation; and second is DoorMan: Opening the Sim-to-Real Door for Humanoid Pixel-to-Action Policy Transfer.

RoboPapers

30,767 Aufrufe • vor 7 Monaten

Excited to share the project that has surprised me the most in the last year! Large-scale RL in simulation, no demos and no reward engineering can solve dynamic, dexterous and contact rich tasks. The learned behaviors are reactive, forceful and use the environment for recovery in ways that are extremely challenging to bake in or teleoperate! You can play with the policies yourself to see: And, the learned behavior transfers to real world robots from RGB camera inputs! So what’s the trick - using simulator resets carefully! Let’s unpack (1/10)

Excited to share the project that has surprised me the most in the last year! Large-scale RL in simulation, no demos and no reward engineering can solve dynamic, dexterous and contact rich tasks. The learned behaviors are reactive, forceful and use the environment for recovery in ways that are extremely challenging to bake in or teleoperate! You can play with the policies yourself to see: And, the learned behavior transfers to real world robots from RGB camera inputs! So what’s the trick - using simulator resets carefully! Let’s unpack (1/10)

Abhishek Gupta

83,018 Aufrufe • vor 5 Monaten

Booster Robotics has launched Booster Studio, the first IDE built specifically for embodied AI. > Booster Studio features code editing, high-precision simulation, real-robot debugging, and real-world deployment. > It shortens the path from the first idea to the working hardware, with the build, test, and deployment loop living in a single environment rather than scattered across separate tools. Vibe Robotics 👀

Booster Robotics has launched Booster Studio, the first IDE built specifically for embodied AI. > Booster Studio features code editing, high-precision simulation, real-robot debugging, and real-world deployment. > It shortens the path from the first idea to the working hardware, with the build, test, and deployment loop living in a single environment rather than scattered across separate tools. Vibe Robotics 👀

🚨 AI News | TestingCatalog

21,276 Aufrufe • vor 2 Monaten

Can vision-language-action (VLA) models generalize to diverse OOD tasks and align with customized objectives? 🤔 🚀 We introduce GRAPE, a plug-and-play algorithm to generalize robot policies via preference alignment. GRAPE unfolds three benefits to boost the generalizability of VLAs: 👉1. GRAPE aligns VLAs on a trajectory level and endows the model with the ability for global decision-making, instead of merely cloning behavior; 👉2. GRAPE implicitly models reward from both successful and failed trials to boost generalizability to diverse tasks; 👉3. GRAPE adopts a scalable preference synthesis algorithm to rank trajectories with preferences that align with arbitrary objectives. Our experiments on a diverse array of real-world and simulated robotic tasks reveal: 1⃣GRAPE enhances the performance of state-of-the-art VLA models, increasing success rates on in-domain and unseen manipulation tasks by 51.79% and 60.36%; 2⃣GRAPE is versatile to be aligned with diverse objectives and reduce collision rates by 44.31% or rollout length by 11.15% when aligning towards safer or more efficient manipulation policy, respectively. Check out our full project for more details: 🔥 Paper: 🔥 Project: 🔥 Code:

Can vision-language-action (VLA) models generalize to diverse OOD tasks and align with customized objectives? 🤔 🚀 We introduce GRAPE, a plug-and-play algorithm to generalize robot policies via preference alignment. GRAPE unfolds three benefits to boost the generalizability of VLAs: 👉1. GRAPE aligns VLAs on a trajectory level and endows the model with the ability for global decision-making, instead of merely cloning behavior; 👉2. GRAPE implicitly models reward from both successful and failed trials to boost generalizability to diverse tasks; 👉3. GRAPE adopts a scalable preference synthesis algorithm to rank trajectories with preferences that align with arbitrary objectives. Our experiments on a diverse array of real-world and simulated robotic tasks reveal: 1⃣GRAPE enhances the performance of state-of-the-art VLA models, increasing success rates on in-domain and unseen manipulation tasks by 51.79% and 60.36%; 2⃣GRAPE is versatile to be aligned with diverse objectives and reduce collision rates by 44.31% or rollout length by 11.15% when aligning towards safer or more efficient manipulation policy, respectively. Check out our full project for more details: 🔥 Paper: 🔥 Project: 🔥 Code:

Huaxiu Yao

20,071 Aufrufe • vor 1 Jahr

Excited to introduce COLA: Learning Human–Humanoid Coordination for Collaborative Object Carrying 🤝🤖 COLA makes humanoids truly helpful in human collaboration — capable of carrying objects, pushing carts, or responding to human push commands. It provides a proprioception-only policy for compliant human–humanoid coordination across diverse movement patterns. The core idea is simple yet effective: 👉 Fine-tune a collaborative policy from a locomotion policy using a residual teacher. 👉 Train in simulation and distill to a real-world student policy for deployment. Paper: Project:

Excited to introduce COLA: Learning Human–Humanoid Coordination for Collaborative Object Carrying 🤝🤖 COLA makes humanoids truly helpful in human collaboration — capable of carrying objects, pushing carts, or responding to human push commands. It provides a proprioception-only policy for compliant human–humanoid coordination across diverse movement patterns. The core idea is simple yet effective: 👉 Fine-tune a collaborative policy from a locomotion policy using a residual teacher. 👉 Train in simulation and distill to a real-world student policy for deployment. Paper: Project:

Siyuan Huang

21,932 Aufrufe • vor 10 Monaten