Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

Zero teleoperation. Zero real-world data. ➔ Autonomous humanoid loco-manipulation in reality. Introducing VIRAL: Visual Sim-to-Real at Scale. We achieved 54 autonomous cycles (walk, stand, place, pick, turn) using a simple recipe: 1. RL 2. Simulation 3. GPUs Website: Arxiv: Deep dive with me: 🧵

Tairan He

15,927 subscribers

270,002 Aufrufe • vor 11 Monaten •via X (Twitter)

Bildung Wissenschaft & Technologie

Anya Rossi• Live Now

Private livecam show

24 Kommentare

Profilbild von Tairan He

Tairan Hevor 11 Monaten

Data is the bottleneck of robot learning. The community relies heavily on expensive, slow, human-collected data. We flipped the script. VIRAL trains entirely inside NVIDIA Isaac Lab. We accelerate physics by 10,000x real-time with photorealistic rendering. No human data required. 2/

Profilbild von Tairan He

Tairan Hevor 11 Monaten

The Pipeline: Teacher: We train a privileged teacher with full state access in sim. Student: We distill a vision-based student policy from the teacher. This uses large-scale simulation with tiled rendering, combining online DAgger and Behavior Cloning across tens of GPUs. 3/

Profilbild von Tairan He

Tairan Hevor 11 Monaten

Scale is not optional. We scaled visual sim-to-real compute up to 64 GPUs. We discovered that for long-horizon tasks (like loco-manipulation), large-scale simulation isn't just "nice to have"—it is necessary for convergence and robustness. 4/

Profilbild von Tairan He

Tairan Hevor 11 Monaten

Bridging the Gap. To enable smooth transfer, we performed System Identification (SysID) on our 3-fingered dexterous hands. The difference is night and day. Below is the overlay of simulation vs. real-world physics before and after SysID. 5/

Profilbild von Tairan He

Tairan Hevor 11 Monaten

The Free Lunch. Sim-to-real is hard. But once you solve it, you get generalization for free. VIRAL handles diverse spatial arrangements and visual variations without any real-world fine-tuning. See the stress tests: 6/

Profilbild von Tairan He

Tairan Hevor 11 Monaten

The Journey. This wasn't an overnight success. It took 6 months of building visual sim, distributed training, and infra from scratch. Robotics is hard. We’ve documented our failures and our path to success here: 7/

Profilbild von Tairan He

Tairan Hevor 11 Monaten

VIRAL: Visual Sim-to-Real at Scale for Humanoid Loco-Manipulation Team: @TairanHe99, Zi Wang, @HaoruXue, @BenQingwei, @zhengyiluo, @_wenlixiao, @_ye_yuan, @XingyeDa, Fernando Castañeda, Shankar Sastry, @ChangliuL, @GuanyaShi, GEAR Leads: @DrJimFan, @yukez Website: Paper:

Profilbild von PrismaX

PrismaXvor 11 Monaten

Training at scale.

Profilbild von Nick

Nickvor 11 Monaten

So, all you need is scale?

Profilbild von Mr 365

Mr 365vor 10 Monaten

Do you know when you'll be able to publish the code? It looks really good and big up all of you! <3

Profilbild von Jed Hoo

Jed Hoovor 11 Monaten

very very cool!

Profilbild von Fairooz Choudhury

Fairooz Choudhuryvor 11 Monaten

Impressive achievement! Excited to learn more.

Profilbild von slicedbread

slicedbreadvor 11 Monaten

Kill all robots

Profilbild von David Lin

David Linvor 11 Monaten

This is insanely impressive. One of the best sim2real zero shot transfer I've seen. Great work team.

Profilbild von 奇怪的知识增加了

奇怪的知识增加了vor 11 Monaten

Sim2real ftw 🙌

Profilbild von ζ Pedram ζ

ζ Pedram ζvor 11 Monaten

lol good subject, arbitrarily silly name ..

Profilbild von Forrest Smith

Forrest Smithvor 11 Monaten

What kind of performance did you get from the tiled renderer? At what resolution?

Profilbild von Altiam Kabir

Altiam Kabirvor 11 Monaten

Impressive strides in sim-to-real tech! Excited to see where this leads.

Profilbild von cosmos

cosmosvor 11 Monaten

🫡

Profilbild von Tian Fang

Tian Fangvor 10 Monaten

Very nice work! Did you give any talk in conferences or universities? A good talk will help a lot.

Profilbild von Amy - AI Girl

Amy - AI Girlvor 11 Monaten

Impressive results with VIRAL! The future of robotics is here.

Profilbild von SoloTech

SoloTechvor 11 Monaten

VIRAL is going to push a lot of teams to rethink their pipelines

Profilbild von Sarthak Bhagat

Sarthak Bhagatvor 11 Monaten

Sim2real ftw 🙌

Profilbild von AI PlanetX

AI PlanetXvor 11 Monaten

Impressive work! Excited to learn more about your approach.

Ähnliche Videos

How to use simulation data for real-world robot manipulation? We present sim-and-real co-training, a simple recipe for manipulation. We demonstrate that sim data can significantly enhance real-world performance, even with notable differences between the sim and the real. (1/n)

How to use simulation data for real-world robot manipulation? We present sim-and-real co-training, a simple recipe for manipulation. We demonstrate that sim data can significantly enhance real-world performance, even with notable differences between the sim and the real. (1/n)

Zhenyu Jiang

45,051 Aufrufe • vor 1 Jahr

🤖 How can we scale up humanoid robot learning? Introducing 🌟VLK🌟: generating large-scale synthetic data with paired egocentric observations, text, and full-body G1 kinematics for learning humanoid loco-manipulation. No teleoperation needed! Website:

🤖 How can we scale up humanoid robot learning? Introducing 🌟VLK🌟: generating large-scale synthetic data with paired egocentric observations, text, and full-body G1 kinematics for learning humanoid loco-manipulation. No teleoperation needed! Website:

Jiaman Li

976,031 Aufrufe • vor 3 Monaten

Robot foundation models are limited by costly real data, while simulation data is plentiful but visually mismatched to reality. We present Point Bridge, a method that enables zero-shot sim-to-real transfer for robot learning with minimal visual alignment.

Robot foundation models are limited by costly real data, while simulation data is plentiful but visually mismatched to reality. We present Point Bridge, a method that enables zero-shot sim-to-real transfer for robot learning with minimal visual alignment.

Siddhant Haldar

19,981 Aufrufe • vor 8 Monaten

🤔 Ever wondered if simulation-based animation/avatar learnings can be applied to real humanoid in real-time? 🤖 Introducing H2O (Human2HumanOid): - 🧠 An RL-based human-to-humanoid real-time whole-body teleoperation framework - 💃 Scalable retargeting and training using large human motion dataset - 🎥 With just an RGB camera, everyone can teleoperate a full-sized humanoid to perform actions like pick and place, walking, kicking, boxing, etc - 💡Unleash the potential of humanoids with human cognitive skills and adaptability 🔗: 📄: 🎬: H2O proposes: - A scalable retargeting framework for obtaining large-scale humanoid motion dataset, intelligently filtering out infeasible motion for the humanoid embodiment. - We train a full-body motion imitator (similar to PHC) and deploy to the real world zero-shot. - Using this framework, we enable real-time teleoperation of a humanoid via a human operator and webcam, performing skills such as pick and place, kicking, walking strollers, etc. Team: Tairan He Zhengyi “Zen” Luo Wenli Xiao @ COLM @ChongZitaZhang Kris Kitani Changliu Liu Guanya Shi

Zhengyi “Zen” Luo

47,358 Aufrufe • vor 2 Jahren

Our 6ft humanoid HumanPlus at Stanford can autonomously put on a Nike skateboard shoe, tie shoelaces, stand up and walk. Using two transformers & dual RGB vision, it integrates two recipes of general robotics end-to-end: - imitating humans in real world - large-scale RL in sim

Our 6ft humanoid HumanPlus at Stanford can autonomously put on a Nike skateboard shoe, tie shoelaces, stand up and walk. Using two transformers & dual RGB vision, it integrates two recipes of general robotics end-to-end: - imitating humans in real world - large-scale RL in sim

Zipeng Fu

246,746 Aufrufe • vor 2 Jahren

🚨 New (Offline) RL Method 🦾 🚨 Introducing Q-Transformer - new RL approach that works at scale with large models and many tasks. This is the best method we found so far that works with demos and autonomous (also negative) data at large scale. 🧵

🚨 New (Offline) RL Method 🦾 🚨 Introducing Q-Transformer - new RL approach that works at scale with large models and many tasks. This is the best method we found so far that works with demos and autonomous (also negative) data at large scale. 🧵

Karol Hausman

56,054 Aufrufe • vor 3 Jahren

Introducing Tether 🪢, a fun little idea to scale data by having our robot “play” in the real world for over 24 hours, throughout the day and overnight—improving policies from zero to mastery with minimal supervision! But play is messy, with out-of-distribution scenarios that are hard to anticipate. To perform autonomous functional play in the real world, from just a handful of demos, we propose a highly robust few-shot imitation method that warps demo trajectories using visual correspondences. Then, continuously running it within a multi-task VLM-guided cycle, we generate a data stream that produces 1000+ expert-level demos. This generated data is finally funneled downstream to train imitation learning policies, which improve from zero to near-perfect success rates. We’ll be presenting Tether at #ICLR2026 in just a few weeks! But before that, deep dive with me… 🧵

Introducing Tether 🪢, a fun little idea to scale data by having our robot “play” in the real world for over 24 hours, throughout the day and overnight—improving policies from zero to mastery with minimal supervision! But play is messy, with out-of-distribution scenarios that are hard to anticipate. To perform autonomous functional play in the real world, from just a handful of demos, we propose a highly robust few-shot imitation method that warps demo trajectories using visual correspondences. Then, continuously running it within a multi-task VLM-guided cycle, we generate a data stream that produces 1000+ expert-level demos. This generated data is finally funneled downstream to train imitation learning policies, which improve from zero to near-perfect success rates. We’ll be presenting Tether at #ICLR2026 in just a few weeks! But before that, deep dive with me… 🧵

Will Liang

46,628 Aufrufe • vor 7 Monaten

A big part of scaling robot learning to solve real-world problems is that we somehow need to get enough diverse, high-quality data to train our robots to perform useful things. GPT and its fellow large language models were bootstrapped and proved out on a massive dataset of real-world language data. Unfortunately, despite our best efforts, similarly massive datasets don’t really exist for robotics — so, in our unending pursuit of high-quality, useful data, we turn to simulation. I compared a couple recent works on sim-to-real robot manipulation, which discuss how to train perception-driven manipulation policies in simulation, in such a way that they’re useful in the real world. - DextraH-RGB, from NVIDIA - Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation, also from NVIDIA — specifically the GEAR lab - Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids, another GEAR lab paper - Local Policies Enable Zero-shot Long-Horizon Manipulation, from CMU (video from DextrAH-RGB)

A big part of scaling robot learning to solve real-world problems is that we somehow need to get enough diverse, high-quality data to train our robots to perform useful things. GPT and its fellow large language models were bootstrapped and proved out on a massive dataset of real-world language data. Unfortunately, despite our best efforts, similarly massive datasets don’t really exist for robotics — so, in our unending pursuit of high-quality, useful data, we turn to simulation. I compared a couple recent works on sim-to-real robot manipulation, which discuss how to train perception-driven manipulation policies in simulation, in such a way that they’re useful in the real world. - DextraH-RGB, from NVIDIA - Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation, also from NVIDIA — specifically the GEAR lab - Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids, another GEAR lab paper - Local Policies Enable Zero-shot Long-Horizon Manipulation, from CMU (video from DextrAH-RGB)

Chris Paxton

20,486 Aufrufe • vor 1 Jahr

For robots to be useful, they must be able to interact with a wide variety of environments; and yet, scaling interaction data is difficult, expensive, and time consuming. Instead, much research revolves around sim-to-real manipulation — but mostly this has not been mobile manipulation. Recently, though, this has begun to change. Two recent papers from Tairan He and Haoru Xue show us how to unlock the potential of this technique, building policies which, without any real data at all, can move objects around in the world and open doors in the real world with a humanoid robot. Watch Episode #60 of RoboPapers now to learn more, hosted by Chris Paxton and Jiafei Duan. In this episode, we cover two papers:. First is VIRAL: Visual Sim-to-Real at Scale for Humanoid Loco-Manipulation; and second is DoorMan: Opening the Sim-to-Real Door for Humanoid Pixel-to-Action Policy Transfer.

For robots to be useful, they must be able to interact with a wide variety of environments; and yet, scaling interaction data is difficult, expensive, and time consuming. Instead, much research revolves around sim-to-real manipulation — but mostly this has not been mobile manipulation. Recently, though, this has begun to change. Two recent papers from Tairan He and Haoru Xue show us how to unlock the potential of this technique, building policies which, without any real data at all, can move objects around in the world and open doors in the real world with a humanoid robot. Watch Episode #60 of RoboPapers now to learn more, hosted by Chris Paxton and Jiafei Duan. In this episode, we cover two papers:. First is VIRAL: Visual Sim-to-Real at Scale for Humanoid Loco-Manipulation; and second is DoorMan: Opening the Sim-to-Real Door for Humanoid Pixel-to-Action Policy Transfer.

RoboPapers

30,767 Aufrufe • vor 9 Monaten

The "reality gap" is narrowing. Sanctuary AI has demonstrated a sim-to-real breakthrough: "zero-shot" transfer for complex in-hand manipulation. By leveraging high-DOF hydraulic hands, the company successfully reoriented a cube in the physical world using a policy trained entirely in simulation.

The "reality gap" is narrowing. Sanctuary AI has demonstrated a sim-to-real breakthrough: "zero-shot" transfer for complex in-hand manipulation. By leveraging high-DOF hydraulic hands, the company successfully reoriented a cube in the physical world using a policy trained entirely in simulation.

Humanoids daily

12,391 Aufrufe • vor 6 Monaten

DiT4DiT is now open source! As the first humanoid-deployable Video-Action Model built on a world model, DiT4DiT continues to surprise us. In our paper last month, we showed its strong data efficiency. Now, with only slight modifications, it enables real-time whole-body autonomous pick-and-place. Paper: Code: Website:

DiT4DiT is now open source! As the first humanoid-deployable Video-Action Model built on a world model, DiT4DiT continues to surprise us. In our paper last month, we showed its strong data efficiency. Now, with only slight modifications, it enables real-time whole-body autonomous pick-and-place. Paper: Code: Website:

Shuo Yang

15,969 Aufrufe • vor 6 Monaten

Advancing dexterous manipulation through scalable visual sim-to-real transfer. We are excited to share our RSS paper, “ViserDex: Visual Sim-to-Real for Robust Dexterous In-hand Reorientation.” 🌐 Project page: 1/N 🧵

Advancing dexterous manipulation through scalable visual sim-to-real transfer. We are excited to share our RSS paper, “ViserDex: Visual Sim-to-Real for Robust Dexterous In-hand Reorientation.” 🌐 Project page: 1/N 🧵

Robotic Systems Lab

40,380 Aufrufe • vor 5 Monaten

Training fully autonomous robots takes more data than the real world can easily provide. Simulation lets us train, test, and deploy autonomous software without being limited by robots, space, or cost. Here’s OM1 running in simulation side-by-side to the real world.

Training fully autonomous robots takes more data than the real world can easily provide. Simulation lets us train, test, and deploy autonomous software without being limited by robots, space, or cost. Here’s OM1 running in simulation side-by-side to the real world.

OpenMind

121,631 Aufrufe • vor 10 Monaten

Reality of robotics: humanoid kung fu is solved before they can open doors with RGB. Here we are. Introducing the frontier of sim2real at NVIDIA GEAR. 100% sim data. RGB input only. Code name: 𝗗𝗼𝗼𝗿𝗠𝗮𝗻. We are opening the sim-to-real door. 🧵

Reality of robotics: humanoid kung fu is solved before they can open doors with RGB. Here we are. Introducing the frontier of sim2real at NVIDIA GEAR. 100% sim data. RGB input only. Code name: 𝗗𝗼𝗼𝗿𝗠𝗮𝗻. We are opening the sim-to-real door. 🧵

Haoru Xue

376,637 Aufrufe • vor 10 Monaten

Announcing RoboFuME🤖💨, a system for autonomous & efficient real-world robot learning that 1. pre-trains a VLM reward model and a multi-task RL policy from diverse off-the-shelf demo data; 2. runs RL fine-tuning online with the VLM reward model. 🔗 🧵↓

Announcing RoboFuME🤖💨, a system for autonomous & efficient real-world robot learning that 1. pre-trains a VLM reward model and a multi-task RL policy from diverse off-the-shelf demo data; 2. runs RL fine-tuning online with the VLM reward model. 🔗 🧵↓

Jingyun Yang

88,312 Aufrufe • vor 3 Jahren

So we did a bunch of projects with real world reinforcement learning - but it was often too inefficient to be practical to train tabula rasa. This suggests we need better priors, but acquiring these from on-robot data can often be expensive as well. In our recent work, we show that despite being fundamentally inaccurate, simulation can guide provide a cheap way to guide real-world RL finetuning to be super efficient! We propose Simulation-Guided Fine-Tuning (SGFT) - a simple paradigm for sim2real finetuning that uses simulation to provide reward shaping that accelerates real world RL finetuning *beyond* just providing an initialization. TLDR: Use value functions from sim to shape rewards for real-world RL, see large sample efficiency improvements 🧵(1/6)

So we did a bunch of projects with real world reinforcement learning - but it was often too inefficient to be practical to train tabula rasa. This suggests we need better priors, but acquiring these from on-robot data can often be expensive as well. In our recent work, we show that despite being fundamentally inaccurate, simulation can guide provide a cheap way to guide real-world RL finetuning to be super efficient! We propose Simulation-Guided Fine-Tuning (SGFT) - a simple paradigm for sim2real finetuning that uses simulation to provide reward shaping that accelerates real world RL finetuning beyond just providing an initialization. TLDR: Use value functions from sim to shape rewards for real-world RL, see large sample efficiency improvements 🧵(1/6)

Abhishek Gupta

13,637 Aufrufe • vor 1 Jahr

A humanoid robot policy trained solely on synthetic data generated by a world model. Research Scientist Joel Jang presents NVIDIA's DreamGen pipeline: ⦿ Post-train the world model Cosmos-Predict2 with a small set of real teleoperation demos. ⦿ Prompt the world model to generate synthetic video data with verbs and scenarios not used in the world model’s post-training. ⦿ Auto-label synthetic video data with action sequences. ⦿ Train robot policies using only synthetic data. That's it. Deploy zero-shot to a real humanoid robot.

A humanoid robot policy trained solely on synthetic data generated by a world model. Research Scientist Joel Jang presents NVIDIA's DreamGen pipeline: ⦿ Post-train the world model Cosmos-Predict2 with a small set of real teleoperation demos. ⦿ Prompt the world model to generate synthetic video data with verbs and scenarios not used in the world model’s post-training. ⦿ Auto-label synthetic video data with action sequences. ⦿ Train robot policies using only synthetic data. That's it. Deploy zero-shot to a real humanoid robot.

The Humanoid Hub

20,968 Aufrufe • vor 1 Jahr

Today, a step forward in open robotics - our results show that sim-to-real zero shot transfer for manipulation is possible. MolmoBot is our open model suite for robotics, trained entirely in simulation on MolmoSpaces.🧵

Today, a step forward in open robotics - our results show that sim-to-real zero shot transfer for manipulation is possible. MolmoBot is our open model suite for robotics, trained entirely in simulation on MolmoSpaces.🧵

Ai2

66,287 Aufrufe • vor 7 Monaten

As robotics teams scale, collecting real-world data can quickly become slow and expensive. 📉 Lightwheel is changing that with a simulation-first platform built on our technology, helping customers generate a 100:1 simulated-to-real data ratio for training autonomous systems. Together, we’re turning synthetic and real data into more capable, more reliable robots. Read the full success story 🔗

As robotics teams scale, collecting real-world data can quickly become slow and expensive. 📉 Lightwheel is changing that with a simulation-first platform built on our technology, helping customers generate a 100:1 simulated-to-real data ratio for training autonomous systems. Together, we’re turning synthetic and real data into more capable, more reliable robots. Read the full success story 🔗

NVIDIA Robotics

10,840 Aufrufe • vor 6 Monaten

Are you opening doors for humanoids… or is your humanoid opening doors for you? Introducing DoorMan, our humanoid door-opening system powered by: • end-to-end RGB pixel-to-action control • large-scale simulation-only training • teacher–student-bootstrap RL This took our team 6 months of relentless iteration. I wasn’t sure it could be done — now it’s real. Website: ArXiv:

Are you opening doors for humanoids… or is your humanoid opening doors for you? Introducing DoorMan, our humanoid door-opening system powered by: • end-to-end RGB pixel-to-action control • large-scale simulation-only training • teacher–student-bootstrap RL This took our team 6 months of relentless iteration. I wasn’t sure it could be done — now it’s real. Website: ArXiv:

Tairan He

32,979 Aufrufe • vor 10 Monaten