Загрузка видео...

Не удалось загрузить видео

Возникла проблема при загрузке этого видео. Это может быть связано с временными проблемами сети или видео может быть недоступно.

На главную

Zero teleoperation. Zero real-world data. ➔ Autonomous humanoid loco-manipulation in reality. Introducing VIRAL: Visual Sim-to-Real at Scale. We achieved 54 autonomous cycles (walk, stand, place, pick, turn) using a simple recipe: 1. RL 2. Simulation 3. GPUs Website: Arxiv: Deep dive with me: 🧵

Tairan He

14,452 subscribers

262,321 просмотров • 7 месяцев назад •via X (Twitter)

Образование Наука и технологии

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

How to use simulation data for real-world robot manipulation? We present sim-and-real co-training, a simple recipe for manipulation. We demonstrate that sim data can significantly enhance real-world performance, even with notable differences between the sim and the real. (1/n)

How to use simulation data for real-world robot manipulation? We present sim-and-real co-training, a simple recipe for manipulation. We demonstrate that sim data can significantly enhance real-world performance, even with notable differences between the sim and the real. (1/n)

Zhenyu Jiang

44,227 просмотров • 1 год назад

Robot foundation models are limited by costly real data, while simulation data is plentiful but visually mismatched to reality. We present Point Bridge, a method that enables zero-shot sim-to-real transfer for robot learning with minimal visual alignment.

Robot foundation models are limited by costly real data, while simulation data is plentiful but visually mismatched to reality. We present Point Bridge, a method that enables zero-shot sim-to-real transfer for robot learning with minimal visual alignment.

Siddhant Haldar

19,765 просмотров • 4 месяцев назад

🤔 Ever wondered if simulation-based animation/avatar learnings can be applied to real humanoid in real-time? 🤖 Introducing H2O (Human2HumanOid): - 🧠 An RL-based human-to-humanoid real-time whole-body teleoperation framework - 💃 Scalable retargeting and training using large human motion dataset - 🎥 With just an RGB camera, everyone can teleoperate a full-sized humanoid to perform actions like pick and place, walking, kicking, boxing, etc - 💡Unleash the potential of humanoids with human cognitive skills and adaptability 🔗: 📄: 🎬: H2O proposes: - A scalable retargeting framework for obtaining large-scale humanoid motion dataset, intelligently filtering out infeasible motion for the humanoid embodiment. - We train a full-body motion imitator (similar to PHC) and deploy to the real world zero-shot. - Using this framework, we enable real-time teleoperation of a humanoid via a human operator and webcam, performing skills such as pick and place, kicking, walking strollers, etc. Team: Tairan He Zhengyi “Zen” Luo Wenli Xiao @ChongZitaZhang Kris Kitani Changliu Liu Guanya Shi

Zhengyi “Zen” Luo

47,287 просмотров • 2 лет назад

Our 6ft humanoid HumanPlus at Stanford can autonomously put on a Nike skateboard shoe, tie shoelaces, stand up and walk. Using two transformers & dual RGB vision, it integrates two recipes of general robotics end-to-end: - imitating humans in real world - large-scale RL in sim

Our 6ft humanoid HumanPlus at Stanford can autonomously put on a Nike skateboard shoe, tie shoelaces, stand up and walk. Using two transformers & dual RGB vision, it integrates two recipes of general robotics end-to-end: - imitating humans in real world - large-scale RL in sim

Zipeng Fu

245,139 просмотров • 2 лет назад

A big part of scaling robot learning to solve real-world problems is that we somehow need to get enough diverse, high-quality data to train our robots to perform useful things. GPT and its fellow large language models were bootstrapped and proved out on a massive dataset of real-world language data. Unfortunately, despite our best efforts, similarly massive datasets don’t really exist for robotics — so, in our unending pursuit of high-quality, useful data, we turn to simulation. I compared a couple recent works on sim-to-real robot manipulation, which discuss how to train perception-driven manipulation policies in simulation, in such a way that they’re useful in the real world. - DextraH-RGB, from NVIDIA - Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation, also from NVIDIA — specifically the GEAR lab - Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids, another GEAR lab paper - Local Policies Enable Zero-shot Long-Horizon Manipulation, from CMU (video from DextrAH-RGB)

A big part of scaling robot learning to solve real-world problems is that we somehow need to get enough diverse, high-quality data to train our robots to perform useful things. GPT and its fellow large language models were bootstrapped and proved out on a massive dataset of real-world language data. Unfortunately, despite our best efforts, similarly massive datasets don’t really exist for robotics — so, in our unending pursuit of high-quality, useful data, we turn to simulation. I compared a couple recent works on sim-to-real robot manipulation, which discuss how to train perception-driven manipulation policies in simulation, in such a way that they’re useful in the real world. - DextraH-RGB, from NVIDIA - Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation, also from NVIDIA — specifically the GEAR lab - Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids, another GEAR lab paper - Local Policies Enable Zero-shot Long-Horizon Manipulation, from CMU (video from DextrAH-RGB)

Chris Paxton

20,486 просмотров • 1 год назад

For robots to be useful, they must be able to interact with a wide variety of environments; and yet, scaling interaction data is difficult, expensive, and time consuming. Instead, much research revolves around sim-to-real manipulation — but mostly this has not been mobile manipulation. Recently, though, this has begun to change. Two recent papers from Tairan He and Haoru Xue show us how to unlock the potential of this technique, building policies which, without any real data at all, can move objects around in the world and open doors in the real world with a humanoid robot. Watch Episode #60 of RoboPapers now to learn more, hosted by Chris Paxton and Jiafei Duan. In this episode, we cover two papers:. First is VIRAL: Visual Sim-to-Real at Scale for Humanoid Loco-Manipulation; and second is DoorMan: Opening the Sim-to-Real Door for Humanoid Pixel-to-Action Policy Transfer.

For robots to be useful, they must be able to interact with a wide variety of environments; and yet, scaling interaction data is difficult, expensive, and time consuming. Instead, much research revolves around sim-to-real manipulation — but mostly this has not been mobile manipulation. Recently, though, this has begun to change. Two recent papers from Tairan He and Haoru Xue show us how to unlock the potential of this technique, building policies which, without any real data at all, can move objects around in the world and open doors in the real world with a humanoid robot. Watch Episode #60 of RoboPapers now to learn more, hosted by Chris Paxton and Jiafei Duan. In this episode, we cover two papers:. First is VIRAL: Visual Sim-to-Real at Scale for Humanoid Loco-Manipulation; and second is DoorMan: Opening the Sim-to-Real Door for Humanoid Pixel-to-Action Policy Transfer.

RoboPapers

30,767 просмотров • 4 месяцев назад

DiT4DiT is now open source! As the first humanoid-deployable Video-Action Model built on a world model, DiT4DiT continues to surprise us. In our paper last month, we showed its strong data efficiency. Now, with only slight modifications, it enables real-time whole-body autonomous pick-and-place. Paper: Code: Website:

DiT4DiT is now open source! As the first humanoid-deployable Video-Action Model built on a world model, DiT4DiT continues to surprise us. In our paper last month, we showed its strong data efficiency. Now, with only slight modifications, it enables real-time whole-body autonomous pick-and-place. Paper: Code: Website:

Shuo Yang

13,846 просмотров • 2 месяцев назад

The "reality gap" is narrowing. Sanctuary AI has demonstrated a sim-to-real breakthrough: "zero-shot" transfer for complex in-hand manipulation. By leveraging high-DOF hydraulic hands, the company successfully reoriented a cube in the physical world using a policy trained entirely in simulation.

The "reality gap" is narrowing. Sanctuary AI has demonstrated a sim-to-real breakthrough: "zero-shot" transfer for complex in-hand manipulation. By leveraging high-DOF hydraulic hands, the company successfully reoriented a cube in the physical world using a policy trained entirely in simulation.

Humanoids daily

12,391 просмотров • 2 месяцев назад

Training fully autonomous robots takes more data than the real world can easily provide. Simulation lets us train, test, and deploy autonomous software without being limited by robots, space, or cost. Here’s OM1 running in simulation side-by-side to the real world.

Training fully autonomous robots takes more data than the real world can easily provide. Simulation lets us train, test, and deploy autonomous software without being limited by robots, space, or cost. Here’s OM1 running in simulation side-by-side to the real world.

OpenMind

120,843 просмотров • 6 месяцев назад

Advancing dexterous manipulation through scalable visual sim-to-real transfer. We are excited to share our RSS paper, “ViserDex: Visual Sim-to-Real for Robust Dexterous In-hand Reorientation.” 🌐 Project page: 1/N 🧵

Advancing dexterous manipulation through scalable visual sim-to-real transfer. We are excited to share our RSS paper, “ViserDex: Visual Sim-to-Real for Robust Dexterous In-hand Reorientation.” 🌐 Project page: 1/N 🧵

Robotic Systems Lab

39,143 просмотров • 1 месяц назад

Reality of robotics: humanoid kung fu is solved before they can open doors with RGB. Here we are. Introducing the frontier of sim2real at NVIDIA GEAR. 100% sim data. RGB input only. Code name: 𝗗𝗼𝗼𝗿𝗠𝗮𝗻. We are opening the sim-to-real door. 🧵

Reality of robotics: humanoid kung fu is solved before they can open doors with RGB. Here we are. Introducing the frontier of sim2real at NVIDIA GEAR. 100% sim data. RGB input only. Code name: 𝗗𝗼𝗼𝗿𝗠𝗮𝗻. We are opening the sim-to-real door. 🧵

Haoru Xue

370,906 просмотров • 6 месяцев назад

So we did a bunch of projects with real world reinforcement learning - but it was often too inefficient to be practical to train tabula rasa. This suggests we need better priors, but acquiring these from on-robot data can often be expensive as well. In our recent work, we show that despite being fundamentally inaccurate, simulation can guide provide a cheap way to guide real-world RL finetuning to be super efficient! We propose Simulation-Guided Fine-Tuning (SGFT) - a simple paradigm for sim2real finetuning that uses simulation to provide reward shaping that accelerates real world RL finetuning *beyond* just providing an initialization. TLDR: Use value functions from sim to shape rewards for real-world RL, see large sample efficiency improvements 🧵(1/6)

So we did a bunch of projects with real world reinforcement learning - but it was often too inefficient to be practical to train tabula rasa. This suggests we need better priors, but acquiring these from on-robot data can often be expensive as well. In our recent work, we show that despite being fundamentally inaccurate, simulation can guide provide a cheap way to guide real-world RL finetuning to be super efficient! We propose Simulation-Guided Fine-Tuning (SGFT) - a simple paradigm for sim2real finetuning that uses simulation to provide reward shaping that accelerates real world RL finetuning beyond just providing an initialization. TLDR: Use value functions from sim to shape rewards for real-world RL, see large sample efficiency improvements 🧵(1/6)

Abhishek Gupta

13,591 просмотров • 1 год назад

A humanoid robot policy trained solely on synthetic data generated by a world model. Research Scientist Joel Jang presents NVIDIA's DreamGen pipeline: ⦿ Post-train the world model Cosmos-Predict2 with a small set of real teleoperation demos. ⦿ Prompt the world model to generate synthetic video data with verbs and scenarios not used in the world model’s post-training. ⦿ Auto-label synthetic video data with action sequences. ⦿ Train robot policies using only synthetic data. That's it. Deploy zero-shot to a real humanoid robot.

A humanoid robot policy trained solely on synthetic data generated by a world model. Research Scientist Joel Jang presents NVIDIA's DreamGen pipeline: ⦿ Post-train the world model Cosmos-Predict2 with a small set of real teleoperation demos. ⦿ Prompt the world model to generate synthetic video data with verbs and scenarios not used in the world model’s post-training. ⦿ Auto-label synthetic video data with action sequences. ⦿ Train robot policies using only synthetic data. That's it. Deploy zero-shot to a real humanoid robot.

The Humanoid Hub

20,968 просмотров • 11 месяцев назад

Today, a step forward in open robotics - our results show that sim-to-real zero shot transfer for manipulation is possible. MolmoBot is our open model suite for robotics, trained entirely in simulation on MolmoSpaces.🧵

Today, a step forward in open robotics - our results show that sim-to-real zero shot transfer for manipulation is possible. MolmoBot is our open model suite for robotics, trained entirely in simulation on MolmoSpaces.🧵

Ai2

64,423 просмотров • 3 месяцев назад

As robotics teams scale, collecting real-world data can quickly become slow and expensive. 📉 Lightwheel is changing that with a simulation-first platform built on our technology, helping customers generate a 100:1 simulated-to-real data ratio for training autonomous systems. Together, we’re turning synthetic and real data into more capable, more reliable robots. Read the full success story 🔗

As robotics teams scale, collecting real-world data can quickly become slow and expensive. 📉 Lightwheel is changing that with a simulation-first platform built on our technology, helping customers generate a 100:1 simulated-to-real data ratio for training autonomous systems. Together, we’re turning synthetic and real data into more capable, more reliable robots. Read the full success story 🔗

NVIDIA Robotics

10,780 просмотров • 2 месяцев назад

Are you opening doors for humanoids… or is your humanoid opening doors for you? Introducing DoorMan, our humanoid door-opening system powered by: • end-to-end RGB pixel-to-action control • large-scale simulation-only training • teacher–student-bootstrap RL This took our team 6 months of relentless iteration. I wasn’t sure it could be done — now it’s real. Website: ArXiv:

Are you opening doors for humanoids… or is your humanoid opening doors for you? Introducing DoorMan, our humanoid door-opening system powered by: • end-to-end RGB pixel-to-action control • large-scale simulation-only training • teacher–student-bootstrap RL This took our team 6 months of relentless iteration. I wasn’t sure it could be done — now it’s real. Website: ArXiv:

Tairan He

31,573 просмотров • 6 месяцев назад

So I heard we need more data for robot learning :) Purely real world teleop is expensive and slow, making large scale data collection challenging. I’ve been excited about getting more data into robot learning, going beyond just real-world teleop data. To this end, we’ve been scaling up data generation with RL in realistic simulations generated on the fly from crowdsourced videos. Enables realistic data collection, much more cheaply than purely real world teleop. Importantly, data collection becomes even*cheaper* with more environments, allowing training with over 100x more data. Transfers to real robots for generalizable manipulation. A 🧵 (1/N)

So I heard we need more data for robot learning :) Purely real world teleop is expensive and slow, making large scale data collection challenging. I’ve been excited about getting more data into robot learning, going beyond just real-world teleop data. To this end, we’ve been scaling up data generation with RL in realistic simulations generated on the fly from crowdsourced videos. Enables realistic data collection, much more cheaply than purely real world teleop. Importantly, data collection becomes evencheaper with more environments, allowing training with over 100x more data. Transfers to real robots for generalizable manipulation. A 🧵 (1/N)

Abhishek Gupta

13,336 просмотров • 1 год назад

We are showing off autonomous humanoid navigation at the WOW Summit. Our Unitree G1 humanoid is connecting to the real-world web, receiving spatial data about the conference, and guiding visitors to the booths they want to see. Probably nothing.

We are showing off autonomous humanoid navigation at the WOW Summit. Our Unitree G1 humanoid is connecting to the real-world web, receiving spatial data about the conference, and guiding visitors to the booths they want to see. Probably nothing.

Auki

63,882 просмотров • 9 месяцев назад

In-hand cylinder rotation with RL. IROS 2025 Different sizes, zero-shot sim-to-real, and full dynamic tactile feedback — all in one hand. #Sharpa #SharpaWave #IROS2025 #Robotics #DexterousHand #Humanoid #TactileSensing #ReinforcementLearning #IssacSim

In-hand cylinder rotation with RL. IROS 2025 Different sizes, zero-shot sim-to-real, and full dynamic tactile feedback — all in one hand. #Sharpa #SharpaWave #IROS2025 #Robotics #DexterousHand #Humanoid #TactileSensing #ReinforcementLearning #IssacSim

Sharpa

112,341 просмотров • 8 месяцев назад

🤖 Introducing H2O (Human2HumanOid): - 🧠 An RL-based human-to-humanoid real-time whole-body teleoperation framework - 💃 Scalable retargeting and training using large human motion dataset - 🎥 With just an RGB camera, everyone can teleoperate a full-sized humanoid to perform actions like pick and place, walking, kicking, boxing, etc - 💡Unleash the potential of humanoids with human cognitive skills and adaptability 🔗:

🤖 Introducing H2O (Human2HumanOid): - 🧠 An RL-based human-to-humanoid real-time whole-body teleoperation framework - 💃 Scalable retargeting and training using large human motion dataset - 🎥 With just an RGB camera, everyone can teleoperate a full-sized humanoid to perform actions like pick and place, walking, kicking, boxing, etc - 💡Unleash the potential of humanoids with human cognitive skills and adaptability 🔗:

Tairan He

96,714 просмотров • 2 лет назад