Загрузка видео...

Не удалось загрузить видео

На главную

Advancing dexterous manipulation through scalable visual sim-to-real transfer. We are excited to share our RSS paper, “ViserDex: Visual Sim-to-Real for Robust Dexterous In-hand Reorientation.” 🌐 Project page: 1/N 🧵

leggedrobotics's profile picture

Robotic Systems Lab

38,283 subscribers

40,347 просмотров • 4 месяцев назад •via X (Twitter)

Новости и политикаНаука и технологии
LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 6

Фото профиля Robotic Systems Lab
Robotic Systems Lab4 месяцев назад

4/N This work highlights 3D Gaussian Splatting as a practical path toward scalable RGB-only dexterous manipulation. For more details: This work was led by Arjun Bhardwaj (@ThougthShot )

Фото профиля Robotic Systems Lab
Robotic Systems Lab4 месяцев назад

2/N ViserDex introduces a sim-to-real framework for RGB-based in-hand reorientation using 3D Gaussian Splatting. We perform domain randomization directly in the Gaussian representation space to generate photorealistic and diverse training data for robust object pose estimation.

Фото профиля Robotic Systems Lab
Robotic Systems Lab4 месяцев назад

3/N Combined with curriculum-based reinforcement learning and teacher–student distillation, ViserDex enables real-world reorientation of diverse objects on a multi-fingered robotic hand, even under challenging lighting conditions.

Фото профиля kache
kache4 месяцев назад

ahahahahaahaha

Фото профиля Patrick Walsh
Patrick Walsh3 месяцев назад

Gaussian splatting seems to really be making changes in the world of robotics. The better it gets the better robots will get

Фото профиля Péter Kvasznay
Péter Kvasznay20 дней назад

High-dexterity manipulation needs minimal latency. Eliminating loop jitter and CPU overhead is key for sim-to-real stability here.

Похожие видео

🕸️ Introducing SPIDER — Scalable Physics-Informed Dexterous Retargeting! A dynamically feasible, cross-embodiment retargeting framework for BOTH humanoids 🤖 and dexterous hands ✋. From human motion → sim → real robots, at scale. 🔗 Website: 🧵 1/n
1:25

🕸️ Introducing SPIDER — Scalable Physics-Informed Dexterous Retargeting! A dynamically feasible, cross-embodiment retargeting framework for BOTH humanoids 🤖 and dexterous hands ✋. From human motion → sim → real robots, at scale. 🔗 Website: 🧵 1/n

Chaoyi Pan

65,804 просмотров • 10 месяцев назад

How to use simulation data for real-world robot manipulation? We present sim-and-real co-training, a simple recipe for manipulation. We demonstrate that sim data can significantly enhance real-world performance, even with notable differences between the sim and the real. (1/n)
1:15

How to use simulation data for real-world robot manipulation? We present sim-and-real co-training, a simple recipe for manipulation. We demonstrate that sim data can significantly enhance real-world performance, even with notable differences between the sim and the real. (1/n)

Zhenyu Jiang

44,813 просмотров • 1 год назад

Introducing tactile skin sim-to-real for dexterous in-hand translation! We propose a simulation model for ReSkin, a magnetic tactile sensing skin. It can simulate ternary shear and binary normal forces. More:
0:38

Introducing tactile skin sim-to-real for dexterous in-hand translation! We propose a simulation model for ReSkin, a magnetic tactile sensing skin. It can simulate ternary shear and binary normal forces. More:

Haozhi Qi

33,562 просмотров • 2 лет назад

Excited to share my final PhD project😀 We show how simple, yet elegant changes enable diffusion transformers to learn SOTA robotic policies on real robots. Our method improves performance by 20% across a wide range of highly dexterous tasks - like cutting sushi! 1/n
0:50

Excited to share my final PhD project😀 We show how simple, yet elegant changes enable diffusion transformers to learn SOTA robotic policies on real robots. Our method improves performance by 20% across a wide range of highly dexterous tasks - like cutting sushi! 1/n

Sudeep Dasari

20,536 просмотров • 1 год назад

Thrilled to share one of my favorite works this year: DexNDM! We bridge the Sim2Real gap for dexterous in-hand rotation, achieving a true "0-to-1" advancement. The key? DexNDM learns from biased, real-world data without needing any successful demonstrations. Now a general-purpose dexterous hand can stably rotate large books, long rods, & complex objects around any axis, from any wrist pose. This powerful primitive enables complex, long-horizon tasks like teleoperated screwing and furniture assembly. 📄 Paper: 🌐 Project Page:
4:04

Thrilled to share one of my favorite works this year: DexNDM! We bridge the Sim2Real gap for dexterous in-hand rotation, achieving a true "0-to-1" advancement. The key? DexNDM learns from biased, real-world data without needing any successful demonstrations. Now a general-purpose dexterous hand can stably rotate large books, long rods, & complex objects around any axis, from any wrist pose. This powerful primitive enables complex, long-horizon tasks like teleoperated screwing and furniture assembly. 📄 Paper: 🌐 Project Page:

Li Yi

38,887 просмотров • 10 месяцев назад

We talked to Ritvik Singh about how you can train sim-to-real dexterous manipulation policies using NVIDIA Isaac. This robot is grasping objects using pure RGB stereo: take in images from a camera pair and predict what to do, all without training in the real world.
1:08

We talked to Ritvik Singh about how you can train sim-to-real dexterous manipulation policies using NVIDIA Isaac. This robot is grasping objects using pure RGB stereo: take in images from a camera pair and predict what to do, all without training in the real world.

Chris Paxton

20,067 просмотров • 1 год назад

Robot foundation models are limited by costly real data, while simulation data is plentiful but visually mismatched to reality. We present Point Bridge, a method that enables zero-shot sim-to-real transfer for robot learning with minimal visual alignment.
0:50

Robot foundation models are limited by costly real data, while simulation data is plentiful but visually mismatched to reality. We present Point Bridge, a method that enables zero-shot sim-to-real transfer for robot learning with minimal visual alignment.

Siddhant Haldar

19,981 просмотров • 7 месяцев назад

Today, a step forward in open robotics - our results show that sim-to-real zero shot transfer for manipulation is possible. MolmoBot is our open model suite for robotics, trained entirely in simulation on MolmoSpaces.🧵
0:33

Today, a step forward in open robotics - our results show that sim-to-real zero shot transfer for manipulation is possible. MolmoBot is our open model suite for robotics, trained entirely in simulation on MolmoSpaces.🧵

Ai2

65,538 просмотров • 6 месяцев назад

Zero teleoperation. Zero real-world data. ➔ Autonomous humanoid loco-manipulation in reality. Introducing VIRAL: Visual Sim-to-Real at Scale. We achieved 54 autonomous cycles (walk, stand, place, pick, turn) using a simple recipe: 1. RL 2. Simulation 3. GPUs Website: Arxiv: Deep dive with me: 🧵
0:24

Zero teleoperation. Zero real-world data. ➔ Autonomous humanoid loco-manipulation in reality. Introducing VIRAL: Visual Sim-to-Real at Scale. We achieved 54 autonomous cycles (walk, stand, place, pick, turn) using a simple recipe: 1. RL 2. Simulation 3. GPUs Website: Arxiv: Deep dive with me: 🧵

Tairan He

267,992 просмотров • 10 месяцев назад

A big part of scaling robot learning to solve real-world problems is that we somehow need to get enough diverse, high-quality data to train our robots to perform useful things. GPT and its fellow large language models were bootstrapped and proved out on a massive dataset of real-world language data. Unfortunately, despite our best efforts, similarly massive datasets don’t really exist for robotics — so, in our unending pursuit of high-quality, useful data, we turn to simulation. I compared a couple recent works on sim-to-real robot manipulation, which discuss how to train perception-driven manipulation policies in simulation, in such a way that they’re useful in the real world. - DextraH-RGB, from NVIDIA - Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation, also from NVIDIA — specifically the GEAR lab - Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids, another GEAR lab paper - Local Policies Enable Zero-shot Long-Horizon Manipulation, from CMU (video from DextrAH-RGB)
0:33

A big part of scaling robot learning to solve real-world problems is that we somehow need to get enough diverse, high-quality data to train our robots to perform useful things. GPT and its fellow large language models were bootstrapped and proved out on a massive dataset of real-world language data. Unfortunately, despite our best efforts, similarly massive datasets don’t really exist for robotics — so, in our unending pursuit of high-quality, useful data, we turn to simulation. I compared a couple recent works on sim-to-real robot manipulation, which discuss how to train perception-driven manipulation policies in simulation, in such a way that they’re useful in the real world. - DextraH-RGB, from NVIDIA - Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation, also from NVIDIA — specifically the GEAR lab - Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids, another GEAR lab paper - Local Policies Enable Zero-shot Long-Horizon Manipulation, from CMU (video from DextrAH-RGB)

Chris Paxton

20,486 просмотров • 1 год назад

📢📢📢 Excited to release ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learning (CVPR25). 🤏🤙✌️With ManipTrans, we can transfer dexterous manipulation skills into robotic hands in simulation and deploy them on a real robot, using a residual policy learned for dex manipulation. 🤖🤖🤖The video below illustrates how the MoCap data can be transferred to Inspire, Shadow, Xhand, Allegro, and Mano. With ManipTrans, we can scale up dex manip data greatly with minimal effort. For more details, please check our -webpage: -code: -huggingface:
1:59

📢📢📢 Excited to release ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learning (CVPR25). 🤏🤙✌️With ManipTrans, we can transfer dexterous manipulation skills into robotic hands in simulation and deploy them on a real robot, using a residual policy learned for dex manipulation. 🤖🤖🤖The video below illustrates how the MoCap data can be transferred to Inspire, Shadow, Xhand, Allegro, and Mano. With ManipTrans, we can scale up dex manip data greatly with minimal effort. For more details, please check our -webpage: -code: -huggingface:

Siyuan Huang

20,983 просмотров • 1 год назад

Excited to share: Figure 01 completing real-world tasks This is end-to-end autonomous We have made advances in our autonomous navigation, learned perception models, manipulation robust to pose variation, & generalizable systems for future applications
1:21

Excited to share: Figure 01 completing real-world tasks This is end-to-end autonomous We have made advances in our autonomous navigation, learned perception models, manipulation robust to pose variation, & generalizable systems for future applications

Brett Adcock

232,949 просмотров • 2 лет назад

🔥 #ICRA2026 Best Paper Finalist The era of "robot VLA = single-arm gripper" is ending. Introducing Dexora — the first open-source Vision-Language-Action system for dual-arm, dual-hand, 36-DoF dexterous manipulation. 🦾 Dual Arms 🖐️ Dual Hands 🎯 36 DoF Control 🌍 Open Source Trained on: • 100K simulated trajectories • 10K real-world demonstrations Dexora achieves: ✓ 90%+ success on basic manipulation ✓ Strong dexterous manipulation performance ✓ Cross-embodiment generalization Our key hypothesis: Train on the hardest embodiment. Transfer to simpler robots later. Instead of scaling up gripper policies, we train directly in the most expressive action space and project downward to simpler embodiments. This may be a practical path toward universal robot controllers. 🎥 Demos: 📄 Paper:
2:58

🔥 #ICRA2026 Best Paper Finalist The era of "robot VLA = single-arm gripper" is ending. Introducing Dexora — the first open-source Vision-Language-Action system for dual-arm, dual-hand, 36-DoF dexterous manipulation. 🦾 Dual Arms 🖐️ Dual Hands 🎯 36 DoF Control 🌍 Open Source Trained on: • 100K simulated trajectories • 10K real-world demonstrations Dexora achieves: ✓ 90%+ success on basic manipulation ✓ Strong dexterous manipulation performance ✓ Cross-embodiment generalization Our key hypothesis: Train on the hardest embodiment. Transfer to simpler robots later. Instead of scaling up gripper policies, we train directly in the most expressive action space and project downward to simpler embodiments. This may be a practical path toward universal robot controllers. 🎥 Demos: 📄 Paper:

Hao Zhao

17,474 просмотров • 3 месяцев назад

After years of working on precise dexterous manipulation, I am thrilled to share our latest milestone in robotic dexterity! 🤖🗑️ Gemini Robotics 2 achieves advanced spatial awareness, precise control & robust VLA capabilities in multi-finger hand control. Great team success🦾💡
0:39

After years of working on precise dexterous manipulation, I am thrilled to share our latest milestone in robotic dexterity! 🤖🗑️ Gemini Robotics 2 achieves advanced spatial awareness, precise control & robust VLA capabilities in multi-finger hand control. Great team success🦾💡

Maria Bauza Villalonga

15,034 просмотров • 1 месяц назад

1/n: We are excited to share that our paper on Chroma, a general purpose diffusion model for proteins, is out today in nature! A couple of my favorite highlights in the 🧵below 👇
0:21

1/n: We are excited to share that our paper on Chroma, a general purpose diffusion model for proteins, is out today in nature! A couple of my favorite highlights in the 🧵below 👇

Andrew Beam

362,118 просмотров • 2 лет назад

Despite great advances in learning dexterity, hardware remains a major bottleneck. Most dexterous hands are either bulky, weak or expensive. I’m thrilled to present the RUKA Hand — a powerful, accessible research tool for dexterous manipulation that overcomes these limitations!
0:40

Despite great advances in learning dexterity, hardware remains a major bottleneck. Most dexterous hands are either bulky, weak or expensive. I’m thrilled to present the RUKA Hand — a powerful, accessible research tool for dexterous manipulation that overcomes these limitations!

Irmak Guzey

111,933 просмотров • 1 год назад

I'm excited to share our new work Diffusion as Shader (DaS), a versatile controllable video generation method for various tasks: object manipulation, camera control, mesh-to-video, and motion transfer. Project page: Github:
0:46

I'm excited to share our new work Diffusion as Shader (DaS), a versatile controllable video generation method for various tasks: object manipulation, camera control, mesh-to-video, and motion transfer. Project page: Github:

Yuan Liu

33,490 просмотров • 1 год назад

We relived Michelangelo in neural network style!! Super excited to finally share Neuralangelo 🛠 -- 3D surface reconstruction from RGB videos with extremely high fidelity! 🤩 To appear at #CVPR2023 project page: paper: 🧵⬇️ (1/4)
0:24

We relived Michelangelo in neural network style!! Super excited to finally share Neuralangelo 🛠 -- 3D surface reconstruction from RGB videos with extremely high fidelity! 🤩 To appear at #CVPR2023 project page: paper: 🧵⬇️ (1/4)

Chen-Hsuan Lin

253,380 просмотров • 3 лет назад

Excited to share our latest work: “Bio-Inspired Plastic Neural Networks for Zero-Shot Out-of-Distribution Generalization in Complex Animal-Inspired Robots” 🪲🦎 We show that Hebbian learning outperforms LSTM-based adaptation for real-world transfer. It even works without domain randomization! It can handle: ✅ Uneven terrain ✅ Morphological damage ✅ Sim-to-real gaps
0:24

Excited to share our latest work: “Bio-Inspired Plastic Neural Networks for Zero-Shot Out-of-Distribution Generalization in Complex Animal-Inspired Robots” 🪲🦎 We show that Hebbian learning outperforms LSTM-based adaptation for real-world transfer. It even works without domain randomization! It can handle: ✅ Uneven terrain ✅ Morphological damage ✅ Sim-to-real gaps

Sebastian Risi

14,390 просмотров • 1 год назад

For robots to be useful, they must be able to interact with a wide variety of environments; and yet, scaling interaction data is difficult, expensive, and time consuming. Instead, much research revolves around sim-to-real manipulation — but mostly this has not been mobile manipulation. Recently, though, this has begun to change. Two recent papers from Tairan He and Haoru Xue show us how to unlock the potential of this technique, building policies which, without any real data at all, can move objects around in the world and open doors in the real world with a humanoid robot. Watch Episode #60 of RoboPapers now to learn more, hosted by Chris Paxton and Jiafei Duan. In this episode, we cover two papers:. First is VIRAL: Visual Sim-to-Real at Scale for Humanoid Loco-Manipulation; and second is DoorMan: Opening the Sim-to-Real Door for Humanoid Pixel-to-Action Policy Transfer.
1:13:32

For robots to be useful, they must be able to interact with a wide variety of environments; and yet, scaling interaction data is difficult, expensive, and time consuming. Instead, much research revolves around sim-to-real manipulation — but mostly this has not been mobile manipulation. Recently, though, this has begun to change. Two recent papers from Tairan He and Haoru Xue show us how to unlock the potential of this technique, building policies which, without any real data at all, can move objects around in the world and open doors in the real world with a humanoid robot. Watch Episode #60 of RoboPapers now to learn more, hosted by Chris Paxton and Jiafei Duan. In this episode, we cover two papers:. First is VIRAL: Visual Sim-to-Real at Scale for Humanoid Loco-Manipulation; and second is DoorMan: Opening the Sim-to-Real Door for Humanoid Pixel-to-Action Policy Transfer.

RoboPapers

30,767 просмотров • 7 месяцев назад