Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Swarm navigation at 20 m/s with no communication, no state estimation, & on a $21 computer? This paper combines deep learning + differentiable sim for zero-shot sim-to-real flight. It is an underrated breakthrough. Paper: #naturemachineintelligence

Yunlong Song

1,285 subscribers

17,515 views • 1 year ago •via X (Twitter)

#naturemachineintelligence

Anya Rossi• Live Now

Private livecam show

5 Comments

Chuck Petras1 year ago

@GrandpaRoy2 @johnrobb @BrianRoemmele

kache1 year ago

thank you for sharing

Irene1 year ago

Does this mean that future drones will be smaller and lighter and less expensive?

MangoMagic™️1 year ago

Remarkable. Just wait until the drones start taking selfies. #FlyingUnderTheRadar #BudgetBillionaires #DeepLearningDisco

Zanir Habib1 year ago

Hmmm....it seems the best methods are going to be those that pipe in the most data about physics/the environment? Was reading that genetic algos are also good at these types of problems

Related Videos

A robot that sees the terrain and predicts its own future… up to 5 seconds ahead? This is real. ❗️Best Systems Paper finalist at #RSS2025 The team introduces a perceptive Forward Dynamics Model that helps legged robots safely navigate rough, complex environments: no manual tuning no rigid-body limits. Why this matters ✅ Predicts future robot states using both perception and proprioception ✅ Trained on years of sim + real data for strong sim-to-real performance ✅ Enables zero-shot navigation with minimal cost tuning ✅ Boosts success rates and safety on rough terrain like never before This isn’t just another planner, it’s a big step toward robust, adaptable locomotion. Paper: Project website:

A robot that sees the terrain and predicts its own future… up to 5 seconds ahead? This is real. ❗️Best Systems Paper finalist at #RSS2025 The team introduces a perceptive Forward Dynamics Model that helps legged robots safely navigate rough, complex environments: no manual tuning no rigid-body limits. Why this matters ✅ Predicts future robot states using both perception and proprioception ✅ Trained on years of sim + real data for strong sim-to-real performance ✅ Enables zero-shot navigation with minimal cost tuning ✅ Boosts success rates and safety on rough terrain like never before This isn’t just another planner, it’s a big step toward robust, adaptable locomotion. Paper: Project website:

Ilir Aliu

43,990 views • 1 year ago

Haven't been to a conference in a while, really excited to be at #NeurIPS2024! I'll be helping present 4 of our group's recent papers: 1. Overcoming the Sim-to-Real Gap: Leveraging Simulation to Learn to Explore for Real-World RL 2. Distributional Successor Features Enable Zero-Shot Policy Optimization 3. Learning to Cooperate with Humans using Generative Agents 4. Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning Find more details on each paper and where to find us in this thread (1/6)

Haven't been to a conference in a while, really excited to be at #NeurIPS2024! I'll be helping present 4 of our group's recent papers: 1. Overcoming the Sim-to-Real Gap: Leveraging Simulation to Learn to Explore for Real-World RL 2. Distributional Successor Features Enable Zero-Shot Policy Optimization 3. Learning to Cooperate with Humans using Generative Agents 4. Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning Find more details on each paper and where to find us in this thread (1/6)

Abhishek Gupta

10,803 views • 1 year ago

Introducing LocoTouch: Quadrupedal robots equipped with tactile sensing can now transport unsecured objects — no mounts, no straps. The tactile policy transfers zero-shot from sim to real. Core Task-Agnostic Features: 1. High-fidelity contact simulation for distributed tactile sensors 2. Adaptive gait reward for symmetric, frequency-adaptive locomotion Website: Paper:

Introducing LocoTouch: Quadrupedal robots equipped with tactile sensing can now transport unsecured objects — no mounts, no straps. The tactile policy transfers zero-shot from sim to real. Core Task-Agnostic Features: 1. High-fidelity contact simulation for distributed tactile sensors 2. Adaptive gait reward for symmetric, frequency-adaptive locomotion Website: Paper:

Changyi Lin

38,494 views • 1 year ago

Robot foundation models are limited by costly real data, while simulation data is plentiful but visually mismatched to reality. We present Point Bridge, a method that enables zero-shot sim-to-real transfer for robot learning with minimal visual alignment.

Robot foundation models are limited by costly real data, while simulation data is plentiful but visually mismatched to reality. We present Point Bridge, a method that enables zero-shot sim-to-real transfer for robot learning with minimal visual alignment.

Siddhant Haldar

19,981 views • 7 months ago

We are thrilled to share our breakthrough research on "Agile Flight from Pixels without State Estimation," to be presented and live-demonstrated at #RSS2024 next week! You heard well: no state estimation means no explicit visual localization, no SLAM, no VIO, and no IMU! Paper: Video (Narrated): Last year, we demonstrated that #ReinforcementLearning (RL) policies could outperform world-champion drone-racing pilots using the same quadrotor hardware; however, unlike human pilots, these policies continuously estimated an explicit state from known gate positions, the camera feed, and inertial measurements (IMU). In this new work, we tackle the challenge of learning vision-based drone racing using an end-to-end reinforcement learning approach that eliminates the need for IMU data or explicit state estimation. Like professional pilots, we go directly from images to control commands. The training is facilitated by an asymmetric actor-critic with access to privileged information. To overcome the computational complexity during image-based RL training, we use an appropriate sensor representation, which can be efficiently simulated during training without rendering images. We achieve agile flight at speeds up to 40 km/h with accelerations up to 2 g's. Although our demonstration focuses on drone racing, we believe that our method has an impact beyond drone racing and can serve as a foundation for future research into real-world applications in structured environments. Besides the paper presentation, we will also give a live demo next Tuesday and Wednesday between and hrs at TU Delft: Reference: Ismail Geles*, Leonard Bauersfeld*, Angel Romero, Jiaxu Xing, Davide Scaramuzza "Demonstrating Agile Flight from Pixels without State Estimation" Robotics: Science and Systems (RSS), 2024. Kudos to Ismail Geles Leonard Bauersfeld Ángel Romero Jiaxu Xing! University of Zurich UZH Science UZH Space Hub Aerial Core AUTOASSESS European Research Council (ERC)

We are thrilled to share our breakthrough research on "Agile Flight from Pixels without State Estimation," to be presented and live-demonstrated at #RSS2024 next week! You heard well: no state estimation means no explicit visual localization, no SLAM, no VIO, and no IMU! Paper: Video (Narrated): Last year, we demonstrated that #ReinforcementLearning (RL) policies could outperform world-champion drone-racing pilots using the same quadrotor hardware; however, unlike human pilots, these policies continuously estimated an explicit state from known gate positions, the camera feed, and inertial measurements (IMU). In this new work, we tackle the challenge of learning vision-based drone racing using an end-to-end reinforcement learning approach that eliminates the need for IMU data or explicit state estimation. Like professional pilots, we go directly from images to control commands. The training is facilitated by an asymmetric actor-critic with access to privileged information. To overcome the computational complexity during image-based RL training, we use an appropriate sensor representation, which can be efficiently simulated during training without rendering images. We achieve agile flight at speeds up to 40 km/h with accelerations up to 2 g's. Although our demonstration focuses on drone racing, we believe that our method has an impact beyond drone racing and can serve as a foundation for future research into real-world applications in structured environments. Besides the paper presentation, we will also give a live demo next Tuesday and Wednesday between and hrs at TU Delft: Reference: Ismail Geles, Leonard Bauersfeld, Angel Romero, Jiaxu Xing, Davide Scaramuzza "Demonstrating Agile Flight from Pixels without State Estimation" Robotics: Science and Systems (RSS), 2024. Kudos to Ismail Geles Leonard Bauersfeld Ángel Romero Jiaxu Xing! University of Zurich UZH Science UZH Space Hub Aerial Core AUTOASSESS European Research Council (ERC)

Davide Scaramuzza

28,002 views • 2 years ago

I think the Supreme Court’s ruling on birthright citizenship was a serious mistake. Most Americans understand the danger: people with zero loyalty to this country come here, have a baby on our soil, then head home. Only for that child to return as an adult with real influence over America’s future, even a shot at the presidency. No deep attachment, no roots, no allegiance. It makes ZERO sense, & I hope this finally forces Congress to act.

I think the Supreme Court’s ruling on birthright citizenship was a serious mistake. Most Americans understand the danger: people with zero loyalty to this country come here, have a baby on our soil, then head home. Only for that child to return as an adult with real influence over America’s future, even a shot at the presidency. No deep attachment, no roots, no allegiance. It makes ZERO sense, & I hope this finally forces Congress to act.

Kari Lake

56,481 views • 2 months ago

Excited to share our #ICML2023 paper ✨LIV✨! Extending VIP, LIV is at once a pre-training, fine-tuning, and (zero-shot!) multi-modal reward method for (real-world!) language-conditioned robotic control. Project: Code & Model: 🧵:

Excited to share our #ICML2023 paper ✨LIV✨! Extending VIP, LIV is at once a pre-training, fine-tuning, and (zero-shot!) multi-modal reward method for (real-world!) language-conditioned robotic control. Project: Code & Model: 🧵:

Jason Ma

55,068 views • 3 years ago

MiniMax M2.7 is real deal man I have made a complete website with proper fan physics and fibre physics this looks so good MiniMax (official) i am waiting for the next update man , i want a big model next cook it for us Prompt is in comment for normal paper receipt physics sim

MiniMax M2.7 is real deal man I have made a complete website with proper fan physics and fibre physics this looks so good MiniMax (official) i am waiting for the next update man , i want a big model next cook it for us Prompt is in comment for normal paper receipt physics sim

Chetaslua

186,910 views • 6 months ago

A big part of scaling robot learning to solve real-world problems is that we somehow need to get enough diverse, high-quality data to train our robots to perform useful things. GPT and its fellow large language models were bootstrapped and proved out on a massive dataset of real-world language data. Unfortunately, despite our best efforts, similarly massive datasets don’t really exist for robotics — so, in our unending pursuit of high-quality, useful data, we turn to simulation. I compared a couple recent works on sim-to-real robot manipulation, which discuss how to train perception-driven manipulation policies in simulation, in such a way that they’re useful in the real world. - DextraH-RGB, from NVIDIA - Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation, also from NVIDIA — specifically the GEAR lab - Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids, another GEAR lab paper - Local Policies Enable Zero-shot Long-Horizon Manipulation, from CMU (video from DextrAH-RGB)

A big part of scaling robot learning to solve real-world problems is that we somehow need to get enough diverse, high-quality data to train our robots to perform useful things. GPT and its fellow large language models were bootstrapped and proved out on a massive dataset of real-world language data. Unfortunately, despite our best efforts, similarly massive datasets don’t really exist for robotics — so, in our unending pursuit of high-quality, useful data, we turn to simulation. I compared a couple recent works on sim-to-real robot manipulation, which discuss how to train perception-driven manipulation policies in simulation, in such a way that they’re useful in the real world. - DextraH-RGB, from NVIDIA - Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation, also from NVIDIA — specifically the GEAR lab - Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids, another GEAR lab paper - Local Policies Enable Zero-shot Long-Horizon Manipulation, from CMU (video from DextrAH-RGB)

Chris Paxton

20,486 views • 1 year ago

Humanoids finally move like humans… and can do more than copy. [Details + demos in thread 👇] A new framework, BeyondMimic, shows how to learn naturalistic whole-body control from human motion. But then goes further by composing those skills into versatile, zero-shot policies. Two key pieces make it work: •Motion tracking pipeline: robustly turns human mocap into dynamic robot motions (jump spins, sprinting, cartwheels) on real hardware •Guided diffusion policy: distills motion primitives and enables zero-shot downstream tasks with simple cost functions The result: ✅ State-of-the-art motion quality from mocap tracking ✅ Versatile task execution; waypoint navigation, joystick teleop, obstacle avoidance ✅ Stable sim-to-real transfer for diverse humanoid skills Thank you for sharing, Qiayuan Liao! 📍 Paper: Video:

Humanoids finally move like humans… and can do more than copy. [Details + demos in thread 👇] A new framework, BeyondMimic, shows how to learn naturalistic whole-body control from human motion. But then goes further by composing those skills into versatile, zero-shot policies. Two key pieces make it work: •Motion tracking pipeline: robustly turns human mocap into dynamic robot motions (jump spins, sprinting, cartwheels) on real hardware •Guided diffusion policy: distills motion primitives and enables zero-shot downstream tasks with simple cost functions The result: ✅ State-of-the-art motion quality from mocap tracking ✅ Versatile task execution; waypoint navigation, joystick teleop, obstacle avoidance ✅ Stable sim-to-real transfer for diverse humanoid skills Thank you for sharing, Qiayuan Liao! 📍 Paper: Video:

Ilir Aliu - eu/acc

72,522 views • 1 year ago

Every empty state, secondary state, and error state imported into an editable canvas automatically. We don't have to map this stuff out manually anymore. Just ask for it, iterate on it, done. I was initially a bit negative on Paper, but this is a game changing workflow!

Every empty state, secondary state, and error state imported into an editable canvas automatically. We don't have to map this stuff out manually anymore. Just ask for it, iterate on it, done. I was initially a bit negative on Paper, but this is a game changing workflow!

Nolan Perkins

16,048 views • 6 months ago

Miley Cyrus reveals that she cancels Christmas every year because of a rare medical condition "Christmas I'm known as a little bit of a Grinch because I have a qualm with Christmas. I hate paper, looking at that makes me want to vomit." "the real problem for me with paper is when people have dry hands and they touch paper, and then it's Christmas and it's cold and everyone's hands are dry and they're all touching paper." "when someone sends me like a lovely letter, I just don't even open it, just don't even worry. Text message no paper, email no paper, phone call no paper." "I can't even talk about it. Newspaper? That, no." "Rolling papers? That's fine. First of all, my hands aren't dry, okay? And neither are the papers, because I know you know you have to lick the paper."

Miley Cyrus reveals that she cancels Christmas every year because of a rare medical condition "Christmas I'm known as a little bit of a Grinch because I have a qualm with Christmas. I hate paper, looking at that makes me want to vomit." "the real problem for me with paper is when people have dry hands and they touch paper, and then it's Christmas and it's cold and everyone's hands are dry and they're all touching paper." "when someone sends me like a lovely letter, I just don't even open it, just don't even worry. Text message no paper, email no paper, phone call no paper." "I can't even talk about it. Newspaper? That, no." "Rolling papers? That's fine. First of all, my hands aren't dry, okay? And neither are the papers, because I know you know you have to lick the paper."

Vince

877,630 views • 27 days ago

Xavier's Special Birthday Collab | Paper Carving Lantern Etching an infinite galaxy onto paper, wish for the soft, radiant starlight to grace each night with its gentle glow. The exclusive interview for Xavier's special birthday collab is now released! ✨The Story Behind Paper Carving Lantern Narrator: Bingcai (Real Name: Yao Yanlu), representative inheritor of the Yao Family's Paper Carving The paper carving lantern is celebrated as an exquisite art form where light and shadow gracefully dance across the paper. It combines the magic of paper with the precision of a blade. Light drifts and dances through each intricately carved line, unveiling the endless splendor of a shimmering galaxy. ✨This special artwork captures a fleeting moment of tenderness within the vastness of the cosmos. Master Bingcai will share the intricate process of carving paper lanterns and reveal the inspiration and journey behind this piece. Come and light the romance of shadow and glow beneath paper lanterns! #LoveandDeepspace #Xavier #HBDXavier

Xavier's Special Birthday Collab | Paper Carving Lantern Etching an infinite galaxy onto paper, wish for the soft, radiant starlight to grace each night with its gentle glow. The exclusive interview for Xavier's special birthday collab is now released! ✨The Story Behind Paper Carving Lantern Narrator: Bingcai (Real Name: Yao Yanlu), representative inheritor of the Yao Family's Paper Carving The paper carving lantern is celebrated as an exquisite art form where light and shadow gracefully dance across the paper. It combines the magic of paper with the precision of a blade. Light drifts and dances through each intricately carved line, unveiling the endless splendor of a shimmering galaxy. ✨This special artwork captures a fleeting moment of tenderness within the vastness of the cosmos. Master Bingcai will share the intricate process of carving paper lanterns and reveal the inspiration and journey behind this piece. Come and light the romance of shadow and glow beneath paper lanterns! #LoveandDeepspace #Xavier #HBDXavier

Love and Deepspace

103,785 views • 1 year ago

From live video to 3D avatar in seconds. This #SIGGRAPH2025 paper from Adobe Research reconstructs a realistic head avatar on the fly, with no pre-cached data, and adapts seamlessly to facial motion for VR, animation, and online communication. 🔗

Adobe Research

29,176 views • 1 year ago