正在加载视频...

视频加载失败

We are thrilled to share our breakthrough research on "Agile Flight from Pixels without State Estimation," to be presented and live-demonstrated at #RSS2024 next week! You heard well: no state estimation means no explicit visual localization, no SLAM, no VIO, and no IMU! Paper: Video (Narrated): Last year, we...

28,002 次观看 • 2 年前 •via X (Twitter)

5 条评论

sam cash 🌎 的头像
sam cash 🌎2 年前

Impressive

Kaltoro ⚡️ 的头像
Kaltoro ⚡️2 年前

The drones are gonna kill us all aren't they? Yes, yes they are.

Sabeer Saeed 的头像
Sabeer Saeed2 年前

Super Outstanding..

Micah Corah 的头像
Micah Corah2 年前

What exactly does "no IMU" constitute here? Is there a low level flight controller that relies on an IMU, or do you mean there is no IMU entirely?

iandanforth 🦋 @iandanforth.bsky.social 的头像
iandanforth 🦋 @iandanforth.bsky.social2 年前

Would it be even better if you gave it access to IMU data?

相关视频

Can an inexpensive, off-the-shelf IMU be the only sensor to estimate the full state (position, velocity, orientation) of a quadrotor flying through a track at high speed and even be on-pair with vision-based localization? The answer is yes, within certain limitations! In this #RAL2023 paper, we propose a learning-based odometry algorithm that couples a model-based filter driven by the inertial measurements with a learning-based module with access to the control commands. Our system outperforms by a large margin the state-of-the-art visual-inertial odometry (#VIO) algorithms and the state-of-the-art learned-inertial odometry algorithm, #TLIO, for the task of drone racing. Additionally, we show that our system is as accurate as a VIO algorithm that uses a camera to localize to a known map of the racing track. The main limitation of our approach is that it cannot generalize to trajectories that have not been seen at training time. However, in drone racing competitions, the track is known beforehand. Human pilots spend hours or even days of practice on the race track before the competition. Similarly, our system can be trained with the data collected during practice time and deployed during the competition. Future work will investigate how to generalize to trajectories not seen at training time. The code is released! Paper: Video: Code: Kudos to Giovanni Cioffi Leonard Bauersfeld Elia Kaufmann European Research Council (ERC) University of Zurich UZH Science UZH Space Hub NCCR Robotics Aerial Core #RAL2023 #IROS2023 #SLAM

Davide Scaramuzza

37,061 次观看 • 3 年前

We are excited to share our latest work, "Superhuman Safe and Agile Racing through Multi-Agent Reinforcement Learning," done in collaboration with Google DeepMind . Autonomous drones have reached superhuman speed in isolation, but what happens when multiple agents share the same airspace? Paper: Website: Video: Using league-based self-play, we train #ReinforcementLearning agents that race against a diverse, evolving population of opponents. Through this competitive training, sophisticated behaviors emerge without explicit programming: strategic overtaking, proactive collision avoidance, and even awareness of aerodynamic downwash from nearby drones. In real-world multi-player races at speeds exceeding 80kph (50 mph) and accelerations up to 7g, our agents outperform a five-time Swiss national drone racing champion while reducing collision rates by 50% compared to single-agent baselines. Crucially, training against diverse artificial opponents enables zero-shot generalization to human pilots, achieving over 90% race completion in mixed human-AI races with up to four competitors. A key insight: human pilots adopt riskier strategies when trailing, leading to more crashes under competitive pressure. Our learned policies, by contrast, maintain consistent safety margins regardless of race standing, a property essential for deploying autonomous systems alongside humans. Also, the multi-agent self-play policies are more robust than those trained independently, suggesting that training in competitive environments is not only key to winning races but also to learning safer, more reliable autonomy for real-world multi-robot systems. Kudos to Ismail Geles, Leonard Bauersfeld, Markus Wulfmeier! Ismail Geles Leonard Bauersfeld Markus Wulfmeier European Research Council (ERC) UZH IfI University of Zurich UZH Science UZH Space Hub Swiss Robotics NCCR Robotics

Davide Scaramuzza

14,713 次观看 • 4 个月前

Check out our latest work, "Actor-Critic Model Predictive Control: Differentiable Optimization meets Reinforcement Learning for Agile Flight," published in the IEEE Transactions on Robotics, where we reconcile #OptimalControl and #ReinforcementLearning, achieving the same super-human performance, but with superior generalizability, as our previous model-free deep RL! Code released! PDF: Code: Full Video: Model-free #ReinforcementLearning (RL) is known for its strong task performance and flexibility in optimizing general reward formulations. On the other hand, #ModelPredictiveControl (MPC) provides robustness, constraint handling, and powerful online replanning capabilities. In this work, we extend our previous AC-MPC paper (Romero, ICRA'24) by taking a deeper look at how both approaches can be unified. We introduce and extend Actor-Critic Model Predictive Control (AC-MPC), a framework that embeds a differentiable MPC inside an Actor-Critic RL architecture. This integration allows the MPC-based actor to perform short-term predictive optimization, while the critic facilitates long-horizon learning and exploration. We conduct a comprehensive study that highlights AC-MPC’s key advantages: - Better out-of-distribution generalization, both against unknown disturbances and changes in the quadrotor dynamics - Improved sample efficiency - A novel empirical analysis uncovering a relationship between the critic’s value function and the MPC cost function, providing deeper insight into their interplay. We validate our method in simulation and the real world on a quadcopter flying at superhuman speeds of up to 21 m/s, matching state-of-the-art model-free RL performance, and retaining the predictive structure of MPC for more reliable out-of-distribution behavior. Reference: Actor-Critic Model Predictive Control: Differentiable Optimization meets Reinforcement Learning for Agile Flight IEEE Transactions on Robotics (T-RO), 2025 PDF: Full Video: Code: Kudos to Ángel Romero, Elie Aljalbout, Yunlong Song! University of Zurich UZH Science UZH Space Hub AUTOASSESS European Research Council (ERC) UZHai

Davide Scaramuzza

27,351 次观看 • 8 个月前

We are excited to share our work “Event-Aided Sharp Radiance Field Reconstruction for Fast-Flying Drones” published in IEEE Transactions on Robotics IEEE Transactions on Robotics (T-RO), which tackles sharp radiance field reconstruction under agile drone motion, where RGB frames are heavily motion-blurred and pose priors become unreliable! 4 years in the making! Code & dataset released! PDF: Code & Dataset: Full Narrated Video: High-speed flight is essential for time- and battery-constrained missions (e.g., inspection, exploration, search & rescue). However, fast motion corrupts visual data with severe motion blur and introduces drift/noise in visual-inertial odometry, making NeRF-based 3D reconstruction particularly brittle. We propose a unified framework that leverages asynchronous #EventCamera streams together with motion-blurred frames to reconstruct high-fidelity radiance fields from agile drone flights. Our key idea is to embed event-image fusion directly into radiance field optimization while jointly refining a shared, continuous-time camera trajectory initialized from event-based VIO. This enables us to recover sharp radiance fields and accurate trajectories without ground-truth supervision during training. We validate our method on synthetic data and on real sequences captured by a drone flying up to 2 m/s. Despite severe blur and noisy pose priors, our method preserves fine scene details and achieves a performance gain of over 50% on real-world data compared to state-of-the-art methods. Kudos to Rong Zou and Marco Cannici! Marco Cannici Reference: Rong Zou*, Marco Cannici*, Davide Scaramuzza Event-Aided Sharp Radiance Field Reconstruction for Fast-Flying Drones IEEE Transactions on Robotics (T-RO), 2026 NCCR Robotics European Research Council (ERC) AUTOASSESS UZH IfI University of Zurich UZH Science Prophesee SynSense UZH Space Hub

Davide Scaramuzza

12,028 次观看 • 6 个月前

Physics-based Motion Retargeting from Sparse Inputs paper page: Avatars are important to create interactive and immersive experiences in virtual worlds. One challenge in animating these characters to mimic a user's motion is that commercial AR/VR products consist only of a headset and controllers, providing very limited sensor data of the user's pose. Another challenge is that an avatar might have a different skeleton structure than a human and the mapping between them is unclear. In this work we address both of these challenges. We introduce a method to retarget motions in real-time from sparse human sensor data to characters of various morphologies. Our method uses reinforcement learning to train a policy to control characters in a physics simulator. We only require human motion capture data for training, without relying on artist-generated animations for each avatar. This allows us to use large motion capture datasets to train general policies that can track unseen users from real and sparse data in real-time. We demonstrate the feasibility of our approach on three characters with different skeleton structure: a dinosaur, a mouse-like creature and a human. We show that the avatar poses often match the user surprisingly well, despite having no sensor information of the lower body available. We discuss and ablate the important components in our framework, specifically the kinematic retargeting step, the imitation, contact and action reward as well as our asymmetric actor-critic observations. We further explore the robustness of our method in a variety of settings including unbalancing, dancing and sports motions.

AK

106,527 次观看 • 3 年前

Today, 14th April 2025, we have just concluded an expanded security council meeting comprising security chiefs and local government chairmen, convened in response to the attack that came to our attention early this morning. At around midnight, into the early hours of today, over a hundred bandits descended on one of the communities in Bassa Local Government Area, leaving behind a trail of destruction numerous lives lost and several houses destroyed. We have received detailed briefings from the heads of the various security agencies regarding the incident. We are working diligently to establish exactly what happened and why, so that we can respond appropriately and prevent such incidents in the future. At the moment, the situation in the area is relatively calm. We have engaged with the affected community, urging the people to remain calm and avoid any retaliatory actions that could further escalate tensions. We are grateful that the youths of the community listened to our appeal. We are currently considering far-reaching decisions aimed at preventing future occurrences. The details of some of these decisions will be shared during my broadcast tomorrow. In the meantime, I want to reassure the people of Plateau State that we are firmly in control of the situation, and we will go to any length necessary to prevent a recurrence. I also want to encourage citizens across the state: if you come across any intelligence no matter how insignificant it may seem please report it to the authorities. We are committed to protecting the sources of information, ensuring that no life is at jeopardy. We recognize that without timely and sufficient intelligence, we cannot respond as effectively as needed. By God’s grace, we are working to strengthen our capacity for intelligence gathering. Meanwhile , we continue to pray that God will comfort the families of those who lost their lives in this tragic incident.

Caleb Mutfwang

47,463 次观看 • 1 年前

I am happy to be finally able to post what I was able to build over the last few weeks. A full real-time high-frequency state estimation and mapping algorithm completely written line by line from scratch in Rust, which can be used by robots to navigate and reason within the 3D world also in complicated scenarios. TBH this took me longer than expected (which was still super fast :D) but you need to get a lot right: From the sensors over the drivers to their respective estimation pipeline and then fusing everything together - a covariance nightmare - and something that can be refined over years to come (currently using Fisher Information from the real measurements). What you see here is not the output of some structure from motion or Gaussian splatting, these are the points of a tight mesh (high res for the video) that a robot can use in real time to plan a path using any open-source planner. The flight you experience through the world is the actual state estimate of the scanner which is published at IMU rate. Yes, currently we have some artefacts of filtered-out humans (GDPR compliant of course :) ) and moving cars and there is still some calibration that could be improved. Offline refinement with SFM and Gaussian splats is possible as well but currently not on the road map. What is on the road map is an exciting step of now being able to collect data from customers at construction sites and in warehouses (currently handheld in the near future with a robot). This data can then be used by our physical agents to reason within this world and automate any customer’s task related to 3D data. If you have anyone who wastes time manually looking 👀 through 3D data, or cannot collect enough 3D data and interpret: Tell me how to reach them!

Benedikt Seidel

16,671 次观看 • 4 个月前