正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

This is a neural network flying a drone at extremely high speed, beating human champions in FPV drone racing. - Reinforcement learning as a tool is so marvelously versatile. It's able to solve both fast, reactive tasks and slow, deliberate tasks (ChatGPT RLHF). - Trained in large-scale simulation, finetuned... show more

Jim Fan

440,718 subscribers

598,499 次观看 • 2 年前 •via X (Twitter)

教育新闻政治科学技术

Anya Rossi• Live Now

Private livecam show

10 条评论

Jim Fan 的头像

Jim Fan2 年前

Original post from the author:

Jeff Holmes 的头像

Jeff Holmes2 年前

Who's building the tiny fly and mosquito catching drone for us?

Amy Robinson Sterling 的头像

Amy Robinson Sterling2 年前

one step closer to house robots

Per-Anders Edwards 的头像

Per-Anders Edwards2 年前

There’s a positive use for deliveries, but there are oh so many more negative uses and you can be sure the DoD and domestic enforcement are going to use every last one of them. So well done that team I guess, you achieved your goal and now you get to reap your reward.

Dave Lavallee 的头像

Dave Lavallee2 年前

The implications of this achievement are just scratching the surface of what will be possible in the future. Kudos to the team.

Harri Hakulinen 🇫🇮🇺🇦🇪🇺 的头像

Harri Hakulinen 🇫🇮🇺🇦🇪🇺2 年前

The (near) future of warplanes / wingmans..

𝙀𝙧𝙣𝙖 𝙆𝙧𝙪𝙢𝙗𝙖𝙘𝙝 的头像

𝙀𝙧𝙣𝙖 𝙆𝙧𝙪𝙢𝙗𝙖𝙘𝙝2 年前

When I saw this simulation it reminded me of Harry Potter movie when Harry playing soccer with the broomstick 🧹. Probably NIMBUS1000 is good name for super high speed drone . Just like Harry’s broomsticks.

Lucas Cooper-Bey 的头像

Lucas Cooper-Bey2 年前

Well it’s been fun. Hope AGI is stoked on humans

Syrsly (bsky:syrsly.com) 的头像

Syrsly (bsky:syrsly.com)2 年前

They're essentially handicapping the race course with standard track shapes instead of random objects. I'm sure it would require a lot more training to handle boxes of different colors, different lighting conditions, rain, triangles... you get the idea.

Φιμπονάτσι 的头像

Φιμπονάτσι2 年前

ww3 will be fought with drones/lasers/ai no question

相关视频

End-to-end neural networks racing drones in Abu Dhabi! 🚁 Check out the drone racing team from Delft University of Technology! A completely end-to-end neural network solution, from pixels to direct motor commands. No Kalman filters. No computer vision feature detectors. Just neurons flying the drone. The challenge is extreme. These drones fly at high speeds and need split-second decisions with minimal onboard resources: a single rolling-shutter camera and an IMU. Their approach is called SkyDreamer, based on the Dreamer-v3 reinforcement learning algorithm. First, a world model is trained in simulation. Then, the neural network learns how to fly in its dreams through reinforcement learning. The network's internal state can be read out to see where it thinks it is on the track or how fast it's going. Even better, the drone estimates some of its own body characteristics during flight, like the camera angle relative to the body, eliminating time-consuming manual calibration. The system uses only a single camera and the gyros from the IMU, ignoring the accelerometers, just like human FPV pilots do. ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

End-to-end neural networks racing drones in Abu Dhabi! 🚁 Check out the drone racing team from Delft University of Technology! A completely end-to-end neural network solution, from pixels to direct motor commands. No Kalman filters. No computer vision feature detectors. Just neurons flying the drone. The challenge is extreme. These drones fly at high speeds and need split-second decisions with minimal onboard resources: a single rolling-shutter camera and an IMU. Their approach is called SkyDreamer, based on the Dreamer-v3 reinforcement learning algorithm. First, a world model is trained in simulation. Then, the neural network learns how to fly in its dreams through reinforcement learning. The network's internal state can be read out to see where it thinks it is on the track or how fast it's going. Even better, the drone estimates some of its own body characteristics during flight, like the camera angle relative to the body, eliminating time-consuming manual calibration. The system uses only a single camera and the gyros from the IMU, ignoring the accelerometers, just like human FPV pilots do. ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

206,924 次观看 • 4 个月前

In order for robots to be deployed in the real world, performing tasks of real value, they must be reliable. Unfortunately, even more, most robotic demos work maybe 70-80% of the time at best. The way to get better reliability is to do real-world reinforcement learning: having the robot teach itself how to perform the task up to a high level of success. The key to doing this is to start with a core of expert human data, use that to train a policy then iteratively improve it, until finally finishing with on-policy reinforcement learning. Kun Lei talks through a unified framework for imitation and reinforcement learning based on PPO, which enables this improvement process. In this episode, Kun Lei explains the theory behind his reinforcement learning method and how it allowed his robot to run in a shopping mall juicing oranges for seven hours at a time, among experiments on a wide variety of tasks and embodiments. Watch episode 58 of RoboPapers now, hosted by Michael Cho - Rbt/Acc and Chris Paxton!

In order for robots to be deployed in the real world, performing tasks of real value, they must be reliable. Unfortunately, even more, most robotic demos work maybe 70-80% of the time at best. The way to get better reliability is to do real-world reinforcement learning: having the robot teach itself how to perform the task up to a high level of success. The key to doing this is to start with a core of expert human data, use that to train a policy then iteratively improve it, until finally finishing with on-policy reinforcement learning. Kun Lei talks through a unified framework for imitation and reinforcement learning based on PPO, which enables this improvement process. In this episode, Kun Lei explains the theory behind his reinforcement learning method and how it allowed his robot to run in a shopping mall juicing oranges for seven hours at a time, among experiments on a wide variety of tasks and embodiments. Watch episode 58 of RoboPapers now, hosted by Michael Cho - Rbt/Acc and Chris Paxton!

RoboPapers

18,813 次观看 • 5 个月前

Wow. This is crazy. A developer trained an AI agent in simulation and deployed it onto a real robotic air hockey table using reinforcement learning. This robot can track the puck with millimeter-level accuracy and react in roughly 20 milliseconds, fast enough to challenge even skilled human players. We’re moving from robots that follow programmed rules to machines that learn strategies in simulation and execute them in the physical world.

Wow. This is crazy. A developer trained an AI agent in simulation and deployed it onto a real robotic air hockey table using reinforcement learning. This robot can track the puck with millimeter-level accuracy and react in roughly 20 milliseconds, fast enough to challenge even skilled human players. We’re moving from robots that follow programmed rules to machines that learn strategies in simulation and execute them in the physical world.

SciTech Era

1,582,819 次观看 • 27 天前

TWIST: a real-time teleoperation system for humanoid robots to mimic whole-body motions. Reference motion data is generated by retargeting human motion-capture data to the robot. Then, the controller is trained in simulation using reinforcement learning and behavior cloning.

TWIST: a real-time teleoperation system for humanoid robots to mimic whole-body motions. Reference motion data is generated by retargeting human motion-capture data to the robot. Then, the controller is trained in simulation using reinforcement learning and behavior cloning.

The Humanoid Hub

46,715 次观看 • 1 年前

The best way to get robust, high-quality robot performance is through reinforcement learning; but RL in either the real world or a traditional simulation has lots of limitations. Instead, Jiazhi Yang in RISE does RL in a compositional world model. Learn more ->

The best way to get robust, high-quality robot performance is through reinforcement learning; but RL in either the real world or a traditional simulation has lots of limitations. Instead, Jiazhi Yang in RISE does RL in a compositional world model. Learn more ->

Chris Paxton

33,766 次观看 • 17 天前

NVIDIA's Jim Fan has verified from reliable sources that the video by EngineAI is a real robot demo and not CGI. The controller is a neural net trained in the NVIDIA Isaac simulator using reinforcement learning. There’s also another video from a different angle.

NVIDIA's Jim Fan has verified from reliable sources that the video by EngineAI is a real robot demo and not CGI. The controller is a neural net trained in the NVIDIA Isaac simulator using reinforcement learning. There’s also another video from a different angle.

The Humanoid Hub

148,535 次观看 • 1 年前

Eric Schmidt says war is no longer constrained by humans. It's AI vs AI. “No human can plan a battle… without reinforcement learning.” The real conflict lives in data centers and drone swarms. No human can follow it in real time.

Eric Schmidt says war is no longer constrained by humans. It's AI vs AI. “No human can plan a battle… without reinforcement learning.” The real conflict lives in data centers and drone swarms. No human can follow it in real time.

vitrupo

112,802 次观看 • 1 年前

Using our brain simulator, we’ve trained a reinforcement learning agent to maximize bits per second. Here is the RL policy converting brain data to cursor control in simulation:

Using our brain simulator, we’ve trained a reinforcement learning agent to maximize bits per second. Here is the RL policy converting brain data to cursor control in simulation:

Neuralink

18,410 次观看 • 1 年前

Motion planning in complex tasks is hard and still done via slow, explicit, traditional planners. We present a generalist Neural Motion Planner -- a single neural network that plans complex dynamic motions quickly and accurately at test time. Building upon our lab's sim2real efforts, the key idea is to create many complex scenes in simulation and then distill classical motion planner trajectories into a single reactive neural network policy. More details in the thread below! 👇 Open-sourced:

Motion planning in complex tasks is hard and still done via slow, explicit, traditional planners. We present a generalist Neural Motion Planner -- a single neural network that plans complex dynamic motions quickly and accurately at test time. Building upon our lab's sim2real efforts, the key idea is to create many complex scenes in simulation and then distill classical motion planner trajectories into a single reactive neural network policy. More details in the thread below! 👇 Open-sourced:

Deepak Pathak

28,642 次观看 • 1 年前

Our project on Reinforcement Learning for versatile exoskeleton controllers is published in Nature. I hope there is more AI assistive technology to augment humans in the physical world. The project is led by NCSU Hao Su's group and the paper is

Our project on Reinforcement Learning for versatile exoskeleton controllers is published in Nature. I hope there is more AI assistive technology to augment humans in the physical world. The project is led by NCSU Hao Su's group and the paper is

Bolei Zhou

16,795 次观看 • 2 年前

Helix is now learning directly from human video data We have already trained on data collected in the real world, including Brookfield residential units To our knowledge, this is the first instance of a humanoid robot learning navigation end-to-end using only human video

Helix is now learning directly from human video data We have already trained on data collected in the real world, including Brookfield residential units To our knowledge, this is the first instance of a humanoid robot learning navigation end-to-end using only human video

Figure

46,041 次观看 • 9 个月前

The future of robotics isn’t being built in factories, it’s being trained in simulation MasterBOT’s proprietary AI model is learning in real-time, powered by reinforcement and continuous feedback loops This is how machines learn to think $BOT

The future of robotics isn’t being built in factories, it’s being trained in simulation MasterBOT’s proprietary AI model is learning in real-time, powered by reinforcement and continuous feedback loops This is how machines learn to think $BOT

MasterBOT

51,156 次观看 • 8 个月前

We are thrilled to share our breakthrough research on "Agile Flight from Pixels without State Estimation," to be presented and live-demonstrated at #RSS2024 next week! You heard well: no state estimation means no explicit visual localization, no SLAM, no VIO, and no IMU! Paper: Video (Narrated): Last year, we demonstrated that #ReinforcementLearning (RL) policies could outperform world-champion drone-racing pilots using the same quadrotor hardware; however, unlike human pilots, these policies continuously estimated an explicit state from known gate positions, the camera feed, and inertial measurements (IMU). In this new work, we tackle the challenge of learning vision-based drone racing using an end-to-end reinforcement learning approach that eliminates the need for IMU data or explicit state estimation. Like professional pilots, we go directly from images to control commands. The training is facilitated by an asymmetric actor-critic with access to privileged information. To overcome the computational complexity during image-based RL training, we use an appropriate sensor representation, which can be efficiently simulated during training without rendering images. We achieve agile flight at speeds up to 40 km/h with accelerations up to 2 g's. Although our demonstration focuses on drone racing, we believe that our method has an impact beyond drone racing and can serve as a foundation for future research into real-world applications in structured environments. Besides the paper presentation, we will also give a live demo next Tuesday and Wednesday between and hrs at TU Delft: Reference: Ismail Geles*, Leonard Bauersfeld*, Angel Romero, Jiaxu Xing, Davide Scaramuzza "Demonstrating Agile Flight from Pixels without State Estimation" Robotics: Science and Systems (RSS), 2024. Kudos to Ismail Geles Leonard Bauersfeld Ángel Romero Jiaxu Xing! University of Zurich UZH Science UZH Space Hub Aerial Core AUTOASSESS European Research Council (ERC)

We are thrilled to share our breakthrough research on "Agile Flight from Pixels without State Estimation," to be presented and live-demonstrated at #RSS2024 next week! You heard well: no state estimation means no explicit visual localization, no SLAM, no VIO, and no IMU! Paper: Video (Narrated): Last year, we demonstrated that #ReinforcementLearning (RL) policies could outperform world-champion drone-racing pilots using the same quadrotor hardware; however, unlike human pilots, these policies continuously estimated an explicit state from known gate positions, the camera feed, and inertial measurements (IMU). In this new work, we tackle the challenge of learning vision-based drone racing using an end-to-end reinforcement learning approach that eliminates the need for IMU data or explicit state estimation. Like professional pilots, we go directly from images to control commands. The training is facilitated by an asymmetric actor-critic with access to privileged information. To overcome the computational complexity during image-based RL training, we use an appropriate sensor representation, which can be efficiently simulated during training without rendering images. We achieve agile flight at speeds up to 40 km/h with accelerations up to 2 g's. Although our demonstration focuses on drone racing, we believe that our method has an impact beyond drone racing and can serve as a foundation for future research into real-world applications in structured environments. Besides the paper presentation, we will also give a live demo next Tuesday and Wednesday between and hrs at TU Delft: Reference: Ismail Geles, Leonard Bauersfeld, Angel Romero, Jiaxu Xing, Davide Scaramuzza "Demonstrating Agile Flight from Pixels without State Estimation" Robotics: Science and Systems (RSS), 2024. Kudos to Ismail Geles Leonard Bauersfeld Ángel Romero Jiaxu Xing! University of Zurich UZH Science UZH Space Hub Aerial Core AUTOASSESS European Research Council (ERC)

Davide Scaramuzza

27,891 次观看 • 1 年前

Have you ever flown a drone while wearing FPV goggles that let you see exactly what the drone sees in real time? My son is a drone enthusiast, and thanks to his goggles, I’ve had the chance to experience First-Person View (FPV) flying. It’s both thrilling and breathtaking!

Have you ever flown a drone while wearing FPV goggles that let you see exactly what the drone sees in real time? My son is a drone enthusiast, and thanks to his goggles, I’ve had the chance to experience First-Person View (FPV) flying. It’s both thrilling and breathtaking!

🇺🇸 Pecan 🇺🇸

54,578 次观看 • 2 个月前

This robot learned to walk without ever actually walking in the real world Like in The Matrix (“I know Kung Fu”) it learned quickly just by SIMULATING walking. It practiced "in its head" Ok so what? Do you realize how suddenly our lil chatbots might go from blinking cursors to very real things? Some people assume that we’ll see robots in the real world slowly getting better over years or decades !BUT! if they can learn 10000x faster via simulation, they could they could blow our expectations out of the water. For example, an autonomous killer drone (like the ones already deployed in Ukraine) could get 1,000 years of aiming experience in 1 hour of human time. It would never miss. (More technically, this is a humanoid transformer that was trained with large-scale reinforcement learning in simulation. It was deployed to the real world zero-shot.)

This robot learned to walk without ever actually walking in the real world Like in The Matrix (“I know Kung Fu”) it learned quickly just by SIMULATING walking. It practiced "in its head" Ok so what? Do you realize how suddenly our lil chatbots might go from blinking cursors to very real things? Some people assume that we’ll see robots in the real world slowly getting better over years or decades !BUT! if they can learn 10000x faster via simulation, they could they could blow our expectations out of the water. For example, an autonomous killer drone (like the ones already deployed in Ukraine) could get 1,000 years of aiming experience in 1 hour of human time. It would never miss. (More technically, this is a humanoid transformer that was trained with large-scale reinforcement learning in simulation. It was deployed to the real world zero-shot.)

AI Notkilleveryoneism Memes ⏸️

128,995 次观看 • 2 年前

Introducing RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learning. 7 real robot tasks, 900/900 successes. Up to 250 consecutive trials in one task, running 2 hours nonstop without failure. High success rate against physical disturbances, zero-shot, and few-shot adaptation Our first step toward a deployable robot learning system.

Introducing RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learning. 7 real robot tasks, 900/900 successes. Up to 250 consecutive trials in one task, running 2 hours nonstop without failure. High success rate against physical disturbances, zero-shot, and few-shot adaptation Our first step toward a deployable robot learning system.

Kun Lei

90,660 次观看 • 8 个月前

GOKU TAKES FLIGHT… ON A DRONE Someone strapped a Kid Goku figure to a racing drone and recreated the Flying Nimbus, in real life. It zips through fields, snow, and sunset skies like it’s straight out of Dragon Ball. You gotta admit: this is some peak nerd engineering. Source: miguellemadj

GOKU TAKES FLIGHT… ON A DRONE Someone strapped a Kid Goku figure to a racing drone and recreated the Flying Nimbus, in real life. It zips through fields, snow, and sunset skies like it’s straight out of Dragon Ball. You gotta admit: this is some peak nerd engineering. Source: miguellemadj

Mario Nawfal

33,771 次观看 • 11 个月前

Can an inexpensive, off-the-shelf IMU be the only sensor to estimate the full state (position, velocity, orientation) of a quadrotor flying through a track at high speed and even be on-pair with vision-based localization? The answer is yes, within certain limitations! In this #RAL2023 paper, we propose a learning-based odometry algorithm that couples a model-based filter driven by the inertial measurements with a learning-based module with access to the control commands. Our system outperforms by a large margin the state-of-the-art visual-inertial odometry (#VIO) algorithms and the state-of-the-art learned-inertial odometry algorithm, #TLIO, for the task of drone racing. Additionally, we show that our system is as accurate as a VIO algorithm that uses a camera to localize to a known map of the racing track. The main limitation of our approach is that it cannot generalize to trajectories that have not been seen at training time. However, in drone racing competitions, the track is known beforehand. Human pilots spend hours or even days of practice on the race track before the competition. Similarly, our system can be trained with the data collected during practice time and deployed during the competition. Future work will investigate how to generalize to trajectories not seen at training time. The code is released! Paper: Video: Code: Kudos to Giovanni Cioffi Leonard Bauersfeld Elia Kaufmann European Research Council (ERC) University of Zurich UZH Science UZH Space Hub NCCR Robotics Aerial Core #RAL2023 #IROS2023 #SLAM

Can an inexpensive, off-the-shelf IMU be the only sensor to estimate the full state (position, velocity, orientation) of a quadrotor flying through a track at high speed and even be on-pair with vision-based localization? The answer is yes, within certain limitations! In this #RAL2023 paper, we propose a learning-based odometry algorithm that couples a model-based filter driven by the inertial measurements with a learning-based module with access to the control commands. Our system outperforms by a large margin the state-of-the-art visual-inertial odometry (#VIO) algorithms and the state-of-the-art learned-inertial odometry algorithm, #TLIO, for the task of drone racing. Additionally, we show that our system is as accurate as a VIO algorithm that uses a camera to localize to a known map of the racing track. The main limitation of our approach is that it cannot generalize to trajectories that have not been seen at training time. However, in drone racing competitions, the track is known beforehand. Human pilots spend hours or even days of practice on the race track before the competition. Similarly, our system can be trained with the data collected during practice time and deployed during the competition. Future work will investigate how to generalize to trajectories not seen at training time. The code is released! Paper: Video: Code: Kudos to Giovanni Cioffi Leonard Bauersfeld Elia Kaufmann European Research Council (ERC) University of Zurich UZH Science UZH Space Hub NCCR Robotics Aerial Core #RAL2023 #IROS2023 #SLAM

Davide Scaramuzza

37,049 次观看 • 2 年前

This is a single uncut video, showing a robot learning several tasks instantly, after just one demonstration each ... This is possible because we've now been able to achieve in-context learning for everyday robotics tasks, and I'm very excited to announce our latest paper: 🎆 Instant Policy: In-Context Imitation Learning via Graph Diffusion 🎆 (1/6) 🧵👇

This is a single uncut video, showing a robot learning several tasks instantly, after just one demonstration each ... This is possible because we've now been able to achieve in-context learning for everyday robotics tasks, and I'm very excited to announce our latest paper: 🎆 Instant Policy: In-Context Imitation Learning via Graph Diffusion 🎆 (1/6) 🧵👇

Edward Johns

74,680 次观看 • 1 年前

Reinforcement learning is used to speed the production of behavior for the Boston Dynamics Atlas humanoid robot. At the heart of the learning process is a physics-based simulator that generates training data for a variety of maneuvers.

Reinforcement learning is used to speed the production of behavior for the Boston Dynamics Atlas humanoid robot. At the heart of the learning process is a physics-based simulator that generates training data for a variety of maneuvers.

RAI Institute

76,576 次观看 • 1 年前