Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Out today in Science Robotics, we introduce a kilometer-scale autonomous navigation system for wheeled-legged quadrupeds. Project website: Paper link: @swiss_mile Joonho Lee Takahiro Miki Marko Bjelonic Lorenz Wellhausen #ScienceRoboticsResearch

leggedrobotics's profile picture

Robotic Systems Lab

34,210 subscribers

25,613 Aufrufe • vor 2 Jahren •via X (Twitter)

Gesundheit & WellnessWissenschaft & Technologie#ScienceRoboticsResearch
LIVE

Anya Rossi• Live Now

Private livecam show

5 Kommentare

Profilbild von Robotic Systems Lab
Robotic Systems Labvor 2 Jahren

Full video is here.

Profilbild von Joonho Lee
Joonho Leevor 2 Jahren

@SciRobotics @swiss_mile @ki_ki_ki1 @MBjelonic @schlurry It was an amazing team work with @swiss_mile and @ki_ki_ki1 . Check the full video:

Profilbild von Joonho Lee
Joonho Leevor 2 Jahren

@SciRobotics @swiss_mile @ki_ki_ki1 @MBjelonic @schlurry Just added the "free-access" link to the Science Robotics version to the project page.

Profilbild von Julian Peterson
Julian Petersonvor 2 Jahren

@SciRobotics @swiss_mile @junja941 @ki_ki_ki1 @MBjelonic @schlurry I can imagine this becoming a waiter serving tables in restaurants. Or delivering packages (if it isn't attacked).

Profilbild von Stissle
Stisslevor 2 Jahren

@SciRobotics @swiss_mile @junja941 @ki_ki_ki1 @MBjelonic @schlurry I see lidar on it so it isn't good enough. Vision (cameras) only

Ähnliche Videos

Scaling 3D scene data is a long-standing challenge in scene understanding, spatial reasoning, and robotics. Since scanning, reconstruction, and labeling are so labor-intensive, data scarcity has remained a major bottleneck. 🛑 To solve this, we propose SceneVerse++: Lifting Unlabeled Internet-level Data for 3D Scene Understanding (CVPR 2026). By reconstructing internet videos and annotating 3D scenes automatically, we’ve created a massive real-world dataset for end-to-end understanding. 🌐📐 SceneVerse++ makes it easy to scale "in-the-wild" 3D scenes toward more capable spatial reasoning systems. This significantly promotes progress in 3D VQA, visual navigation, and broader tasks in Embodied AI and Robotics. 🤖🦾 We are fully open-sourced! Check out the paper, code, and data here: 🌐 Project: 📄 Paper: 📊 Dataset: Code:
2:13

Scaling 3D scene data is a long-standing challenge in scene understanding, spatial reasoning, and robotics. Since scanning, reconstruction, and labeling are so labor-intensive, data scarcity has remained a major bottleneck. 🛑 To solve this, we propose SceneVerse++: Lifting Unlabeled Internet-level Data for 3D Scene Understanding (CVPR 2026). By reconstructing internet videos and annotating 3D scenes automatically, we’ve created a massive real-world dataset for end-to-end understanding. 🌐📐 SceneVerse++ makes it easy to scale "in-the-wild" 3D scenes toward more capable spatial reasoning systems. This significantly promotes progress in 3D VQA, visual navigation, and broader tasks in Embodied AI and Robotics. 🤖🦾 We are fully open-sourced! Check out the paper, code, and data here: 🌐 Project: 📄 Paper: 📊 Dataset: Code:

Siyuan Huang

12,612 Aufrufe • vor 3 Monaten

LIGHTYEAR ROBOTICS has officially launched the M1, their first wheeled-legged robot. The M1 utilizes a self-developed parallel joint drive module, achieving a 40% efficiency increase over traditional structures while keeping the body lightweight. The design balances two core performance features: off-road mobility and high-speed cruising, enabling it to navigate complex environments like grass, gravel, and steep slopes using its all-scenario intelligent perception and autonomous navigation system. The robot is primarily aimed at enterprise applications, providing unmanned and highly reliable solutions for tasks ranging from outdoor inspection and emergency detection to material transport in challenging environments. #AI #HumanoidRobot #EmbodiedAI
1:23

LIGHTYEAR ROBOTICS has officially launched the M1, their first wheeled-legged robot. The M1 utilizes a self-developed parallel joint drive module, achieving a 40% efficiency increase over traditional structures while keeping the body lightweight. The design balances two core performance features: off-road mobility and high-speed cruising, enabling it to navigate complex environments like grass, gravel, and steep slopes using its all-scenario intelligent perception and autonomous navigation system. The robot is primarily aimed at enterprise applications, providing unmanned and highly reliable solutions for tasks ranging from outdoor inspection and emergency detection to material transport in challenging environments. #AI #HumanoidRobot #EmbodiedAI

RoboHub🤖

25,689 Aufrufe • vor 10 Monaten

New episode tomorrow on the path to fully autonomous robots with Sergey Levine One question I had for him: why won't robotics be like self driving cars? Google started their self driving project in 2009. And for over a decade and half, we’ve had the kinds of cool demos we sometimes now see for robotics, without any commercially viable and widely deployed machines. Sergey thinks we're in a different regime with robotics today, for 2 reasons: 1. Mistakes are way less costly (killing someone versus breaking a dish), so you can deploy much earlier and actually learn from those mistakes in deployement. 2. LLMs give robots a level of common sense which significantly boosts their generality. "You can say, 'Hey, there's a sign that says slippery floor. What's going to happen when I walk over that?' No autonomous car in 2009 would have been able to answer that question." Full episode out tomorrow!
3:13

New episode tomorrow on the path to fully autonomous robots with Sergey Levine One question I had for him: why won't robotics be like self driving cars? Google started their self driving project in 2009. And for over a decade and half, we’ve had the kinds of cool demos we sometimes now see for robotics, without any commercially viable and widely deployed machines. Sergey thinks we're in a different regime with robotics today, for 2 reasons: 1. Mistakes are way less costly (killing someone versus breaking a dish), so you can deploy much earlier and actually learn from those mistakes in deployement. 2. LLMs give robots a level of common sense which significantly boosts their generality. "You can say, 'Hey, there's a sign that says slippery floor. What's going to happen when I walk over that?' No autonomous car in 2009 would have been able to answer that question." Full episode out tomorrow!

Dwarkesh Patel

50,960 Aufrufe • vor 11 Monaten

How do you give a humanoid the general motion capability? Not just single motions, but all motion? Introducing SONIC, our new work on supersizing motion tracking for natural humanoid control. We argue that motion tracking is the scalable foundation task for humanoids. So we "supersized" it: 9k+ GPU hours and 100M+ motion frames. But tracking alone is not enough; we show how to make a useful control system out of it: - Universal Kinematic Planner: Enables game-like gamepad control and high-level teleoperation, just like controlling a character in a game. - VR Full-Body Teleop: Direct, real-time whole-body control by a human wearing a VR headset. - VR Keypoint Teleop: Control the upper body (hands/head) while our planner handles robust locomotion automatically. - VLA Integration: We connect this motion tracker to autonomous Visual-Language-Action (VLA) models for autonomous task execution! We use a Universal Token Space to UNIFY this command space, turning our robust tracker into a general-purpose, programmable humanoid brain. This is the generalist "System 1" for humanoids. 🚀 Project: #Humanoids #Robotics #AI #FoundationModels #NVIDIAResearch 🧠🔥
1:32

How do you give a humanoid the general motion capability? Not just single motions, but all motion? Introducing SONIC, our new work on supersizing motion tracking for natural humanoid control. We argue that motion tracking is the scalable foundation task for humanoids. So we "supersized" it: 9k+ GPU hours and 100M+ motion frames. But tracking alone is not enough; we show how to make a useful control system out of it: - Universal Kinematic Planner: Enables game-like gamepad control and high-level teleoperation, just like controlling a character in a game. - VR Full-Body Teleop: Direct, real-time whole-body control by a human wearing a VR headset. - VR Keypoint Teleop: Control the upper body (hands/head) while our planner handles robust locomotion automatically. - VLA Integration: We connect this motion tracker to autonomous Visual-Language-Action (VLA) models for autonomous task execution! We use a Universal Token Space to UNIFY this command space, turning our robust tracker into a general-purpose, programmable humanoid brain. This is the generalist "System 1" for humanoids. 🚀 Project: #Humanoids #Robotics #AI #FoundationModels #NVIDIAResearch 🧠🔥

Zhengyi “Zen” Luo

62,322 Aufrufe • vor 9 Monaten

Check out our #ICRA2024 paper "Contrastive Initial State Buffer for Reinforcement Learning," which tackles the sample inefficiency in #ReinforcementLearning head-on. Code released! We introduce an approach agnostic to the underlying RL algorithm: the Contrastive Initial State Buffer. This tool strategically selects states from past experiences and uses them to initialize the agent in the environment to guide it toward more informative states. Our experiments on drone racing and legged locomotion show that our method achieves higher task performance while also speeding up training convergence. Reference: Nico Messikommer, Yunlong Song, Davide Scaramuzza Contrastive Initial State Buffer for Reinforcement Learning IEEE International Conference on Robotics and Automation (ICRA), 2024. PDF: Code: Video: Kudos to Messikommer Yunlong Song Aerial Core European Research Council (ERC) University of Zurich UZH Space Hub IEEE ICRA UZH Science
1:12

Check out our #ICRA2024 paper "Contrastive Initial State Buffer for Reinforcement Learning," which tackles the sample inefficiency in #ReinforcementLearning head-on. Code released! We introduce an approach agnostic to the underlying RL algorithm: the Contrastive Initial State Buffer. This tool strategically selects states from past experiences and uses them to initialize the agent in the environment to guide it toward more informative states. Our experiments on drone racing and legged locomotion show that our method achieves higher task performance while also speeding up training convergence. Reference: Nico Messikommer, Yunlong Song, Davide Scaramuzza Contrastive Initial State Buffer for Reinforcement Learning IEEE International Conference on Robotics and Automation (ICRA), 2024. PDF: Code: Video: Kudos to Messikommer Yunlong Song Aerial Core European Research Council (ERC) University of Zurich UZH Space Hub IEEE ICRA UZH Science

Davide Scaramuzza

13,846 Aufrufe • vor 2 Jahren

Check out our #RSS2024 paper "#MPCC++: Model Predictive Contouring Control for Time-Optimal Flight with Safety Constraints." Model Predictive Contouring Control (MPCC) has shown promising results for agile robotics applications, including car and drone racing. Existing approaches struggle to introduce safety considerations, often resulting in crashes. What does it take to drive or fly fast and safe? We enhance our former MPCC by incorporating spatial constraints that reliably prevent obstacle collisions, allowing planning the fastest trajectory within these safety limits. To improve performance, we leverage real-world data to refine the dynamic model. Our approach is the first to achieve a 100% success rate in real-world experiments. This safety benefit comes without compromising performance, as our method achieves lap times comparable to the best-performing state-based #ReinforcementLearning (RL) policies. Reference M. Krinner, A. Romero, L. Bauersfeld, M. Zeilinger, A. Carron, D. Scaramuzza, "MPCC++: Model Predictive Contouring Control for Time-Optimal Flight with Safety Constraints" Robotics, Science and Systems, 2024 PDF: Video: Kudos to Maria Krinner, Angel Romero Aguilar, Leonard Bauersfeld, Melanie Zeilinger, Andrea Carron! Ángel Romero Leonard Bauersfeld University of Zurich UZH Science UZH Space Hub AUTOASSESS European Research Council (ERC) #MPC #ModelPredictiveControl
0:47

Check out our #RSS2024 paper "#MPCC++: Model Predictive Contouring Control for Time-Optimal Flight with Safety Constraints." Model Predictive Contouring Control (MPCC) has shown promising results for agile robotics applications, including car and drone racing. Existing approaches struggle to introduce safety considerations, often resulting in crashes. What does it take to drive or fly fast and safe? We enhance our former MPCC by incorporating spatial constraints that reliably prevent obstacle collisions, allowing planning the fastest trajectory within these safety limits. To improve performance, we leverage real-world data to refine the dynamic model. Our approach is the first to achieve a 100% success rate in real-world experiments. This safety benefit comes without compromising performance, as our method achieves lap times comparable to the best-performing state-based #ReinforcementLearning (RL) policies. Reference M. Krinner, A. Romero, L. Bauersfeld, M. Zeilinger, A. Carron, D. Scaramuzza, "MPCC++: Model Predictive Contouring Control for Time-Optimal Flight with Safety Constraints" Robotics, Science and Systems, 2024 PDF: Video: Kudos to Maria Krinner, Angel Romero Aguilar, Leonard Bauersfeld, Melanie Zeilinger, Andrea Carron! Ángel Romero Leonard Bauersfeld University of Zurich UZH Science UZH Space Hub AUTOASSESS European Research Council (ERC) #MPC #ModelPredictiveControl

Davide Scaramuzza

17,903 Aufrufe • vor 2 Jahren

Today we are excited to open up Neuracore to the academic community! Neuracore is a new data foundation built to accelerate robot learning by removing one of the field’s biggest bottlenecks: capturing and working with high-fidelity multimodal robotics data. For the first time, researchers can store, view, and work with robotics data in a cloud-native system built specifically for large-scale learning, and we are making this core platform completely free for academia. The platform lets teams capture every sensor at its native rate, store and visualize data without loss, and then train and deploy models locally using our open-source code (Link in the comments). We are rolling out access to select academic institutions first. Anyone with an academic email can sign up, and if your institution is not part of the initial rollout, you will be able to join the waitlist directly. Beyond providing this infrastructure, we see an opportunity to build a global community where engineers and researchers can share, collaborate, and advance the frontier of robot learning together. Supported by our recent $3M pre-seed round led by Earlybird VC, we are excited to take this mission even further. Our long-term goal is for Neuracore to become the natural home for cutting-edge robot learning algorithms and real-world robotics experimentation, helping accelerate the next wave of Physical AI.
1:00

Today we are excited to open up Neuracore to the academic community! Neuracore is a new data foundation built to accelerate robot learning by removing one of the field’s biggest bottlenecks: capturing and working with high-fidelity multimodal robotics data. For the first time, researchers can store, view, and work with robotics data in a cloud-native system built specifically for large-scale learning, and we are making this core platform completely free for academia. The platform lets teams capture every sensor at its native rate, store and visualize data without loss, and then train and deploy models locally using our open-source code (Link in the comments). We are rolling out access to select academic institutions first. Anyone with an academic email can sign up, and if your institution is not part of the initial rollout, you will be able to join the waitlist directly. Beyond providing this infrastructure, we see an opportunity to build a global community where engineers and researchers can share, collaborate, and advance the frontier of robot learning together. Supported by our recent $3M pre-seed round led by Earlybird VC, we are excited to take this mission even further. Our long-term goal is for Neuracore to become the natural home for cutting-edge robot learning algorithms and real-world robotics experimentation, helping accelerate the next wave of Physical AI.

Neuracore

40,620 Aufrufe • vor 8 Monaten

What do you do for a living today? AI and intelligent robots will increasingly take over many tasks from humans. And they will do it better, faster and cheaper. Yes, yours too. Many people will work much less or not at all in normal jobs. And earn less or even nothing. What will you live on then? And how will our companies survive when their customers don't have the money to buy their products and services? This is the key question for your financial future! We will take a look at three scenarios for the future of our life and the economy. If AI and robots work for us on a large scale, our future could be great. However, it could also turn out quite differently. We could also wake up in a horror scenario. Because we have a huge and catastrophic flaw in our economic system that could lead to disaster in a world of AI and robotics. Correcting this mistake is the biggest challenge. What the flaw is and how we can fix it, you'll learn in this video.
25:39

What do you do for a living today? AI and intelligent robots will increasingly take over many tasks from humans. And they will do it better, faster and cheaper. Yes, yours too. Many people will work much less or not at all in normal jobs. And earn less or even nothing. What will you live on then? And how will our companies survive when their customers don't have the money to buy their products and services? This is the key question for your financial future! We will take a look at three scenarios for the future of our life and the economy. If AI and robots work for us on a large scale, our future could be great. However, it could also turn out quite differently. We could also wake up in a horror scenario. Because we have a huge and catastrophic flaw in our economic system that could lead to disaster in a world of AI and robotics. Correcting this mistake is the biggest challenge. What the flaw is and how we can fix it, you'll learn in this video.

Dr Pero Micic

12,010,196 Aufrufe • vor 1 Jahr

How can generative AI and Robotics help advance drug discovery？ 🚀 Excited to introduce LUMI-lab! A foundation model-driven Self-Driving Lab (SDL) for autonomous ionizable lipid discovery in mRNA delivery 🤖🔍 🔬 What is LUMI-lab? LUMI-lab integrates molecular foundation models with autonomous robotic experiments to efficiently explore new LNPs (lipid nanoparticles, mRNA delivery vehicles) with minimal wet-lab data. 🔥 Key Highlights: - 🧠 Foundation model trained on 28M molecules using a three-step strategy: - Unsupervised pretraining to capture broad molecular knowledge - Continual pretraining to specialize in lipid-like molecules - Active learning fine-tuning within a closed-loop experimental system - 🤖 1,700+ new LNPs synthesized & tested across 10 iterative cycles - 🧪 Brominated lipids autonomously identified as a novel structural feature that enhances mRNA transfection—an insight previously unrecognized in LNP design - 🏆 20.3% in vivo CRISPR gene editing efficiency in lung epithelial cells—the highest reported for inhaled LNPs 🚀 Why it matters? LNPs are the backbone of mRNA therapeutics, yet discovery has been slow due to data scarcity. LUMI-lab shows that AI-powered autonomous labs can accelerate mRNA delivery innovation🚀💡 🌐 Beyond mRNA drugs, LUMI-lab exemplifies a scalable framework for AI-driven molecular discovery, pushing boundaries in material science & drug delivery. 📜 Read the preprint: 🔗 💻 Code available on GitHub: 🔗 #AI #DrugDiscovery #mRNA #LNP #SyntheticBiology 🙏 A huge team effort behind this work, with special appreciation to Bowen LI for driving the project. Kudos to Haotian Cui, @YueXu1995, Kuan Pang, Gen Li, and Bettycat567!
1:05

How can generative AI and Robotics help advance drug discovery？ 🚀 Excited to introduce LUMI-lab! A foundation model-driven Self-Driving Lab (SDL) for autonomous ionizable lipid discovery in mRNA delivery 🤖🔍 🔬 What is LUMI-lab? LUMI-lab integrates molecular foundation models with autonomous robotic experiments to efficiently explore new LNPs (lipid nanoparticles, mRNA delivery vehicles) with minimal wet-lab data. 🔥 Key Highlights: - 🧠 Foundation model trained on 28M molecules using a three-step strategy: - Unsupervised pretraining to capture broad molecular knowledge - Continual pretraining to specialize in lipid-like molecules - Active learning fine-tuning within a closed-loop experimental system - 🤖 1,700+ new LNPs synthesized & tested across 10 iterative cycles - 🧪 Brominated lipids autonomously identified as a novel structural feature that enhances mRNA transfection—an insight previously unrecognized in LNP design - 🏆 20.3% in vivo CRISPR gene editing efficiency in lung epithelial cells—the highest reported for inhaled LNPs 🚀 Why it matters? LNPs are the backbone of mRNA therapeutics, yet discovery has been slow due to data scarcity. LUMI-lab shows that AI-powered autonomous labs can accelerate mRNA delivery innovation🚀💡 🌐 Beyond mRNA drugs, LUMI-lab exemplifies a scalable framework for AI-driven molecular discovery, pushing boundaries in material science & drug delivery. 📜 Read the preprint: 🔗 💻 Code available on GitHub: 🔗 #AI #DrugDiscovery #mRNA #LNP #SyntheticBiology 🙏 A huge team effort behind this work, with special appreciation to Bowen LI for driving the project. Kudos to Haotian Cui, @YueXu1995, Kuan Pang, Gen Li, and Bettycat567!

Bo Wang

27,626 Aufrufe • vor 1 Jahr

ELON: MANUFACTURING AT SCALE IS CRITICAL TO AMERICA Elon’s full introductory remarks during Pete Hegseth's visit at Starbase: “Welcome to Starbase, Texas. This is actually legally a city that, thanks to the hard work of the SpaceX team, we built out of nothing and is now a gigantic rocket manufacturing system. For people out there who are curious to see it, we're actually on a public highway, so you can come and visit and drive down the road and see the epic hardware. I think this is the first time that a rocket development program has actually been on a public highway, so you can come and see the epic stuff. We're honored today to have the Secretary of War, Pete Hegseth, and much of the senior leadership of the Pentagon here. It's an honor to have them visit. We just did a tour of the factory, and I think this really helps illustrate how manufacturing and manufacturing at scale are critical to the strength of America. I'll tell you a little bit just about the purpose of SpaceX. It's like we want to make Star Trek real. We want to make Starfleet Academy real so that it's not always science fiction, but one day the science fiction turns to science fact. And we have spaceships going through space, big spaceships with people going to other planets, going to the moon, and ultimately going beyond our star system to other star systems where we may meet aliens or discover long-dead alien civilizations. I don't know, but we want to go, and we want to see what's happening, and we want to have epic futuristic spaceships with lots of people in them traveling to places we've never been to before. Yeah, that's the goal. And that's what I think the public thinks of when they think of Space Force. So on that note, I'd like to introduce the Secretary of War, Pete Hegseth.” Source: Elon Musk SpaceX, Department of War, January 12, 2025
1:59

ELON: MANUFACTURING AT SCALE IS CRITICAL TO AMERICA Elon’s full introductory remarks during Pete Hegseth's visit at Starbase: “Welcome to Starbase, Texas. This is actually legally a city that, thanks to the hard work of the SpaceX team, we built out of nothing and is now a gigantic rocket manufacturing system. For people out there who are curious to see it, we're actually on a public highway, so you can come and visit and drive down the road and see the epic hardware. I think this is the first time that a rocket development program has actually been on a public highway, so you can come and see the epic stuff. We're honored today to have the Secretary of War, Pete Hegseth, and much of the senior leadership of the Pentagon here. It's an honor to have them visit. We just did a tour of the factory, and I think this really helps illustrate how manufacturing and manufacturing at scale are critical to the strength of America. I'll tell you a little bit just about the purpose of SpaceX. It's like we want to make Star Trek real. We want to make Starfleet Academy real so that it's not always science fiction, but one day the science fiction turns to science fact. And we have spaceships going through space, big spaceships with people going to other planets, going to the moon, and ultimately going beyond our star system to other star systems where we may meet aliens or discover long-dead alien civilizations. I don't know, but we want to go, and we want to see what's happening, and we want to have epic futuristic spaceships with lots of people in them traveling to places we've never been to before. Yeah, that's the goal. And that's what I think the public thinks of when they think of Space Force. So on that note, I'd like to introduce the Secretary of War, Pete Hegseth.” Source: Elon Musk SpaceX, Department of War, January 12, 2025

Mario Nawfal

18,097 Aufrufe • vor 7 Monaten

Check out our latest work, "Actor-Critic Model Predictive Control: Differentiable Optimization meets Reinforcement Learning for Agile Flight," published in the IEEE Transactions on Robotics, where we reconcile #OptimalControl and #ReinforcementLearning, achieving the same super-human performance, but with superior generalizability, as our previous model-free deep RL! Code released! PDF: Code: Full Video: Model-free #ReinforcementLearning (RL) is known for its strong task performance and flexibility in optimizing general reward formulations. On the other hand, #ModelPredictiveControl (MPC) provides robustness, constraint handling, and powerful online replanning capabilities. In this work, we extend our previous AC-MPC paper (Romero, ICRA'24) by taking a deeper look at how both approaches can be unified. We introduce and extend Actor-Critic Model Predictive Control (AC-MPC), a framework that embeds a differentiable MPC inside an Actor-Critic RL architecture. This integration allows the MPC-based actor to perform short-term predictive optimization, while the critic facilitates long-horizon learning and exploration. We conduct a comprehensive study that highlights AC-MPC’s key advantages: - Better out-of-distribution generalization, both against unknown disturbances and changes in the quadrotor dynamics - Improved sample efficiency - A novel empirical analysis uncovering a relationship between the critic’s value function and the MPC cost function, providing deeper insight into their interplay. We validate our method in simulation and the real world on a quadcopter flying at superhuman speeds of up to 21 m/s, matching state-of-the-art model-free RL performance, and retaining the predictive structure of MPC for more reliable out-of-distribution behavior. Reference: Actor-Critic Model Predictive Control: Differentiable Optimization meets Reinforcement Learning for Agile Flight IEEE Transactions on Robotics (T-RO), 2025 PDF: Full Video: Code: Kudos to Ángel Romero, Elie Aljalbout, Yunlong Song! University of Zurich UZH Science UZH Space Hub AUTOASSESS European Research Council (ERC) UZHai
1:46

Check out our latest work, "Actor-Critic Model Predictive Control: Differentiable Optimization meets Reinforcement Learning for Agile Flight," published in the IEEE Transactions on Robotics, where we reconcile #OptimalControl and #ReinforcementLearning, achieving the same super-human performance, but with superior generalizability, as our previous model-free deep RL! Code released! PDF: Code: Full Video: Model-free #ReinforcementLearning (RL) is known for its strong task performance and flexibility in optimizing general reward formulations. On the other hand, #ModelPredictiveControl (MPC) provides robustness, constraint handling, and powerful online replanning capabilities. In this work, we extend our previous AC-MPC paper (Romero, ICRA'24) by taking a deeper look at how both approaches can be unified. We introduce and extend Actor-Critic Model Predictive Control (AC-MPC), a framework that embeds a differentiable MPC inside an Actor-Critic RL architecture. This integration allows the MPC-based actor to perform short-term predictive optimization, while the critic facilitates long-horizon learning and exploration. We conduct a comprehensive study that highlights AC-MPC’s key advantages: - Better out-of-distribution generalization, both against unknown disturbances and changes in the quadrotor dynamics - Improved sample efficiency - A novel empirical analysis uncovering a relationship between the critic’s value function and the MPC cost function, providing deeper insight into their interplay. We validate our method in simulation and the real world on a quadcopter flying at superhuman speeds of up to 21 m/s, matching state-of-the-art model-free RL performance, and retaining the predictive structure of MPC for more reliable out-of-distribution behavior. Reference: Actor-Critic Model Predictive Control: Differentiable Optimization meets Reinforcement Learning for Agile Flight IEEE Transactions on Robotics (T-RO), 2025 PDF: Full Video: Code: Kudos to Ángel Romero, Elie Aljalbout, Yunlong Song! University of Zurich UZH Science UZH Space Hub AUTOASSESS European Research Council (ERC) UZHai

Davide Scaramuzza

27,090 Aufrufe • vor 6 Monaten

Here are Elon Musk's full introductory remarks during Pete Hegseth's visit at Starbase earlier today. “Welcome to Starbase, Texas. This actually legally a city that thanks to the hard work of the SpaceX team, we built out of nothing and is now a gigantic rocket manufacturing system. And for people out there who are curious to see it, we're actually on a public highway so you can come and visit and drive down the road and see the epic hardware. I think this is the first time that a rocket development program has actually been on a public highway, so you can come and see the epic stuff. We're honored today to have the Secretary of War, Pete Hegseth, and much of the senior leadership of the Pentagon here. It's an honor to have them visit. We just did a tour of the factory and, and I think this, it really helps illustrate how manufacturing and manufacturing at scale is critical to the strength of America. I'll tell you a little bit just about the purpose of SpaceX. It's like we want to make Star Trek real. We want to make Starfleet Academy real so that it's not always science fiction, but one day the science fiction turns to science fact. And we have spaceships going through space, big spaceships with people going to other planets, going to the moon, and ultimately going beyond our star system to other star systems where we may meet aliens or discover long dead alien civilizations. I don't know, but we want to go and we want to see what's happening and we want to have epic futuristic spaceships with lots of people in them traveling to places we've never been to before. Yeah, that's the goal. And that's what I think the public thinks of when they think of Space Force. So on that note, I'd like to introduce the Secretary of War, Pete Hegseth.” From: Starbase Texas: SpaceX's Rocket City, January 13, 2026
1:59

Here are Elon Musk's full introductory remarks during Pete Hegseth's visit at Starbase earlier today. “Welcome to Starbase, Texas. This actually legally a city that thanks to the hard work of the SpaceX team, we built out of nothing and is now a gigantic rocket manufacturing system. And for people out there who are curious to see it, we're actually on a public highway so you can come and visit and drive down the road and see the epic hardware. I think this is the first time that a rocket development program has actually been on a public highway, so you can come and see the epic stuff. We're honored today to have the Secretary of War, Pete Hegseth, and much of the senior leadership of the Pentagon here. It's an honor to have them visit. We just did a tour of the factory and, and I think this, it really helps illustrate how manufacturing and manufacturing at scale is critical to the strength of America. I'll tell you a little bit just about the purpose of SpaceX. It's like we want to make Star Trek real. We want to make Starfleet Academy real so that it's not always science fiction, but one day the science fiction turns to science fact. And we have spaceships going through space, big spaceships with people going to other planets, going to the moon, and ultimately going beyond our star system to other star systems where we may meet aliens or discover long dead alien civilizations. I don't know, but we want to go and we want to see what's happening and we want to have epic futuristic spaceships with lots of people in them traveling to places we've never been to before. Yeah, that's the goal. And that's what I think the public thinks of when they think of Space Force. So on that note, I'd like to introduce the Secretary of War, Pete Hegseth.” From: Starbase Texas: SpaceX's Rocket City, January 13, 2026

ELON CLIPS

12,477 Aufrufe • vor 7 Monaten

Hello, Ghana 🇬🇭 Your Excellency, John Dramani Mahama Hello Africa and the world. We 3Farmate are proud to announce the official launch of FAMA—Ghana's FIRST AI-powered autonomous farming robot for large-scale crop production. Founded in 2021 by Clinton 🛸 (CEO) and Koffi-Cobbin (CTO), the company began in a dorm room at Kwame Nkrumah University of Science and Technology, where its first prototype was developed. Since then, the team has engineered FAMA into a full-scale autonomous robot capable of planting seeds, applying fertilizer, weeding, and spraying across real farm environments. FAMA navigates using a vision-based AI system instead of GPS, allowing it to operate reliably in areas where GPS is unavailable or inconsistent. The robot runs on batteries charged by solar panels while in the field and can operate across uneven terrain, loose and muddy soils, and variable weather conditions. A single operator can oversee multiple robots, each covering 27 to 35 acres per day with sub-85mm planting precision. 3Farmate targets large-scale staple crop producers in Ghana, starting with corn and soybeans. We operate a service model, charging farmers per acre and removing the need for upfront equipment investment. Over 70 farmers and several large-scale crop production companies are currently in discussions, with commercial deployments beginning in the 2026 planting season. Approximately $200,000 has been raised to date, including investment from 776 Foundation (Alexis Ohanian, Reddit co-founder) and a grant from Kosmos Innovation Center Ghana. Our team consists of young engineers specializing in robotics, embedded systems, software, and mechanical design. With 8 major iterations, 60+ field test runs, 100+ cumulative acres covered, and thousands of runtime hours in real farm conditions, FAMA is market-ready to become the ultimate farmer-assistant. We built FAMA right here in Ghana, inspired by Dr. Kwame Nkrumah’s belief that “Africa must industrialize to achieve true independence.” We built FAMA as a true testament to every young engineer in Africa that “IT IS POSSIBLE.” FAMA is designed to operate seamlessly on Ghanaian soil and is adaptable to diverse agricultural environments worldwide. Join us as we drive innovation across global agriculture. We call on the government of Ghana, stakeholders, international organizations, and agri-industry leaders to partner with us in transforming agriculture together. This is not history in the making, because history has already been made, and we thank you for being a part of our journey. On this note, we are officially launched! For more information, visit 3FARMATE – The future is here.
1:50

Hello, Ghana 🇬🇭 Your Excellency, John Dramani Mahama Hello Africa and the world. We 3Farmate are proud to announce the official launch of FAMA—Ghana's FIRST AI-powered autonomous farming robot for large-scale crop production. Founded in 2021 by Clinton 🛸 (CEO) and Koffi-Cobbin (CTO), the company began in a dorm room at Kwame Nkrumah University of Science and Technology, where its first prototype was developed. Since then, the team has engineered FAMA into a full-scale autonomous robot capable of planting seeds, applying fertilizer, weeding, and spraying across real farm environments. FAMA navigates using a vision-based AI system instead of GPS, allowing it to operate reliably in areas where GPS is unavailable or inconsistent. The robot runs on batteries charged by solar panels while in the field and can operate across uneven terrain, loose and muddy soils, and variable weather conditions. A single operator can oversee multiple robots, each covering 27 to 35 acres per day with sub-85mm planting precision. 3Farmate targets large-scale staple crop producers in Ghana, starting with corn and soybeans. We operate a service model, charging farmers per acre and removing the need for upfront equipment investment. Over 70 farmers and several large-scale crop production companies are currently in discussions, with commercial deployments beginning in the 2026 planting season. Approximately $200,000 has been raised to date, including investment from 776 Foundation (Alexis Ohanian, Reddit co-founder) and a grant from Kosmos Innovation Center Ghana. Our team consists of young engineers specializing in robotics, embedded systems, software, and mechanical design. With 8 major iterations, 60+ field test runs, 100+ cumulative acres covered, and thousands of runtime hours in real farm conditions, FAMA is market-ready to become the ultimate farmer-assistant. We built FAMA right here in Ghana, inspired by Dr. Kwame Nkrumah’s belief that “Africa must industrialize to achieve true independence.” We built FAMA as a true testament to every young engineer in Africa that “IT IS POSSIBLE.” FAMA is designed to operate seamlessly on Ghanaian soil and is adaptable to diverse agricultural environments worldwide. Join us as we drive innovation across global agriculture. We call on the government of Ghana, stakeholders, international organizations, and agri-industry leaders to partner with us in transforming agriculture together. This is not history in the making, because history has already been made, and we thank you for being a part of our journey. On this note, we are officially launched! For more information, visit 3FARMATE – The future is here.

MARK OFORIQUAYE

148,540 Aufrufe • vor 4 Monaten

🎉 The best way to start the week is to find out that our MedSAM is finally published today in Nature Communications! **Segment anything in medical images** Paper: arXiv: Data & Code: MedSAM is the first promotable foundation model for medical image segmentation. **Highlights**: ⭐ Before its formal publication, we have received 220 citations and 1400+ GitHub stars 🙏🙏❤️‍🔥❤️‍🔥❤️‍🔥 📊 We curated a large-scale medical image dataset with 1,570,263 image-mask pairs, covering 10 imaging modalities and over 30 cancer types. 🚀 Built on top of SAM (AI at Meta ) with transfer learning, we have significantly enhanced its segmentation performance of medical images. 📈 Comprehensive evaluations of 86 internal validation tasks and 60 external validation tasks demonstrate its better accuracy and robustness than modality-wise specialist models. **What is Next? --- Clinical Translation!!** 🍕Our next goal is to make the model deployable on laptops (CPUs) or other edge devices without reliance on GPUs. We have distilled a lightweight model, LiteMedSAM, offering a speed boost of 10x while maintaining accuracy. Plus, we have integrated it into the 3D Slicer plugin, providing an efficient tool for medical image segmentation. 🌐 To further promote developments in this field, we organize a competition on #CVPR2026: Segment Anything in Medical Images on Laptop! An out-of-the-box baseline has been released to reduce the entry barriers. Welcome to join us to push the boundary further: 🙏 Massive thanks to MetaAI AI at Meta for their open-source project SAM and many reviewers/users for their invaluable feedback. A huge shoutout to my postdoc Jun Ma (JunMa) for his leadership on this project!! UHN AI Hub Vector Institute Peter Munk Cardiac Centre AI Department of Laboratory Medicine & Pathobiology U of T Department of Computer Science University of Toronto University Health Network Brad Wouters 🇨🇦 Barry Rubin MD, PhD, FRCSC Shaf Keshavjee
0:59

🎉 The best way to start the week is to find out that our MedSAM is finally published today in Nature Communications! **Segment anything in medical images** Paper: arXiv: Data & Code: MedSAM is the first promotable foundation model for medical image segmentation. **Highlights**: ⭐ Before its formal publication, we have received 220 citations and 1400+ GitHub stars 🙏🙏❤️‍🔥❤️‍🔥❤️‍🔥 📊 We curated a large-scale medical image dataset with 1,570,263 image-mask pairs, covering 10 imaging modalities and over 30 cancer types. 🚀 Built on top of SAM (AI at Meta ) with transfer learning, we have significantly enhanced its segmentation performance of medical images. 📈 Comprehensive evaluations of 86 internal validation tasks and 60 external validation tasks demonstrate its better accuracy and robustness than modality-wise specialist models. **What is Next? --- Clinical Translation!!** 🍕Our next goal is to make the model deployable on laptops (CPUs) or other edge devices without reliance on GPUs. We have distilled a lightweight model, LiteMedSAM, offering a speed boost of 10x while maintaining accuracy. Plus, we have integrated it into the 3D Slicer plugin, providing an efficient tool for medical image segmentation. 🌐 To further promote developments in this field, we organize a competition on #CVPR2026: Segment Anything in Medical Images on Laptop! An out-of-the-box baseline has been released to reduce the entry barriers. Welcome to join us to push the boundary further: 🙏 Massive thanks to MetaAI AI at Meta for their open-source project SAM and many reviewers/users for their invaluable feedback. A huge shoutout to my postdoc Jun Ma (JunMa) for his leadership on this project!! UHN AI Hub Vector Institute Peter Munk Cardiac Centre AI Department of Laboratory Medicine & Pathobiology U of T Department of Computer Science University of Toronto University Health Network Brad Wouters 🇨🇦 Barry Rubin MD, PhD, FRCSC Shaf Keshavjee

Bo Wang

140,229 Aufrufe • vor 2 Jahren

We are excited to share our latest work, "Superhuman Safe and Agile Racing through Multi-Agent Reinforcement Learning," done in collaboration with Google DeepMind . Autonomous drones have reached superhuman speed in isolation, but what happens when multiple agents share the same airspace? Paper: Website: Video: Using league-based self-play, we train #ReinforcementLearning agents that race against a diverse, evolving population of opponents. Through this competitive training, sophisticated behaviors emerge without explicit programming: strategic overtaking, proactive collision avoidance, and even awareness of aerodynamic downwash from nearby drones. In real-world multi-player races at speeds exceeding 80kph (50 mph) and accelerations up to 7g, our agents outperform a five-time Swiss national drone racing champion while reducing collision rates by 50% compared to single-agent baselines. Crucially, training against diverse artificial opponents enables zero-shot generalization to human pilots, achieving over 90% race completion in mixed human-AI races with up to four competitors. A key insight: human pilots adopt riskier strategies when trailing, leading to more crashes under competitive pressure. Our learned policies, by contrast, maintain consistent safety margins regardless of race standing, a property essential for deploying autonomous systems alongside humans. Also, the multi-agent self-play policies are more robust than those trained independently, suggesting that training in competitive environments is not only key to winning races but also to learning safer, more reliable autonomy for real-world multi-robot systems. Kudos to Ismail Geles, Leonard Bauersfeld, Markus Wulfmeier! Ismail Geles Leonard Bauersfeld Markus Wulfmeier European Research Council (ERC) UZH IfI University of Zurich UZH Science UZH Space Hub Swiss Robotics NCCR Robotics
1:47

We are excited to share our latest work, "Superhuman Safe and Agile Racing through Multi-Agent Reinforcement Learning," done in collaboration with Google DeepMind . Autonomous drones have reached superhuman speed in isolation, but what happens when multiple agents share the same airspace? Paper: Website: Video: Using league-based self-play, we train #ReinforcementLearning agents that race against a diverse, evolving population of opponents. Through this competitive training, sophisticated behaviors emerge without explicit programming: strategic overtaking, proactive collision avoidance, and even awareness of aerodynamic downwash from nearby drones. In real-world multi-player races at speeds exceeding 80kph (50 mph) and accelerations up to 7g, our agents outperform a five-time Swiss national drone racing champion while reducing collision rates by 50% compared to single-agent baselines. Crucially, training against diverse artificial opponents enables zero-shot generalization to human pilots, achieving over 90% race completion in mixed human-AI races with up to four competitors. A key insight: human pilots adopt riskier strategies when trailing, leading to more crashes under competitive pressure. Our learned policies, by contrast, maintain consistent safety margins regardless of race standing, a property essential for deploying autonomous systems alongside humans. Also, the multi-agent self-play policies are more robust than those trained independently, suggesting that training in competitive environments is not only key to winning races but also to learning safer, more reliable autonomy for real-world multi-robot systems. Kudos to Ismail Geles, Leonard Bauersfeld, Markus Wulfmeier! Ismail Geles Leonard Bauersfeld Markus Wulfmeier European Research Council (ERC) UZH IfI University of Zurich UZH Science UZH Space Hub Swiss Robotics NCCR Robotics

Davide Scaramuzza

14,637 Aufrufe • vor 2 Monaten

In my past research experience, finding or developing an appropriate simulation environment, dataset, and benchmark has always been a challenge. Missing features, limited support, or unexpected bugs often occupied my days and nights. Moreover, current simulation platforms are relatively fragmented—making it challenging to replicate the success of the RT-X dataset in unifying community efforts. Introducing RoboVerse, we provide a unified platform, dataset, and benchmark for scalable and generalizable robot learning. We hope to build a shared foundation to combine the community efforts. RoboVerse includes: MetaSim: We carefully designed a configuration system and a universal interface to align current robotic simulators. With MetaSim, you can use any simulator with the same code—bringing together the community’s diverse efforts under one framework! RoboVerse Dataset and Benchmark: We unify popular simulation environments and benchmarks into a single cohesive system and introduce the RoboVerse dataset—a large-scale, high-quality synthetic dataset. Additionally, we propose a standardized benchmark across both imitation learning and reinforcement learning. A cool feature enabled by our unified framework: Hybrid Simulation! You can now integrate physics engines and renderers from different simulators—e.g., using MuJoCo precise physics with Isaac photorealistic rendering. This not only elevates simulation fidelity but also significantly enhances real-world transfer performance across complex robotic applications. Hopefully, our team’s efforts could serve the robotic community to thrive vibrantly in the years to come. RoboVerse is open-sourced🥳!!! Project Page: Documentation: Github Repo: Paper:
2:21

In my past research experience, finding or developing an appropriate simulation environment, dataset, and benchmark has always been a challenge. Missing features, limited support, or unexpected bugs often occupied my days and nights. Moreover, current simulation platforms are relatively fragmented—making it challenging to replicate the success of the RT-X dataset in unifying community efforts. Introducing RoboVerse, we provide a unified platform, dataset, and benchmark for scalable and generalizable robot learning. We hope to build a shared foundation to combine the community efforts. RoboVerse includes: MetaSim: We carefully designed a configuration system and a universal interface to align current robotic simulators. With MetaSim, you can use any simulator with the same code—bringing together the community’s diverse efforts under one framework! RoboVerse Dataset and Benchmark: We unify popular simulation environments and benchmarks into a single cohesive system and introduce the RoboVerse dataset—a large-scale, high-quality synthetic dataset. Additionally, we propose a standardized benchmark across both imitation learning and reinforcement learning. A cool feature enabled by our unified framework: Hybrid Simulation! You can now integrate physics engines and renderers from different simulators—e.g., using MuJoCo precise physics with Isaac photorealistic rendering. This not only elevates simulation fidelity but also significantly enhances real-world transfer performance across complex robotic applications. Hopefully, our team’s efforts could serve the robotic community to thrive vibrantly in the years to come. RoboVerse is open-sourced🥳!!! Project Page: Documentation: Github Repo: Paper:

Haoran Geng

84,318 Aufrufe • vor 1 Jahr

#NewPaper The first microscope, invented in the 16th century, was designed to unlock the secrets of the microscopic world. Today, as many fields become increasingly data-driven, there is a pressing need for new types of microscopes---tools that help us zoom in, explore, and understand complex data. We call these tools "algorithmic microscopes." Introducing the Vendiscope: The first algorithmic microscope for data collections. 🔬 The Vendiscope maximizes the probability-weighted Vendi Score of a dataset to assign a weight to each element in the collection. This weight represents a data point's contribution to the overall diversity of the collection. These weights enable high-resolution data analysis at scale. We use them to zoom in on datasets across three domains: biology, materials science, & AI. 🧬 Biology: We used the Vendiscope on the protein universe, which contains nearly 250 million proteins. We found that nearly 200 million of the proteins are near-duplicates of each other and that AlphaFold fails on proteins that contribute most to the diversity of the protein universe. (See GIF below). 🪜 Materials Science: We used the Vendiscope on the Materials Project database, which contains 170K materials as of today. We found that 85% of crystals with formation energy data are near-duplicates of each other and that ML models for materials property prediction struggle with materials that contribute most to diversity. 🤖 Artificial Intelligence: We applied the Vendiscope to CIFAR-10, a benchmark dataset containing 50K images. We found duplicates. We applied the Vendiscope to analyze state-of-the-art generative models trained on this dataset. We found the best generative models memorize training data, as is known in the AI literature. However, we can do more with the Vendiscope and characterize the type of samples that get memorized. We found that data points contributing least to diversity are more prone to memorization by these generative models. 🧠 "Our findings demonstrate that the Vendiscope can serve as a powerful tool for data-driven science, providing a systematic and scalable way to identify duplicates and outliers, as well as pinpointing samples prone to memorization and those that models may struggle to predict---even before training." 💫 "The Vendiscope provides a unified framework for analyzing complex data at scale. Researchers, engineers, and data auditors can use the Vendiscope to audit datasets, identify potential biases, and refine data collection practices. For AI ethicists, the Vendiscope offers a critical lens to understand how models interact with data, particularly in the context of bias, memorization, and data fairness, enabling better mitigation strategies to prevent undesirable outcomes in AI deployment. For scientists, the Vendiscope represents a new companion in the discovery process." #VendiScoring #AlgorithmicMicroscopy Link to paper: Authors: Amey Pasarkar (Amey Pasarkar) and Adji Bousso Dieng (@adjiboussodieng)
0:30

#NewPaper The first microscope, invented in the 16th century, was designed to unlock the secrets of the microscopic world. Today, as many fields become increasingly data-driven, there is a pressing need for new types of microscopes---tools that help us zoom in, explore, and understand complex data. We call these tools "algorithmic microscopes." Introducing the Vendiscope: The first algorithmic microscope for data collections. 🔬 The Vendiscope maximizes the probability-weighted Vendi Score of a dataset to assign a weight to each element in the collection. This weight represents a data point's contribution to the overall diversity of the collection. These weights enable high-resolution data analysis at scale. We use them to zoom in on datasets across three domains: biology, materials science, & AI. 🧬 Biology: We used the Vendiscope on the protein universe, which contains nearly 250 million proteins. We found that nearly 200 million of the proteins are near-duplicates of each other and that AlphaFold fails on proteins that contribute most to the diversity of the protein universe. (See GIF below). 🪜 Materials Science: We used the Vendiscope on the Materials Project database, which contains 170K materials as of today. We found that 85% of crystals with formation energy data are near-duplicates of each other and that ML models for materials property prediction struggle with materials that contribute most to diversity. 🤖 Artificial Intelligence: We applied the Vendiscope to CIFAR-10, a benchmark dataset containing 50K images. We found duplicates. We applied the Vendiscope to analyze state-of-the-art generative models trained on this dataset. We found the best generative models memorize training data, as is known in the AI literature. However, we can do more with the Vendiscope and characterize the type of samples that get memorized. We found that data points contributing least to diversity are more prone to memorization by these generative models. 🧠 "Our findings demonstrate that the Vendiscope can serve as a powerful tool for data-driven science, providing a systematic and scalable way to identify duplicates and outliers, as well as pinpointing samples prone to memorization and those that models may struggle to predict---even before training." 💫 "The Vendiscope provides a unified framework for analyzing complex data at scale. Researchers, engineers, and data auditors can use the Vendiscope to audit datasets, identify potential biases, and refine data collection practices. For AI ethicists, the Vendiscope offers a critical lens to understand how models interact with data, particularly in the context of bias, memorization, and data fairness, enabling better mitigation strategies to prevent undesirable outcomes in AI deployment. For scientists, the Vendiscope represents a new companion in the discovery process." #VendiScoring #AlgorithmicMicroscopy Link to paper: Authors: Amey Pasarkar (Amey Pasarkar) and Adji Bousso Dieng (@adjiboussodieng)

Vertaix® (AI & Science)

34,762 Aufrufe • vor 1 Jahr

The hardest problems in AI aren't research problems anymore. They're deployment problems. It’s how we actually deliver real value, today, to build the future people want. That’s why, after 20 years in AI, my next step was inevitable: make robots do useful work for and alongside people, right now. Today, I am delighted to announce the launch of Walden Robotics to tackle just that. We started this year and are coming out of stealth today with a $300M seed round backed by some of the most serious companies and investors in the world. They have seen firsthand our general-purpose robots being useful in production on day one, and getting better every day after. You can see a glimpse of what we've been building in the video below. Physical AI has gone through a rapid phase transition, in part thanks to pioneering research from my friends and co-founders Russ Tedrake , Ben Burchfiel , Siyuan Feng, Rareș Ambruș , and many others at Walden. But from our long experience working together with co-founders Kerri Fetzer-Borelli and Dave Johnson, we learned how hard it is to deploy cutting-edge AI in a real, live, incredibly sophisticated production environment with an intricate ballet of automation and human ingenuity. That’s why we deliberately created Walden Robotics as a full-stack, human-centric, customer-focused robotics company from the start: we seeded the company with a world-class team across hardware, software, AI, deployment, operations, product, and business talent, so we could continuously optimize our whole system end-to-end, deeply and purposefully, from real-world experience with real customers. The efficacy of this strategy speaks for itself: since February, our general-purpose robots have been doing useful work in production at a Toyota plant in North America, moving from first pilot to real work in under two months. Not a lab. Not a demo. Not a future promise. Real work on a real line, today, at one of the best large-scale manufacturers in the world, with general-purpose robots that get better every day. And this is just the beginning. Two ways to find us: If you run a manufacturing or logistics business and want robots that are widely useful now, not someday, let's talk. We own “ for a reason! And if you want to build them: we're hiring across the company, from software, to hardware, AI, ops, product, business, and more. In particular, as the Chief Strategy Officer at Walden, I am recruiting for three incredibly impactful founding roles to fuel our agent-native go-to-market engine. Check out Let’s build together!
2:31

The hardest problems in AI aren't research problems anymore. They're deployment problems. It’s how we actually deliver real value, today, to build the future people want. That’s why, after 20 years in AI, my next step was inevitable: make robots do useful work for and alongside people, right now. Today, I am delighted to announce the launch of Walden Robotics to tackle just that. We started this year and are coming out of stealth today with a $300M seed round backed by some of the most serious companies and investors in the world. They have seen firsthand our general-purpose robots being useful in production on day one, and getting better every day after. You can see a glimpse of what we've been building in the video below. Physical AI has gone through a rapid phase transition, in part thanks to pioneering research from my friends and co-founders Russ Tedrake , Ben Burchfiel , Siyuan Feng, Rareș Ambruș , and many others at Walden. But from our long experience working together with co-founders Kerri Fetzer-Borelli and Dave Johnson, we learned how hard it is to deploy cutting-edge AI in a real, live, incredibly sophisticated production environment with an intricate ballet of automation and human ingenuity. That’s why we deliberately created Walden Robotics as a full-stack, human-centric, customer-focused robotics company from the start: we seeded the company with a world-class team across hardware, software, AI, deployment, operations, product, and business talent, so we could continuously optimize our whole system end-to-end, deeply and purposefully, from real-world experience with real customers. The efficacy of this strategy speaks for itself: since February, our general-purpose robots have been doing useful work in production at a Toyota plant in North America, moving from first pilot to real work in under two months. Not a lab. Not a demo. Not a future promise. Real work on a real line, today, at one of the best large-scale manufacturers in the world, with general-purpose robots that get better every day. And this is just the beginning. Two ways to find us: If you run a manufacturing or logistics business and want robots that are widely useful now, not someday, let's talk. We own “ for a reason! And if you want to build them: we're hiring across the company, from software, to hardware, AI, ops, product, business, and more. In particular, as the Chief Strategy Officer at Walden, I am recruiting for three incredibly impactful founding roles to fuel our agent-native go-to-market engine. Check out Let’s build together!

Adrien Gaidon

63,982 Aufrufe • vor 1 Monat

Today, we're joined by Nikita Rudin, co-founder and CEO of Flexion to discuss the gap between current robotic capabilities and what’s required to deploy fully autonomous robots in the real world. Nikita explains how reinforcement learning and simulation have driven rapid progress in robot locomotion—and why locomotion is still far from “solved.” We dig into the sim2real gap, and how adding visual inputs introduces noise and significantly complicates sim-to-real transfer. We also explore the debate between end-to-end models and modular approaches, and why separating locomotion, planning, and semantics remains a pragmatic approach today. Nikita also introduces the concept of "real-to-sim", which uses real-world data to refine simulation parameters for higher fidelity training, discusses how reinforcement learning, imitation learning, and teleoperation data are combined to train robust policies for both quadruped and humanoid robots, and introduces Flexion's hierarchical approach that utilizes pre-trained Vision-Language Models (VLMs) for high-level task orchestration with Vision-Language-Action (VLA) models and low-level whole-body trackers. Finally, Nikita shares the behind-the-scenes in humanoid robot demos, his take on reinforcement learning in simulation versus the real world, the nuances of reward tuning, and offers practical advice for researchers and practitioners looking to get started in robotics today. 🗒️ For the full list of resources for this episode, visit the show notes page: 📖 CHAPTERS =============================== 00:00 - Introduction 04:07 - Is robot locomotion solved? 06:04 - Sim-to-real gap 08:58 - Adding semantics to policies 09:42 - Modular vs end-to-end architectures 10:29 - Planner model 12:21 - Adapting RL techniques from quadrupeds to humanoids 15:39 - Behind robot demos 18:09 - Humanoid robots in home environments 22:03 - Training approach 23:56 - VLA models 27:59 - Closing the sim-to-real gap 32:55 - Task orchestration using VLMs 36:38 - Tool use 38:10 - Model hierarchy 43:37 - Simulator versus simulation environment 44:57 - Combining imitation learning and reinforcement learning 46:42 - RL in real world versus RL in simulation 52:58 - Reward tuning and value functions in robotics 56:38 - Predictions 1:00:10 - Humanoids, quadropeds, and wheeled platforms 1:02:45 - Advice, recommended robot kits, and community pla
1:06:07

Today, we're joined by Nikita Rudin, co-founder and CEO of Flexion to discuss the gap between current robotic capabilities and what’s required to deploy fully autonomous robots in the real world. Nikita explains how reinforcement learning and simulation have driven rapid progress in robot locomotion—and why locomotion is still far from “solved.” We dig into the sim2real gap, and how adding visual inputs introduces noise and significantly complicates sim-to-real transfer. We also explore the debate between end-to-end models and modular approaches, and why separating locomotion, planning, and semantics remains a pragmatic approach today. Nikita also introduces the concept of "real-to-sim", which uses real-world data to refine simulation parameters for higher fidelity training, discusses how reinforcement learning, imitation learning, and teleoperation data are combined to train robust policies for both quadruped and humanoid robots, and introduces Flexion's hierarchical approach that utilizes pre-trained Vision-Language Models (VLMs) for high-level task orchestration with Vision-Language-Action (VLA) models and low-level whole-body trackers. Finally, Nikita shares the behind-the-scenes in humanoid robot demos, his take on reinforcement learning in simulation versus the real world, the nuances of reward tuning, and offers practical advice for researchers and practitioners looking to get started in robotics today. 🗒️ For the full list of resources for this episode, visit the show notes page: 📖 CHAPTERS =============================== 00:00 - Introduction 04:07 - Is robot locomotion solved? 06:04 - Sim-to-real gap 08:58 - Adding semantics to policies 09:42 - Modular vs end-to-end architectures 10:29 - Planner model 12:21 - Adapting RL techniques from quadrupeds to humanoids 15:39 - Behind robot demos 18:09 - Humanoid robots in home environments 22:03 - Training approach 23:56 - VLA models 27:59 - Closing the sim-to-real gap 32:55 - Task orchestration using VLMs 36:38 - Tool use 38:10 - Model hierarchy 43:37 - Simulator versus simulation environment 44:57 - Combining imitation learning and reinforcement learning 46:42 - RL in real world versus RL in simulation 52:58 - Reward tuning and value functions in robotics 56:38 - Predictions 1:00:10 - Humanoids, quadropeds, and wheeled platforms 1:02:45 - Advice, recommended robot kits, and community pla

The TWIML AI Podcast

22,592 Aufrufe • vor 7 Monaten