Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Can we get robots to improve at long-horizon tasks without supervision? Our latest work tackles this problem by planning to practice! Here's a teaser showing initial task -> autonomous practice -> eval (+ interference by a gremlin👿)

Nishanth Kumar

1,993 subscribers

18,722 views • 2 years ago •via X (Twitter)

Education Health & Wellness Science & Technology

Anya Rossi• Live Now

Private livecam show

5 Comments

Nishanth Kumar2 years ago

The robot repeatedly chooses a skill to practice and then plans to practice it. To select a skill, the robot asks: "How much would the skill improve through practice?", and as a result, "How much would I improve at solving human-given tasks?"

Nishanth Kumar2 years ago

Here are full videos of the robot practicing and improving in two real-world environments!

Nishanth Kumar2 years ago

Check out our website for links to the paper and code, technical details, links to uncut and unedited (2+ hours!) robot videos, and of course assumptions and limitations!

Nishanth Kumar2 years ago

This was so much fun to work on during an (extended) internship at the Boston Dynamics AI Institute! Special thanks to @tomssilver, @williebeit, @LinfengZhaoZLF, Steve Proulx, Tomás Lozano-Pérez, Leslie Kaelbling, and Jenny Barry for incredible dedication and support throughout!!

Nishanth Kumar2 years ago

@williebeit @LinfengZhaoZLF This project also wouldn’t have been possible without open-source models (specifically DETIC and SAM from @AIatMeta ), which form the backbone of our robot’s vision pipeline!!

Related Videos

Can VLMs enable robots to autonomously improve? In our new work we ran a fleet of robot arms to collect autonomous data with VLM-proposed tasks and showed that robots can keep getting better as they are deployed, without supervision: 🧵👇

Can VLMs enable robots to autonomously improve? In our new work we ran a fleet of robot arms to collect autonomous data with VLM-proposed tasks and showed that robots can keep getting better as they are deployed, without supervision: 🧵👇

Sergey Levine

19,449 views • 2 years ago

Learn how to build a Gemini 3 Deep Agents with this video from LangChain. These agents can reason through complex, long horizon tasks by: > Breaking goals into actionable steps > Delegating specific work for specialized models > Leveraging file systems and code execution

Learn how to build a Gemini 3 Deep Agents with this video from LangChain. These agents can reason through complex, long horizon tasks by: > Breaking goals into actionable steps > Delegating specific work for specialized models > Leveraging file systems and code execution

Google AI Developers

19,484 views • 9 months ago

A few weeks ago, we shared our progress on articulated objects and long-horizon tasks. Here are two representative examples: - We've been steadily expanding our asset library to cover more articulated objects. Articulated objects have always been a challenging asset class to handle in simulation. Interacting with them requires robots to master atomic skills such as pushing, pulling, opening, and closing, and to understand part structure, interaction constraints, and how the object moves. - Long-horizon tasks can now be generated at scale. Long-horizon tasks are the other hard category: they require chaining multiple sub-goals in sequence. A failure early in the task can cascade and make the rest unrecoverable. Axis is scaling along three dimensions at once: data volume, data quality, and task difficulty.

A few weeks ago, we shared our progress on articulated objects and long-horizon tasks. Here are two representative examples: - We've been steadily expanding our asset library to cover more articulated objects. Articulated objects have always been a challenging asset class to handle in simulation. Interacting with them requires robots to master atomic skills such as pushing, pulling, opening, and closing, and to understand part structure, interaction constraints, and how the object moves. - Long-horizon tasks can now be generated at scale. Long-horizon tasks are the other hard category: they require chaining multiple sub-goals in sequence. A failure early in the task can cascade and make the rest unrecoverable. Axis is scaling along three dimensions at once: data volume, data quality, and task difficulty.

Axis Robotics

13,700 views • 1 month ago

Wouldn't it be great if we could train robots without any teleoperation! In our latest paper, we train robots to mimic a human video of the task by simply matching the object features using RL. We only need one video and under an hour of robot training.

Wouldn't it be great if we could train robots without any teleoperation! In our latest paper, we train robots to mimic a human video of the task by simply matching the object features using RL. We only need one video and under an hour of robot training.

Lerrel Pinto

46,221 views • 1 year ago

🚨Is it possible to devise an intuitive approach for crowdsourcing trainable data for robots without requiring a physical robot🤖? Can we democratize robot learning for all?🧑‍🤝‍🧑 Check out our latest #CoRL2023 paper-> AR2-D2: Training a Robot Without a Robot

🚨Is it possible to devise an intuitive approach for crowdsourcing trainable data for robots without requiring a physical robot🤖? Can we democratize robot learning for all?🧑‍🤝‍🧑 Check out our latest #CoRL2023 paper-> AR2-D2: Training a Robot Without a Robot

Jiafei Duan

38,894 views • 3 years ago

We sat down with Dr. Sasho McKenzie on our latest pod to dive into the science of putting. From information and research to methods and practice strategies, Sasho shares a wealth of information about putting and how golfers can work to improve at one of the most challenging parts of the game.

We sat down with Dr. Sasho McKenzie on our latest pod to dive into the science of putting. From information and research to methods and practice strategies, Sasho shares a wealth of information about putting and how golfers can work to improve at one of the most challenging parts of the game.

No Laying Up

43,541 views • 10 months ago

Introducing Yell At Your Robot (YAY Robot!) 🗣️- a fun collaboration b/w Stanford University and UC Berkeley 🤖 We enable robots to improve on-the-fly from language corrections: robots rapidly adapt in real-time and continuously improve from human verbal feedback. YAY Robot enables long-horizon, dexterous manipulation tasks like preparing trail-mix, packing a ziploc bag, and cleaning dishes:

Introducing Yell At Your Robot (YAY Robot!) 🗣️- a fun collaboration b/w Stanford University and UC Berkeley 🤖 We enable robots to improve on-the-fly from language corrections: robots rapidly adapt in real-time and continuously improve from human verbal feedback. YAY Robot enables long-horizon, dexterous manipulation tasks like preparing trail-mix, packing a ziploc bag, and cleaning dishes:

Lucy Shi

122,827 views • 2 years ago

What are the largest software engineering tasks AI can perform? To answer this, we built MirrorCode, our long-horizon SWE benchmark that lets AI code autonomously for days at a time. The best models complete some tasks we estimate would take human engineers several weeks.

What are the largest software engineering tasks AI can perform? To answer this, we built MirrorCode, our long-horizon SWE benchmark that lets AI code autonomously for days at a time. The best models complete some tasks we estimate would take human engineers several weeks.

Epoch AI

74,854 views • 1 month ago

Every wondered if we can model motion as a language? can we tokenize this new language? is it useful? Turns out tremendously! 🚀 In out latest #NeurIPS2024 paper on QueST: Self-Supervised Skill Abstractions for Learning Continuous Control, we find that action tokenization matters a lot! We can learn skill encodings by representing temporal action abstractions with a discrete codebook. This enables 2 things 1. Better Behaviour Cloning: we can better assimilate multi-task data (>9%) over best paper. This is currently best in class BC method! 2. generalization of this language to represent new tasks in 5-shot transfer to longer horizon tasks! Check out the thread by Atharva Mete for more details. And check out more details at: Joint work with Atharva Mete Albert Wilcox Haotian Xue Yongxin Chen Georgia Tech School of Interactive Computing Machine Learning at Georgia Tech Robotics@GT NVIDIA Robotics

Every wondered if we can model motion as a language? can we tokenize this new language? is it useful? Turns out tremendously! 🚀 In out latest #NeurIPS2024 paper on QueST: Self-Supervised Skill Abstractions for Learning Continuous Control, we find that action tokenization matters a lot! We can learn skill encodings by representing temporal action abstractions with a discrete codebook. This enables 2 things 1. Better Behaviour Cloning: we can better assimilate multi-task data (>9%) over best paper. This is currently best in class BC method! 2. generalization of this language to represent new tasks in 5-shot transfer to longer horizon tasks! Check out the thread by Atharva Mete for more details. And check out more details at: Joint work with Atharva Mete Albert Wilcox Haotian Xue Yongxin Chen Georgia Tech School of Interactive Computing Machine Learning at Georgia Tech Robotics@GT NVIDIA Robotics

Animesh Garg

26,218 views • 1 year ago

We just solved the long-horizon planning & execution issue with Agents 🤯! Excited to announce that MultiOn AI can now take actions well over 500+ steps without loosing context & cross-operate on 10+ websites as part of a single task 🤩!! Our latest upgrade cracks the goal divergence & looping issues that have long plagued agents like AutoGPT 🔥 Here's a demo of it working as a AI news research agent, continuously searching for the latest AI news and tweeting it out everyday, all autonomously of course 🚀: 🔊

We just solved the long-horizon planning & execution issue with Agents 🤯! Excited to announce that MultiOn AI can now take actions well over 500+ steps without loosing context & cross-operate on 10+ websites as part of a single task 🤩!! Our latest upgrade cracks the goal divergence & looping issues that have long plagued agents like AutoGPT 🔥 Here's a demo of it working as a AI news research agent, continuously searching for the latest AI news and tweeting it out everyday, all autonomously of course 🚀: 🔊

Div Garg

309,967 views • 2 years ago

Feels quite topical - so here's some more chat about practice starts 😅 Scott went trackside to get a proper look at how they work with this new set of regulations:

Feels quite topical - so here's some more chat about practice starts 😅 Scott went trackside to get a proper look at how they work with this new set of regulations:

The Race

22,287 views • 6 months ago

Introducing EdgeBench, a benchmark designed to study how agents learn from environments over at least 12~72-hour runs. We find that performance follows a log-sigmoid function of environment interaction time with high precision. EdgeBench is built with three ingredients: - 🌍 Real & Diverse: 134 real-world tasks across 6 task categories, spanning scientific problems, professional knowledge work, software engineering, optimization, formal math, and games. - ⏳ Ultra-Long-Horizon: Each task supports 12–72 hours of agent work. Recorded human effort averages 57.2 hours. - 🔁 Informative Feedback: Agents receive real-world feedback for continuous improvement. After 38,000 hours of agent runs on EdgeBench, a scaling law for learning from environments emerges: - 📈 As agents interact with task environments over time, their aggregate performance is precisely fit by a log-sigmoid function. - 🧠 This phenomenon can be explained by an elegant theory of graph exploration. We are releasing an initial 51 of the 134 tasks, together with the full evaluation framework, to help advance long-horizon agent research. Check our blog & paper for more findings! Blog Paper GitHub Dataset Details below 👇🧵

Introducing EdgeBench, a benchmark designed to study how agents learn from environments over at least 12~72-hour runs. We find that performance follows a log-sigmoid function of environment interaction time with high precision. EdgeBench is built with three ingredients: - 🌍 Real & Diverse: 134 real-world tasks across 6 task categories, spanning scientific problems, professional knowledge work, software engineering, optimization, formal math, and games. - ⏳ Ultra-Long-Horizon: Each task supports 12–72 hours of agent work. Recorded human effort averages 57.2 hours. - 🔁 Informative Feedback: Agents receive real-world feedback for continuous improvement. After 38,000 hours of agent runs on EdgeBench, a scaling law for learning from environments emerges: - 📈 As agents interact with task environments over time, their aggregate performance is precisely fit by a log-sigmoid function. - 🧠 This phenomenon can be explained by an elegant theory of graph exploration. We are releasing an initial 51 of the 134 tasks, together with the full evaluation framework, to help advance long-horizon agent research. Check our blog & paper for more findings! Blog Paper GitHub Dataset Details below 👇🧵

Deyao Zhu

361,163 views • 1 month ago

Hermes Agent can use Autobrowse to make better browser skills. In this HN example, 2 iterations take us from: 102 seconds -> 35 23 turns -> 8 $1.46 -> $0.28 Instead of clicking step by step it decides to eval JS directly on the page and save that workflow to a skill.

Hermes Agent can use Autobrowse to make better browser skills. In this HN example, 2 iterations take us from: 102 seconds -> 35 23 turns -> 8 $1.46 -> $0.28 Instead of clicking step by step it decides to eval JS directly on the page and save that workflow to a skill.

Kyle Jeong

319,804 views • 3 months ago

While we work on training Master Agent, here's a video of our Master Agent at work. This demo video is for initial proof of work purposes. You will be able to test Master Agent on our website shortly. $Nexus

While we work on training Master Agent, here's a video of our Master Agent at work. This demo video is for initial proof of work purposes. You will be able to test Master Agent on our website shortly. $Nexus

Nexus AI Network

14,496 views • 1 year ago

Can video generative models exhibit visuospatial intelligence? 🤔 Introducing Video4Spatial — a video-only framework that tackles spatial tasks. With just video context, our model can: 🔍 Ground objects by planning geometry-consistent paths 📸 Follow camera-pose instructions for scene navigation 🌐 Generalize to long contexts & unseen outdoor scenes A step toward video models as visual-spatial reasoners. Project: arXiv:

Can video generative models exhibit visuospatial intelligence? 🤔 Introducing Video4Spatial — a video-only framework that tackles spatial tasks. With just video context, our model can: 🔍 Ground objects by planning geometry-consistent paths 📸 Follow camera-pose instructions for scene navigation 🌐 Generalize to long contexts & unseen outdoor scenes A step toward video models as visual-spatial reasoners. Project: arXiv:

Xingang Pan

15,946 views • 8 months ago

Autonomous dodging off-roads without any human presence. This is a demonstration of pushing the limits of what is possible in off-road #autonomousdriving. In this demo, we performed autonomous dodging without any human presence in our autonomous vehicle, that too off-roads. In this mode, collision avoidance against aggressive-stochastic-adversarial traffic is the sole responsibility of #AutonomousVehicles. Our autonomous vehicle, Deep Xplorer, was tasked with navigating an off-road trail, avoiding both the static and dynamic obstacles. Furthermore, team members on bikes introduced random cross-traffic interactions by cutting its path at random, creating highly adversarial and stochastic traffic scenarios. To increase the stakes even further, I personally drove our other autonomous vehicle (Xplorer), cutting the path of Deep Xplorer at random --leaving the task of negotiation, motion planning, and decision making, to ensure collision avoidance and safety of the vehicles and the environment as a sole responsibility of our autonomous driving stack in Deep Xplorer. This is the first time in the history of Swaayatt Robots that we performed autonomous dodging without human presence in our autonomous vehicle, pushing the limits of what is possible in autonomous driving landscape. #reinforcementlearning #deeplearning

Autonomous dodging off-roads without any human presence. This is a demonstration of pushing the limits of what is possible in off-road #autonomousdriving. In this demo, we performed autonomous dodging without any human presence in our autonomous vehicle, that too off-roads. In this mode, collision avoidance against aggressive-stochastic-adversarial traffic is the sole responsibility of #AutonomousVehicles. Our autonomous vehicle, Deep Xplorer, was tasked with navigating an off-road trail, avoiding both the static and dynamic obstacles. Furthermore, team members on bikes introduced random cross-traffic interactions by cutting its path at random, creating highly adversarial and stochastic traffic scenarios. To increase the stakes even further, I personally drove our other autonomous vehicle (Xplorer), cutting the path of Deep Xplorer at random --leaving the task of negotiation, motion planning, and decision making, to ensure collision avoidance and safety of the vehicles and the environment as a sole responsibility of our autonomous driving stack in Deep Xplorer. This is the first time in the history of Swaayatt Robots that we performed autonomous dodging without human presence in our autonomous vehicle, pushing the limits of what is possible in autonomous driving landscape. #reinforcementlearning #deeplearning

Sanjeev Sharma

68,342 views • 6 months ago

Sick of Cursor mess up your code? This new flow reduced 90% error for my Cursor - by adopting a task system: 1. Break down PRD into small tasks with dependencies 2. Maintain & update task list as we go 3. Self improve with cursor rules With tools like claude task master & Boomerang tasks, this process is a no brainer I broken down my step by step process in the video If you want to learn more about vibe coding beyond basics, join my community here: (also get free access to 10xCoder.Dev)

Sick of Cursor mess up your code? This new flow reduced 90% error for my Cursor - by adopting a task system: 1. Break down PRD into small tasks with dependencies 2. Maintain & update task list as we go 3. Self improve with cursor rules With tools like claude task master & Boomerang tasks, this process is a no brainer I broken down my step by step process in the video If you want to learn more about vibe coding beyond basics, join my community here: (also get free access to 10xCoder.Dev)

Jason Zhou

43,014 views • 1 year ago

Kind of cool how we get a major step up in AI video every time image models improve. Insane work here with GPT Image 2 -> Seedance 2. Can't wait until we can each stream our own game like this 👀

Kind of cool how we get a major step up in AI video every time image models improve. Insane work here with GPT Image 2 -> Seedance 2. Can't wait until we can each stream our own game like this 👀

Justine Moore

380,972 views • 4 months ago