Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Having humans annotate data to pre-train robots is expensive and time-consuming! Introducing SPRINT: A pre-training approach using LLMs and offline RL to equip robots w/ many language-annotated skills while minimizing human annotation effort! URL: 🧵👇

Jesse_Y_Zhang's profile picture

Jesse Zhang

1,677 subscribers

24,662 Aufrufe • vor 3 Jahren •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

7 Kommentare

Profilbild von Jesse Zhang
Jesse Zhangvor 3 Jahren

Labeling demonstrations with natural language instructions in hindsight is standard, but it is tedious and expensive to scale. We propose automatically (1) **relabeling** language instructions and (2) **chaining** trajectories together to generate more training data.

Profilbild von Jesse Zhang
Jesse Zhangvor 3 Jahren

(1) Relabeling: If we have two skills, "Put mug in coffee machine" and "Press brew button," we could call this "Make Coffee." In SPRINT, we do this relabeling **automatically** by prompting an LLM to summarize nearby instructions. This gives us 2-2.5X more pre-training data!

Profilbild von Jesse Zhang
Jesse Zhangvor 3 Jahren

(2) Chaining: Offline RL can "stitch" trajectories to learn new behaviors. We carefully relabel rewards with offline RL and modified language instructions to allow stitching even with language conditioning!

Profilbild von Jesse Zhang
Jesse Zhangvor 3 Jahren

Results Overall, this allows us to achieve 2-8X better zero-shot long horizon task execution in ALFRED, a realistic household simulator, and on a real robot setup! SPRINT agents also fine-tune more efficiently to new tasks in unseen environments! ALFRED results:

Profilbild von Jesse Zhang
Jesse Zhangvor 3 Jahren

Real Robot Results With offline fine-tuning, SPRINT achieves superior performance on new, long-horizon manipulation tasks in previously unseen environments!

Profilbild von Jesse Zhang
Jesse Zhangvor 3 Jahren

For more details about SPRINT and experiment results, please see our paper or website. Paper: Website: Work done in collaboration with @KarlPertsch, @JiahuiZhang_32, @JosephLim_AI. @JiahuiZhang_32 is applying for PhD this year!

Profilbild von OliviaLi
OliviaLivor 2 Jahren

It sounds great, so that humans don't have to complete such a large amount of work every day, just let the robot do it

Ähnliche Videos

How to teach robots to perform long-horizon tasks efficiently and robustly🦾? Introducing MimicPlay - an imitation learning algorithm that uses "cheap human play data". Our approach unlocks both real-time planning through raw perception and strong robustness to disturbances!🧵👇
1:15

How to teach robots to perform long-horizon tasks efficiently and robustly🦾? Introducing MimicPlay - an imitation learning algorithm that uses "cheap human play data". Our approach unlocks both real-time planning through raw perception and strong robustness to disturbances!🧵👇

Chen Wang

288,642 Aufrufe • vor 3 Jahren

Wouldn't it be great if we could train robots without any teleoperation! In our latest paper, we train robots to mimic a human video of the task by simply matching the object features using RL. We only need one video and under an hour of robot training.
0:22

Wouldn't it be great if we could train robots without any teleoperation! In our latest paper, we train robots to mimic a human video of the task by simply matching the object features using RL. We only need one video and under an hour of robot training.

Lerrel Pinto

46,221 Aufrufe • vor 1 Jahr

HDMI (HumanoiD iMitation for Interaction) is a framework enabling humanoid robots to learn whole-body object interaction skills from monocular RGB human videos. It extracts and retargets human poses and object trajectories using GVHMR and LocoMujoco, building reference datasets with contact annotations. The data is used to train an RL policy via robot-object co-tracking. HDMI achieved 67 consecutive door traversals.
0:45

HDMI (HumanoiD iMitation for Interaction) is a framework enabling humanoid robots to learn whole-body object interaction skills from monocular RGB human videos. It extracts and retargets human poses and object trajectories using GVHMR and LocoMujoco, building reference datasets with contact annotations. The data is used to train an RL policy via robot-object co-tracking. HDMI achieved 67 consecutive door traversals.

The Humanoid Hub

17,395 Aufrufe • vor 9 Monaten

Amazon is training humanoids to move boxes. Makes sense! OmniRetarget is a data generation engine that enables complex loco-manipulation for humanoids. It uses offline retargeting from human MoCap datasets and augments data from single demos to produce 8 hours of trajectories that train RL policies in simulation, transferred zero-shot to real robots. Amazon FAR is the primary affiliation for the paper:
1:37

Amazon is training humanoids to move boxes. Makes sense! OmniRetarget is a data generation engine that enables complex loco-manipulation for humanoids. It uses offline retargeting from human MoCap datasets and augments data from single demos to produce 8 hours of trajectories that train RL policies in simulation, transferred zero-shot to real robots. Amazon FAR is the primary affiliation for the paper:

The Humanoid Hub

106,447 Aufrufe • vor 9 Monaten

One of the biggest bottlenecks in deploying visual AI and computer vision is annotation, which can be both costly and time-consuming. Today, we’re introducing Verified Auto Labeling, a new approach to AI-assisted annotation that achieves up to 95% of human-level performance while cutting labeling costs by up to 100,000x and time by 5,000x. Read the full paper:
2:39

One of the biggest bottlenecks in deploying visual AI and computer vision is annotation, which can be both costly and time-consuming. Today, we’re introducing Verified Auto Labeling, a new approach to AI-assisted annotation that achieves up to 95% of human-level performance while cutting labeling costs by up to 100,000x and time by 5,000x. Read the full paper:

Voxel51

12,689 Aufrufe • vor 1 Jahr

Introducing GRID: the General Robot Intelligence Development platform, designed for prototyping smart and safe robots rapidly using foundation models, LLMs, and simulation. Paper: Try now: GitHub: 🧵👇(1/N)
0:22

Introducing GRID: the General Robot Intelligence Development platform, designed for prototyping smart and safe robots rapidly using foundation models, LLMs, and simulation. Paper: Try now: GitHub: 🧵👇(1/N)

Sai Vemprala

277,281 Aufrufe • vor 2 Jahren

Our course recommendation of the day is “Post-training of LLMs, ” where you’ll learn how to customize pre-trained language models using Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Online Reinforcement Learning (RL). You'll learn when to use each method, how to curate training data, and implement them in code to shape model behavior effectively. Enroll here:
1:05

Our course recommendation of the day is “Post-training of LLMs, ” where you’ll learn how to customize pre-trained language models using Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Online Reinforcement Learning (RL). You'll learn when to use each method, how to curate training data, and implement them in code to shape model behavior effectively. Enroll here:

DeepLearning.AI

29,369 Aufrufe • vor 9 Monaten

A reward model that works, zero-shot, across robots, tasks, and scenes? Introducing Robometer: Scaling general-purpose robotic reward models with 1M+ trajectories. Enables zero-shot: online/offline/model-based RL, data retrieval + IL, automatic failure detection, and more! 🧵 (1/12)
2:01

A reward model that works, zero-shot, across robots, tasks, and scenes? Introducing Robometer: Scaling general-purpose robotic reward models with 1M+ trajectories. Enables zero-shot: online/offline/model-based RL, data retrieval + IL, automatic failure detection, and more! 🧵 (1/12)

Jesse Zhang @ ICML 2026 ✈️

100,947 Aufrufe • vor 4 Monaten

The most frustrating part of imitation learning is collecting huge amounts of teleop data. But why teleop robots when robots can learn by watching us? Introducing Point Policy, a novel framework that enables robots to learn from human videos without any teleop, sim2real, or RL.
0:40

The most frustrating part of imitation learning is collecting huge amounts of teleop data. But why teleop robots when robots can learn by watching us? Introducing Point Policy, a novel framework that enables robots to learn from human videos without any teleop, sim2real, or RL.

Siddhant Haldar

69,056 Aufrufe • vor 1 Jahr

Imagine robots learning new skills—without any robot data. Today, we're excited to release EgoZero: our first steps in training robot policies that operate in unseen environments, solely from data collected through humans wearing Aria smart glasses. 🧵👇
1:24

Imagine robots learning new skills—without any robot data. Today, we're excited to release EgoZero: our first steps in training robot policies that operate in unseen environments, solely from data collected through humans wearing Aria smart glasses. 🧵👇

Lerrel Pinto

42,576 Aufrufe • vor 1 Jahr

Cooking in kitchens is fun. BUT doing it collaboratively with two robots is even more satisfying! We introduce MOSAIC, a modular framework that coordinates multiple robots to closely collaborate and cook with humans via natural language interaction and a repository of skills.
1:14

Cooking in kitchens is fun. BUT doing it collaboratively with two robots is even more satisfying! We introduce MOSAIC, a modular framework that coordinates multiple robots to closely collaborate and cook with humans via natural language interaction and a repository of skills.

Sanjiban Choudhury

26,373 Aufrufe • vor 2 Jahren

Brain-controlled exoskeletons to train humanoid robots! 🧠 Fourier just presented human tele-operators using brain control interfaces and exoskeletal arms to train humanoid robots on home tasks. The brain control interface is the interesting part. Instead of using a controller or joystick to teleoperate, the operator's movements and intentions are captured more naturally through the exoskeleton and BCI. This means the demonstrations are more fluid, more human-like, and better suited for training robots to perform delicate home tasks. Multiple tele-operators are simultaneously generating training data across multiple robots. This is how you build the dataset needed for eventual full autonomy, without waiting years for it to arrive. This might be the bridge between "robots that work in controlled environments" and "robots that work in homes." Not full autonomy right away, but trusted human intelligence operating through a robot body, getting better with every task completed. ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →
0:56

Brain-controlled exoskeletons to train humanoid robots! 🧠 Fourier just presented human tele-operators using brain control interfaces and exoskeletal arms to train humanoid robots on home tasks. The brain control interface is the interesting part. Instead of using a controller or joystick to teleoperate, the operator's movements and intentions are captured more naturally through the exoskeleton and BCI. This means the demonstrations are more fluid, more human-like, and better suited for training robots to perform delicate home tasks. Multiple tele-operators are simultaneously generating training data across multiple robots. This is how you build the dataset needed for eventual full autonomy, without waiting years for it to arrive. This might be the bridge between "robots that work in controlled environments" and "robots that work in homes." Not full autonomy right away, but trusted human intelligence operating through a robot body, getting better with every task completed. ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

20,898 Aufrufe • vor 5 Monaten

Train the AI behind real-world robots We’re hiring Data Labelers to annotate images and videos to train Optimus & autonomous systems. You’ll work with real production data & directly impact how these systems learn and operate in the real world Come join!
0:33

Train the AI behind real-world robots We’re hiring Data Labelers to annotate images and videos to train Optimus & autonomous systems. You’ll work with real production data & directly impact how these systems learn and operate in the real world Come join!

Tesla Recruiting

646,513 Aufrufe • vor 2 Monaten

I need to annotate some images for training a computer vision model. There are many powerful annotation platforms available, but I want to keep my images local. I added a new section to my CV Streamlit app to quickly annotate images and train a YOLO model in a few clicks.
0:32

I need to annotate some images for training a computer vision model. There are many powerful annotation platforms available, but I want to keep my images local. I added a new section to my CV Streamlit app to quickly annotate images and train a YOLO model in a few clicks.

Marco Franzon

29,530 Aufrufe • vor 5 Monaten

We discovered an emergent property of VLAs like π0/π0.5/π0.6: as we scale up pre-training, the model learns to align human videos and robot data! This gives us a simple way to leverage human videos. Once π0.5 knows how to control robots, it can naturally learn from human video.
2:02

We discovered an emergent property of VLAs like π0/π0.5/π0.6: as we scale up pre-training, the model learns to align human videos and robot data! This gives us a simple way to leverage human videos. Once π0.5 knows how to control robots, it can naturally learn from human video.

Physical Intelligence

1,182,649 Aufrufe • vor 6 Monaten

Robots are the bottleneck in scaling robotics, and learning from human video promises to solve it. But how can chaotic human data ever measure up to sanitized, lab-made teleoperation data? Introducing Do as I Do: establishing a much needed correspondence between human videos and dexterous robot data. Some fun insights below: 🧵
0:55

Robots are the bottleneck in scaling robotics, and learning from human video promises to solve it. But how can chaotic human data ever measure up to sanitized, lab-made teleoperation data? Introducing Do as I Do: establishing a much needed correspondence between human videos and dexterous robot data. Some fun insights below: 🧵

Mahi Shafiullah 🏠🤖

91,113 Aufrufe • vor 21 Tagen

Introducing LDA, a latent world action foundation model that, for the first time, unifies the utilization of heterogeneous embodied data across simulation and reality, humans and robots, and varying levels of action quality and annotation. By breaking long-standing data silos in embodied intelligence, LDA enables the field, much like GPT did for language, to benefit continuously from scaling data, marking the transition into a new era of scalable learning. #Galbot #Robotics #Innovation #AI #Technology #Humanoid #WorldModel
3:44

Introducing LDA, a latent world action foundation model that, for the first time, unifies the utilization of heterogeneous embodied data across simulation and reality, humans and robots, and varying levels of action quality and annotation. By breaking long-standing data silos in embodied intelligence, LDA enables the field, much like GPT did for language, to benefit continuously from scaling data, marking the transition into a new era of scalable learning. #Galbot #Robotics #Innovation #AI #Technology #Humanoid #WorldModel

Galbot

38,131 Aufrufe • vor 2 Monaten

🤖WE HUMANS ARE TRAINING OUR AI REPLACEMENTS Humanoid robots are learning by watching humans work. Workers wear devices that capture egocentric data, showing robots exactly how tasks look from a human perspective. We are not just building robots. We are handing them our job's playbook.
0:29

🤖WE HUMANS ARE TRAINING OUR AI REPLACEMENTS Humanoid robots are learning by watching humans work. Workers wear devices that capture egocentric data, showing robots exactly how tasks look from a human perspective. We are not just building robots. We are handing them our job's playbook.

Coin Bureau

20,173 Aufrufe • vor 1 Monat

Let’s think about humanoid robots outside carrying the box. How about having the humanoid come out the door, interact with humans, and even dance? Introducing Expressive Whole-Body Control for Humanoid Robots: See how our robot performs rich, diverse, and expressive motions in the real world 👇🧵
2:37

Let’s think about humanoid robots outside carrying the box. How about having the humanoid come out the door, interact with humans, and even dance? Introducing Expressive Whole-Body Control for Humanoid Robots: See how our robot performs rich, diverse, and expressive motions in the real world 👇🧵

Xiaolong Wang

309,104 Aufrufe • vor 2 Jahren