Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

CMU Vision-Language-Autonomy update: The team just released SORT3D, the first general spatial relation toolbox for autonomous vision-language navigation that is fully integrated into real-robot systems! 🤖👀 Simulation and real-robot data is provided!:

CMU Robotics Institute

19,827 subscribers

18,234 views • 1 year ago •via X (Twitter)

Arts Science & Technology Education

Anya Rossi• Live Now

Private livecam show

4 Comments

SaoAI1 year ago

cool

MightyBot1 year ago

🧠 Unified Search. Smarter Meetings. Effortless CRM. MightyBot is your AI agent platform for seamless workflows—record meetings, automate CRM updates, and find answers across apps in seconds. 🌟 Focus on what matters. We'll handle the grind.

Leila Anime1 year ago

Impressive innovation! SORT3D is pushing the boundaries of vision-language navigation, making autonomous robots smarter. Kudos to the CMU Robotics team! 🤖

LECCA Intern (Ø,G) 🍜1 year ago

The real-world applications of SORT3D are crazy. Home robots, warehouses, even search-and-rescue just leveled up.

Related Videos

As part of the ongoing Vision-Language-Autonomy project, CMU researchers have developed a general navigation platform with an onboard autonomy stack to support Vision-Language-Navigation. It functions in simulation and on real robots 🤖💥 Check it out!

As part of the ongoing Vision-Language-Autonomy project, CMU researchers have developed a general navigation platform with an onboard autonomy stack to support Vision-Language-Navigation. It functions in simulation and on real robots 🤖💥 Check it out!

CMU Robotics Institute

23,706 views • 1 year ago

Qwen-VLA feels like one of the first real robotics foundation models. A single system trained across robot manipulation, navigation, egocentric human video, simulation, and vision-language reasoning instead of isolated robot policies.

Qwen-VLA feels like one of the first real robotics foundation models. A single system trained across robot manipulation, navigation, egocentric human video, simulation, and vision-language reasoning instead of isolated robot policies.

Robots Digest 🤖

14,601 views • 29 days ago

What happens when we train the largest vision-language model and add in robot experiences? The result is PaLM-E 🌴🤖, a 562-billion parameter, general-purpose, embodied visual-language generalist - across robotics, vision, and language. Website:

What happens when we train the largest vision-language model and add in robot experiences? The result is PaLM-E 🌴🤖, a 562-billion parameter, general-purpose, embodied visual-language generalist - across robotics, vision, and language. Website:

Danny Driess

1,272,487 views • 3 years ago

🚨CMU Vision-Language-Autonomy update: The team released a video to "find the refrigerator in the lounge"–– they are looking for new PhD & Master's students to work on long-horizon navigation & instruction! Contact Ji Zhang for more information:

🚨CMU Vision-Language-Autonomy update: The team released a video to "find the refrigerator in the lounge"–– they are looking for new PhD & Master's students to work on long-horizon navigation & instruction! Contact Ji Zhang for more information:

CMU Robotics Institute

20,362 views • 9 months ago

Real-world robot data is expensive and slow to collect, creating a major challenge for humanoid development. 🤖 The NVIDIA GR00T N1.6 open vision language action model is pre-trained on a diverse mix of data, including thousands of hours of Stanford Vision and Learning Lab’s BEHAVIOR simulation data, which covers long-horizon everyday manipulation tasks. This diverse training is the key to robust cross-embodiment performance and real-world adaptability. 🌍 Read the blog 🔗

Real-world robot data is expensive and slow to collect, creating a major challenge for humanoid development. 🤖 The NVIDIA GR00T N1.6 open vision language action model is pre-trained on a diverse mix of data, including thousands of hours of Stanford Vision and Learning Lab’s BEHAVIOR simulation data, which covers long-horizon everyday manipulation tasks. This diverse training is the key to robust cross-embodiment performance and real-world adaptability. 🌍 Read the blog 🔗

NVIDIA Robotics

13,421 views • 5 months ago

Physical Intelligence's π₀, a general-purpose robot foundation model that combines Internet-scale vision-language pretraining with robot interaction data to execute tasks. They aim "to develop foundation models that can control any robot to perform any task" Autonomous demos:

Physical Intelligence's π₀, a general-purpose robot foundation model that combines Internet-scale vision-language pretraining with robot interaction data to execute tasks. They aim "to develop foundation models that can control any robot to perform any task" Autonomous demos:

The Humanoid Hub

196,055 views • 1 year ago

🎤🎤 Excited to introduce COME-robot🤖🤖, Closed-Loop Open-Vocabulary Mobile Manipulation with GPT-4V. It is the first closed-loop framework utilizing the vision-language foundation model for open-ended reasoning and adaptive planning in real-world scenarios. COME-robot demonstrates a significant improvement in task success rate (~25%) compared to SOTA methods. Project: Arxiv:

🎤🎤 Excited to introduce COME-robot🤖🤖, Closed-Loop Open-Vocabulary Mobile Manipulation with GPT-4V. It is the first closed-loop framework utilizing the vision-language foundation model for open-ended reasoning and adaptive planning in real-world scenarios. COME-robot demonstrates a significant improvement in task success rate (~25%) compared to SOTA methods. Project: Arxiv:

Siyuan Huang

22,291 views • 2 years ago

QACE Dynamics | Autonomous Navigation Preview Today we are showing autonomous navigation running live inside QACE, the robot plans its route on the map, then adjusts in real time as lidar and vision pick up new obstacles. All of this runs through a single navigation block inside QACE, ready to plug into different robots without touching low-level control. This navigation block is one of the core building pieces of QACE autonomy, and it’s just the first of many we will be rolling out.

QACE Dynamics | Autonomous Navigation Preview Today we are showing autonomous navigation running live inside QACE, the robot plans its route on the map, then adjusts in real time as lidar and vision pick up new obstacles. All of this runs through a single navigation block inside QACE, ready to plug into different robots without touching low-level control. This navigation block is one of the core building pieces of QACE autonomy, and it’s just the first of many we will be rolling out.

Qace Dynamics

12,654 views • 7 months ago

🤖 What if a humanoid robot could make a hamburger from raw ingredients—all the way to your plate? 🔥 Excited to announce ViTacFormer: our new pipeline for next-level dexterous manipulation with active vision + high-resolution touch. 🎯 For the first time ever, we demonstrate ~2.5 minutes of continuous, autonomous control—combining active vision, high-res touch, and high-DoF robot hands SharpaWave — to complete complex, real-world tasks. Code is fully released; check out our: Homepage: Paper link: Github:

🤖 What if a humanoid robot could make a hamburger from raw ingredients—all the way to your plate? 🔥 Excited to announce ViTacFormer: our new pipeline for next-level dexterous manipulation with active vision + high-resolution touch. 🎯 For the first time ever, we demonstrate ~2.5 minutes of continuous, autonomous control—combining active vision, high-res touch, and high-DoF robot hands SharpaWave — to complete complex, real-world tasks. Code is fully released; check out our: Homepage: Paper link: Github:

Haoran Geng

95,044 views • 11 months ago

Check out this Circulus Pion robot at Computex 2026. It’s running on a Panther Lake processor, using P-cores for real-time control, E-cores to collect sensor data, the GPU for vision-language-action models, and the NPU for continuous perception.

Check out this Circulus Pion robot at Computex 2026. It’s running on a Panther Lake processor, using P-cores for real-time control, E-cores to collect sensor data, the GPU for vision-language-action models, and the NPU for continuous perception.

Intel Technology

10,752 views • 22 days ago

Introducing TraceVLA: a fully open-source Vision-Language-Action model reimagining spatial-temporal awareness: ✨ 3.5x gains on real robots, SOTA in simulation 💡 Fine-tunes on just 150K trajectories ⚡ Compact 4B model = 7B performance

Introducing TraceVLA: a fully open-source Vision-Language-Action model reimagining spatial-temporal awareness: ✨ 3.5x gains on real robots, SOTA in simulation 💡 Fine-tunes on just 150K trajectories ⚡ Compact 4B model = 7B performance

Yongyuan Liang

39,500 views • 1 year ago

A big part of scaling robot learning to solve real-world problems is that we somehow need to get enough diverse, high-quality data to train our robots to perform useful things. GPT and its fellow large language models were bootstrapped and proved out on a massive dataset of real-world language data. Unfortunately, despite our best efforts, similarly massive datasets don’t really exist for robotics — so, in our unending pursuit of high-quality, useful data, we turn to simulation. I compared a couple recent works on sim-to-real robot manipulation, which discuss how to train perception-driven manipulation policies in simulation, in such a way that they’re useful in the real world. - DextraH-RGB, from NVIDIA - Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation, also from NVIDIA — specifically the GEAR lab - Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids, another GEAR lab paper - Local Policies Enable Zero-shot Long-Horizon Manipulation, from CMU (video from DextrAH-RGB)

A big part of scaling robot learning to solve real-world problems is that we somehow need to get enough diverse, high-quality data to train our robots to perform useful things. GPT and its fellow large language models were bootstrapped and proved out on a massive dataset of real-world language data. Unfortunately, despite our best efforts, similarly massive datasets don’t really exist for robotics — so, in our unending pursuit of high-quality, useful data, we turn to simulation. I compared a couple recent works on sim-to-real robot manipulation, which discuss how to train perception-driven manipulation policies in simulation, in such a way that they’re useful in the real world. - DextraH-RGB, from NVIDIA - Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation, also from NVIDIA — specifically the GEAR lab - Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids, another GEAR lab paper - Local Policies Enable Zero-shot Long-Horizon Manipulation, from CMU (video from DextrAH-RGB)

Chris Paxton

20,486 views • 1 year ago

Excited to announce Tau Robotics (Tau Robotics). We are building a general AI for robots. We start by building millions of robot arms that learn in the real world. In the video, two robot arms are fully autonomous and controlled by a single neural network conditioned on different language instructions (four axes and five axes robot arms). The other two arms are teleoperated. The entire hardware cost in the video is about $1400. The video is at 1.5x speed.

Excited to announce Tau Robotics (Tau Robotics). We are building a general AI for robots. We start by building millions of robot arms that learn in the real world. In the video, two robot arms are fully autonomous and controlled by a single neural network conditioned on different language instructions (four axes and five axes robot arms). The other two arms are teleoperated. The entire hardware cost in the video is about $1400. The video is at 1.5x speed.

Alexander Koch

437,797 views • 2 years ago

Vision-language models can control robots, but what if the prompt is too complex for the robot to follow directly? We developed a way to get robots to “think through” complex instructions, feedback, and interjections. We call it the Hierarchical Interactive Robot (Hi Robot).

Vision-language models can control robots, but what if the prompt is too complex for the robot to follow directly? We developed a way to get robots to “think through” complex instructions, feedback, and interjections. We call it the Hierarchical Interactive Robot (Hi Robot).

Physical Intelligence

116,845 views • 1 year ago

Developing general-purpose robot models is costly due to the need for extensive real-world data. 🤖 #NVIDIAIsaac GR00T addresses these challenges by providing #humanoidrobot developers with robot foundation models & more. #CES2025 📹

Developing general-purpose robot models is costly due to the need for extensive real-world data. 🤖 #NVIDIAIsaac GR00T addresses these challenges by providing #humanoidrobot developers with robot foundation models & more. #CES2025 📹

NVIDIA Robotics

19,055 views • 1 year ago

POV: you’re a delivery robot navigating the real world 🤖 We’ll call it Dot. Real streets. Real traffic. Real decisions happening in milliseconds. Dot’s autonomy stack was designed for real-world uncertainty. This is DoorDash Labs autonomy in action.

POV: you’re a delivery robot navigating the real world 🤖 We’ll call it Dot. Real streets. Real traffic. Real decisions happening in milliseconds. Dot’s autonomy stack was designed for real-world uncertainty. This is DoorDash Labs autonomy in action.

DoorDash

129,505 views • 3 months ago

Our CoRL 2024 paper shows Reinforcement Learning can allow robots to learn skills via real-world practice, without any demonstrations or simulation engineering. Rewards are provided using language/vision models, and mobility of robots enables autonomous exploration. 1/N

Our CoRL 2024 paper shows Reinforcement Learning can allow robots to learn skills via real-world practice, without any demonstrations or simulation engineering. Rewards are provided using language/vision models, and mobility of robots enables autonomous exploration. 1/N

Russell Mendonca

38,454 views • 1 year ago

Robot foundation models are limited by costly real data, while simulation data is plentiful but visually mismatched to reality. We present Point Bridge, a method that enables zero-shot sim-to-real transfer for robot learning with minimal visual alignment.

Robot foundation models are limited by costly real data, while simulation data is plentiful but visually mismatched to reality. We present Point Bridge, a method that enables zero-shot sim-to-real transfer for robot learning with minimal visual alignment.

Siddhant Haldar

19,818 views • 4 months ago

Launching Dimensional: the operating system for physical space Command any robot in natural language, with universal abilities like spatial memory, autonomous navigation, manipulation, person tracking, more. Open source. Single pip install It’s time to program atoms, not bits

Launching Dimensional: the operating system for physical space Command any robot in natural language, with universal abilities like spatial memory, autonomous navigation, manipulation, person tracking, more. Open source. Single pip install It’s time to program atoms, not bits

stash

251,136 views • 7 months ago