Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

We (Pantheon) have been working on a robotics data quality pipeline that uncovered a series of major problems in public robotics datasets, especially for world modeling. To improve the quality of data available to open-source robotics, we're publishing annotations for four of the most popular datasets. Some examples of... show more

calix

12,273 subscribers

83,197 views • 8 hours ago •via X (Twitter)

Science & Technology

Anya Rossi• Live Now

Private livecam show

37 Comments

calix8 hours ago

1. Camera disconnection One of the wrist cameras in this episode froze, then disconnected. This would show up as pure noise to a world model.

calix8 hours ago

2. Missing gripper actions Despite the left gripper opening and closing, no actions are recorded. The world model is penalized for failing to predict the gripper's actions.

calix8 hours ago

3. Spiky actions In some datasets, action data contains sharp spikes. These are hard for a world action model with broad smoothness priors to fit.

calix8 hours ago

4. Shifted actions Action data is misaligned by a few frames for the grasp, but not the movement. This tricks the world model into predicting state changes too early or late.

calix8 hours ago

Read our full report: Credits to @sambhavgupta159 for uncovering and publishing this analysis. We're hiring for robotics foundation model research, software, data, and hardware engineering. We're also open to data collaborations.

"Leigh Marie" Braswell7 hours ago

@PantheonInc Such important work! Data is king

Calvin Chen8 hours ago

@PantheonInc bullish for looking at the data you’d be surprised how many don’t

Jannik Schilling7 hours ago

@PantheonInc Crazy how many issues we found in just two weeks...

carlo agostinelli8 hours ago

@PantheonInc I’ve been waiting for this

Aryan Shah7 hours ago

@PantheonInc Dream team!!

Nikhil Suresh7 hours ago

@hamostaf04 @PantheonInc Great work on the data!

Amir Fischer7 hours ago

@PantheonInc @Jannikschg @arminsommer_ the european robotics mafia!

Kushal Thaman7 hours ago

@PantheonInc bot see good data, bot happy :)

Yang Fan Yun7 hours ago

@PantheonInc let’s goooooo @calixo888 @nettedjay !!!!!! 🚀

Keval Shah7 hours ago

@PantheonInc Interesting read! Congrats to the team @_varunnair great to see you in action :)

arminsommer7 hours ago

@PantheonInc @sambhavgupta159 is a beast! very nice work!

devansh7 hours ago

@PantheonInc cool!!

Sonith7 hours ago

@PantheonInc 🙏

nathan chen7 hours ago

@PantheonInc amazing read! cong to the team :D @Jannikschg

Sarah Su7 hours ago

@PantheonInc this is super cool!

Henry Yin7 hours ago

@PantheonInc KALIX!!

Amulya Harish7 hours ago

@PantheonInc The beginning

Victor Li7 hours ago

@PantheonInc 🔥🔥

Henry Lien7 hours ago

@PantheonInc 🚀🚀🚀

npjd8 hours ago

@PantheonInc so amazing calix

Osmond7 hours ago

@PantheonInc quietly cooking!!

OnchainLoving.ETH7 hours ago

@PantheonInc finally some real data rigor

Varun Nair7 hours ago

@PantheonInc Data Interpretability is huge. These are fixable issues and the human data community should know the aspects that must be improved. Overall great for the community as a whole!

Benedict Wolff7 hours ago

@PantheonInc Crushing it!

Eric Li7 hours ago

@PantheonInc Well done Mr Gupta @sambhavgupta159

hamza mostafa7 hours ago

@PantheonInc very cool work!!

Kole Lee7 hours ago

@PantheonInc letsgoo

Daniel Ceballos7 hours ago

@PantheonInc Awesome results!!!

Zain Javaid7 hours ago

@PantheonInc awesome work! congrats pantheon

joseph7 hours ago

@PantheonInc long time coming 💪

Alexander Yevchenko7 hours ago

@PantheonInc thank you!!

Zuhair Lakhani7 hours ago

@PantheonInc Thank you for looking at the data

Related Videos

Today we're launching Hebbian Robotics (YC S26) Hebbian Robotics (YC S26) . We build APIs for evaluating data quality, without training a robotics model. Data vendors use our API to search and analyze Physical AI data at scale, getting on-demand quality signals and metrics without managing infrastructure. High-quality data is the bottleneck to the next frontier of intelligence in robotics. Buyers want data they can trust. Data providers are focused on operational excellence. They don't have the bandwidth to understand the quality of their data at scale. We're fixing this. Vendors use our APIs to understand the impact of their decisions on their data quality: - Buyers demanding per-collector hour limits? Prove compliance while we stay ahead of the workarounds. - Changed your collection SOPs? Monitor for quality drift and regression. - Suspect duplicate samples? Verify with a single API call. Before Hebbian, I trained AI models for industrial cleaning robots, and Kingston Kuan spent years building large-scale infrastructure at Jane Street.

Today we're launching Hebbian Robotics (YC S26) Hebbian Robotics (YC S26) . We build APIs for evaluating data quality, without training a robotics model. Data vendors use our API to search and analyze Physical AI data at scale, getting on-demand quality signals and metrics without managing infrastructure. High-quality data is the bottleneck to the next frontier of intelligence in robotics. Buyers want data they can trust. Data providers are focused on operational excellence. They don't have the bandwidth to understand the quality of their data at scale. We're fixing this. Vendors use our APIs to understand the impact of their decisions on their data quality: - Buyers demanding per-collector hour limits? Prove compliance while we stay ahead of the workarounds. - Changed your collection SOPs? Monitor for quality drift and regression. - Suspect duplicate samples? Verify with a single API call. Before Hebbian, I trained AI models for industrial cleaning robots, and Kingston Kuan spent years building large-scale infrastructure at Jane Street.

Brandon Ong

166,623 views • 1 month ago

1/ The data parsing market for robotics is just emerging, and it’s already valued in the tens of billions. Data parsing to build data models and software for Robotics is a $20B market in 2024 and is expected to reach $150B+ by 2034 Cherry AI Robotics has built a data parsing tool for robotics, which goes live next week, and will effectively make this project one of the pioneers in data parsing for robotics.

1/ The data parsing market for robotics is just emerging, and it’s already valued in the tens of billions. Data parsing to build data models and software for Robotics is a $20B market in 2024 and is expected to reach $150B+ by 2034 Cherry AI Robotics has built a data parsing tool for robotics, which goes live next week, and will effectively make this project one of the pioneers in data parsing for robotics.

CherryAI Robotics

90,786 views • 1 year ago

Most capable generalist robotics models today are closed or at best, open weights. But robotics won’t reach its ChatGPT moment without real openness. That GPT moment was built on years of open tools and datasets such as Python, PyTorch, ImageNet and more, that let researchers inspect, reproduce, and build. Today, we’re introducing MolmoAct 2: a fully open-source action reasoning model for real-world robotics. We rethought and reshaped everything! 🧵👇

Most capable generalist robotics models today are closed or at best, open weights. But robotics won’t reach its ChatGPT moment without real openness. That GPT moment was built on years of open tools and datasets such as Python, PyTorch, ImageNet and more, that let researchers inspect, reproduce, and build. Today, we’re introducing MolmoAct 2: a fully open-source action reasoning model for real-world robotics. We rethought and reshaped everything! 🧵👇

Jiafei Duan

117,636 views • 4 months ago

Building valuable datasets takes dedication, skill and expertise. From robotics data to trend forecasting, we incubate cutting-edge datasets that generate lasting real-world value. Vana connects passionate entrepreneurs with our network of top backers, experts and data buyers. Thank you to Hub71, ecosystem mentors, and partners for joining us at the Vana Founders' Intensive in Abu Dhabi. Real builders. Real business. That's the Vana way.

Building valuable datasets takes dedication, skill and expertise. From robotics data to trend forecasting, we incubate cutting-edge datasets that generate lasting real-world value. Vana connects passionate entrepreneurs with our network of top backers, experts and data buyers. Thank you to Hub71, ecosystem mentors, and partners for joining us at the Vana Founders' Intensive in Abu Dhabi. Real builders. Real business. That's the Vana way.

vana

15,816 views • 1 year ago

World Labs CEO Dr. Fei-Fei Li & SceniX Co-Founder Yunzhu Li on the data bottleneck in robotics and how world models help: "The lack of data in training, the lack of data in evaluation, this is very, very different from language models, where data is abundant on the internet." "And we know that in order for robotics to work, we have to somehow unlock the power of scaling law. But where does that come from?" "It's a profound problem that everybody's battling with in robotics." "We see a lot of unlock in being able to do this whole process through the modeling of the environments." "Being able to create these digital worlds... that's just going to unlock so much more potential for being able to replace all the costly and the unsafe data in the real environments with the data generated from the worlds for robots to be able to do scalable learning and evaluations." Fei-Fei Li Yunzhu Li martin_casado

World Labs CEO Dr. Fei-Fei Li & SceniX Co-Founder Yunzhu Li on the data bottleneck in robotics and how world models help: "The lack of data in training, the lack of data in evaluation, this is very, very different from language models, where data is abundant on the internet." "And we know that in order for robotics to work, we have to somehow unlock the power of scaling law. But where does that come from?" "It's a profound problem that everybody's battling with in robotics." "We see a lot of unlock in being able to do this whole process through the modeling of the environments." "Being able to create these digital worlds... that's just going to unlock so much more potential for being able to replace all the costly and the unsafe data in the real environments with the data generated from the worlds for robots to be able to do scalable learning and evaluations." Fei-Fei Li Yunzhu Li martin_casado

a16z

77,065 views • 1 month ago

New meta on the horizon - AI x Robotics?👀 "The ChatGPT moment for general robotics is just around the corner..." – Jensen Huang, CEO of NVIDIA Most profitable wallet on 7D leaderboard by Nansen 🧭 loaded a few Robotic plays: 🔷 $SAM - Crowdsourcing real-world datasets with robotic gaming 🔷 $HIBER - robust AI Infrastructure 🔷 $BREW - Intersection of robotics × AI × crypto 🔷 $MROBOTICS - mobile gaming and robotics company Another quality project to research: $XRT What else did I miss?🤔

New meta on the horizon - AI x Robotics?👀 "The ChatGPT moment for general robotics is just around the corner..." – Jensen Huang, CEO of NVIDIA Most profitable wallet on 7D leaderboard by Nansen 🧭 loaded a few Robotic plays: 🔷 $SAM - Crowdsourcing real-world datasets with robotic gaming 🔷 $HIBER - robust AI Infrastructure 🔷 $BREW - Intersection of robotics × AI × crypto 🔷 $MROBOTICS - mobile gaming and robotics company Another quality project to research: $XRT What else did I miss?🤔

Crypto Gideon(🧠,🧠)

15,027 views • 1 year ago

Very excited to share what our research team at Covariant has been working on: RFM-1, our latest Robotics Foundation Model. Built on top of high-quality multimodal data the Covariant robotics fleet has collected over the years, this project embodies our commitment to pushing the boundaries of how AI can interact with and understand the physical world. We're eager to share this milestone with the community and invite feedback and collaboration. Read more at

Very excited to share what our research team at Covariant has been working on: RFM-1, our latest Robotics Foundation Model. Built on top of high-quality multimodal data the Covariant robotics fleet has collected over the years, this project embodies our commitment to pushing the boundaries of how AI can interact with and understand the physical world. We're eager to share this milestone with the community and invite feedback and collaboration. Read more at

Rocky Duan

31,887 views • 2 years ago

Co-training has become a key part of the recipe for training large robotics models; it means that you mix some proportion of real robot data with other data sources, like simulation or egocentric human video data. This is especially important because robotics data tends to lack diversity which can be somewhat compensated for by the inclusion of these other modalities. And yet there has not been a sizable study on what constitute good practices for cotraining until now! We talk to Fanqi Lin and Jose Barreiros about their new work, a massive study which evaluated 89 policies over thousands of rollouts to tell us which forms of co-training were most useful for robotics. Watch episode 70 of RoboPapers, with Michael Cho and Chris Paxton, now!

Co-training has become a key part of the recipe for training large robotics models; it means that you mix some proportion of real robot data with other data sources, like simulation or egocentric human video data. This is especially important because robotics data tends to lack diversity which can be somewhat compensated for by the inclusion of these other modalities. And yet there has not been a sizable study on what constitute good practices for cotraining until now! We talk to Fanqi Lin and Jose Barreiros about their new work, a massive study which evaluated 89 policies over thousands of rollouts to tell us which forms of co-training were most useful for robotics. Watch episode 70 of RoboPapers, with Michael Cho and Chris Paxton, now!

RoboPapers

19,357 views • 5 months ago

Robotics in the age of FMs, cloud and Starlink. We've been rethinking robot AI — cloud-first, LLM-first approach. Fast to deploy. Modular. Data-centric. Built for long-horizon reasoning. Paper: Blog: 🧵 #Robotics #AI #LLM #VLM

Robotics in the age of FMs, cloud and Starlink. We've been rethinking robot AI — cloud-first, LLM-first approach. Fast to deploy. Modular. Data-centric. Built for long-horizon reasoning. Paper: Blog: 🧵 #Robotics #AI #LLM #VLM

Ashish Kapoor

10,185 views • 11 months ago

The Open Robotics AI Forum came to life in Seoul, thanks to the energy of our contributors and participants. Together we’re shaping an open future for robotics.

The Open Robotics AI Forum came to life in Seoul, thanks to the energy of our contributors and participants. Together we’re shaping an open future for robotics.

OpenMind

23,633 views • 11 months ago

A few months ago we set out to understand exactly how our models can be used to accelerate progress in robotics and other AI systems that don’t just represent the world, but are actually able to interact with it. We’ve been collaborating with leading companies in the space and we’ve developed an initial approach that we're now opening up to accelerate development across the industry. Our fifth announcement is GWM Robotics. GWM Robotics is a learned simulator that generates synthetic data for scalable robot training and policy evaluation, removing the bottlenecks of physical hardware.

A few months ago we set out to understand exactly how our models can be used to accelerate progress in robotics and other AI systems that don’t just represent the world, but are actually able to interact with it. We’ve been collaborating with leading companies in the space and we’ve developed an initial approach that we're now opening up to accelerate development across the industry. Our fifth announcement is GWM Robotics. GWM Robotics is a learned simulator that generates synthetic data for scalable robot training and policy evaluation, removing the bottlenecks of physical hardware.

Runway

35,040 views • 9 months ago

Sergey Levine (Sergey Levine) is one of the world's top robotics researchers and co-founder of Physical Intelligence. We talked about where humanoid robotics is today, thoughts on the Chinese robotics ecosystem, and his predictions for future timelines. In this episode: • Current state of robotics and surprising capabilities so far • Chinese robotics compared to US ecosystem • If OpenAI and Anthropic got into robotics • His top robotics research paper recommendation • Predictions for when humanoid robotics will land Where to watch: • YouTube - • Spotify - • Apple Podcasts - • Transcript - Thank you to the sponsor of this episode for supporting my work: • WorkOS: makes your app Enterprise Ready with easy to use APIs to add SSO, SCIM, RBAC, and more in just a few lines of code, check them out at Chapters: 00:00 Intro 00:37 Where are we today 04:20 Most surprising capabilities so far 07:03 The most inspiring real world robotics 08:36 If OpenAI or Anthropic got into robotics 10:22 Chinese robotics 13:15 Will one lab breakout from the rest 16:59 Thoughts on a concrete roadmap 21:03 Generalization and demonstrating it 26:04 Types of data and which is best for robotics 34:34 Why humanoid robotics differs from Waymo 37:10 If humanoid robotics failed here is why 39:55 Are there hot take modeling architectures in robotics 42:05 Thoughts on AI safety in robotics 46:44 Top robotics research paper recommendation 49:35 Why is Boston Dynamics less top of mind 53:47 Advice for his younger self 56:42 Outro

Sergey Levine (Sergey Levine) is one of the world's top robotics researchers and co-founder of Physical Intelligence. We talked about where humanoid robotics is today, thoughts on the Chinese robotics ecosystem, and his predictions for future timelines. In this episode: • Current state of robotics and surprising capabilities so far • Chinese robotics compared to US ecosystem • If OpenAI and Anthropic got into robotics • His top robotics research paper recommendation • Predictions for when humanoid robotics will land Where to watch: • YouTube - • Spotify - • Apple Podcasts - • Transcript - Thank you to the sponsor of this episode for supporting my work: • WorkOS: makes your app Enterprise Ready with easy to use APIs to add SSO, SCIM, RBAC, and more in just a few lines of code, check them out at Chapters: 00:00 Intro 00:37 Where are we today 04:20 Most surprising capabilities so far 07:03 The most inspiring real world robotics 08:36 If OpenAI or Anthropic got into robotics 10:22 Chinese robotics 13:15 Will one lab breakout from the rest 16:59 Thoughts on a concrete roadmap 21:03 Generalization and demonstrating it 26:04 Types of data and which is best for robotics 34:34 Why humanoid robotics differs from Waymo 37:10 If humanoid robotics failed here is why 39:55 Are there hot take modeling architectures in robotics 42:05 Thoughts on AI safety in robotics 46:44 Top robotics research paper recommendation 49:35 Why is Boston Dynamics less top of mind 53:47 Advice for his younger self 56:42 Outro

Ryan Peterman

70,752 views • 1 month ago

Unreal Engine 5 is becoming the go-to platform for robotics teams, not just as a simulator but as a synthetic data factory powering the next generation of robotics systems🦾 From synthetic data to sim-to-real transfer, read our piece on how UE5 is quietly reshaping how robots learn to see, move, and act in the real world:

Unreal Engine 5 is becoming the go-to platform for robotics teams, not just as a simulator but as a synthetic data factory powering the next generation of robotics systems🦾 From synthetic data to sim-to-real transfer, read our piece on how UE5 is quietly reshaping how robots learn to see, move, and act in the real world:

Unreal Engine

61,187 views • 5 months ago

The open-source AI community just got a new home for their data workflows. 🤗 Hugging Face is now available in Adaptive Data. Pull datasets directly into a platform that evolves with the problems you're solving.

The open-source AI community just got a new home for their data workflows. 🤗 Hugging Face is now available in Adaptive Data. Pull datasets directly into a platform that evolves with the problems you're solving.

adaption

55,194 views • 5 months ago

🤖 OrionX Robotics is actively engaged in a fundraising round to accelerate growth, scale operations, and advance the development of next-generation autonomous robotics systems. We look forward to partnering with investors who share our vision for the future of intelligent robotics.

🤖 OrionX Robotics is actively engaged in a fundraising round to accelerate growth, scale operations, and advance the development of next-generation autonomous robotics systems. We look forward to partnering with investors who share our vision for the future of intelligent robotics.

OrionX Robotics

12,345 views • 3 months ago

Figure just crossed 86,000 weekly active users uploading data We're building the largest and most diverse robotics dataset in the world The website has a live map viewer showing each pixel of data as it's uploaded around the world

Figure just crossed 86,000 weekly active users uploading data We're building the largest and most diverse robotics dataset in the world The website has a live map viewer showing each pixel of data as it's uploaded around the world

Brett Adcock

113,943 views • 13 days ago

NVIDIA CEO Jensen Huang: Elon Musk is in a phenomenal position to lead the future of AI, autonomous vehicles, and humanoid robotics. “Collecting world data is very expensive, and Elon has a great advantage because, number one, his AI factory for his cars is fantastic, has a lot of NVIDIA gear in it, and he has a very large fleet of cars on the road that allows him to collect a lot of data. So I think he has just a phenomenal position, and he's been working on this for a long time, and so he's going to be in a great position to take advantage of it. I know Elon's attitude towards AI, and he's very optimistic about its future, and obviously he's working on some of the most important AI areas. xAI is working on foundation, cognitive intelligence AI. Tesla's working on autonomous vehicles, and Optimus is for humanoid robotics. These three areas of AI are the three most important areas.”

NVIDIA CEO Jensen Huang: Elon Musk is in a phenomenal position to lead the future of AI, autonomous vehicles, and humanoid robotics. “Collecting world data is very expensive, and Elon has a great advantage because, number one, his AI factory for his cars is fantastic, has a lot of NVIDIA gear in it, and he has a very large fleet of cars on the road that allows him to collect a lot of data. So I think he has just a phenomenal position, and he's been working on this for a long time, and so he's going to be in a great position to take advantage of it. I know Elon's attitude towards AI, and he's very optimistic about its future, and obviously he's working on some of the most important AI areas. xAI is working on foundation, cognitive intelligence AI. Tesla's working on autonomous vehicles, and Optimus is for humanoid robotics. These three areas of AI are the three most important areas.”

DogeDesigner

142,845 views • 1 month ago

A big part of scaling robot learning to solve real-world problems is that we somehow need to get enough diverse, high-quality data to train our robots to perform useful things. GPT and its fellow large language models were bootstrapped and proved out on a massive dataset of real-world language data. Unfortunately, despite our best efforts, similarly massive datasets don’t really exist for robotics — so, in our unending pursuit of high-quality, useful data, we turn to simulation. I compared a couple recent works on sim-to-real robot manipulation, which discuss how to train perception-driven manipulation policies in simulation, in such a way that they’re useful in the real world. - DextraH-RGB, from NVIDIA - Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation, also from NVIDIA — specifically the GEAR lab - Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids, another GEAR lab paper - Local Policies Enable Zero-shot Long-Horizon Manipulation, from CMU (video from DextrAH-RGB)

A big part of scaling robot learning to solve real-world problems is that we somehow need to get enough diverse, high-quality data to train our robots to perform useful things. GPT and its fellow large language models were bootstrapped and proved out on a massive dataset of real-world language data. Unfortunately, despite our best efforts, similarly massive datasets don’t really exist for robotics — so, in our unending pursuit of high-quality, useful data, we turn to simulation. I compared a couple recent works on sim-to-real robot manipulation, which discuss how to train perception-driven manipulation policies in simulation, in such a way that they’re useful in the real world. - DextraH-RGB, from NVIDIA - Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation, also from NVIDIA — specifically the GEAR lab - Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids, another GEAR lab paper - Local Policies Enable Zero-shot Long-Horizon Manipulation, from CMU (video from DextrAH-RGB)

Chris Paxton

20,486 views • 1 year ago