正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

We (Pantheon) have been working on a robotics data quality pipeline that uncovered a series of major problems in public robotics datasets, especially for world modeling. To improve the quality of data available to open-source robotics, we're publishing annotations for four of the most popular datasets. Some examples of... show more

calix

12,273 subscribers

83,197 次观看 • 6 小时前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

37 条评论

calix 的头像

calix6 小时前

1. Camera disconnection One of the wrist cameras in this episode froze, then disconnected. This would show up as pure noise to a world model.

calix 的头像

calix6 小时前

2. Missing gripper actions Despite the left gripper opening and closing, no actions are recorded. The world model is penalized for failing to predict the gripper's actions.

calix 的头像

calix6 小时前

3. Spiky actions In some datasets, action data contains sharp spikes. These are hard for a world action model with broad smoothness priors to fit.

calix 的头像

calix6 小时前

4. Shifted actions Action data is misaligned by a few frames for the grasp, but not the movement. This tricks the world model into predicting state changes too early or late.

calix 的头像

calix6 小时前

Read our full report: Credits to @sambhavgupta159 for uncovering and publishing this analysis. We're hiring for robotics foundation model research, software, data, and hardware engineering. We're also open to data collaborations.

"Leigh Marie" Braswell 的头像

"Leigh Marie" Braswell6 小时前

@PantheonInc Such important work! Data is king

Calvin Chen 的头像

Calvin Chen6 小时前

@PantheonInc bullish for looking at the data you’d be surprised how many don’t

Jannik Schilling 的头像

Jannik Schilling6 小时前

@PantheonInc Crazy how many issues we found in just two weeks...

carlo agostinelli 的头像

carlo agostinelli6 小时前

@PantheonInc I’ve been waiting for this

Aryan Shah 的头像

Aryan Shah6 小时前

@PantheonInc Dream team!!

Nikhil Suresh 的头像

Nikhil Suresh6 小时前

@hamostaf04 @PantheonInc Great work on the data!

Amir Fischer 的头像

Amir Fischer6 小时前

@PantheonInc @Jannikschg @arminsommer_ the european robotics mafia!

Kushal Thaman 的头像

Kushal Thaman6 小时前

@PantheonInc bot see good data, bot happy :)

Yang Fan Yun 的头像

Yang Fan Yun6 小时前

@PantheonInc let’s goooooo @calixo888 @nettedjay !!!!!! 🚀

Keval Shah 的头像

Keval Shah6 小时前

@PantheonInc Interesting read! Congrats to the team @_varunnair great to see you in action :)

arminsommer 的头像

arminsommer6 小时前

@PantheonInc @sambhavgupta159 is a beast! very nice work!

devansh 的头像

devansh6 小时前

@PantheonInc cool!!

Sonith 的头像

Sonith6 小时前

@PantheonInc 🙏

nathan chen 的头像

nathan chen6 小时前

@PantheonInc amazing read! cong to the team :D @Jannikschg

Sarah Su 的头像

Sarah Su6 小时前

@PantheonInc this is super cool!

Henry Yin 的头像

Henry Yin6 小时前

@PantheonInc KALIX!!

Amulya Harish 的头像

Amulya Harish6 小时前

@PantheonInc The beginning

Victor Li 的头像

Victor Li6 小时前

@PantheonInc 🔥🔥

Henry Lien 的头像

Henry Lien6 小时前

@PantheonInc 🚀🚀🚀

npjd 的头像

npjd6 小时前

@PantheonInc so amazing calix

Osmond 的头像

Osmond6 小时前

@PantheonInc quietly cooking!!

OnchainLoving.ETH 的头像

OnchainLoving.ETH6 小时前

@PantheonInc finally some real data rigor

Varun Nair 的头像

Varun Nair6 小时前

@PantheonInc Data Interpretability is huge. These are fixable issues and the human data community should know the aspects that must be improved. Overall great for the community as a whole!

Benedict Wolff 的头像

Benedict Wolff6 小时前

@PantheonInc Crushing it!

Eric Li 的头像

Eric Li6 小时前

@PantheonInc Well done Mr Gupta @sambhavgupta159

hamza mostafa 的头像

hamza mostafa6 小时前

@PantheonInc very cool work!!

Kole Lee 的头像

Kole Lee6 小时前

@PantheonInc letsgoo

Daniel Ceballos 的头像

Daniel Ceballos6 小时前

@PantheonInc Awesome results!!!

Zain Javaid 的头像

Zain Javaid6 小时前

@PantheonInc awesome work! congrats pantheon

joseph 的头像

joseph6 小时前

@PantheonInc long time coming 💪

Alexander Yevchenko 的头像

Alexander Yevchenko6 小时前

@PantheonInc thank you!!

Zuhair Lakhani 的头像

Zuhair Lakhani6 小时前

@PantheonInc Thank you for looking at the data

相关视频

Today we're launching Hebbian Robotics (YC S26) Hebbian Robotics (YC S26) . We build APIs for evaluating data quality, without training a robotics model. Data vendors use our API to search and analyze Physical AI data at scale, getting on-demand quality signals and metrics without managing infrastructure. High-quality data is the bottleneck to the next frontier of intelligence in robotics. Buyers want data they can trust. Data providers are focused on operational excellence. They don't have the bandwidth to understand the quality of their data at scale. We're fixing this. Vendors use our APIs to understand the impact of their decisions on their data quality: - Buyers demanding per-collector hour limits? Prove compliance while we stay ahead of the workarounds. - Changed your collection SOPs? Monitor for quality drift and regression. - Suspect duplicate samples? Verify with a single API call. Before Hebbian, I trained AI models for industrial cleaning robots, and Kingston Kuan spent years building large-scale infrastructure at Jane Street.

Today we're launching Hebbian Robotics (YC S26) Hebbian Robotics (YC S26) . We build APIs for evaluating data quality, without training a robotics model. Data vendors use our API to search and analyze Physical AI data at scale, getting on-demand quality signals and metrics without managing infrastructure. High-quality data is the bottleneck to the next frontier of intelligence in robotics. Buyers want data they can trust. Data providers are focused on operational excellence. They don't have the bandwidth to understand the quality of their data at scale. We're fixing this. Vendors use our APIs to understand the impact of their decisions on their data quality: - Buyers demanding per-collector hour limits? Prove compliance while we stay ahead of the workarounds. - Changed your collection SOPs? Monitor for quality drift and regression. - Suspect duplicate samples? Verify with a single API call. Before Hebbian, I trained AI models for industrial cleaning robots, and Kingston Kuan spent years building large-scale infrastructure at Jane Street.

Brandon Ong

166,623 次观看 • 1 个月前

1/ The data parsing market for robotics is just emerging, and it’s already valued in the tens of billions. Data parsing to build data models and software for Robotics is a $20B market in 2024 and is expected to reach $150B+ by 2034 Cherry AI Robotics has built a data parsing tool for robotics, which goes live next week, and will effectively make this project one of the pioneers in data parsing for robotics.

1/ The data parsing market for robotics is just emerging, and it’s already valued in the tens of billions. Data parsing to build data models and software for Robotics is a $20B market in 2024 and is expected to reach $150B+ by 2034 Cherry AI Robotics has built a data parsing tool for robotics, which goes live next week, and will effectively make this project one of the pioneers in data parsing for robotics.

CherryAI Robotics

90,786 次观看 • 1 年前

Most capable generalist robotics models today are closed or at best, open weights. But robotics won’t reach its ChatGPT moment without real openness. That GPT moment was built on years of open tools and datasets such as Python, PyTorch, ImageNet and more, that let researchers inspect, reproduce, and build. Today, we’re introducing MolmoAct 2: a fully open-source action reasoning model for real-world robotics. We rethought and reshaped everything! 🧵👇

Most capable generalist robotics models today are closed or at best, open weights. But robotics won’t reach its ChatGPT moment without real openness. That GPT moment was built on years of open tools and datasets such as Python, PyTorch, ImageNet and more, that let researchers inspect, reproduce, and build. Today, we’re introducing MolmoAct 2: a fully open-source action reasoning model for real-world robotics. We rethought and reshaped everything! 🧵👇

Jiafei Duan

117,636 次观看 • 4 个月前

Building valuable datasets takes dedication, skill and expertise. From robotics data to trend forecasting, we incubate cutting-edge datasets that generate lasting real-world value. Vana connects passionate entrepreneurs with our network of top backers, experts and data buyers. Thank you to Hub71, ecosystem mentors, and partners for joining us at the Vana Founders' Intensive in Abu Dhabi. Real builders. Real business. That's the Vana way.

Building valuable datasets takes dedication, skill and expertise. From robotics data to trend forecasting, we incubate cutting-edge datasets that generate lasting real-world value. Vana connects passionate entrepreneurs with our network of top backers, experts and data buyers. Thank you to Hub71, ecosystem mentors, and partners for joining us at the Vana Founders' Intensive in Abu Dhabi. Real builders. Real business. That's the Vana way.

vana

15,816 次观看 • 1 年前

World Labs CEO Dr. Fei-Fei Li & SceniX Co-Founder Yunzhu Li on the data bottleneck in robotics and how world models help: "The lack of data in training, the lack of data in evaluation, this is very, very different from language models, where data is abundant on the internet." "And we know that in order for robotics to work, we have to somehow unlock the power of scaling law. But where does that come from?" "It's a profound problem that everybody's battling with in robotics." "We see a lot of unlock in being able to do this whole process through the modeling of the environments." "Being able to create these digital worlds... that's just going to unlock so much more potential for being able to replace all the costly and the unsafe data in the real environments with the data generated from the worlds for robots to be able to do scalable learning and evaluations." Fei-Fei Li Yunzhu Li martin_casado

World Labs CEO Dr. Fei-Fei Li & SceniX Co-Founder Yunzhu Li on the data bottleneck in robotics and how world models help: "The lack of data in training, the lack of data in evaluation, this is very, very different from language models, where data is abundant on the internet." "And we know that in order for robotics to work, we have to somehow unlock the power of scaling law. But where does that come from?" "It's a profound problem that everybody's battling with in robotics." "We see a lot of unlock in being able to do this whole process through the modeling of the environments." "Being able to create these digital worlds... that's just going to unlock so much more potential for being able to replace all the costly and the unsafe data in the real environments with the data generated from the worlds for robots to be able to do scalable learning and evaluations." Fei-Fei Li Yunzhu Li martin_casado

a16z

77,065 次观看 • 1 个月前

New meta on the horizon - AI x Robotics?👀 "The ChatGPT moment for general robotics is just around the corner..." – Jensen Huang, CEO of NVIDIA Most profitable wallet on 7D leaderboard by Nansen 🧭 loaded a few Robotic plays: 🔷 $SAM - Crowdsourcing real-world datasets with robotic gaming 🔷 $HIBER - robust AI Infrastructure 🔷 $BREW - Intersection of robotics × AI × crypto 🔷 $MROBOTICS - mobile gaming and robotics company Another quality project to research: $XRT What else did I miss?🤔

New meta on the horizon - AI x Robotics?👀 "The ChatGPT moment for general robotics is just around the corner..." – Jensen Huang, CEO of NVIDIA Most profitable wallet on 7D leaderboard by Nansen 🧭 loaded a few Robotic plays: 🔷 $SAM - Crowdsourcing real-world datasets with robotic gaming 🔷 $HIBER - robust AI Infrastructure 🔷 $BREW - Intersection of robotics × AI × crypto 🔷 $MROBOTICS - mobile gaming and robotics company Another quality project to research: $XRT What else did I miss?🤔

Crypto Gideon(🧠,🧠)

15,027 次观看 • 1 年前

Very excited to share what our research team at Covariant has been working on: RFM-1, our latest Robotics Foundation Model. Built on top of high-quality multimodal data the Covariant robotics fleet has collected over the years, this project embodies our commitment to pushing the boundaries of how AI can interact with and understand the physical world. We're eager to share this milestone with the community and invite feedback and collaboration. Read more at

Very excited to share what our research team at Covariant has been working on: RFM-1, our latest Robotics Foundation Model. Built on top of high-quality multimodal data the Covariant robotics fleet has collected over the years, this project embodies our commitment to pushing the boundaries of how AI can interact with and understand the physical world. We're eager to share this milestone with the community and invite feedback and collaboration. Read more at

Rocky Duan

31,887 次观看 • 2 年前

Co-training has become a key part of the recipe for training large robotics models; it means that you mix some proportion of real robot data with other data sources, like simulation or egocentric human video data. This is especially important because robotics data tends to lack diversity which can be somewhat compensated for by the inclusion of these other modalities. And yet there has not been a sizable study on what constitute good practices for cotraining until now! We talk to Fanqi Lin and Jose Barreiros about their new work, a massive study which evaluated 89 policies over thousands of rollouts to tell us which forms of co-training were most useful for robotics. Watch episode 70 of RoboPapers, with Michael Cho and Chris Paxton, now!

Co-training has become a key part of the recipe for training large robotics models; it means that you mix some proportion of real robot data with other data sources, like simulation or egocentric human video data. This is especially important because robotics data tends to lack diversity which can be somewhat compensated for by the inclusion of these other modalities. And yet there has not been a sizable study on what constitute good practices for cotraining until now! We talk to Fanqi Lin and Jose Barreiros about their new work, a massive study which evaluated 89 policies over thousands of rollouts to tell us which forms of co-training were most useful for robotics. Watch episode 70 of RoboPapers, with Michael Cho and Chris Paxton, now!

RoboPapers

19,357 次观看 • 5 个月前

Robotics in the age of FMs, cloud and Starlink. We've been rethinking robot AI — cloud-first, LLM-first approach. Fast to deploy. Modular. Data-centric. Built for long-horizon reasoning. Paper: Blog: 🧵 #Robotics #AI #LLM #VLM

Robotics in the age of FMs, cloud and Starlink. We've been rethinking robot AI — cloud-first, LLM-first approach. Fast to deploy. Modular. Data-centric. Built for long-horizon reasoning. Paper: Blog: 🧵 #Robotics #AI #LLM #VLM

Ashish Kapoor

10,185 次观看 • 11 个月前

The Open Robotics AI Forum came to life in Seoul, thanks to the energy of our contributors and participants. Together we’re shaping an open future for robotics.

The Open Robotics AI Forum came to life in Seoul, thanks to the energy of our contributors and participants. Together we’re shaping an open future for robotics.

OpenMind

23,633 次观看 • 11 个月前

A few months ago we set out to understand exactly how our models can be used to accelerate progress in robotics and other AI systems that don’t just represent the world, but are actually able to interact with it. We’ve been collaborating with leading companies in the space and we’ve developed an initial approach that we're now opening up to accelerate development across the industry. Our fifth announcement is GWM Robotics. GWM Robotics is a learned simulator that generates synthetic data for scalable robot training and policy evaluation, removing the bottlenecks of physical hardware.

A few months ago we set out to understand exactly how our models can be used to accelerate progress in robotics and other AI systems that don’t just represent the world, but are actually able to interact with it. We’ve been collaborating with leading companies in the space and we’ve developed an initial approach that we're now opening up to accelerate development across the industry. Our fifth announcement is GWM Robotics. GWM Robotics is a learned simulator that generates synthetic data for scalable robot training and policy evaluation, removing the bottlenecks of physical hardware.

Runway

35,040 次观看 • 9 个月前

Sergey Levine (Sergey Levine) is one of the world's top robotics researchers and co-founder of Physical Intelligence. We talked about where humanoid robotics is today, thoughts on the Chinese robotics ecosystem, and his predictions for future timelines. In this episode: • Current state of robotics and surprising capabilities so far • Chinese robotics compared to US ecosystem • If OpenAI and Anthropic got into robotics • His top robotics research paper recommendation • Predictions for when humanoid robotics will land Where to watch: • YouTube - • Spotify - • Apple Podcasts - • Transcript - Thank you to the sponsor of this episode for supporting my work: • WorkOS: makes your app Enterprise Ready with easy to use APIs to add SSO, SCIM, RBAC, and more in just a few lines of code, check them out at Chapters: 00:00 Intro 00:37 Where are we today 04:20 Most surprising capabilities so far 07:03 The most inspiring real world robotics 08:36 If OpenAI or Anthropic got into robotics 10:22 Chinese robotics 13:15 Will one lab breakout from the rest 16:59 Thoughts on a concrete roadmap 21:03 Generalization and demonstrating it 26:04 Types of data and which is best for robotics 34:34 Why humanoid robotics differs from Waymo 37:10 If humanoid robotics failed here is why 39:55 Are there hot take modeling architectures in robotics 42:05 Thoughts on AI safety in robotics 46:44 Top robotics research paper recommendation 49:35 Why is Boston Dynamics less top of mind 53:47 Advice for his younger self 56:42 Outro

Sergey Levine (Sergey Levine) is one of the world's top robotics researchers and co-founder of Physical Intelligence. We talked about where humanoid robotics is today, thoughts on the Chinese robotics ecosystem, and his predictions for future timelines. In this episode: • Current state of robotics and surprising capabilities so far • Chinese robotics compared to US ecosystem • If OpenAI and Anthropic got into robotics • His top robotics research paper recommendation • Predictions for when humanoid robotics will land Where to watch: • YouTube - • Spotify - • Apple Podcasts - • Transcript - Thank you to the sponsor of this episode for supporting my work: • WorkOS: makes your app Enterprise Ready with easy to use APIs to add SSO, SCIM, RBAC, and more in just a few lines of code, check them out at Chapters: 00:00 Intro 00:37 Where are we today 04:20 Most surprising capabilities so far 07:03 The most inspiring real world robotics 08:36 If OpenAI or Anthropic got into robotics 10:22 Chinese robotics 13:15 Will one lab breakout from the rest 16:59 Thoughts on a concrete roadmap 21:03 Generalization and demonstrating it 26:04 Types of data and which is best for robotics 34:34 Why humanoid robotics differs from Waymo 37:10 If humanoid robotics failed here is why 39:55 Are there hot take modeling architectures in robotics 42:05 Thoughts on AI safety in robotics 46:44 Top robotics research paper recommendation 49:35 Why is Boston Dynamics less top of mind 53:47 Advice for his younger self 56:42 Outro

Ryan Peterman

70,752 次观看 • 1 个月前

Unreal Engine 5 is becoming the go-to platform for robotics teams, not just as a simulator but as a synthetic data factory powering the next generation of robotics systems🦾 From synthetic data to sim-to-real transfer, read our piece on how UE5 is quietly reshaping how robots learn to see, move, and act in the real world:

Unreal Engine 5 is becoming the go-to platform for robotics teams, not just as a simulator but as a synthetic data factory powering the next generation of robotics systems🦾 From synthetic data to sim-to-real transfer, read our piece on how UE5 is quietly reshaping how robots learn to see, move, and act in the real world:

Unreal Engine

61,187 次观看 • 5 个月前

The open-source AI community just got a new home for their data workflows. 🤗 Hugging Face is now available in Adaptive Data. Pull datasets directly into a platform that evolves with the problems you're solving.

The open-source AI community just got a new home for their data workflows. 🤗 Hugging Face is now available in Adaptive Data. Pull datasets directly into a platform that evolves with the problems you're solving.

adaption

55,194 次观看 • 5 个月前

🤖 OrionX Robotics is actively engaged in a fundraising round to accelerate growth, scale operations, and advance the development of next-generation autonomous robotics systems. We look forward to partnering with investors who share our vision for the future of intelligent robotics.

🤖 OrionX Robotics is actively engaged in a fundraising round to accelerate growth, scale operations, and advance the development of next-generation autonomous robotics systems. We look forward to partnering with investors who share our vision for the future of intelligent robotics.

OrionX Robotics

12,345 次观看 • 3 个月前

Figure just crossed 86,000 weekly active users uploading data We're building the largest and most diverse robotics dataset in the world The website has a live map viewer showing each pixel of data as it's uploaded around the world

Figure just crossed 86,000 weekly active users uploading data We're building the largest and most diverse robotics dataset in the world The website has a live map viewer showing each pixel of data as it's uploaded around the world

Brett Adcock

113,943 次观看 • 13 天前

NVIDIA CEO Jensen Huang: Elon Musk is in a phenomenal position to lead the future of AI, autonomous vehicles, and humanoid robotics. “Collecting world data is very expensive, and Elon has a great advantage because, number one, his AI factory for his cars is fantastic, has a lot of NVIDIA gear in it, and he has a very large fleet of cars on the road that allows him to collect a lot of data. So I think he has just a phenomenal position, and he's been working on this for a long time, and so he's going to be in a great position to take advantage of it. I know Elon's attitude towards AI, and he's very optimistic about its future, and obviously he's working on some of the most important AI areas. xAI is working on foundation, cognitive intelligence AI. Tesla's working on autonomous vehicles, and Optimus is for humanoid robotics. These three areas of AI are the three most important areas.”

NVIDIA CEO Jensen Huang: Elon Musk is in a phenomenal position to lead the future of AI, autonomous vehicles, and humanoid robotics. “Collecting world data is very expensive, and Elon has a great advantage because, number one, his AI factory for his cars is fantastic, has a lot of NVIDIA gear in it, and he has a very large fleet of cars on the road that allows him to collect a lot of data. So I think he has just a phenomenal position, and he's been working on this for a long time, and so he's going to be in a great position to take advantage of it. I know Elon's attitude towards AI, and he's very optimistic about its future, and obviously he's working on some of the most important AI areas. xAI is working on foundation, cognitive intelligence AI. Tesla's working on autonomous vehicles, and Optimus is for humanoid robotics. These three areas of AI are the three most important areas.”

DogeDesigner

142,845 次观看 • 1 个月前

A big part of scaling robot learning to solve real-world problems is that we somehow need to get enough diverse, high-quality data to train our robots to perform useful things. GPT and its fellow large language models were bootstrapped and proved out on a massive dataset of real-world language data. Unfortunately, despite our best efforts, similarly massive datasets don’t really exist for robotics — so, in our unending pursuit of high-quality, useful data, we turn to simulation. I compared a couple recent works on sim-to-real robot manipulation, which discuss how to train perception-driven manipulation policies in simulation, in such a way that they’re useful in the real world. - DextraH-RGB, from NVIDIA - Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation, also from NVIDIA — specifically the GEAR lab - Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids, another GEAR lab paper - Local Policies Enable Zero-shot Long-Horizon Manipulation, from CMU (video from DextrAH-RGB)

A big part of scaling robot learning to solve real-world problems is that we somehow need to get enough diverse, high-quality data to train our robots to perform useful things. GPT and its fellow large language models were bootstrapped and proved out on a massive dataset of real-world language data. Unfortunately, despite our best efforts, similarly massive datasets don’t really exist for robotics — so, in our unending pursuit of high-quality, useful data, we turn to simulation. I compared a couple recent works on sim-to-real robot manipulation, which discuss how to train perception-driven manipulation policies in simulation, in such a way that they’re useful in the real world. - DextraH-RGB, from NVIDIA - Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation, also from NVIDIA — specifically the GEAR lab - Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids, another GEAR lab paper - Local Policies Enable Zero-shot Long-Horizon Manipulation, from CMU (video from DextrAH-RGB)

Chris Paxton

20,486 次观看 • 1 年前