Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

We (Pantheon) have been working on a robotics data quality pipeline that uncovered a series of major problems in public robotics datasets, especially for world modeling. To improve the quality of data available to open-source robotics, we're publishing annotations for four of the most popular datasets. Some examples of... show more

calix

12,273 subscribers

83,197 Aufrufe • vor 9 Stunden •via X (Twitter)

Wissenschaft & Technologie

Anya Rossi• Live Now

Private livecam show

37 Kommentare

Profilbild von calix

calixvor 9 Stunden

1. Camera disconnection One of the wrist cameras in this episode froze, then disconnected. This would show up as pure noise to a world model.

Profilbild von calix

calixvor 9 Stunden

2. Missing gripper actions Despite the left gripper opening and closing, no actions are recorded. The world model is penalized for failing to predict the gripper's actions.

Profilbild von calix

calixvor 9 Stunden

3. Spiky actions In some datasets, action data contains sharp spikes. These are hard for a world action model with broad smoothness priors to fit.

Profilbild von calix

calixvor 9 Stunden

4. Shifted actions Action data is misaligned by a few frames for the grasp, but not the movement. This tricks the world model into predicting state changes too early or late.

Profilbild von calix

calixvor 9 Stunden

Read our full report: Credits to @sambhavgupta159 for uncovering and publishing this analysis. We're hiring for robotics foundation model research, software, data, and hardware engineering. We're also open to data collaborations.

Profilbild von "Leigh Marie" Braswell

"Leigh Marie" Braswellvor 9 Stunden

@PantheonInc Such important work! Data is king

Profilbild von Calvin Chen

Calvin Chenvor 9 Stunden

@PantheonInc bullish for looking at the data you’d be surprised how many don’t

Profilbild von Jannik Schilling

Jannik Schillingvor 8 Stunden

@PantheonInc Crazy how many issues we found in just two weeks...

Profilbild von carlo agostinelli

carlo agostinellivor 9 Stunden

@PantheonInc I’ve been waiting for this

Profilbild von Aryan Shah

Aryan Shahvor 9 Stunden

@PantheonInc Dream team!!

Profilbild von Nikhil Suresh

Nikhil Sureshvor 8 Stunden

@hamostaf04 @PantheonInc Great work on the data!

Profilbild von Amir Fischer

Amir Fischervor 9 Stunden

@PantheonInc @Jannikschg @arminsommer_ the european robotics mafia!

Profilbild von Kushal Thaman

Kushal Thamanvor 8 Stunden

@PantheonInc bot see good data, bot happy :)

Profilbild von Yang Fan Yun

Yang Fan Yunvor 9 Stunden

@PantheonInc let’s goooooo @calixo888 @nettedjay !!!!!! 🚀

Profilbild von Keval Shah

Keval Shahvor 8 Stunden

@PantheonInc Interesting read! Congrats to the team @_varunnair great to see you in action :)

Profilbild von arminsommer

arminsommervor 8 Stunden

@PantheonInc @sambhavgupta159 is a beast! very nice work!

Profilbild von devansh

devanshvor 9 Stunden

@PantheonInc cool!!

Profilbild von Sonith

Sonithvor 8 Stunden

@PantheonInc 🙏

Profilbild von nathan chen

nathan chenvor 8 Stunden

@PantheonInc amazing read! cong to the team :D @Jannikschg

Profilbild von Sarah Su

Sarah Suvor 9 Stunden

@PantheonInc this is super cool!

Profilbild von Henry Yin

Henry Yinvor 8 Stunden

@PantheonInc KALIX!!

Profilbild von Amulya Harish

Amulya Harishvor 9 Stunden

@PantheonInc The beginning

Profilbild von Victor Li

Victor Livor 9 Stunden

@PantheonInc 🔥🔥

Profilbild von Henry Lien

Henry Lienvor 9 Stunden

@PantheonInc 🚀🚀🚀

Profilbild von npjd

npjdvor 9 Stunden

@PantheonInc so amazing calix

Profilbild von Osmond

Osmondvor 8 Stunden

@PantheonInc quietly cooking!!

Profilbild von OnchainLoving.ETH

OnchainLoving.ETHvor 8 Stunden

@PantheonInc finally some real data rigor

Profilbild von Varun Nair

Varun Nairvor 8 Stunden

@PantheonInc Data Interpretability is huge. These are fixable issues and the human data community should know the aspects that must be improved. Overall great for the community as a whole!

Profilbild von Benedict Wolff

Benedict Wolffvor 8 Stunden

@PantheonInc Crushing it!

Profilbild von Eric Li

Eric Livor 8 Stunden

@PantheonInc Well done Mr Gupta @sambhavgupta159

Profilbild von hamza mostafa

hamza mostafavor 8 Stunden

@PantheonInc very cool work!!

Profilbild von Kole Lee

Kole Leevor 8 Stunden

@PantheonInc letsgoo

Profilbild von Daniel Ceballos

Daniel Ceballosvor 9 Stunden

@PantheonInc Awesome results!!!

Profilbild von Zain Javaid

Zain Javaidvor 8 Stunden

@PantheonInc awesome work! congrats pantheon

Profilbild von joseph

josephvor 8 Stunden

@PantheonInc long time coming 💪

Profilbild von Alexander Yevchenko

Alexander Yevchenkovor 8 Stunden

@PantheonInc thank you!!

Profilbild von Zuhair Lakhani

Zuhair Lakhanivor 8 Stunden

@PantheonInc Thank you for looking at the data

Ähnliche Videos

Today we're launching Hebbian Robotics (YC S26) Hebbian Robotics (YC S26) . We build APIs for evaluating data quality, without training a robotics model. Data vendors use our API to search and analyze Physical AI data at scale, getting on-demand quality signals and metrics without managing infrastructure. High-quality data is the bottleneck to the next frontier of intelligence in robotics. Buyers want data they can trust. Data providers are focused on operational excellence. They don't have the bandwidth to understand the quality of their data at scale. We're fixing this. Vendors use our APIs to understand the impact of their decisions on their data quality: - Buyers demanding per-collector hour limits? Prove compliance while we stay ahead of the workarounds. - Changed your collection SOPs? Monitor for quality drift and regression. - Suspect duplicate samples? Verify with a single API call. Before Hebbian, I trained AI models for industrial cleaning robots, and Kingston Kuan spent years building large-scale infrastructure at Jane Street.

Today we're launching Hebbian Robotics (YC S26) Hebbian Robotics (YC S26) . We build APIs for evaluating data quality, without training a robotics model. Data vendors use our API to search and analyze Physical AI data at scale, getting on-demand quality signals and metrics without managing infrastructure. High-quality data is the bottleneck to the next frontier of intelligence in robotics. Buyers want data they can trust. Data providers are focused on operational excellence. They don't have the bandwidth to understand the quality of their data at scale. We're fixing this. Vendors use our APIs to understand the impact of their decisions on their data quality: - Buyers demanding per-collector hour limits? Prove compliance while we stay ahead of the workarounds. - Changed your collection SOPs? Monitor for quality drift and regression. - Suspect duplicate samples? Verify with a single API call. Before Hebbian, I trained AI models for industrial cleaning robots, and Kingston Kuan spent years building large-scale infrastructure at Jane Street.

Brandon Ong

166,623 Aufrufe • vor 1 Monat

1/ The data parsing market for robotics is just emerging, and it’s already valued in the tens of billions. Data parsing to build data models and software for Robotics is a $20B market in 2024 and is expected to reach $150B+ by 2034 Cherry AI Robotics has built a data parsing tool for robotics, which goes live next week, and will effectively make this project one of the pioneers in data parsing for robotics.

1/ The data parsing market for robotics is just emerging, and it’s already valued in the tens of billions. Data parsing to build data models and software for Robotics is a $20B market in 2024 and is expected to reach $150B+ by 2034 Cherry AI Robotics has built a data parsing tool for robotics, which goes live next week, and will effectively make this project one of the pioneers in data parsing for robotics.

CherryAI Robotics

90,786 Aufrufe • vor 1 Jahr

Most capable generalist robotics models today are closed or at best, open weights. But robotics won’t reach its ChatGPT moment without real openness. That GPT moment was built on years of open tools and datasets such as Python, PyTorch, ImageNet and more, that let researchers inspect, reproduce, and build. Today, we’re introducing MolmoAct 2: a fully open-source action reasoning model for real-world robotics. We rethought and reshaped everything! 🧵👇

Most capable generalist robotics models today are closed or at best, open weights. But robotics won’t reach its ChatGPT moment without real openness. That GPT moment was built on years of open tools and datasets such as Python, PyTorch, ImageNet and more, that let researchers inspect, reproduce, and build. Today, we’re introducing MolmoAct 2: a fully open-source action reasoning model for real-world robotics. We rethought and reshaped everything! 🧵👇

Jiafei Duan

117,636 Aufrufe • vor 4 Monaten

Building valuable datasets takes dedication, skill and expertise. From robotics data to trend forecasting, we incubate cutting-edge datasets that generate lasting real-world value. Vana connects passionate entrepreneurs with our network of top backers, experts and data buyers. Thank you to Hub71, ecosystem mentors, and partners for joining us at the Vana Founders' Intensive in Abu Dhabi. Real builders. Real business. That's the Vana way.

Building valuable datasets takes dedication, skill and expertise. From robotics data to trend forecasting, we incubate cutting-edge datasets that generate lasting real-world value. Vana connects passionate entrepreneurs with our network of top backers, experts and data buyers. Thank you to Hub71, ecosystem mentors, and partners for joining us at the Vana Founders' Intensive in Abu Dhabi. Real builders. Real business. That's the Vana way.

vana

15,816 Aufrufe • vor 1 Jahr

World Labs CEO Dr. Fei-Fei Li & SceniX Co-Founder Yunzhu Li on the data bottleneck in robotics and how world models help: "The lack of data in training, the lack of data in evaluation, this is very, very different from language models, where data is abundant on the internet." "And we know that in order for robotics to work, we have to somehow unlock the power of scaling law. But where does that come from?" "It's a profound problem that everybody's battling with in robotics." "We see a lot of unlock in being able to do this whole process through the modeling of the environments." "Being able to create these digital worlds... that's just going to unlock so much more potential for being able to replace all the costly and the unsafe data in the real environments with the data generated from the worlds for robots to be able to do scalable learning and evaluations." Fei-Fei Li Yunzhu Li martin_casado

World Labs CEO Dr. Fei-Fei Li & SceniX Co-Founder Yunzhu Li on the data bottleneck in robotics and how world models help: "The lack of data in training, the lack of data in evaluation, this is very, very different from language models, where data is abundant on the internet." "And we know that in order for robotics to work, we have to somehow unlock the power of scaling law. But where does that come from?" "It's a profound problem that everybody's battling with in robotics." "We see a lot of unlock in being able to do this whole process through the modeling of the environments." "Being able to create these digital worlds... that's just going to unlock so much more potential for being able to replace all the costly and the unsafe data in the real environments with the data generated from the worlds for robots to be able to do scalable learning and evaluations." Fei-Fei Li Yunzhu Li martin_casado

a16z

77,065 Aufrufe • vor 1 Monat

New meta on the horizon - AI x Robotics?👀 "The ChatGPT moment for general robotics is just around the corner..." – Jensen Huang, CEO of NVIDIA Most profitable wallet on 7D leaderboard by Nansen 🧭 loaded a few Robotic plays: 🔷 $SAM - Crowdsourcing real-world datasets with robotic gaming 🔷 $HIBER - robust AI Infrastructure 🔷 $BREW - Intersection of robotics × AI × crypto 🔷 $MROBOTICS - mobile gaming and robotics company Another quality project to research: $XRT What else did I miss?🤔

New meta on the horizon - AI x Robotics?👀 "The ChatGPT moment for general robotics is just around the corner..." – Jensen Huang, CEO of NVIDIA Most profitable wallet on 7D leaderboard by Nansen 🧭 loaded a few Robotic plays: 🔷 $SAM - Crowdsourcing real-world datasets with robotic gaming 🔷 $HIBER - robust AI Infrastructure 🔷 $BREW - Intersection of robotics × AI × crypto 🔷 $MROBOTICS - mobile gaming and robotics company Another quality project to research: $XRT What else did I miss?🤔

Crypto Gideon(🧠,🧠)

15,027 Aufrufe • vor 1 Jahr

Very excited to share what our research team at Covariant has been working on: RFM-1, our latest Robotics Foundation Model. Built on top of high-quality multimodal data the Covariant robotics fleet has collected over the years, this project embodies our commitment to pushing the boundaries of how AI can interact with and understand the physical world. We're eager to share this milestone with the community and invite feedback and collaboration. Read more at

Very excited to share what our research team at Covariant has been working on: RFM-1, our latest Robotics Foundation Model. Built on top of high-quality multimodal data the Covariant robotics fleet has collected over the years, this project embodies our commitment to pushing the boundaries of how AI can interact with and understand the physical world. We're eager to share this milestone with the community and invite feedback and collaboration. Read more at

Rocky Duan

31,887 Aufrufe • vor 2 Jahren

Co-training has become a key part of the recipe for training large robotics models; it means that you mix some proportion of real robot data with other data sources, like simulation or egocentric human video data. This is especially important because robotics data tends to lack diversity which can be somewhat compensated for by the inclusion of these other modalities. And yet there has not been a sizable study on what constitute good practices for cotraining until now! We talk to Fanqi Lin and Jose Barreiros about their new work, a massive study which evaluated 89 policies over thousands of rollouts to tell us which forms of co-training were most useful for robotics. Watch episode 70 of RoboPapers, with Michael Cho and Chris Paxton, now!

Co-training has become a key part of the recipe for training large robotics models; it means that you mix some proportion of real robot data with other data sources, like simulation or egocentric human video data. This is especially important because robotics data tends to lack diversity which can be somewhat compensated for by the inclusion of these other modalities. And yet there has not been a sizable study on what constitute good practices for cotraining until now! We talk to Fanqi Lin and Jose Barreiros about their new work, a massive study which evaluated 89 policies over thousands of rollouts to tell us which forms of co-training were most useful for robotics. Watch episode 70 of RoboPapers, with Michael Cho and Chris Paxton, now!

RoboPapers

19,357 Aufrufe • vor 5 Monaten

Robotics in the age of FMs, cloud and Starlink. We've been rethinking robot AI — cloud-first, LLM-first approach. Fast to deploy. Modular. Data-centric. Built for long-horizon reasoning. Paper: Blog: 🧵 #Robotics #AI #LLM #VLM

Robotics in the age of FMs, cloud and Starlink. We've been rethinking robot AI — cloud-first, LLM-first approach. Fast to deploy. Modular. Data-centric. Built for long-horizon reasoning. Paper: Blog: 🧵 #Robotics #AI #LLM #VLM

Ashish Kapoor

10,185 Aufrufe • vor 11 Monaten

The Open Robotics AI Forum came to life in Seoul, thanks to the energy of our contributors and participants. Together we’re shaping an open future for robotics.

The Open Robotics AI Forum came to life in Seoul, thanks to the energy of our contributors and participants. Together we’re shaping an open future for robotics.

OpenMind

23,633 Aufrufe • vor 11 Monaten

A few months ago we set out to understand exactly how our models can be used to accelerate progress in robotics and other AI systems that don’t just represent the world, but are actually able to interact with it. We’ve been collaborating with leading companies in the space and we’ve developed an initial approach that we're now opening up to accelerate development across the industry. Our fifth announcement is GWM Robotics. GWM Robotics is a learned simulator that generates synthetic data for scalable robot training and policy evaluation, removing the bottlenecks of physical hardware.

A few months ago we set out to understand exactly how our models can be used to accelerate progress in robotics and other AI systems that don’t just represent the world, but are actually able to interact with it. We’ve been collaborating with leading companies in the space and we’ve developed an initial approach that we're now opening up to accelerate development across the industry. Our fifth announcement is GWM Robotics. GWM Robotics is a learned simulator that generates synthetic data for scalable robot training and policy evaluation, removing the bottlenecks of physical hardware.

Runway

35,040 Aufrufe • vor 9 Monaten

Sergey Levine (Sergey Levine) is one of the world's top robotics researchers and co-founder of Physical Intelligence. We talked about where humanoid robotics is today, thoughts on the Chinese robotics ecosystem, and his predictions for future timelines. In this episode: • Current state of robotics and surprising capabilities so far • Chinese robotics compared to US ecosystem • If OpenAI and Anthropic got into robotics • His top robotics research paper recommendation • Predictions for when humanoid robotics will land Where to watch: • YouTube - • Spotify - • Apple Podcasts - • Transcript - Thank you to the sponsor of this episode for supporting my work: • WorkOS: makes your app Enterprise Ready with easy to use APIs to add SSO, SCIM, RBAC, and more in just a few lines of code, check them out at Chapters: 00:00 Intro 00:37 Where are we today 04:20 Most surprising capabilities so far 07:03 The most inspiring real world robotics 08:36 If OpenAI or Anthropic got into robotics 10:22 Chinese robotics 13:15 Will one lab breakout from the rest 16:59 Thoughts on a concrete roadmap 21:03 Generalization and demonstrating it 26:04 Types of data and which is best for robotics 34:34 Why humanoid robotics differs from Waymo 37:10 If humanoid robotics failed here is why 39:55 Are there hot take modeling architectures in robotics 42:05 Thoughts on AI safety in robotics 46:44 Top robotics research paper recommendation 49:35 Why is Boston Dynamics less top of mind 53:47 Advice for his younger self 56:42 Outro

Sergey Levine (Sergey Levine) is one of the world's top robotics researchers and co-founder of Physical Intelligence. We talked about where humanoid robotics is today, thoughts on the Chinese robotics ecosystem, and his predictions for future timelines. In this episode: • Current state of robotics and surprising capabilities so far • Chinese robotics compared to US ecosystem • If OpenAI and Anthropic got into robotics • His top robotics research paper recommendation • Predictions for when humanoid robotics will land Where to watch: • YouTube - • Spotify - • Apple Podcasts - • Transcript - Thank you to the sponsor of this episode for supporting my work: • WorkOS: makes your app Enterprise Ready with easy to use APIs to add SSO, SCIM, RBAC, and more in just a few lines of code, check them out at Chapters: 00:00 Intro 00:37 Where are we today 04:20 Most surprising capabilities so far 07:03 The most inspiring real world robotics 08:36 If OpenAI or Anthropic got into robotics 10:22 Chinese robotics 13:15 Will one lab breakout from the rest 16:59 Thoughts on a concrete roadmap 21:03 Generalization and demonstrating it 26:04 Types of data and which is best for robotics 34:34 Why humanoid robotics differs from Waymo 37:10 If humanoid robotics failed here is why 39:55 Are there hot take modeling architectures in robotics 42:05 Thoughts on AI safety in robotics 46:44 Top robotics research paper recommendation 49:35 Why is Boston Dynamics less top of mind 53:47 Advice for his younger self 56:42 Outro

Ryan Peterman

70,752 Aufrufe • vor 1 Monat

Unreal Engine 5 is becoming the go-to platform for robotics teams, not just as a simulator but as a synthetic data factory powering the next generation of robotics systems🦾 From synthetic data to sim-to-real transfer, read our piece on how UE5 is quietly reshaping how robots learn to see, move, and act in the real world:

Unreal Engine 5 is becoming the go-to platform for robotics teams, not just as a simulator but as a synthetic data factory powering the next generation of robotics systems🦾 From synthetic data to sim-to-real transfer, read our piece on how UE5 is quietly reshaping how robots learn to see, move, and act in the real world:

Unreal Engine

61,187 Aufrufe • vor 5 Monaten

The open-source AI community just got a new home for their data workflows. 🤗 Hugging Face is now available in Adaptive Data. Pull datasets directly into a platform that evolves with the problems you're solving.

The open-source AI community just got a new home for their data workflows. 🤗 Hugging Face is now available in Adaptive Data. Pull datasets directly into a platform that evolves with the problems you're solving.

adaption

55,194 Aufrufe • vor 5 Monaten

🤖 OrionX Robotics is actively engaged in a fundraising round to accelerate growth, scale operations, and advance the development of next-generation autonomous robotics systems. We look forward to partnering with investors who share our vision for the future of intelligent robotics.

🤖 OrionX Robotics is actively engaged in a fundraising round to accelerate growth, scale operations, and advance the development of next-generation autonomous robotics systems. We look forward to partnering with investors who share our vision for the future of intelligent robotics.

OrionX Robotics

12,345 Aufrufe • vor 3 Monaten

Figure just crossed 86,000 weekly active users uploading data We're building the largest and most diverse robotics dataset in the world The website has a live map viewer showing each pixel of data as it's uploaded around the world

Figure just crossed 86,000 weekly active users uploading data We're building the largest and most diverse robotics dataset in the world The website has a live map viewer showing each pixel of data as it's uploaded around the world

Brett Adcock

113,943 Aufrufe • vor 13 Tagen

NVIDIA CEO Jensen Huang: Elon Musk is in a phenomenal position to lead the future of AI, autonomous vehicles, and humanoid robotics. “Collecting world data is very expensive, and Elon has a great advantage because, number one, his AI factory for his cars is fantastic, has a lot of NVIDIA gear in it, and he has a very large fleet of cars on the road that allows him to collect a lot of data. So I think he has just a phenomenal position, and he's been working on this for a long time, and so he's going to be in a great position to take advantage of it. I know Elon's attitude towards AI, and he's very optimistic about its future, and obviously he's working on some of the most important AI areas. xAI is working on foundation, cognitive intelligence AI. Tesla's working on autonomous vehicles, and Optimus is for humanoid robotics. These three areas of AI are the three most important areas.”

NVIDIA CEO Jensen Huang: Elon Musk is in a phenomenal position to lead the future of AI, autonomous vehicles, and humanoid robotics. “Collecting world data is very expensive, and Elon has a great advantage because, number one, his AI factory for his cars is fantastic, has a lot of NVIDIA gear in it, and he has a very large fleet of cars on the road that allows him to collect a lot of data. So I think he has just a phenomenal position, and he's been working on this for a long time, and so he's going to be in a great position to take advantage of it. I know Elon's attitude towards AI, and he's very optimistic about its future, and obviously he's working on some of the most important AI areas. xAI is working on foundation, cognitive intelligence AI. Tesla's working on autonomous vehicles, and Optimus is for humanoid robotics. These three areas of AI are the three most important areas.”

DogeDesigner

142,845 Aufrufe • vor 1 Monat

A big part of scaling robot learning to solve real-world problems is that we somehow need to get enough diverse, high-quality data to train our robots to perform useful things. GPT and its fellow large language models were bootstrapped and proved out on a massive dataset of real-world language data. Unfortunately, despite our best efforts, similarly massive datasets don’t really exist for robotics — so, in our unending pursuit of high-quality, useful data, we turn to simulation. I compared a couple recent works on sim-to-real robot manipulation, which discuss how to train perception-driven manipulation policies in simulation, in such a way that they’re useful in the real world. - DextraH-RGB, from NVIDIA - Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation, also from NVIDIA — specifically the GEAR lab - Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids, another GEAR lab paper - Local Policies Enable Zero-shot Long-Horizon Manipulation, from CMU (video from DextrAH-RGB)

A big part of scaling robot learning to solve real-world problems is that we somehow need to get enough diverse, high-quality data to train our robots to perform useful things. GPT and its fellow large language models were bootstrapped and proved out on a massive dataset of real-world language data. Unfortunately, despite our best efforts, similarly massive datasets don’t really exist for robotics — so, in our unending pursuit of high-quality, useful data, we turn to simulation. I compared a couple recent works on sim-to-real robot manipulation, which discuss how to train perception-driven manipulation policies in simulation, in such a way that they’re useful in the real world. - DextraH-RGB, from NVIDIA - Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation, also from NVIDIA — specifically the GEAR lab - Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids, another GEAR lab paper - Local Policies Enable Zero-shot Long-Horizon Manipulation, from CMU (video from DextrAH-RGB)

Chris Paxton

20,486 Aufrufe • vor 1 Jahr