Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Imitation learning has a data scarcity problem. Introducing EgoDex from Apple, the largest and most diverse dataset of dexterous human manipulation to date — 829 hours of egocentric video + paired 3D hand poses across 194 tasks. Now on arxiv: (1/4)

Ryan Hoque

1,976 subscribers

113,849 views • 1 year ago •via X (Twitter)

Science & Technology News & Politics Education

Anya Rossi• Live Now

Private livecam show

11 Comments

Ryan Hoque1 year ago

Unlike teleoperation, egocentric video is passively scalable - like text and images on the Internet. We use Apple Vision Pro to collect video + precise pose annotations (unlike Ego4D, which lacks native pose data). This unlocks 5x the scale of existing large datasets like DROID.

Ryan Hoque1 year ago

We also propose new benchmarks and train imitation learning policies for dexterous trajectory prediction. Below are 30 Hz wrist and fingertip trajectories on the test set, where blue = ground truth, red = model predictions, and points get lighter up to 2 seconds in the future.

Ryan Hoque1 year ago

The full dataset is now publicly available to the community, access details are in the paper. Sample code for data loading is coming soon. Enjoy!

SecBriefs | Making Cybersecurity Simple1 year ago

⚠️The average person generates 2.5 quintillion bytes of data annually. That's enough to fill 575,000 libraries!📚 This data is used to track, target, and manipulate you. #Cybersecurity matters.💡 Cybersecurity Dictionary for Everyone is on Apple Books:

Raul1 year ago

Perfect for Optimus to learn new skills

RL1 year ago

Fyi the dataset links dont work: “ NoSuchKeyThe specified key does not exist.datasets/egodex/[filename].zip9C2FBJJ7FJHKHDT3Uk3l1oHoR9NeaNJdC7gInDjt5u8slFtW5lRt9wFR0MQIWNXIk4sTWiLGEYF22KUPQQ9X6CVC+UU=”

Michael Black1 year ago

Looks great. You mention that it’s now public but I don’t find the link anywhere.

Hussein Lezzaik1 year ago

excellent work, congrats!

Soroush Nasiriany1 year ago

Congrats Ryan! Awesome work as always!!

Humanoids daily1 year ago

Impressive! Interesting use of the Apple Vision Pro.

Idriel Vermillion1 year ago

So hype

Related Videos

Impressive on every dimension! Genesis AI has announced GENE-26.5, its first robotic foundation model, aimed at achieving human-level dexterous manipulation: - Single model, shared weights, handles egg cracking, lab pipetting, bimanual Rubik's Cube, smoothie making, and wire harnessing. Most tasks: under 1 hour of task-specific data, 1x real-world speed. - Genesis Hand 1.0: human-size, 20 active DoF, soft-contact skin. Paired glove gives a 1:1:1 mapping (glove, human hand, robot hand). 100x cheaper than teleop, 5x more data-efficient internally. - Data engine pulls from three sources: glove data, egocentric (head-cam) video, and large-scale internet video. The 1:1 hand-to-human match closes the embodiment gap, letting Genesis use video data more effectively than rivals. - Flow matching across vision, tactile, proprioception, and language inputs. Per Genesis, scaling data and compute directly improves zero-shot

Impressive on every dimension! Genesis AI has announced GENE-26.5, its first robotic foundation model, aimed at achieving human-level dexterous manipulation: - Single model, shared weights, handles egg cracking, lab pipetting, bimanual Rubik's Cube, smoothie making, and wire harnessing. Most tasks: under 1 hour of task-specific data, 1x real-world speed. - Genesis Hand 1.0: human-size, 20 active DoF, soft-contact skin. Paired glove gives a 1:1:1 mapping (glove, human hand, robot hand). 100x cheaper than teleop, 5x more data-efficient internally. - Data engine pulls from three sources: glove data, egocentric (head-cam) video, and large-scale internet video. The 1:1 hand-to-human match closes the embodiment gap, letting Genesis use video data more effectively than rivals. - Flow matching across vision, tactile, proprioception, and language inputs. Per Genesis, scaling data and compute directly improves zero-shot

The Humanoid Hub

94,935 views • 1 month ago

Happy to share what I’ve been working on since joining Genesis! GENE-26.5 is a one-of-a-kind, robotics-native multimodal foundation model that learns from diverse, in-the-wild data across modalities and outputs actions enabling a 54-DoF robot system to perform the most dexterous, long-horizon manipulation tasks to date—approaching human-level capability. This is the result of innovations across the full stack—data collection and processing, robot systems, model architecture, training strategies, and scalable evaluation infrastructure.

Happy to share what I’ve been working on since joining Genesis! GENE-26.5 is a one-of-a-kind, robotics-native multimodal foundation model that learns from diverse, in-the-wild data across modalities and outputs actions enabling a 54-DoF robot system to perform the most dexterous, long-horizon manipulation tasks to date—approaching human-level capability. This is the result of innovations across the full stack—data collection and processing, robot systems, model architecture, training strategies, and scalable evaluation infrastructure.

Zu Wang

17,823 views • 1 month ago

Introducing EgoVerse: an ecosystem for robot learning from egocentric human data. Built and tested by 4 research labs + 3 industry partners, EgoVerse enables both science and scaling 1300+ hrs, 240 scenes, 2000+ tasks, and growing Dataset design, findings, and ecosystem 🧵

Introducing EgoVerse: an ecosystem for robot learning from egocentric human data. Built and tested by 4 research labs + 3 industry partners, EgoVerse enables both science and scaling 1300+ hrs, 240 scenes, 2000+ tasks, and growing Dataset design, findings, and ecosystem 🧵

Danfei Xu

267,447 views • 2 months ago

⚡️EgoVerse is a first-of-its-kind, collaborative ecosystem for human-to-robot learning. The consortium leverages Project Aria to capture high-fidelity, egocentric human data — including 3D hand and head poses — to train next-gen robot manipulation policies. With over 1,300 hours of data across 2,000+ tasks, EgoVerse is a prime example of how the Aria Research Kit is being used by our partners to accelerate the future of embodied AI. Learn more: 🔗 📰 Apply for the Aria Research Kit: #MachineLearning #Robotics #ProjectAria #EgoVerse #ComputerVision Simar Kareer , Ryan Punamiya , Roger Qiu, Xiongyi Cai , Alexey Gavryushin

⚡️EgoVerse is a first-of-its-kind, collaborative ecosystem for human-to-robot learning. The consortium leverages Project Aria to capture high-fidelity, egocentric human data — including 3D hand and head poses — to train next-gen robot manipulation policies. With over 1,300 hours of data across 2,000+ tasks, EgoVerse is a prime example of how the Aria Research Kit is being used by our partners to accelerate the future of embodied AI. Learn more: 🔗 📰 Apply for the Aria Research Kit: #MachineLearning #Robotics #ProjectAria #EgoVerse #ComputerVision Simar Kareer , Ryan Punamiya , Roger Qiu, Xiongyi Cai , Alexey Gavryushin

Project Aria @Meta

17,847 views • 1 month ago

The Clone hand is a compact, biomimetic robotic hand designed to mimic human anatomy and perform a diverse array of manipulation tasks. Credit: Clone Robotics

The Clone hand is a compact, biomimetic robotic hand designed to mimic human anatomy and perform a diverse array of manipulation tasks. Credit: Clone Robotics

Tech Burrito

13,799 views • 1 year ago

introducing Egocentric-1M. the largest egocentric video dataset in the world, and our next step in building the internet for physical AI.

introducing Egocentric-1M. the largest egocentric video dataset in the world, and our next step in building the internet for physical AI.

Eddy Xu

326,317 views • 2 months ago

Egocentric Videos for Robot Training 🧵 Researchers at NYU and UC Berkeley published research where they developed a system called EgoZero that trains robots using human demonstration videos recorded with smart glasses. 👓 The system converts first-person human actions into 3D point-based state-action representations. The policies executed on a gripper-equipped robot, achieved a 70% zero-shot success rate across seven manipulation tasks, with only 20 minutes of human data per task. EgoZero stands as one of the solid empirical proofs that egocentric smart-glass video collected from everyday human behavior can serve as powerful, scalable training data for real robot learning. Vincent Liu Ademi Adeniji

Egocentric Videos for Robot Training 🧵 Researchers at NYU and UC Berkeley published research where they developed a system called EgoZero that trains robots using human demonstration videos recorded with smart glasses. 👓 The system converts first-person human actions into 3D point-based state-action representations. The policies executed on a gripper-equipped robot, achieved a 70% zero-shot success rate across seven manipulation tasks, with only 20 minutes of human data per task. EgoZero stands as one of the solid empirical proofs that egocentric smart-glass video collected from everyday human behavior can serve as powerful, scalable training data for real robot learning. Vincent Liu Ademi Adeniji

VaderResearch

23,055 views • 9 months ago

Tactile feedback is one of the most important modalities in manipulation, but has been underutilized in dexterous hands. T-Dex is a framework for learning dexterous policies from tactile play data, beating vision and torque-based methods by 1.7x. 🧵👇

Tactile feedback is one of the most important modalities in manipulation, but has been underutilized in dexterous hands. T-Dex is a framework for learning dexterous policies from tactile play data, beating vision and torque-based methods by 1.7x. 🧵👇

Lerrel Pinto

84,258 views • 3 years ago

Teaching robots to perform dexterous manipulation tasks currently requires teleoperation, which limits demonstration quality, speed, and scalability. Instead, why not use human videos? The problem is that a human hand isn’t a robot hand, so data must be retargeted using simulation to resolve issues like collisions and interpenetration when controlling the hand. In VideoManip, Hongyi Chen and co-authors built a system to solve this problem, taking in RGB videos of humans performing manipulation tasks and using them to create accurate simulations with which to learn robot policies. Watch episode #73 of RoboPapers, hosted by Michael Cho - Rbt/Acc and Chris Paxton, now to learn more!

Teaching robots to perform dexterous manipulation tasks currently requires teleoperation, which limits demonstration quality, speed, and scalability. Instead, why not use human videos? The problem is that a human hand isn’t a robot hand, so data must be retargeted using simulation to resolve issues like collisions and interpenetration when controlling the hand. In VideoManip, Hongyi Chen and co-authors built a system to solve this problem, taking in RGB videos of humans performing manipulation tasks and using them to create accurate simulations with which to learn robot policies. Watch episode #73 of RoboPapers, hosted by Michael Cho - Rbt/Acc and Chris Paxton, now to learn more!

RoboPapers

27,249 views • 1 month ago

Gen2Act: Casting language-conditioned manipulation as *human video generation* followed by *closed-loop policy execution conditioned on the generated video* enables solving diverse real-world tasks unseen in the robot dataset! 1/n

Gen2Act: Casting language-conditioned manipulation as human video generation followed by closed-loop policy execution conditioned on the generated video enables solving diverse real-world tasks unseen in the robot dataset! 1/n

Homanga Bharadhwaj @ CVPR

71,132 views • 1 year ago

Sim2Real RL for Vision-Based Dexterous Manipulation on Humanoids TLDR - we train a humanoid robot with two multifingered hands to perform a range of dexterous manipulation tasks robust generalization and high performance without human demonstration :D

Sim2Real RL for Vision-Based Dexterous Manipulation on Humanoids TLDR - we train a humanoid robot with two multifingered hands to perform a range of dexterous manipulation tasks robust generalization and high performance without human demonstration :D

Toru

49,425 views • 1 year ago

Scaling vision-language-action (VLA) models to high-DoF dexterous hands has long been a "holy grail" challenge due to the high-dimensional action space and data scarcity. As a wrap up of the year 2025, we are releasing GR-Dexter, a holistic hardware-model-data framework for generalist manipulation on a bimanual dexterous-hand robot. This is the first VLA system to achieve: ✅ High-DoF Control: Managing a 56-DoF bimanual system (21-DoF per hand). ✅ Long-Horizon Tasks with tool use: Vacuuming, bread serving with tongs, and table decluttering. ✅ Open-World Generalization: Robust performance with unseen objects and abstract instructions. Project page: ArXiv:

Scaling vision-language-action (VLA) models to high-DoF dexterous hands has long been a "holy grail" challenge due to the high-dimensional action space and data scarcity. As a wrap up of the year 2025, we are releasing GR-Dexter, a holistic hardware-model-data framework for generalist manipulation on a bimanual dexterous-hand robot. This is the first VLA system to achieve: ✅ High-DoF Control: Managing a 56-DoF bimanual system (21-DoF per hand). ✅ Long-Horizon Tasks with tool use: Vacuuming, bread serving with tongs, and table decluttering. ✅ Open-World Generalization: Robust performance with unseen objects and abstract instructions. Project page: ArXiv:

Xiao Ma

93,641 views • 5 months ago

YouTube is a LARGE dataset of demonstration videos to train Generalist robot agents, but lacks action data. How can we learn DEXTEROUS skills from them? In #CoRL2024, we explore the problem of learning a Generalist Piano Playing agent from YouTube videos.

YouTube is a LARGE dataset of demonstration videos to train Generalist robot agents, but lacks action data. How can we learn DEXTEROUS skills from them? In #CoRL2024, we explore the problem of learning a Generalist Piano Playing agent from YouTube videos.

Julen Urain

42,434 views • 1 year ago

Happy to announce mimic-one: a Scalable Model Recipe for General Purpose Robot Dexterity, the culmination of years of research work in dexterous manipulation with imitation learning.

Happy to announce mimic-one: a Scalable Model Recipe for General Purpose Robot Dexterity, the culmination of years of research work in dexterous manipulation with imitation learning.

Elvis Nava

20,232 views • 11 months ago

Most of today's AI can see the world, but it doesn’t **feel** it. Capturing the sense of touch is crucial for dexterous robotic manipulation, user modeling, and understanding physical interactions. Introducing OpenTouch: bringing full-hand tactile sensing into real-world AI🖐️ OpenTouch is collected in-the-wild using tactile sensing gloves, hand pose tracking gloves, and egocentric glasses. It includes: • 5 hours of real-world data, • 3 hours densely annotated contact-rich interactions, • 2,900 curated interaction clips, • across 800 objects, 14 environments, and 29 grasp types. all open at:

Most of today's AI can see the world, but it doesn’t feel it. Capturing the sense of touch is crucial for dexterous robotic manipulation, user modeling, and understanding physical interactions. Introducing OpenTouch: bringing full-hand tactile sensing into real-world AI🖐️ OpenTouch is collected in-the-wild using tactile sensing gloves, hand pose tracking gloves, and egocentric glasses. It includes: • 5 hours of real-world data, • 3 hours densely annotated contact-rich interactions, • 2,900 curated interaction clips, • across 800 objects, 14 environments, and 29 grasp types. all open at:

Paul Liang

46,779 views • 2 months ago

today, we’re open sourcing the largest egocentric dataset in history. - 10,000 hours - 2,153 factory workers - 1,080,000,000 frames the era of data scaling in robotics is here. (thread)

today, we’re open sourcing the largest egocentric dataset in history. - 10,000 hours - 2,153 factory workers - 1,080,000,000 frames the era of data scaling in robotics is here. (thread)

Eddy Xu

2,320,429 views • 7 months ago

Introducing Nymeria, a dataset for improving egocentric human motion understanding for AR and VR devices from Project Aria @Meta

Introducing Nymeria, a dataset for improving egocentric human motion understanding for AR and VR devices from Project Aria @Meta

Reality Labs at Meta

14,576 views • 1 year ago

Mecka is excited to power EgoVerse: a growing ecosystem for robot learning from egocentric human data Proven across 4 leading research labs, EgoVerse data consistently boosts robot performance We release the tools for anyone to collect data, inference/train, and contribute! 🧵

Mecka is excited to power EgoVerse: a growing ecosystem for robot learning from egocentric human data Proven across 4 leading research labs, EgoVerse data consistently boosts robot performance We release the tools for anyone to collect data, inference/train, and contribute! 🧵

Josh

101,838 views • 2 months ago

To scale GEN-0 capabilities, we are constructing the largest and most diverse real-world manipulation dataset ever built 🌍 including every manipulation task humans can think of – from peeling potatoes, to threading bolts – spanning homes, bakeries, warehouses, factories, & more

To scale GEN-0 capabilities, we are constructing the largest and most diverse real-world manipulation dataset ever built 🌍 including every manipulation task humans can think of – from peeling potatoes, to threading bolts – spanning homes, bakeries, warehouses, factories, & more

Generalist

51,615 views • 7 months ago

Real-world robot data is expensive and slow to collect, creating a major challenge for humanoid development. 🤖 The NVIDIA GR00T N1.6 open vision language action model is pre-trained on a diverse mix of data, including thousands of hours of Stanford Vision and Learning Lab’s BEHAVIOR simulation data, which covers long-horizon everyday manipulation tasks. This diverse training is the key to robust cross-embodiment performance and real-world adaptability. 🌍 Read the blog 🔗

Real-world robot data is expensive and slow to collect, creating a major challenge for humanoid development. 🤖 The NVIDIA GR00T N1.6 open vision language action model is pre-trained on a diverse mix of data, including thousands of hours of Stanford Vision and Learning Lab’s BEHAVIOR simulation data, which covers long-horizon everyday manipulation tasks. This diverse training is the key to robust cross-embodiment performance and real-world adaptability. 🌍 Read the blog 🔗

NVIDIA Robotics

13,390 views • 4 months ago