Video yükleniyor...

Video Yüklenemedi

Bu video yüklenirken bir sorun oluştu. Bu geçici bir ağ sorunundan kaynaklanıyor olabilir veya video kullanılamıyor olabilir.

Ana Sayfaya Dön

Imitation learning has a data scarcity problem. Introducing EgoDex from Apple, the largest and most diverse dataset of dexterous human manipulation to date — 829 hours of egocentric video + paired 3D hand poses across 194 tasks. Now on arxiv: (1/4)

Ryan Hoque

1,976 subscribers

113,815 görüntüleme • 1 yıl önce •via X (Twitter)

Bilim & Teknoloji Haberler & Politika Eğitim

Anya Rossi• Live Now

Private livecam show

11 Yorum

Ryan Hoque profil fotoğrafı

Ryan Hoque1 yıl önce

Unlike teleoperation, egocentric video is passively scalable - like text and images on the Internet. We use Apple Vision Pro to collect video + precise pose annotations (unlike Ego4D, which lacks native pose data). This unlocks 5x the scale of existing large datasets like DROID.

Ryan Hoque profil fotoğrafı

Ryan Hoque1 yıl önce

We also propose new benchmarks and train imitation learning policies for dexterous trajectory prediction. Below are 30 Hz wrist and fingertip trajectories on the test set, where blue = ground truth, red = model predictions, and points get lighter up to 2 seconds in the future.

Ryan Hoque profil fotoğrafı

Ryan Hoque1 yıl önce

The full dataset is now publicly available to the community, access details are in the paper. Sample code for data loading is coming soon. Enjoy!

SecBriefs | Making Cybersecurity Simple profil fotoğrafı

SecBriefs | Making Cybersecurity Simple1 yıl önce

⚠️The average person generates 2.5 quintillion bytes of data annually. That's enough to fill 575,000 libraries!📚 This data is used to track, target, and manipulate you. #Cybersecurity matters.💡 Cybersecurity Dictionary for Everyone is on Apple Books:

Raul profil fotoğrafı

Raul1 yıl önce

Perfect for Optimus to learn new skills

RL profil fotoğrafı

RL1 yıl önce

Fyi the dataset links dont work: “ NoSuchKeyThe specified key does not exist.datasets/egodex/[filename].zip9C2FBJJ7FJHKHDT3Uk3l1oHoR9NeaNJdC7gInDjt5u8slFtW5lRt9wFR0MQIWNXIk4sTWiLGEYF22KUPQQ9X6CVC+UU=”

Michael Black profil fotoğrafı

Michael Black1 yıl önce

Looks great. You mention that it’s now public but I don’t find the link anywhere.

Hussein Lezzaik profil fotoğrafı

Hussein Lezzaik1 yıl önce

excellent work, congrats!

Soroush Nasiriany profil fotoğrafı

Soroush Nasiriany1 yıl önce

Congrats Ryan! Awesome work as always!!

Humanoids daily profil fotoğrafı

Humanoids daily1 yıl önce

Impressive! Interesting use of the Apple Vision Pro.

Idriel Vermillion profil fotoğrafı

Idriel Vermillion1 yıl önce

So hype

Benzer Videolar

Impressive on every dimension! Genesis AI has announced GENE-26.5, its first robotic foundation model, aimed at achieving human-level dexterous manipulation: - Single model, shared weights, handles egg cracking, lab pipetting, bimanual Rubik's Cube, smoothie making, and wire harnessing. Most tasks: under 1 hour of task-specific data, 1x real-world speed. - Genesis Hand 1.0: human-size, 20 active DoF, soft-contact skin. Paired glove gives a 1:1:1 mapping (glove, human hand, robot hand). 100x cheaper than teleop, 5x more data-efficient internally. - Data engine pulls from three sources: glove data, egocentric (head-cam) video, and large-scale internet video. The 1:1 hand-to-human match closes the embodiment gap, letting Genesis use video data more effectively than rivals. - Flow matching across vision, tactile, proprioception, and language inputs. Per Genesis, scaling data and compute directly improves zero-shot

Impressive on every dimension! Genesis AI has announced GENE-26.5, its first robotic foundation model, aimed at achieving human-level dexterous manipulation: - Single model, shared weights, handles egg cracking, lab pipetting, bimanual Rubik's Cube, smoothie making, and wire harnessing. Most tasks: under 1 hour of task-specific data, 1x real-world speed. - Genesis Hand 1.0: human-size, 20 active DoF, soft-contact skin. Paired glove gives a 1:1:1 mapping (glove, human hand, robot hand). 100x cheaper than teleop, 5x more data-efficient internally. - Data engine pulls from three sources: glove data, egocentric (head-cam) video, and large-scale internet video. The 1:1 hand-to-human match closes the embodiment gap, letting Genesis use video data more effectively than rivals. - Flow matching across vision, tactile, proprioception, and language inputs. Per Genesis, scaling data and compute directly improves zero-shot

The Humanoid Hub

94,935 görüntüleme • 1 ay önce

Happy to share what I’ve been working on since joining Genesis! GENE-26.5 is a one-of-a-kind, robotics-native multimodal foundation model that learns from diverse, in-the-wild data across modalities and outputs actions enabling a 54-DoF robot system to perform the most dexterous, long-horizon manipulation tasks to date—approaching human-level capability. This is the result of innovations across the full stack—data collection and processing, robot systems, model architecture, training strategies, and scalable evaluation infrastructure.

Happy to share what I’ve been working on since joining Genesis! GENE-26.5 is a one-of-a-kind, robotics-native multimodal foundation model that learns from diverse, in-the-wild data across modalities and outputs actions enabling a 54-DoF robot system to perform the most dexterous, long-horizon manipulation tasks to date—approaching human-level capability. This is the result of innovations across the full stack—data collection and processing, robot systems, model architecture, training strategies, and scalable evaluation infrastructure.

Zu Wang

17,823 görüntüleme • 1 ay önce

Introducing EgoVerse: an ecosystem for robot learning from egocentric human data. Built and tested by 4 research labs + 3 industry partners, EgoVerse enables both science and scaling 1300+ hrs, 240 scenes, 2000+ tasks, and growing Dataset design, findings, and ecosystem 🧵

Introducing EgoVerse: an ecosystem for robot learning from egocentric human data. Built and tested by 4 research labs + 3 industry partners, EgoVerse enables both science and scaling 1300+ hrs, 240 scenes, 2000+ tasks, and growing Dataset design, findings, and ecosystem 🧵

Danfei Xu

266,835 görüntüleme • 2 ay önce

⚡️EgoVerse is a first-of-its-kind, collaborative ecosystem for human-to-robot learning. The consortium leverages Project Aria to capture high-fidelity, egocentric human data — including 3D hand and head poses — to train next-gen robot manipulation policies. With over 1,300 hours of data across 2,000+ tasks, EgoVerse is a prime example of how the Aria Research Kit is being used by our partners to accelerate the future of embodied AI. Learn more: 🔗 📰 Apply for the Aria Research Kit: #MachineLearning #Robotics #ProjectAria #EgoVerse #ComputerVision Simar Kareer , Ryan Punamiya , Roger Qiu, Xiongyi Cai , Alexey Gavryushin

⚡️EgoVerse is a first-of-its-kind, collaborative ecosystem for human-to-robot learning. The consortium leverages Project Aria to capture high-fidelity, egocentric human data — including 3D hand and head poses — to train next-gen robot manipulation policies. With over 1,300 hours of data across 2,000+ tasks, EgoVerse is a prime example of how the Aria Research Kit is being used by our partners to accelerate the future of embodied AI. Learn more: 🔗 📰 Apply for the Aria Research Kit: #MachineLearning #Robotics #ProjectAria #EgoVerse #ComputerVision Simar Kareer , Ryan Punamiya , Roger Qiu, Xiongyi Cai , Alexey Gavryushin

Project Aria @Meta

17,750 görüntüleme • 1 ay önce

The Clone hand is a compact, biomimetic robotic hand designed to mimic human anatomy and perform a diverse array of manipulation tasks. Credit: Clone Robotics

The Clone hand is a compact, biomimetic robotic hand designed to mimic human anatomy and perform a diverse array of manipulation tasks. Credit: Clone Robotics

Tech Burrito

13,799 görüntüleme • 1 yıl önce

introducing Egocentric-1M. the largest egocentric video dataset in the world, and our next step in building the internet for physical AI.

introducing Egocentric-1M. the largest egocentric video dataset in the world, and our next step in building the internet for physical AI.

Eddy Xu @ CVPR

325,271 görüntüleme • 1 ay önce

Egocentric Videos for Robot Training 🧵 Researchers at NYU and UC Berkeley published research where they developed a system called EgoZero that trains robots using human demonstration videos recorded with smart glasses. 👓 The system converts first-person human actions into 3D point-based state-action representations. The policies executed on a gripper-equipped robot, achieved a 70% zero-shot success rate across seven manipulation tasks, with only 20 minutes of human data per task. EgoZero stands as one of the solid empirical proofs that egocentric smart-glass video collected from everyday human behavior can serve as powerful, scalable training data for real robot learning. Vincent Liu Ademi Adeniji

Egocentric Videos for Robot Training 🧵 Researchers at NYU and UC Berkeley published research where they developed a system called EgoZero that trains robots using human demonstration videos recorded with smart glasses. 👓 The system converts first-person human actions into 3D point-based state-action representations. The policies executed on a gripper-equipped robot, achieved a 70% zero-shot success rate across seven manipulation tasks, with only 20 minutes of human data per task. EgoZero stands as one of the solid empirical proofs that egocentric smart-glass video collected from everyday human behavior can serve as powerful, scalable training data for real robot learning. Vincent Liu Ademi Adeniji

VaderResearch

18,674 görüntüleme • 9 ay önce

Tactile feedback is one of the most important modalities in manipulation, but has been underutilized in dexterous hands. T-Dex is a framework for learning dexterous policies from tactile play data, beating vision and torque-based methods by 1.7x. 🧵👇

Tactile feedback is one of the most important modalities in manipulation, but has been underutilized in dexterous hands. T-Dex is a framework for learning dexterous policies from tactile play data, beating vision and torque-based methods by 1.7x. 🧵👇

Lerrel Pinto

84,258 görüntüleme • 3 yıl önce

Teaching robots to perform dexterous manipulation tasks currently requires teleoperation, which limits demonstration quality, speed, and scalability. Instead, why not use human videos? The problem is that a human hand isn’t a robot hand, so data must be retargeted using simulation to resolve issues like collisions and interpenetration when controlling the hand. In VideoManip, Hongyi Chen and co-authors built a system to solve this problem, taking in RGB videos of humans performing manipulation tasks and using them to create accurate simulations with which to learn robot policies. Watch episode #73 of RoboPapers, hosted by Michael Cho - Rbt/Acc and Chris Paxton, now to learn more!

Teaching robots to perform dexterous manipulation tasks currently requires teleoperation, which limits demonstration quality, speed, and scalability. Instead, why not use human videos? The problem is that a human hand isn’t a robot hand, so data must be retargeted using simulation to resolve issues like collisions and interpenetration when controlling the hand. In VideoManip, Hongyi Chen and co-authors built a system to solve this problem, taking in RGB videos of humans performing manipulation tasks and using them to create accurate simulations with which to learn robot policies. Watch episode #73 of RoboPapers, hosted by Michael Cho - Rbt/Acc and Chris Paxton, now to learn more!

RoboPapers

27,249 görüntüleme • 1 ay önce

Gen2Act: Casting language-conditioned manipulation as *human video generation* followed by *closed-loop policy execution conditioned on the generated video* enables solving diverse real-world tasks unseen in the robot dataset! 1/n

Gen2Act: Casting language-conditioned manipulation as human video generation followed by closed-loop policy execution conditioned on the generated video enables solving diverse real-world tasks unseen in the robot dataset! 1/n

Homanga Bharadhwaj

71,127 görüntüleme • 1 yıl önce

Scaling vision-language-action (VLA) models to high-DoF dexterous hands has long been a "holy grail" challenge due to the high-dimensional action space and data scarcity. As a wrap up of the year 2025, we are releasing GR-Dexter, a holistic hardware-model-data framework for generalist manipulation on a bimanual dexterous-hand robot. This is the first VLA system to achieve: ✅ High-DoF Control: Managing a 56-DoF bimanual system (21-DoF per hand). ✅ Long-Horizon Tasks with tool use: Vacuuming, bread serving with tongs, and table decluttering. ✅ Open-World Generalization: Robust performance with unseen objects and abstract instructions. Project page: ArXiv:

Scaling vision-language-action (VLA) models to high-DoF dexterous hands has long been a "holy grail" challenge due to the high-dimensional action space and data scarcity. As a wrap up of the year 2025, we are releasing GR-Dexter, a holistic hardware-model-data framework for generalist manipulation on a bimanual dexterous-hand robot. This is the first VLA system to achieve: ✅ High-DoF Control: Managing a 56-DoF bimanual system (21-DoF per hand). ✅ Long-Horizon Tasks with tool use: Vacuuming, bread serving with tongs, and table decluttering. ✅ Open-World Generalization: Robust performance with unseen objects and abstract instructions. Project page: ArXiv:

Xiao Ma

93,641 görüntüleme • 5 ay önce

Sim2Real RL for Vision-Based Dexterous Manipulation on Humanoids TLDR - we train a humanoid robot with two multifingered hands to perform a range of dexterous manipulation tasks robust generalization and high performance without human demonstration :D

Sim2Real RL for Vision-Based Dexterous Manipulation on Humanoids TLDR - we train a humanoid robot with two multifingered hands to perform a range of dexterous manipulation tasks robust generalization and high performance without human demonstration :D

Toru

49,425 görüntüleme • 1 yıl önce

YouTube is a LARGE dataset of demonstration videos to train Generalist robot agents, but lacks action data. How can we learn DEXTEROUS skills from them? In #CoRL2024, we explore the problem of learning a Generalist Piano Playing agent from YouTube videos.

YouTube is a LARGE dataset of demonstration videos to train Generalist robot agents, but lacks action data. How can we learn DEXTEROUS skills from them? In #CoRL2024, we explore the problem of learning a Generalist Piano Playing agent from YouTube videos.

Julen Urain

42,434 görüntüleme • 1 yıl önce

Happy to announce mimic-one: a Scalable Model Recipe for General Purpose Robot Dexterity, the culmination of years of research work in dexterous manipulation with imitation learning.

Happy to announce mimic-one: a Scalable Model Recipe for General Purpose Robot Dexterity, the culmination of years of research work in dexterous manipulation with imitation learning.

Elvis Nava

20,232 görüntüleme • 11 ay önce

Most of today's AI can see the world, but it doesn’t **feel** it. Capturing the sense of touch is crucial for dexterous robotic manipulation, user modeling, and understanding physical interactions. Introducing OpenTouch: bringing full-hand tactile sensing into real-world AI🖐️ OpenTouch is collected in-the-wild using tactile sensing gloves, hand pose tracking gloves, and egocentric glasses. It includes: • 5 hours of real-world data, • 3 hours densely annotated contact-rich interactions, • 2,900 curated interaction clips, • across 800 objects, 14 environments, and 29 grasp types. all open at:

Most of today's AI can see the world, but it doesn’t feel it. Capturing the sense of touch is crucial for dexterous robotic manipulation, user modeling, and understanding physical interactions. Introducing OpenTouch: bringing full-hand tactile sensing into real-world AI🖐️ OpenTouch is collected in-the-wild using tactile sensing gloves, hand pose tracking gloves, and egocentric glasses. It includes: • 5 hours of real-world data, • 3 hours densely annotated contact-rich interactions, • 2,900 curated interaction clips, • across 800 objects, 14 environments, and 29 grasp types. all open at:

Paul Liang

46,779 görüntüleme • 1 ay önce

today, we’re open sourcing the largest egocentric dataset in history. - 10,000 hours - 2,153 factory workers - 1,080,000,000 frames the era of data scaling in robotics is here. (thread)

today, we’re open sourcing the largest egocentric dataset in history. - 10,000 hours - 2,153 factory workers - 1,080,000,000 frames the era of data scaling in robotics is here. (thread)

Eddy Xu @ CVPR

2,318,589 görüntüleme • 6 ay önce

Introducing Nymeria, a dataset for improving egocentric human motion understanding for AR and VR devices from Project Aria @Meta

Introducing Nymeria, a dataset for improving egocentric human motion understanding for AR and VR devices from Project Aria @Meta

Reality Labs at Meta

14,576 görüntüleme • 1 yıl önce

Mecka is excited to power EgoVerse: a growing ecosystem for robot learning from egocentric human data Proven across 4 leading research labs, EgoVerse data consistently boosts robot performance We release the tools for anyone to collect data, inference/train, and contribute! 🧵

Mecka is excited to power EgoVerse: a growing ecosystem for robot learning from egocentric human data Proven across 4 leading research labs, EgoVerse data consistently boosts robot performance We release the tools for anyone to collect data, inference/train, and contribute! 🧵

Josh

101,695 görüntüleme • 2 ay önce

To scale GEN-0 capabilities, we are constructing the largest and most diverse real-world manipulation dataset ever built 🌍 including every manipulation task humans can think of – from peeling potatoes, to threading bolts – spanning homes, bakeries, warehouses, factories, & more

To scale GEN-0 capabilities, we are constructing the largest and most diverse real-world manipulation dataset ever built 🌍 including every manipulation task humans can think of – from peeling potatoes, to threading bolts – spanning homes, bakeries, warehouses, factories, & more

Generalist

51,595 görüntüleme • 7 ay önce

The problem with humanoid teleoperation is that it is expensive and difficult to scale Enter NVIDIA's EgoScale: - A VLA model pretrained on thousands hours of egocentric human videos. - Mid-trained via 50 hours of human + 4 hours of robot "play" data for human-robot alignment. - Fine-tuned with very few examples of task-specific robot teleoperation (100 or fewer per task). - Successfully transfers across 5-finger (Sharpa) and 3-finger (Unitree G1) robot hands. - Performance scales predictably as data increases.

The problem with humanoid teleoperation is that it is expensive and difficult to scale Enter NVIDIA's EgoScale: - A VLA model pretrained on thousands hours of egocentric human videos. - Mid-trained via 50 hours of human + 4 hours of robot "play" data for human-robot alignment. - Fine-tuned with very few examples of task-specific robot teleoperation (100 or fewer per task). - Successfully transfers across 5-finger (Sharpa) and 3-finger (Unitree G1) robot hands. - Performance scales predictably as data increases.

The Humanoid Hub

44,441 görüntüleme • 3 ay önce