Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Introduce EgoInfinity: a web-scale (14.6 yrs, 142M clips) data engine that automatically lifts Youtube videos into 4D hand-object interaction, and retargets for robot learning Not only a dataset, but a modular, upgradable engine HF Space:

Vector Wang

2,529 subscribers

108,164 views • 4 months ago •via X (Twitter)

Education Science & Technology

Anya Rossi• Live Now

Private livecam show

20 Comments

Vector Wang4 months ago

The dataset is open to anyone to use and contribute. The pipeline is modular and upgradable. Come scale things up even without any hardwares! Project Page: Paper: Codes: Dataset:

Damian @ The Lighthouse 🤖💡🏠4 months ago

@antoinemarcel just shared this in our gc. Looks great. Passing this on to a handful of people now.

Jerry Chéng4 months ago

Absolute game changer! Thanks for the open-sourcing

Vector Wang4 months ago

Thanks Jerry! Let’s build it together!

Michael Cho - Rbt/Acc4 months ago

Mind blown! Tks for putting this together!!

Vector Wang4 months ago

Just a start! As a modular platform, it still can be and needs to be upgraded and improved by us and everyone in the community! Just like XLeRobot, I am just offering a path for everyone to build upon.

ahad4 months ago

fire!!!

clankr4 months ago

The demos look fantastic. Creating an SE(3)-compliant coordinate space for every camera angle is the key. Here is our take:

Damian @ The Lighthouse 🤖💡🏠4 months ago

@nicks_robots

Tolu4 months ago

❤️

Damian @ The Lighthouse 🤖💡🏠4 months ago

@astridwilde

Damian @ The Lighthouse 🤖💡🏠4 months ago

@grantg07

wasabee_dai4 months ago

Ming blowing, it's a game changer! Thank you for it

Mahid4 months ago

NIce work! How long did you work on this?

🐱🐱💥4 months ago

Nice work! Super excited to see what comes ahead!

bnw4 months ago

vaporware or fr? most such repos hide a critical component, are you fr fr?

Damian @ The Lighthouse 🤖💡🏠4 months ago

@krithikas12

Milton Calderon Donefer4 months ago

very interesting!

Kavin Raj4 months ago

amazing work!

Alen Gabs4 months ago

@tommiekerssies

Related Videos

HDMI (HumanoiD iMitation for Interaction) is a framework enabling humanoid robots to learn whole-body object interaction skills from monocular RGB human videos. It extracts and retargets human poses and object trajectories using GVHMR and LocoMujoco, building reference datasets with contact annotations. The data is used to train an RL policy via robot-object co-tracking. HDMI achieved 67 consecutive door traversals.

HDMI (HumanoiD iMitation for Interaction) is a framework enabling humanoid robots to learn whole-body object interaction skills from monocular RGB human videos. It extracts and retargets human poses and object trajectories using GVHMR and LocoMujoco, building reference datasets with contact annotations. The data is used to train an RL policy via robot-object co-tracking. HDMI achieved 67 consecutive door traversals.

The Humanoid Hub

17,395 views • 1 year ago

YouTube is a LARGE dataset of demonstration videos to train Generalist robot agents, but lacks action data. How can we learn DEXTEROUS skills from them? In #CoRL2024, we explore the problem of learning a Generalist Piano Playing agent from YouTube videos.

YouTube is a LARGE dataset of demonstration videos to train Generalist robot agents, but lacks action data. How can we learn DEXTEROUS skills from them? In #CoRL2024, we explore the problem of learning a Generalist Piano Playing agent from YouTube videos.

Julen Urain

42,451 views • 2 years ago

HOI-DETR (Hand-Object Interaction) is a new zero-shot framework that integrates hand-object and object-object interaction -Hand + tool detections -Interaction links (+secondary object) Huge for robotics training 🔥🔥

HOI-DETR (Hand-Object Interaction) is a new zero-shot framework that integrates hand-object and object-object interaction -Hand + tool detections -Interaction links (+secondary object) Huge for robotics training 🔥🔥

mirrash.eth 🦇🔊

302,374 views • 4 months ago

Introducing Robo360 dataset 🚀, the first real-world omnispective multi-view and multi-material robotic manipulation dataset. Robo360 captures synchronized multi-modal robot-object interaction data (video, audio, proprioception, control) to facilitate research in dynamic novel-view synthesis, robot learning, physics understanding, world models, and more! Code and data coming soon!

Introducing Robo360 dataset 🚀, the first real-world omnispective multi-view and multi-material robotic manipulation dataset. Robo360 captures synchronized multi-modal robot-object interaction data (video, audio, proprioception, control) to facilitate research in dynamic novel-view synthesis, robot learning, physics understanding, world models, and more! Code and data coming soon!

Litian Liang

23,477 views • 2 years ago

Sikorsky H-34 unusual engine seen here in a video by AviationWalkarounds on YouTube. Unusual not only for its forward placement on an Helicopter, but also for being a radial engine.

Sikorsky H-34 unusual engine seen here in a video by AviationWalkarounds on YouTube. Unusual not only for its forward placement on an Helicopter, but also for being a radial engine.

Air Safety #OTD by Francisco Cunha

21,491 views • 8 months ago

Humans grasp objects with a purpose! Web2Grasp enables such functional grasping for dexterous robot hands via hand-object reconstruction from web images - without *any* robot teleop data collection 1/n

Humans grasp objects with a purpose! Web2Grasp enables such functional grasping for dexterous robot hands via hand-object reconstruction from web images - without any robot teleop data collection 1/n

Homanga Bharadhwaj @ CVPR

28,408 views • 1 year ago

Robot Learning needs 4D world models! Robot Learning needs 4D world models! Robot Learning needs 4D world models! We introduce TesserAct, a 4D embodied world model that can simulate how agents interact with the 3D world over time! We achieve this by simply extending a pre-trained 2D video generation model to jointly predict RGB, depth, and surface normals. It enables: 1️⃣ Much better policy learning in the wild 2️⃣ Temporal + spatial coherence in 4D dynamic prediction 3️⃣ Novel view synthesis for embodied scenes Code: Paper Link: Project page:

Robot Learning needs 4D world models! Robot Learning needs 4D world models! Robot Learning needs 4D world models! We introduce TesserAct, a 4D embodied world model that can simulate how agents interact with the 3D world over time! We achieve this by simply extending a pre-trained 2D video generation model to jointly predict RGB, depth, and surface normals. It enables: 1️⃣ Much better policy learning in the wild 2️⃣ Temporal + spatial coherence in 4D dynamic prediction 3️⃣ Novel view synthesis for embodied scenes Code: Paper Link: Project page:

Chuang Gan

43,265 views • 1 year ago

We built a robotics data engine with Astra-level performance at 125x lower cost. And we are open-sourcing the weights today! Introducing Flora, a suite of open models to parse the physical world and turn raw footage into rich training signals. Flora already powers our in-house data engine, labeling 200K hours of videos a day on 500 GPUs. We created it to scale the frontier for robot learning. Now you own the weights too. Flora gives you: - High-quality, timestamped captions to curate and filter human egocentric and robot data; - Dense progress rewards for quality control and RL post-training; - Rubrics scoring for real-world benchmarking Try it today, links in thread:

We built a robotics data engine with Astra-level performance at 125x lower cost. And we are open-sourcing the weights today! Introducing Flora, a suite of open models to parse the physical world and turn raw footage into rich training signals. Flora already powers our in-house data engine, labeling 200K hours of videos a day on 500 GPUs. We created it to scale the frontier for robot learning. Now you own the weights too. Flora gives you: - High-quality, timestamped captions to curate and filter human egocentric and robot data; - Dense progress rewards for quality control and RL post-training; - Rubrics scoring for real-world benchmarking Try it today, links in thread:

Jim Fan

161,827 views • 24 days ago

We can convert human videos to robot hand-object interaction trajectories in 4D. Enjoy! Paper: Website: Code: Authors:Bhawna Paliwal,Haritheja,Will Liang, Pieter Abbeel , Mahi Shafiullah 🏠🤖 , Jitendra MALIK

We can convert human videos to robot hand-object interaction trajectories in 4D. Enjoy! Paper: Website: Code: Authors:Bhawna Paliwal,Haritheja,Will Liang, Pieter Abbeel , Mahi Shafiullah 🏠🤖 , Jitendra MALIK

Jitendra MALIK

73,877 views • 4 months ago

HumanX: a scalable framework that converts single monocular human videos into agile, generalizable interaction skills for humanoid robots without task-specific rewards. Core parts: - XGen: retargets human motion + synthesizes diverse physically plausible training data (via physics-driven object trajectories and augmentation). - XMimic: unified imitation pipeline for robust generalization. Shows promising results for both Blind (no external sensors) and MoCap modes.

HumanX: a scalable framework that converts single monocular human videos into agile, generalizable interaction skills for humanoid robots without task-specific rewards. Core parts: - XGen: retargets human motion + synthesizes diverse physically plausible training data (via physics-driven object trajectories and augmentation). - XMimic: unified imitation pipeline for robust generalization. Shows promising results for both Blind (no external sensors) and MoCap modes.

The Humanoid Hub

20,358 views • 9 months ago

Perhaps the "GPT-3 moment" for humanoid robots lies within vast amounts of human video data. UC Berkeley (Abbeel + Malik group) just released Do as I Do: a pipeline that takes monocular RGB human videos (ego + exo)->reconstructs hand-object interactions->retargets them to multi-fingered robotic hands, generating executable robot trajectories. It pushes success rates on real-world clips from ~25% to 71%, outperforming prior SOTA. This kind of human video->robot-complete data approach feels increasingly important for closing the data flywheel in humanoid/embodied AI.

Perhaps the "GPT-3 moment" for humanoid robots lies within vast amounts of human video data. UC Berkeley (Abbeel + Malik group) just released Do as I Do: a pipeline that takes monocular RGB human videos (ego + exo)->reconstructs hand-object interactions->retargets them to multi-fingered robotic hands, generating executable robot trajectories. It pushes success rates on real-world clips from ~25% to 71%, outperforming prior SOTA. This kind of human video->robot-complete data approach feels increasingly important for closing the data flywheel in humanoid/embodied AI.

CyberRobo

12,107 views • 4 months ago

ViPE: Video Pose Engine for 3D Geometric Perception Contributions: • A robust and efficient framework, ViPE, for estimating camera parameters and dense depth from diverse, in-the-wild videos. • A system design that integrates the strengths of classical SLAM (efficiency, scalability) and learned models (robustness), with key improvements in efficiency, dynamic object handling, and depth quality over prior work. • A large-scale dataset of annotated videos, created using ViPE, to facilitate future research in 3D computer vision.

ViPE: Video Pose Engine for 3D Geometric Perception Contributions: • A robust and efficient framework, ViPE, for estimating camera parameters and dense depth from diverse, in-the-wild videos. • A system design that integrates the strengths of classical SLAM (efficiency, scalability) and learned models (robustness), with key improvements in efficiency, dynamic object handling, and depth quality over prior work. • A large-scale dataset of annotated videos, created using ViPE, to facilitate future research in 3D computer vision.

MrNeRF

42,688 views • 1 year ago

🔥🇳🇬💪🏾He don’t need no doggone engine jack! Bro power lifts that engine like it’s a case of beer! Straight up boss! He IS the engine!

🔥🇳🇬💪🏾He don’t need no doggone engine jack! Bro power lifts that engine like it’s a case of beer! Straight up boss! He IS the engine!

Mario Nawfal

125,825 views • 2 months ago

Introducing HO-Cap: A Capture System and Dataset for 3D Reconstruction and Pose Tracking of Hand-Object Interaction. We built a multi-camera system and a semi-automatic method for annotating the shape and pose of hands and objects Project page:

Introducing HO-Cap: A Capture System and Dataset for 3D Reconstruction and Pose Tracking of Hand-Object Interaction. We built a multi-camera system and a semi-automatic method for annotating the shape and pose of hands and objects Project page:

Yu Xiang

57,604 views • 1 year ago

Humanoid robotics is hitting a data wall. Teleop and mocap took us far, but they don’t scale to every object, terrain, and behavior. We’re releasing GRAIL: — a fully digital pipeline for generating loco-manipulation data before the robot moves. 🧵(1/8)

Humanoid robotics is hitting a data wall. Teleop and mocap took us far, but they don’t scale to every object, terrain, and behavior. We’re releasing GRAIL: — a fully digital pipeline for generating loco-manipulation data before the robot moves. 🧵(1/8)

Ye Yuan

44,468 views • 5 months ago

Large-scale 3D Scene Generation (all scenes are real-time rendered)!! Physically-grounded generative data without hallucinations is the missing link for robot learning and testing at scale. We introduce a method that directly generates large-scale 3D driving scenes with accurate geometry, allowing for causal view synthesis and generation with object permanence and explicit 3D geometry. This also allows for extreme trajectory extrapolation without failure! We also show that we can build fully data-driven simulators for end-to-end learning with this approach. Project: with the amazing team of Julian Ost, Amogh Joshi , Andrea Ramazzina, Maximilian Bömer, Mario Bijelic.

Large-scale 3D Scene Generation (all scenes are real-time rendered)!! Physically-grounded generative data without hallucinations is the missing link for robot learning and testing at scale. We introduce a method that directly generates large-scale 3D driving scenes with accurate geometry, allowing for causal view synthesis and generation with object permanence and explicit 3D geometry. This also allows for extreme trajectory extrapolation without failure! We also show that we can build fully data-driven simulators for end-to-end learning with this approach. Project: with the amazing team of Julian Ost, Amogh Joshi , Andrea Ramazzina, Maximilian Bömer, Mario Bijelic.

Felix Heide

27,853 views • 1 year ago

A hydraulic lifter is a small but powerful component that plays a huge role in keeping your engine smooth, quiet, and efficient. It sits between the camshaft and the engine's valves, automatically maintaining the perfect valve clearance while the engine runs.

A hydraulic lifter is a small but powerful component that plays a huge role in keeping your engine smooth, quiet, and efficient. It sits between the camshaft and the engine's valves, automatically maintaining the perfect valve clearance while the engine runs.

Mechanical Knowledge

11,373 views • 8 months ago