正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Introduce EgoInfinity: a web-scale (14.6 yrs, 142M clips) data engine that automatically lifts Youtube videos into 4D hand-object interaction, and retargets for robot learning Not only a dataset, but a modular, upgradable engine HF Space:

Vector Wang

2,529 subscribers

108,164 次观看 • 4 个月前 •via X (Twitter)

教育科学技术

Anya Rossi• Live Now

Private livecam show

20 条评论

Vector Wang 的头像

Vector Wang4 个月前

The dataset is open to anyone to use and contribute. The pipeline is modular and upgradable. Come scale things up even without any hardwares! Project Page: Paper: Codes: Dataset:

Damian @ The Lighthouse 🤖💡🏠 的头像

Damian @ The Lighthouse 🤖💡🏠4 个月前

@antoinemarcel just shared this in our gc. Looks great. Passing this on to a handful of people now.

Jerry Chéng 的头像

Jerry Chéng4 个月前

Absolute game changer! Thanks for the open-sourcing

Vector Wang 的头像

Vector Wang4 个月前

Thanks Jerry! Let’s build it together!

Michael Cho - Rbt/Acc 的头像

Michael Cho - Rbt/Acc4 个月前

Mind blown! Tks for putting this together!!

Vector Wang 的头像

Vector Wang4 个月前

Just a start! As a modular platform, it still can be and needs to be upgraded and improved by us and everyone in the community! Just like XLeRobot, I am just offering a path for everyone to build upon.

ahad 的头像

ahad4 个月前

fire!!!

clankr 的头像

clankr4 个月前

The demos look fantastic. Creating an SE(3)-compliant coordinate space for every camera angle is the key. Here is our take:

Damian @ The Lighthouse 🤖💡🏠 的头像

Damian @ The Lighthouse 🤖💡🏠4 个月前

@nicks_robots

Tolu 的头像

Tolu4 个月前

❤️

Damian @ The Lighthouse 🤖💡🏠 的头像

Damian @ The Lighthouse 🤖💡🏠4 个月前

@astridwilde

Damian @ The Lighthouse 🤖💡🏠 的头像

Damian @ The Lighthouse 🤖💡🏠4 个月前

@grantg07

wasabee_dai 的头像

wasabee_dai4 个月前

Ming blowing, it's a game changer! Thank you for it

Mahid 的头像

Mahid4 个月前

NIce work! How long did you work on this?

🐱🐱💥 的头像

🐱🐱💥4 个月前

Nice work! Super excited to see what comes ahead!

bnw 的头像

bnw4 个月前

vaporware or fr? most such repos hide a critical component, are you fr fr?

Damian @ The Lighthouse 🤖💡🏠 的头像

Damian @ The Lighthouse 🤖💡🏠4 个月前

@krithikas12

Milton Calderon Donefer 的头像

Milton Calderon Donefer4 个月前

very interesting!

Kavin Raj 的头像

Kavin Raj4 个月前

amazing work!

Alen Gabs 的头像

Alen Gabs4 个月前

@tommiekerssies

相关视频

HDMI (HumanoiD iMitation for Interaction) is a framework enabling humanoid robots to learn whole-body object interaction skills from monocular RGB human videos. It extracts and retargets human poses and object trajectories using GVHMR and LocoMujoco, building reference datasets with contact annotations. The data is used to train an RL policy via robot-object co-tracking. HDMI achieved 67 consecutive door traversals.

HDMI (HumanoiD iMitation for Interaction) is a framework enabling humanoid robots to learn whole-body object interaction skills from monocular RGB human videos. It extracts and retargets human poses and object trajectories using GVHMR and LocoMujoco, building reference datasets with contact annotations. The data is used to train an RL policy via robot-object co-tracking. HDMI achieved 67 consecutive door traversals.

The Humanoid Hub

17,395 次观看 • 1 年前

YouTube is a LARGE dataset of demonstration videos to train Generalist robot agents, but lacks action data. How can we learn DEXTEROUS skills from them? In #CoRL2024, we explore the problem of learning a Generalist Piano Playing agent from YouTube videos.

YouTube is a LARGE dataset of demonstration videos to train Generalist robot agents, but lacks action data. How can we learn DEXTEROUS skills from them? In #CoRL2024, we explore the problem of learning a Generalist Piano Playing agent from YouTube videos.

Julen Urain

42,451 次观看 • 2 年前

HOI-DETR (Hand-Object Interaction) is a new zero-shot framework that integrates hand-object and object-object interaction -Hand + tool detections -Interaction links (+secondary object) Huge for robotics training 🔥🔥

HOI-DETR (Hand-Object Interaction) is a new zero-shot framework that integrates hand-object and object-object interaction -Hand + tool detections -Interaction links (+secondary object) Huge for robotics training 🔥🔥

mirrash.eth 🦇🔊

302,374 次观看 • 4 个月前

Introducing Robo360 dataset 🚀, the first real-world omnispective multi-view and multi-material robotic manipulation dataset. Robo360 captures synchronized multi-modal robot-object interaction data (video, audio, proprioception, control) to facilitate research in dynamic novel-view synthesis, robot learning, physics understanding, world models, and more! Code and data coming soon!

Introducing Robo360 dataset 🚀, the first real-world omnispective multi-view and multi-material robotic manipulation dataset. Robo360 captures synchronized multi-modal robot-object interaction data (video, audio, proprioception, control) to facilitate research in dynamic novel-view synthesis, robot learning, physics understanding, world models, and more! Code and data coming soon!

Litian Liang

23,477 次观看 • 2 年前

Sikorsky H-34 unusual engine seen here in a video by AviationWalkarounds on YouTube. Unusual not only for its forward placement on an Helicopter, but also for being a radial engine.

Sikorsky H-34 unusual engine seen here in a video by AviationWalkarounds on YouTube. Unusual not only for its forward placement on an Helicopter, but also for being a radial engine.

Air Safety #OTD by Francisco Cunha

21,491 次观看 • 8 个月前

Humans grasp objects with a purpose! Web2Grasp enables such functional grasping for dexterous robot hands via hand-object reconstruction from web images - without *any* robot teleop data collection 1/n

Humans grasp objects with a purpose! Web2Grasp enables such functional grasping for dexterous robot hands via hand-object reconstruction from web images - without any robot teleop data collection 1/n

Homanga Bharadhwaj @ CVPR

28,408 次观看 • 1 年前

Robot Learning needs 4D world models! Robot Learning needs 4D world models! Robot Learning needs 4D world models! We introduce TesserAct, a 4D embodied world model that can simulate how agents interact with the 3D world over time! We achieve this by simply extending a pre-trained 2D video generation model to jointly predict RGB, depth, and surface normals. It enables: 1️⃣ Much better policy learning in the wild 2️⃣ Temporal + spatial coherence in 4D dynamic prediction 3️⃣ Novel view synthesis for embodied scenes Code: Paper Link: Project page:

Robot Learning needs 4D world models! Robot Learning needs 4D world models! Robot Learning needs 4D world models! We introduce TesserAct, a 4D embodied world model that can simulate how agents interact with the 3D world over time! We achieve this by simply extending a pre-trained 2D video generation model to jointly predict RGB, depth, and surface normals. It enables: 1️⃣ Much better policy learning in the wild 2️⃣ Temporal + spatial coherence in 4D dynamic prediction 3️⃣ Novel view synthesis for embodied scenes Code: Paper Link: Project page:

Chuang Gan

43,265 次观看 • 1 年前

We built a robotics data engine with Astra-level performance at 125x lower cost. And we are open-sourcing the weights today! Introducing Flora, a suite of open models to parse the physical world and turn raw footage into rich training signals. Flora already powers our in-house data engine, labeling 200K hours of videos a day on 500 GPUs. We created it to scale the frontier for robot learning. Now you own the weights too. Flora gives you: - High-quality, timestamped captions to curate and filter human egocentric and robot data; - Dense progress rewards for quality control and RL post-training; - Rubrics scoring for real-world benchmarking Try it today, links in thread:

We built a robotics data engine with Astra-level performance at 125x lower cost. And we are open-sourcing the weights today! Introducing Flora, a suite of open models to parse the physical world and turn raw footage into rich training signals. Flora already powers our in-house data engine, labeling 200K hours of videos a day on 500 GPUs. We created it to scale the frontier for robot learning. Now you own the weights too. Flora gives you: - High-quality, timestamped captions to curate and filter human egocentric and robot data; - Dense progress rewards for quality control and RL post-training; - Rubrics scoring for real-world benchmarking Try it today, links in thread:

Jim Fan

161,827 次观看 • 24 天前

We can convert human videos to robot hand-object interaction trajectories in 4D. Enjoy! Paper: Website: Code: Authors:Bhawna Paliwal,Haritheja,Will Liang, Pieter Abbeel , Mahi Shafiullah 🏠🤖 , Jitendra MALIK

We can convert human videos to robot hand-object interaction trajectories in 4D. Enjoy! Paper: Website: Code: Authors:Bhawna Paliwal,Haritheja,Will Liang, Pieter Abbeel , Mahi Shafiullah 🏠🤖 , Jitendra MALIK

Jitendra MALIK

73,877 次观看 • 4 个月前

HumanX: a scalable framework that converts single monocular human videos into agile, generalizable interaction skills for humanoid robots without task-specific rewards. Core parts: - XGen: retargets human motion + synthesizes diverse physically plausible training data (via physics-driven object trajectories and augmentation). - XMimic: unified imitation pipeline for robust generalization. Shows promising results for both Blind (no external sensors) and MoCap modes.

HumanX: a scalable framework that converts single monocular human videos into agile, generalizable interaction skills for humanoid robots without task-specific rewards. Core parts: - XGen: retargets human motion + synthesizes diverse physically plausible training data (via physics-driven object trajectories and augmentation). - XMimic: unified imitation pipeline for robust generalization. Shows promising results for both Blind (no external sensors) and MoCap modes.

The Humanoid Hub

20,358 次观看 • 9 个月前

Perhaps the "GPT-3 moment" for humanoid robots lies within vast amounts of human video data. UC Berkeley (Abbeel + Malik group) just released Do as I Do: a pipeline that takes monocular RGB human videos (ego + exo)->reconstructs hand-object interactions->retargets them to multi-fingered robotic hands, generating executable robot trajectories. It pushes success rates on real-world clips from ~25% to 71%, outperforming prior SOTA. This kind of human video->robot-complete data approach feels increasingly important for closing the data flywheel in humanoid/embodied AI.

Perhaps the "GPT-3 moment" for humanoid robots lies within vast amounts of human video data. UC Berkeley (Abbeel + Malik group) just released Do as I Do: a pipeline that takes monocular RGB human videos (ego + exo)->reconstructs hand-object interactions->retargets them to multi-fingered robotic hands, generating executable robot trajectories. It pushes success rates on real-world clips from ~25% to 71%, outperforming prior SOTA. This kind of human video->robot-complete data approach feels increasingly important for closing the data flywheel in humanoid/embodied AI.

CyberRobo

12,107 次观看 • 4 个月前

ViPE: Video Pose Engine for 3D Geometric Perception Contributions: • A robust and efficient framework, ViPE, for estimating camera parameters and dense depth from diverse, in-the-wild videos. • A system design that integrates the strengths of classical SLAM (efficiency, scalability) and learned models (robustness), with key improvements in efficiency, dynamic object handling, and depth quality over prior work. • A large-scale dataset of annotated videos, created using ViPE, to facilitate future research in 3D computer vision.

ViPE: Video Pose Engine for 3D Geometric Perception Contributions: • A robust and efficient framework, ViPE, for estimating camera parameters and dense depth from diverse, in-the-wild videos. • A system design that integrates the strengths of classical SLAM (efficiency, scalability) and learned models (robustness), with key improvements in efficiency, dynamic object handling, and depth quality over prior work. • A large-scale dataset of annotated videos, created using ViPE, to facilitate future research in 3D computer vision.

MrNeRF

42,688 次观看 • 1 年前

🔥🇳🇬💪🏾He don’t need no doggone engine jack! Bro power lifts that engine like it’s a case of beer! Straight up boss! He IS the engine!

🔥🇳🇬💪🏾He don’t need no doggone engine jack! Bro power lifts that engine like it’s a case of beer! Straight up boss! He IS the engine!

Mario Nawfal

125,825 次观看 • 2 个月前

Introducing HO-Cap: A Capture System and Dataset for 3D Reconstruction and Pose Tracking of Hand-Object Interaction. We built a multi-camera system and a semi-automatic method for annotating the shape and pose of hands and objects Project page:

Introducing HO-Cap: A Capture System and Dataset for 3D Reconstruction and Pose Tracking of Hand-Object Interaction. We built a multi-camera system and a semi-automatic method for annotating the shape and pose of hands and objects Project page:

Yu Xiang

57,604 次观看 • 1 年前

Humanoid robotics is hitting a data wall. Teleop and mocap took us far, but they don’t scale to every object, terrain, and behavior. We’re releasing GRAIL: — a fully digital pipeline for generating loco-manipulation data before the robot moves. 🧵(1/8)

Humanoid robotics is hitting a data wall. Teleop and mocap took us far, but they don’t scale to every object, terrain, and behavior. We’re releasing GRAIL: — a fully digital pipeline for generating loco-manipulation data before the robot moves. 🧵(1/8)

Ye Yuan

44,468 次观看 • 5 个月前

Large-scale 3D Scene Generation (all scenes are real-time rendered)!! Physically-grounded generative data without hallucinations is the missing link for robot learning and testing at scale. We introduce a method that directly generates large-scale 3D driving scenes with accurate geometry, allowing for causal view synthesis and generation with object permanence and explicit 3D geometry. This also allows for extreme trajectory extrapolation without failure! We also show that we can build fully data-driven simulators for end-to-end learning with this approach. Project: with the amazing team of Julian Ost, Amogh Joshi , Andrea Ramazzina, Maximilian Bömer, Mario Bijelic.

Large-scale 3D Scene Generation (all scenes are real-time rendered)!! Physically-grounded generative data without hallucinations is the missing link for robot learning and testing at scale. We introduce a method that directly generates large-scale 3D driving scenes with accurate geometry, allowing for causal view synthesis and generation with object permanence and explicit 3D geometry. This also allows for extreme trajectory extrapolation without failure! We also show that we can build fully data-driven simulators for end-to-end learning with this approach. Project: with the amazing team of Julian Ost, Amogh Joshi , Andrea Ramazzina, Maximilian Bömer, Mario Bijelic.

Felix Heide

27,853 次观看 • 1 年前

A hydraulic lifter is a small but powerful component that plays a huge role in keeping your engine smooth, quiet, and efficient. It sits between the camshaft and the engine's valves, automatically maintaining the perfect valve clearance while the engine runs.

A hydraulic lifter is a small but powerful component that plays a huge role in keeping your engine smooth, quiet, and efficient. It sits between the camshaft and the engine's valves, automatically maintaining the perfect valve clearance while the engine runs.

Mechanical Knowledge

11,373 次观看 • 8 个月前