Loading video...

Video Failed to Load

Go Home

Introduce EgoInfinity: a web-scale (14.6 yrs, 142M clips) data engine that automatically lifts Youtube videos into 4D hand-object interaction, and retargets for robot learning Not only a dataset, but a modular, upgradable engine HF Space:

108,060 views • 3 months ago •via X (Twitter)

20 Comments

Vector Wang's profile picture
Vector Wang3 months ago

The dataset is open to anyone to use and contribute. The pipeline is modular and upgradable. Come scale things up even without any hardwares! Project Page: Paper: Codes: Dataset:

Damian @ The Lighthouse 🤖💡🏠's profile picture
Damian @ The Lighthouse 🤖💡🏠3 months ago

@antoinemarcel just shared this in our gc. Looks great. Passing this on to a handful of people now.

Jerry Chéng's profile picture
Jerry Chéng3 months ago

Absolute game changer! Thanks for the open-sourcing

Vector Wang's profile picture
Vector Wang3 months ago

Thanks Jerry! Let’s build it together!

Michael Cho - Rbt/Acc's profile picture
Michael Cho - Rbt/Acc3 months ago

Mind blown! Tks for putting this together!!

Vector Wang's profile picture
Vector Wang3 months ago

Just a start! As a modular platform, it still can be and needs to be upgraded and improved by us and everyone in the community! Just like XLeRobot, I am just offering a path for everyone to build upon.

ahad's profile picture
ahad3 months ago

fire!!!

clankr's profile picture
clankr3 months ago

The demos look fantastic. Creating an SE(3)-compliant coordinate space for every camera angle is the key. Here is our take:

Damian @ The Lighthouse 🤖💡🏠's profile picture
Damian @ The Lighthouse 🤖💡🏠3 months ago

@nicks_robots

Tolu's profile picture
Tolu3 months ago

❤️

Damian @ The Lighthouse 🤖💡🏠's profile picture
Damian @ The Lighthouse 🤖💡🏠3 months ago

@astridwilde

Damian @ The Lighthouse 🤖💡🏠's profile picture
Damian @ The Lighthouse 🤖💡🏠3 months ago

@grantg07

wasabee_dai's profile picture
wasabee_dai3 months ago

Ming blowing, it's a game changer! Thank you for it

Mahid's profile picture
Mahid3 months ago

NIce work! How long did you work on this?

🐱🐱💥's profile picture
🐱🐱💥3 months ago

Nice work! Super excited to see what comes ahead!

bnw's profile picture
bnw3 months ago

vaporware or fr? most such repos hide a critical component, are you fr fr?

Damian @ The Lighthouse 🤖💡🏠's profile picture
Damian @ The Lighthouse 🤖💡🏠3 months ago

@krithikas12

Milton Calderon Donefer's profile picture
Milton Calderon Donefer3 months ago

very interesting!

Kavin Raj's profile picture
Kavin Raj3 months ago

amazing work!

Alen Gabs's profile picture
Alen Gabs3 months ago

@tommiekerssies

Related Videos

I don’t know if we live in a Matrix, but I know for sure that robots will spend most of their lives in simulation. Let machines train machines. I’m excited to introduce DexMimicGen, a massive-scale synthetic data generator that enables a humanoid robot to learn complex skills from only a handful of human demonstrations. Yes, as few as 5! DexMimicGen addresses the biggest pain point in robotics: where do we get data? Unlike with LLMs, where vast amounts of texts are readily available, you cannot simply download motor control signals from the internet. So researchers teleoperate the robots to collect motion data via XR headsets. They have to repeat the same skill over and over and over again, because neural nets are data hungry. This is a very slow and uncomfortable process. At NVIDIA, we believe the majority of high-quality tokens for robot foundation models will come from simulation. What DexMimicGen does is to trade GPU compute time for human time. It takes one motion trajectory from human, and multiplies into 1000s of new trajectories. A robot brain trained on this augmented dataset will generalize far better in the real world. Think of DexMimicGen as a learning signal amplifier. It maps a small dataset to a large (de facto infinite) dataset, using physics simulation in the loop. In this way, we free humans from babysitting the bots all day. The future of robot data is generative. The future of the entire robot learning pipeline will also be generative. 🧵

Jim Fan

165,246 views • 1 year ago

The Machine That Learns The Law Behind The Data A very very interesting US Patent US10963540B2 - Physics Informed Learning Machine describes a learning system that does not begin with data alone. It begins with a physical model, usually written as a differential equation (or PDE) dx/dt = f(x,t) A normal Machine Learning model sees scattered data and tries to fit it. A physics-informed learning machine starts with a law. Then it treats the data as evidence that updates what the model believes about the physical system. For this application, I use the patent idea on NASA C-MAPSS Turbofan engine data. The machine watches multivariate telemetry from a degrading engine and infers a hidden health state that is not measured directly. From that posterior belief, it estimates the engine’s remaining useful life. In the main 3D scene, the engine lifetime is turned into a tunnel. The spiral ribbons are real sensor channels evolving over cycle-time. The glowing core is the inferred health state. The surrounding cloud is uncertainty. The orange wall ahead is the predicted failure horizon. So the big picture is: sensor evidence comes in, posterior belief tightens, and the machine moves from uncertainty toward a concrete failure prediction. The inset posteriors make that explicit. The health posterior shows where the model believes the hidden engine condition sits at the current moment, and how sharply it believes it. The RUL posterior shows the same idea for remaining life... early on it is broad, later it shifts left and narrows as the machine becomes more certain about how close failure is. This idea is not limited to engines. The same idea can apply to data centers, CPUs, GPUs, cooling systems, power grids, robotics, batteries, and any machine that produces telemetry while obeying physical constraints. In an age where machine learning runs on massive hardware infrastructure, this kind of model matters: it can turn noisy sensor streams into early warnings before expensive systems fail.

Mathelirium

17,843 views • 4 months ago