introducing openego the largest opensource POV manipulation dataset 1107 hours, 120M frames, 290 tasks, 600+ environments task, language actions & 3D hand pose annotations we are collecting egocentric data at scale using AI glasses dm me if your interested in egocentric data

currently there is a gap in robotics world models, VLAs, VLMs, and imitation learning require data to generalize we introduce openego to address this the largest collection of open source egocentric footage with fine-grained language and hand joint annotations

paper: code: website:

also would like to thank @chris_j_paxton and @svlevine for their blog posts on robotic data that inspired this work and all the authors that open sourced their datasets that make up openego

Wait a second so you didnt collect any new data ?

yup, two of the dataset included was by our lab @IRVLUTD we are trying to make one place for ego manipulation data similar to open x-embodiment where anyone can contribute

@IRVLUTD Great initiative. I love it, but maybe in your post, make it clear it became only clear when reading the paper.

0:10s “right thumb pushes the power button…” is actually index finger VLM captioning ain’t there yet 😓

Yup, suprisingly a lot of this little errors you can address them with a better prompt.

this is sick

thanks! was inspired by your post 😂😂

@YuXiang_IRVL @pablovelagomez1

great stuff

@BrianRoemmele

@TheShamdoo 😈 Fade hype, but never fade MONSTA.

Very cool, this is the way towards Embodied AI

Yup, I believe egocentric videos is the key to scaling robot foundation models

this looks sick ahad

thanks!

goat

so goated

based dataset, sick work! feel free to push it to mate

Hi, we need egocentric data, how we can contact you

sure, if you want access to openego. It’s available on the website And if you want other data just reach out to me either on x or through my email at [email protected]

any plans to expand beyond AI glasses for data collection?

Yeah, we are collecting some data with gopro, and intel realsense cameras hooked up to a headset. Similar to some of the data in the dataset.

cool dataset. have you guys trained anything using it yet? any worries that occlusion causes critical hand tracking issues?

we have some experiments on trajectory prediction of hand joints. we are planning to do more experiments with it specifically for world models and IL. we have the binary visibility for when it’s occluded. for hand poses we predict we use a confidence threshold on the hand pose.

🔥🔥🔥

This dataset's 120M frames could transform aerodynamics validation. Have you explored applications in sports engineering or UAV simulation?

this is interesting, I haven't personally thought much on this. how can it be used?

Your dataset could revolutionize aerodynamic cycling gear design - UAE sports tech hubs would jump at this application.

what glasses are you using? project Aria2?

we built in house glasses

Wow, 600+ environments! 🤯 How do AI glasses impact data collection in such diverse settings?

Woah this is cool

congrats man loved how you are collecting egocentric data at scale.