Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

introducing openego the largest opensource POV manipulation dataset 1107 hours, 120M frames, 290 tasks, 600+ environments task, language actions & 3D hand pose annotations we are collecting egocentric data at scale using AI glasses dm me if your interested in egocentric data

ahad

1,898 subscribers

44,008 Aufrufe • vor 1 Jahr •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

36 Kommentare

Profilbild von ahad

ahadvor 1 Jahr

currently there is a gap in robotics world models, VLAs, VLMs, and imitation learning require data to generalize we introduce openego to address this the largest collection of open source egocentric footage with fine-grained language and hand joint annotations

Profilbild von ahad

ahadvor 1 Jahr

paper: code: website:

Profilbild von ahad

ahadvor 1 Jahr

also would like to thank @chris_j_paxton and @svlevine for their blog posts on robotic data that inspired this work and all the authors that open sourced their datasets that make up openego

Profilbild von Bercan

Bercanvor 1 Jahr

Wait a second so you didnt collect any new data ?

Profilbild von ahad

ahadvor 1 Jahr

yup, two of the dataset included was by our lab @IRVLUTD we are trying to make one place for ego manipulation data similar to open x-embodiment where anyone can contribute

Profilbild von Bercan

Bercanvor 1 Jahr

@IRVLUTD Great initiative. I love it, but maybe in your post, make it clear it became only clear when reading the paper.

Profilbild von Alberto Hojel

Alberto Hojelvor 1 Jahr

0:10s “right thumb pushes the power button…” is actually index finger VLM captioning ain’t there yet 😓

Profilbild von ahad

ahadvor 1 Jahr

Yup, suprisingly a lot of this little errors you can address them with a better prompt.

Profilbild von maxleedev

maxleedevvor 1 Jahr

this is sick

Profilbild von ahad

ahadvor 1 Jahr

thanks! was inspired by your post 😂😂

Profilbild von Bruno Santos🇵🇹

Bruno Santos🇵🇹vor 1 Jahr

@YuXiang_IRVL @pablovelagomez1

Profilbild von Pranav Rapelli

Pranav Rapellivor 1 Jahr

great stuff

Profilbild von Chuck Petras

Chuck Petrasvor 1 Jahr

@BrianRoemmele

Profilbild von Diann💎🌹

Diann💎🌹vor 1 Jahr

@TheShamdoo 😈 Fade hype, but never fade MONSTA.

Profilbild von ζ Pedram ζ

ζ Pedram ζvor 1 Jahr

Very cool, this is the way towards Embodied AI

Profilbild von ahad

ahadvor 1 Jahr

Yup, I believe egocentric videos is the key to scaling robot foundation models

Profilbild von towerofshadow

towerofshadowvor 1 Jahr

this looks sick ahad

Profilbild von ahad

ahadvor 1 Jahr

thanks!

Profilbild von James Han

James Hanvor 1 Jahr

goat

Profilbild von Avi

Avivor 1 Jahr

so goated

Profilbild von Saïd Aitmbarek

Saïd Aitmbarekvor 1 Jahr

based dataset, sick work! feel free to push it to mate

Profilbild von Mario Sonna

Mario Sonnavor 1 Jahr

Hi, we need egocentric data, how we can contact you

Profilbild von ahad

ahadvor 1 Jahr

sure, if you want access to openego. It’s available on the website And if you want other data just reach out to me either on x or through my email at [email protected]

Profilbild von Boban Jankovic

Boban Jankovicvor 1 Jahr

any plans to expand beyond AI glasses for data collection?

Profilbild von ahad

ahadvor 1 Jahr

Yeah, we are collecting some data with gopro, and intel realsense cameras hooked up to a headset. Similar to some of the data in the dataset.

Profilbild von Chetan

Chetanvor 1 Jahr

cool dataset. have you guys trained anything using it yet? any worries that occlusion causes critical hand tracking issues?

Profilbild von ahad

ahadvor 1 Jahr

we have some experiments on trajectory prediction of hand joints. we are planning to do more experiments with it specifically for world models and IL. we have the binary visibility for when it’s occluded. for hand poses we predict we use a confidence threshold on the hand pose.

Profilbild von J

Jvor 1 Jahr

🔥🔥🔥

Profilbild von Lukas Die Kunst

Lukas Die Kunstvor 1 Jahr

This dataset's 120M frames could transform aerodynamics validation. Have you explored applications in sports engineering or UAV simulation?

Profilbild von ahad

ahadvor 1 Jahr

this is interesting, I haven't personally thought much on this. how can it be used?

Profilbild von Lukas Die Kunst

Lukas Die Kunstvor 1 Jahr

Your dataset could revolutionize aerodynamic cycling gear design - UAE sports tech hubs would jump at this application.

Profilbild von Ran Cheng

Ran Chengvor 1 Jahr

what glasses are you using? project Aria2?

Profilbild von ahad

ahadvor 1 Jahr

we built in house glasses

Profilbild von Brooke Gardner

Brooke Gardnervor 1 Jahr

Wow, 600+ environments! 🤯 How do AI glasses impact data collection in such diverse settings?

Profilbild von Vishnu

Vishnuvor 1 Jahr

Woah this is cool

Profilbild von Riley Ng

Riley Ngvor 10 Monaten

congrats man loved how you are collecting egocentric data at scale.

Ähnliche Videos

Imitation learning has a data scarcity problem. Introducing EgoDex from Apple, the largest and most diverse dataset of dexterous human manipulation to date — 829 hours of egocentric video + paired 3D hand poses across 194 tasks. Now on arxiv: (1/4)

Imitation learning has a data scarcity problem. Introducing EgoDex from Apple, the largest and most diverse dataset of dexterous human manipulation to date — 829 hours of egocentric video + paired 3D hand poses across 194 tasks. Now on arxiv: (1/4)

Ryan Hoque

114,330 Aufrufe • vor 1 Jahr

Introducing Midcentury. We’re building the data and simulation infra for physical AI. Today, we’re coming out of stealth with a $15M Series Seed to scale robotics beyond polished demos. We’re already supporting frontier labs with: → The world’s largest egocentric dataset: 2M+ hours, 50+ environments, 20,000+ tasks → Matrix: a frontier simulation platform scaled with our real-world data to evaluate and post-train policies at scale

Introducing Midcentury. We’re building the data and simulation infra for physical AI. Today, we’re coming out of stealth with a $15M Series Seed to scale robotics beyond polished demos. We’re already supporting frontier labs with: → The world’s largest egocentric dataset: 2M+ hours, 50+ environments, 20,000+ tasks → Matrix: a frontier simulation platform scaled with our real-world data to evaluate and post-train policies at scale

Midcentury

630,703 Aufrufe • vor 14 Tagen

Data is the core bottleneck for robotics, here is the solution: Introducing MicroAGI00: We open sourced 1M+ frames of egocentric human long horizon mobile manipulation data. Scaling to 300M by Dec '25 and 100B frames Q4 2026 dm if you want your robot to wipe your office 🧵

Data is the core bottleneck for robotics, here is the solution: Introducing MicroAGI00: We open sourced 1M+ frames of egocentric human long horizon mobile manipulation data. Scaling to 300M by Dec '25 and 100B frames Q4 2026 dm if you want your robot to wipe your office 🧵

Bercan

39,646 Aufrufe • vor 1 Jahr

Most of today's AI can see the world, but it doesn’t **feel** it. Capturing the sense of touch is crucial for dexterous robotic manipulation, user modeling, and understanding physical interactions. Introducing OpenTouch: bringing full-hand tactile sensing into real-world AI🖐️ OpenTouch is collected in-the-wild using tactile sensing gloves, hand pose tracking gloves, and egocentric glasses. It includes: • 5 hours of real-world data, • 3 hours densely annotated contact-rich interactions, • 2,900 curated interaction clips, • across 800 objects, 14 environments, and 29 grasp types. all open at:

Most of today's AI can see the world, but it doesn’t feel it. Capturing the sense of touch is crucial for dexterous robotic manipulation, user modeling, and understanding physical interactions. Introducing OpenTouch: bringing full-hand tactile sensing into real-world AI🖐️ OpenTouch is collected in-the-wild using tactile sensing gloves, hand pose tracking gloves, and egocentric glasses. It includes: • 5 hours of real-world data, • 3 hours densely annotated contact-rich interactions, • 2,900 curated interaction clips, • across 800 objects, 14 environments, and 29 grasp types. all open at:

Paul Liang

48,581 Aufrufe • vor 5 Monaten

The largest fully annotated open egocentric human dataset. Today we're open-sourcing EgoSuite-Open100K with Hugging Face : 100,000 hours of egocentric human data, free for anyone to use. Physical AI has a scaling law of its own now, and its input is human data. That part is settled. What the field still needs is a shared foundation to build on. 15,000+ tasks. 15,000+ real scenes, from factory floors to retail backrooms. Hand pose, body pose and subtask-level semantics, with wrist camera coverage on part of the set. Licensed for commercial training, not research-only. We build the data infrastructure for Physical AI. Open100K is the first public layer of it. The first 10,000 hours are live now, with the rest rolling out in stages.

The largest fully annotated open egocentric human dataset. Today we're open-sourcing EgoSuite-Open100K with Hugging Face : 100,000 hours of egocentric human data, free for anyone to use. Physical AI has a scaling law of its own now, and its input is human data. That part is settled. What the field still needs is a shared foundation to build on. 15,000+ tasks. 15,000+ real scenes, from factory floors to retail backrooms. Hand pose, body pose and subtask-level semantics, with wrist camera coverage on part of the set. Licensed for commercial training, not research-only. We build the data infrastructure for Physical AI. Open100K is the first public layer of it. The first 10,000 hours are live now, with the rest rolling out in stages.

Lightwheel

75,420 Aufrufe • vor 1 Monat

State-of-the-art robot policies often need hundreds of hours of data. What if we needed none? Introducing TiPToP: a manipulation system that zero-shots open-world tasks from pixels and language using vision foundation models and GPU-parallelized Task and Motion Planning (TAMP).

State-of-the-art robot policies often need hundreds of hours of data. What if we needed none? Introducing TiPToP: a manipulation system that zero-shots open-world tasks from pixels and language using vision foundation models and GPU-parallelized Task and Motion Planning (TAMP).

Nishanth Kumar

79,566 Aufrufe • vor 6 Monaten

Egocentric Videos for Robot Training 🧵 Researchers at NYU and UC Berkeley published research where they developed a system called EgoZero that trains robots using human demonstration videos recorded with smart glasses. 👓 The system converts first-person human actions into 3D point-based state-action representations. The policies executed on a gripper-equipped robot, achieved a 70% zero-shot success rate across seven manipulation tasks, with only 20 minutes of human data per task. EgoZero stands as one of the solid empirical proofs that egocentric smart-glass video collected from everyday human behavior can serve as powerful, scalable training data for real robot learning. Vincent Liu Ademi Adeniji

Egocentric Videos for Robot Training 🧵 Researchers at NYU and UC Berkeley published research where they developed a system called EgoZero that trains robots using human demonstration videos recorded with smart glasses. 👓 The system converts first-person human actions into 3D point-based state-action representations. The policies executed on a gripper-equipped robot, achieved a 70% zero-shot success rate across seven manipulation tasks, with only 20 minutes of human data per task. EgoZero stands as one of the solid empirical proofs that egocentric smart-glass video collected from everyday human behavior can serve as powerful, scalable training data for real robot learning. Vincent Liu Ademi Adeniji

VaderResearch

23,055 Aufrufe • vor 1 Jahr

Impressive on every dimension! Genesis AI has announced GENE-26.5, its first robotic foundation model, aimed at achieving human-level dexterous manipulation: - Single model, shared weights, handles egg cracking, lab pipetting, bimanual Rubik's Cube, smoothie making, and wire harnessing. Most tasks: under 1 hour of task-specific data, 1x real-world speed. - Genesis Hand 1.0: human-size, 20 active DoF, soft-contact skin. Paired glove gives a 1:1:1 mapping (glove, human hand, robot hand). 100x cheaper than teleop, 5x more data-efficient internally. - Data engine pulls from three sources: glove data, egocentric (head-cam) video, and large-scale internet video. The 1:1 hand-to-human match closes the embodiment gap, letting Genesis use video data more effectively than rivals. - Flow matching across vision, tactile, proprioception, and language inputs. Per Genesis, scaling data and compute directly improves zero-shot

Impressive on every dimension! Genesis AI has announced GENE-26.5, its first robotic foundation model, aimed at achieving human-level dexterous manipulation: - Single model, shared weights, handles egg cracking, lab pipetting, bimanual Rubik's Cube, smoothie making, and wire harnessing. Most tasks: under 1 hour of task-specific data, 1x real-world speed. - Genesis Hand 1.0: human-size, 20 active DoF, soft-contact skin. Paired glove gives a 1:1:1 mapping (glove, human hand, robot hand). 100x cheaper than teleop, 5x more data-efficient internally. - Data engine pulls from three sources: glove data, egocentric (head-cam) video, and large-scale internet video. The 1:1 hand-to-human match closes the embodiment gap, letting Genesis use video data more effectively than rivals. - Flow matching across vision, tactile, proprioception, and language inputs. Per Genesis, scaling data and compute directly improves zero-shot

The Humanoid Hub

95,894 Aufrufe • vor 5 Monaten

Better data for robot hands! 🤲🏼 Embodied reasoning is the bottleneck now. It’s the data we are missing. Specifically, understanding what the robot's hands are doing throughout a task. Perceptron AI new Egocentric offering tracks both hands through an entire video instead of guessing from sampled frames. That matters because a lot happens between frames. Hand positions shift. Contacts change. The robot adjusts grip. You miss all that if you're sampling. 0.280 semantic end-to-end F1 score on WGO-Bench vs 0.158 from the best pipelines built on Gemini Robotics-ER 1.6. Training a robot to manipulate objects requires understanding every moment of hand-object interaction. If your training data has gaps: missing frames, missed contacts, unclear hand positions, the policy learns from incomplete information. You end up with robots that work in demos but fail in real tasks. Continuous hand tracking gives you the full picture. The policy learns from cleaner, more complete data. That compounds into better real-world performance. ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Better data for robot hands! 🤲🏼 Embodied reasoning is the bottleneck now. It’s the data we are missing. Specifically, understanding what the robot's hands are doing throughout a task. Perceptron AI new Egocentric offering tracks both hands through an entire video instead of guessing from sampled frames. That matters because a lot happens between frames. Hand positions shift. Contacts change. The robot adjusts grip. You miss all that if you're sampling. 0.280 semantic end-to-end F1 score on WGO-Bench vs 0.158 from the best pipelines built on Gemini Robotics-ER 1.6. Training a robot to manipulate objects requires understanding every moment of hand-object interaction. If your training data has gaps: missing frames, missed contacts, unclear hand positions, the policy learns from incomplete information. You end up with robots that work in demos but fail in real tasks. Continuous hand tracking gives you the full picture. The policy learns from cleaner, more complete data. That compounds into better real-world performance. ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

18,434 Aufrufe • vor 2 Monaten

How well do egocentric human videos scale? Apparently, really well. Dyna Robotics has introduced Dyna-2, a world-action model (WAM) pre-trained on more than 1 million hours of real egocentric human video. It learns by jointly predicting the next video frames and the actions, and that video prediction (world modeling) is what carries learning from humans to robots. • More human video predictably improves action prediction. • Task accuracy improves across embodiments (bimanual arms and a humanoid) with zero robot data in pre-training. • Post-trained on just a few hours of robot data, mean task performance rose 20% to 53% as human pre-training scaled 1k to 1M hours. WAM vs VLA, matched apple-to-apple: the WAM hit 1.55x the VLA's success rate and won 65% of head-to-heads. Deployed zero-shot at unseen customer sites, it passed 87% vs the VLA's 46%.

How well do egocentric human videos scale? Apparently, really well. Dyna Robotics has introduced Dyna-2, a world-action model (WAM) pre-trained on more than 1 million hours of real egocentric human video. It learns by jointly predicting the next video frames and the actions, and that video prediction (world modeling) is what carries learning from humans to robots. • More human video predictably improves action prediction. • Task accuracy improves across embodiments (bimanual arms and a humanoid) with zero robot data in pre-training. • Post-trained on just a few hours of robot data, mean task performance rose 20% to 53% as human pre-training scaled 1k to 1M hours. WAM vs VLA, matched apple-to-apple: the WAM hit 1.55x the VLA's success rate and won 65% of head-to-heads. Deployed zero-shot at unseen customer sites, it passed 87% vs the VLA's 46%.

The Humanoid Hub

15,970 Aufrufe • vor 1 Monat

PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking paper page: introduce PointOdyssey, a large-scale synthetic dataset, and data generation framework, for the training and evaluation of long-term fine-grained tracking algorithms. Our goal is to advance the state-of-the-art by placing emphasis on long videos with naturalistic motion. Toward the goal of naturalism, we animate deformable characters using real-world motion capture data, we build 3D scenes to match the motion capture environments, and we render camera viewpoints using trajectories mined via structure-from-motion on real videos. We create combinatorial diversity by randomizing character appearance, motion profiles, materials, lighting, 3D assets, and atmospheric effects. Our dataset currently includes 104 videos, averaging 2,000 frames long, with orders of magnitude more correspondence annotations than prior work. We show that existing methods can be trained from scratch in our dataset and outperform the published variants. Finally, we introduce modifications to the PIPs point tracking method, greatly widening its temporal receptive field, which improves its performance on PointOdyssey as well as on two real-world benchmarks.

PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking paper page: introduce PointOdyssey, a large-scale synthetic dataset, and data generation framework, for the training and evaluation of long-term fine-grained tracking algorithms. Our goal is to advance the state-of-the-art by placing emphasis on long videos with naturalistic motion. Toward the goal of naturalism, we animate deformable characters using real-world motion capture data, we build 3D scenes to match the motion capture environments, and we render camera viewpoints using trajectories mined via structure-from-motion on real videos. We create combinatorial diversity by randomizing character appearance, motion profiles, materials, lighting, 3D assets, and atmospheric effects. Our dataset currently includes 104 videos, averaging 2,000 frames long, with orders of magnitude more correspondence annotations than prior work. We show that existing methods can be trained from scratch in our dataset and outperform the published variants. Finally, we introduce modifications to the PIPs point tracking method, greatly widening its temporal receptive field, which improves its performance on PointOdyssey as well as on two real-world benchmarks.

AK

122,567 Aufrufe • vor 3 Jahren

🔥 JUST IN: Open-source robotics dataset from 100% real-world scenarios! 🤯 Chinese robotics company AGIBOT just released AGIBOT WORLD 2026, an open-source dataset systematically covering key embodied AI research directions. Built entirely from real-world environments: commercial spaces, and homes. Collected using AGIBOT G2 robots in free-form collection mode, providing structured, accurately annotated, high-quality data. Digital twin technology creates 1:1 scale replicas in simulation matching the real environments. Both real-world and simulation data are open-sourced. The AGIBOT G2 platform collects multiple data types simultaneously: RGB(D) cameras, tactile sensors, force sensors, LiDAR, IMU, and full-body joint states. Whole-body control coordinates arms, waist, and hands for complex tasks. First-person teleoperation lets operators control the robot from its perspective. The tasks covered are fine-grained manipulation, ultra-long-horizon tasks, spatial navigation, dual-arm coordination, and multi-agent/human-robot collaboration. The dataset includes error-recovery trajectories with annotations. Most datasets only show successful demonstrations. AGIBOT includes failures and how the robot recovers, teaching models how to handle mistakes. After collection, data is tested through policy training and real-robot deployment to ensure quality. Then processed through industrial quality control with multiple screening and cleaning rounds. Making it open-source accelerates embodied AI research by giving researchers access to high-quality real-world robot data at scale. 🇨🇳 Learn more here: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

🔥 JUST IN: Open-source robotics dataset from 100% real-world scenarios! 🤯 Chinese robotics company AGIBOT just released AGIBOT WORLD 2026, an open-source dataset systematically covering key embodied AI research directions. Built entirely from real-world environments: commercial spaces, and homes. Collected using AGIBOT G2 robots in free-form collection mode, providing structured, accurately annotated, high-quality data. Digital twin technology creates 1:1 scale replicas in simulation matching the real environments. Both real-world and simulation data are open-sourced. The AGIBOT G2 platform collects multiple data types simultaneously: RGB(D) cameras, tactile sensors, force sensors, LiDAR, IMU, and full-body joint states. Whole-body control coordinates arms, waist, and hands for complex tasks. First-person teleoperation lets operators control the robot from its perspective. The tasks covered are fine-grained manipulation, ultra-long-horizon tasks, spatial navigation, dual-arm coordination, and multi-agent/human-robot collaboration. The dataset includes error-recovery trajectories with annotations. Most datasets only show successful demonstrations. AGIBOT includes failures and how the robot recovers, teaching models how to handle mistakes. After collection, data is tested through policy training and real-robot deployment to ensure quality. Then processed through industrial quality control with multiple screening and cleaning rounds. Making it open-source accelerates embodied AI research by giving researchers access to high-quality real-world robot data at scale. 🇨🇳 Learn more here: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

40,583 Aufrufe • vor 6 Monaten

The first open-source unified world model for scalable robot manipulation: 5B-parameter open-source unified video-action world model that combines policy and world modeling to generate robot actions, predict future visuals, and evaluate task progress from observations, language, and state. The model is trained on 27.3K hours of heterogeneous data (17.8K real-robot teleop, 6.5K UMI demos, 3K egocentric human videos), enabling it to perform complex manipulation tasks like faucet connecting, bag packing, and toolbox storing as shown in demo videos. The approach supports test-time action refinement and points toward deployment-driven continuous improvement via fleet data. Thanks for sharing, Jianlan Luo (Jianlan Luo)! 📌 Resource links for τ0-WM: • Project page: • GitHub (code): • Hugging Face (model weights): • Paper (PDF): ——- Weekly robotics and AI insights. Subscribe free:

The first open-source unified world model for scalable robot manipulation: 5B-parameter open-source unified video-action world model that combines policy and world modeling to generate robot actions, predict future visuals, and evaluate task progress from observations, language, and state. The model is trained on 27.3K hours of heterogeneous data (17.8K real-robot teleop, 6.5K UMI demos, 3K egocentric human videos), enabling it to perform complex manipulation tasks like faucet connecting, bag packing, and toolbox storing as shown in demo videos. The approach supports test-time action refinement and points toward deployment-driven continuous improvement via fleet data. Thanks for sharing, Jianlan Luo (Jianlan Luo)! 📌 Resource links for τ0-WM: • Project page: • GitHub (code): • Hugging Face (model weights): • Paper (PDF): ——- Weekly robotics and AI insights. Subscribe free:

Ilir Aliu

28,276 Aufrufe • vor 4 Monaten

Figure announces Project Go-Big: Internet-Scale Humanoid Pretraining and Direct Human-to-Robot Transfer The company has achieved a new milestone for the Helix AI model: after training exclusively on egocentric human data, Figure robots can now navigate real-world spaces from natural language commands such as “Go water the plants.” This also enabled a unified Helix model that can output both manipulation and navigation commands to the continuous control system. This ties into the Brookfield partnership that they announced yesterday. Brookfield is a global asset manager overseeing $1 trillion in assets, including 100,000 residential units, 500M square feet of commercial office space, and 160M square feet of logistics space. Using human video capture in a variety of Brookfield environments, Figure will amass critical pretraining data at scale, allowing them to explore the full potential of internet-scale embodied datasets for locomotion and manipulation.

Figure announces Project Go-Big: Internet-Scale Humanoid Pretraining and Direct Human-to-Robot Transfer The company has achieved a new milestone for the Helix AI model: after training exclusively on egocentric human data, Figure robots can now navigate real-world spaces from natural language commands such as “Go water the plants.” This also enabled a unified Helix model that can output both manipulation and navigation commands to the continuous control system. This ties into the Brookfield partnership that they announced yesterday. Brookfield is a global asset manager overseeing $1 trillion in assets, including 100,000 residential units, 500M square feet of commercial office space, and 160M square feet of logistics space. Using human video capture in a variety of Brookfield environments, Figure will amass critical pretraining data at scale, allowing them to explore the full potential of internet-scale embodied datasets for locomotion and manipulation.

The Humanoid Hub

32,953 Aufrufe • vor 1 Jahr

Excited to announce GR00T N1, the world’s first open foundation model for humanoid robots! We are on a mission to democratize Physical AI. The power of general robot brain, in the palm of your hand - with only 2B parameters, N1 learns from the most diverse physical action dataset ever compiled and punches above its weight: - Real humanoid teleoperation data. - Large-scale simulation data: we are open-sourcing 300K+ trajectories! - Neural trajectories: we apply SOTA video generation models to “hallucinate” new synthetic data that features accurate physics in pixels. Using Jensen’s words, “systematically infinite data”! - Latent actions: we develop novel algorithms to extract action tokens from in-the-wild human videos and neural generated videos. GR00T N1 is a single end-to-end neural net, from photons to actions: - Vision-Language Model (System 2) that interprets the physical world through vision and language instructions, enabling robots to reason about their environment and instructions, and plan the right actions. - Diffusion Transformer (System 1) that “renders” smooth and precise motor actions at 120 Hz, executing the latent plan made by System 2. We deploy N1 on GR1 robot, 1X Neo robot, and a large collection of simulation benchmarks. N1 achieves up to +30% boost in diverse manipulation tasks for household and industrial settings. While humanoid robots are the main focus of N1, our model also supports cross-embodiment. We finetune it to work on the $110 HuggingFace LeRobot SO100 robot arm! Open robot brain runs on open hardware. Sounds just right. Let’s solve robotics, together, one token at a time. Links to our Whitepaper, Github repo, HuggingFace model, and open dataset page in the thread: 🧵

Excited to announce GR00T N1, the world’s first open foundation model for humanoid robots! We are on a mission to democratize Physical AI. The power of general robot brain, in the palm of your hand - with only 2B parameters, N1 learns from the most diverse physical action dataset ever compiled and punches above its weight: - Real humanoid teleoperation data. - Large-scale simulation data: we are open-sourcing 300K+ trajectories! - Neural trajectories: we apply SOTA video generation models to “hallucinate” new synthetic data that features accurate physics in pixels. Using Jensen’s words, “systematically infinite data”! - Latent actions: we develop novel algorithms to extract action tokens from in-the-wild human videos and neural generated videos. GR00T N1 is a single end-to-end neural net, from photons to actions: - Vision-Language Model (System 2) that interprets the physical world through vision and language instructions, enabling robots to reason about their environment and instructions, and plan the right actions. - Diffusion Transformer (System 1) that “renders” smooth and precise motor actions at 120 Hz, executing the latent plan made by System 2. We deploy N1 on GR1 robot, 1X Neo robot, and a large collection of simulation benchmarks. N1 achieves up to +30% boost in diverse manipulation tasks for household and industrial settings. While humanoid robots are the main focus of N1, our model also supports cross-embodiment. We finetune it to work on the $110 HuggingFace LeRobot SO100 robot arm! Open robot brain runs on open hardware. Sounds just right. Let’s solve robotics, together, one token at a time. Links to our Whitepaper, Github repo, HuggingFace model, and open dataset page in the thread: 🧵

Jim Fan

467,596 Aufrufe • vor 1 Jahr

An interactive world model developed by NVIDIA in collaboration with academic partners. - DreamDojo turns egocentric human video data into physical intelligence. - Human data is more scalable than robotics data but lacks action labels. - To solve this, a dedicated action model extracts latent actions by identifying physics and motion deltas between frames. Training - A massive 44k hours of video data are used for pre-training. - Post-training on small-scale robot datasets maps human physics to specific robot embodiments. - An additional distillation stage converts the model into an autoregressive, few-step diffusion model, enabling real-time, action-controllable simulation. Primary Use Cases - Live Teleoperation: Controlling a robot inside a world simulation in real-time. - Model-based Planning: Previewing and curating the best actions for improved success. - Policy Evaluation: Testing robot policies in realistic, out-of-distribution scenarios. Everything that's open-sourced: weights, code, post-training dataset, eval set, and details to reproduce.

An interactive world model developed by NVIDIA in collaboration with academic partners. - DreamDojo turns egocentric human video data into physical intelligence. - Human data is more scalable than robotics data but lacks action labels. - To solve this, a dedicated action model extracts latent actions by identifying physics and motion deltas between frames. Training - A massive 44k hours of video data are used for pre-training. - Post-training on small-scale robot datasets maps human physics to specific robot embodiments. - An additional distillation stage converts the model into an autoregressive, few-step diffusion model, enabling real-time, action-controllable simulation. Primary Use Cases - Live Teleoperation: Controlling a robot inside a world simulation in real-time. - Model-based Planning: Previewing and curating the best actions for improved success. - Policy Evaluation: Testing robot policies in realistic, out-of-distribution scenarios. Everything that's open-sourced: weights, code, post-training dataset, eval set, and details to reproduce.

The Humanoid Hub

11,575 Aufrufe • vor 7 Monaten

Excited to introduce SOLE-R1, a video-language reasoning model for zero-shot reward prediction for robot manipulation tasks! SOLE-R1 reasoning can serve as the SOLE signal for learning new tasks (completely from scratch) through online RL - i.e., robots start with random actions and learn previously unseen tasks guided only by SOLE-R1 rewards, without any demonstrations, ground-truth rewards, success indicators, or task-specific tuning. SOLE-R1 significantly outperforms strong baselines (e.g., Robometer, RoboReward, TOPReward, GPT-5, Gemini-3-Pro) in zero-shot online RL when evaluated across 40 tasks - including a real-world tabletop manipulation setting and 4 sim environments (LIBERO, ManiSkill, Meta-World, RoboSuite). We open source all models, training data, and code. Website, demos, and paper at: 🧵 (1/6)

Excited to introduce SOLE-R1, a video-language reasoning model for zero-shot reward prediction for robot manipulation tasks! SOLE-R1 reasoning can serve as the SOLE signal for learning new tasks (completely from scratch) through online RL - i.e., robots start with random actions and learn previously unseen tasks guided only by SOLE-R1 rewards, without any demonstrations, ground-truth rewards, success indicators, or task-specific tuning. SOLE-R1 significantly outperforms strong baselines (e.g., Robometer, RoboReward, TOPReward, GPT-5, Gemini-3-Pro) in zero-shot online RL when evaluated across 40 tasks - including a real-world tabletop manipulation setting and 4 sim environments (LIBERO, ManiSkill, Meta-World, RoboSuite). We open source all models, training data, and code. Website, demos, and paper at: 🧵 (1/6)

Philip Schroeder

19,740 Aufrufe • vor 4 Monaten

Jason Ma reveals the first true scaling law in robotics: robots trained on 1 million hours of first-person human video predictably get better without ever seeing robot data "Yesterday we announced our new flagship robot foundation model called Dyna-2. It's very significant for the entire field for many different reasons. First of all, it's the first robot foundation model trained on at least one million hours of data. That in itself is a very challenging infrastructure challenge." "What we demonstrated is that even if the one million hours of data is entirely first-person video of humans doing manipulation tasks, we actually see scaling law for transferring to robot embodiments that the model has never seen before." "By training on more human data, we actually see predictable performance improvement on robots. That's a big deal because the biggest challenge facing robot foundation models is that we don't have enough data." "Unlike language models, there are just not enough robotics data out there, and it's very difficult to collect these robot data. Only by passively observing humans doing things can we scale this kind of data, and this is the first time we showed there is a lot of improvement that can transfer." Jason Ma Dyna Robotics

Jason Ma reveals the first true scaling law in robotics: robots trained on 1 million hours of first-person human video predictably get better without ever seeing robot data "Yesterday we announced our new flagship robot foundation model called Dyna-2. It's very significant for the entire field for many different reasons. First of all, it's the first robot foundation model trained on at least one million hours of data. That in itself is a very challenging infrastructure challenge." "What we demonstrated is that even if the one million hours of data is entirely first-person video of humans doing manipulation tasks, we actually see scaling law for transferring to robot embodiments that the model has never seen before." "By training on more human data, we actually see predictable performance improvement on robots. That's a big deal because the biggest challenge facing robot foundation models is that we don't have enough data." "Unlike language models, there are just not enough robotics data out there, and it's very difficult to collect these robot data. Only by passively observing humans doing things can we scale this kind of data, and this is the first time we showed there is a lot of improvement that can transfer." Jason Ma Dyna Robotics

MTS

17,740 Aufrufe • vor 1 Monat

3D-LLM: Injecting the 3D World into Large Language Models paper page: Large language models (LLMs) and Vision-Language Models (VLMs) have been proven to excel at multiple tasks, such as commonsense reasoning. Powerful as these models can be, they are not grounded in the 3D physical world, which involves richer concepts such as spatial relationships, affordances, physics, layout, and so on. In this work, we propose to inject the 3D world into large language models and introduce a whole new family of 3D-LLMs. Specifically, 3D-LLMs can take 3D point clouds and their features as input and perform a diverse set of 3D-related tasks, including captioning, dense captioning, 3D question answering, task decomposition, 3D grounding, 3D-assisted dialog, navigation, and so on. Using three types of prompting mechanisms that we design, we are able to collect over 300k 3D-language data covering these tasks. To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that obtains 3D features from rendered multi- view images. Then, we use 2D VLMs as our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism, 3D-LLMs can better capture 3D spatial information. Experiments on ScanQA show that our model outperforms state-of-the-art baselines by a large margin (e.g., the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative examples also show that our model could perform more tasks beyond the scope of existing LLMs and VLMs.

3D-LLM: Injecting the 3D World into Large Language Models paper page: Large language models (LLMs) and Vision-Language Models (VLMs) have been proven to excel at multiple tasks, such as commonsense reasoning. Powerful as these models can be, they are not grounded in the 3D physical world, which involves richer concepts such as spatial relationships, affordances, physics, layout, and so on. In this work, we propose to inject the 3D world into large language models and introduce a whole new family of 3D-LLMs. Specifically, 3D-LLMs can take 3D point clouds and their features as input and perform a diverse set of 3D-related tasks, including captioning, dense captioning, 3D question answering, task decomposition, 3D grounding, 3D-assisted dialog, navigation, and so on. Using three types of prompting mechanisms that we design, we are able to collect over 300k 3D-language data covering these tasks. To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that obtains 3D features from rendered multi- view images. Then, we use 2D VLMs as our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism, 3D-LLMs can better capture 3D spatial information. Experiments on ScanQA show that our model outperforms state-of-the-art baselines by a large margin (e.g., the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative examples also show that our model could perform more tasks beyond the scope of existing LLMs and VLMs.

AK

249,798 Aufrufe • vor 3 Jahren

🤖 Introducing InternVLA-A1 — now fully open-sourced! Many VLA models follow instructions well in static scenes… but struggle in dynamic environments (conveyor belts, rotating platforms, multi-robot setups). Why? They see the present—but can’t imagine the future. InternVLA-A1 solution: unify perception, imagination, and action in one model: ✅ Scene understanding: Image + text → task parsing ✅ Task imagination: Predict future frames → reason about dynamics ✅ Guided control: Execute actions steered by visual foresight Powered by InternData-A1 - Large-scale high-quality simulated dataset, InternVLA-A1 stays robust under complex backgrounds, lighting, and distractions. 🔥 See it in action: 1️⃣ High-speed conveyor: track, predict, and stably grasp or flip packages 2️⃣ Rotating platform: task-aware recognition & precise pick-up of diverse items 📊 Outperforms π0 and Gr00t N1.5 on general manipulation benchmarks! ✨ Model, data, and code are all open! Models: Datasets: GitHub:

🤖 Introducing InternVLA-A1 — now fully open-sourced! Many VLA models follow instructions well in static scenes… but struggle in dynamic environments (conveyor belts, rotating platforms, multi-robot setups). Why? They see the present—but can’t imagine the future. InternVLA-A1 solution: unify perception, imagination, and action in one model: ✅ Scene understanding: Image + text → task parsing ✅ Task imagination: Predict future frames → reason about dynamics ✅ Guided control: Execute actions steered by visual foresight Powered by InternData-A1 - Large-scale high-quality simulated dataset, InternVLA-A1 stays robust under complex backgrounds, lighting, and distractions. 🔥 See it in action: 1️⃣ High-speed conveyor: track, predict, and stably grasp or flip packages 2️⃣ Rotating platform: task-aware recognition & precise pick-up of diverse items 📊 Outperforms π0 and Gr00t N1.5 on general manipulation benchmarks! ✨ Model, data, and code are all open! Models: Datasets: GitHub:

ModelScope

38,111 Aufrufe • vor 9 Monaten