Loading video...

Video Failed to Load

Go Home

If you want a vision encoder for dexterous manipulation, what should be the most important part to model? ๐Ÿค” Current standard models like CLIP, SigLIP, and DINOv2 have an incredible grasp of semantics and spatial details. But they lack the action-centric structure needed for downstream visuomotor control. But collecting...

roeiherzig's profile picture

Roei Herzig

1,890 subscribers

22,201 views โ€ข 1 month ago โ€ขvia X (Twitter)

LIVE

Anya Rossiโ€ข Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

๐Ÿ”ฅ #ICRA2026 Best Paper Finalist The era of "robot VLA = single-arm gripper" is ending. Introducing Dexora โ€” the first open-source Vision-Language-Action system for dual-arm, dual-hand, 36-DoF dexterous manipulation. ๐Ÿฆพ Dual Arms ๐Ÿ–๏ธ Dual Hands ๐ŸŽฏ 36 DoF Control ๐ŸŒ Open Source Trained on: โ€ข 100K simulated trajectories โ€ข 10K real-world demonstrations Dexora achieves: โœ“ 90%+ success on basic manipulation โœ“ Strong dexterous manipulation performance โœ“ Cross-embodiment generalization Our key hypothesis: Train on the hardest embodiment. Transfer to simpler robots later. Instead of scaling up gripper policies, we train directly in the most expressive action space and project downward to simpler embodiments. This may be a practical path toward universal robot controllers. ๐ŸŽฅ Demos: ๐Ÿ“„ Paper:
2:58

๐Ÿ”ฅ #ICRA2026 Best Paper Finalist The era of "robot VLA = single-arm gripper" is ending. Introducing Dexora โ€” the first open-source Vision-Language-Action system for dual-arm, dual-hand, 36-DoF dexterous manipulation. ๐Ÿฆพ Dual Arms ๐Ÿ–๏ธ Dual Hands ๐ŸŽฏ 36 DoF Control ๐ŸŒ Open Source Trained on: โ€ข 100K simulated trajectories โ€ข 10K real-world demonstrations Dexora achieves: โœ“ 90%+ success on basic manipulation โœ“ Strong dexterous manipulation performance โœ“ Cross-embodiment generalization Our key hypothesis: Train on the hardest embodiment. Transfer to simpler robots later. Instead of scaling up gripper policies, we train directly in the most expressive action space and project downward to simpler embodiments. This may be a practical path toward universal robot controllers. ๐ŸŽฅ Demos: ๐Ÿ“„ Paper:

Hao Zhao

17,048 views โ€ข 1 month ago

We trained a humanoid with 22-DoF dexterous hands to assemble model cars, operate syringes, sort poker cards, fold/roll shirts, all learned primarily from 20,000+ hours of egocentric human video with no robot in the loop. Humans are the most scalable embodiment on the planet. We discovered a near-perfect log-linear scaling law (Rยฒ = 0.998) between human video volume and action prediction loss, and this loss directly predicts real-robot success rate. Humanoid robots will be the end game, because they are the practical form factor with minimal embodiment gap from humans. Call it the Bitter Lesson of robot hardware: the kinematic similarity lets us simply retarget human finger motion onto dexterous robot hand joints. No learned embeddings, no fancy transfer algorithms needed. Relative wrist motion + retargeted 22-DoF finger actions serve as a unified action space that carries through from pre-training to robot execution. Our recipe is called "EgoScale": - Pre-train GR00T N1.5 on 20K hours of human video, mid-train with only 4 hours (!) of robot play data with Sharpa hands. 54% gains over training from scratch across 5 highly dexterous tasks. - Most surprising result: a *single* teleop demo is sufficient to learn a never-before-seen task. Our recipe enables extreme data efficiency. - Although we pre-train in 22-DoF hand joint space, the policy transfers to a Unitree G1 with 7-DoF tri-finger hands. 30%+ gains over training on G1 data alone. The scalable path to robot dexterity was never more robots. It was always us. Deep dives in thread:
1:17

We trained a humanoid with 22-DoF dexterous hands to assemble model cars, operate syringes, sort poker cards, fold/roll shirts, all learned primarily from 20,000+ hours of egocentric human video with no robot in the loop. Humans are the most scalable embodiment on the planet. We discovered a near-perfect log-linear scaling law (Rยฒ = 0.998) between human video volume and action prediction loss, and this loss directly predicts real-robot success rate. Humanoid robots will be the end game, because they are the practical form factor with minimal embodiment gap from humans. Call it the Bitter Lesson of robot hardware: the kinematic similarity lets us simply retarget human finger motion onto dexterous robot hand joints. No learned embeddings, no fancy transfer algorithms needed. Relative wrist motion + retargeted 22-DoF finger actions serve as a unified action space that carries through from pre-training to robot execution. Our recipe is called "EgoScale": - Pre-train GR00T N1.5 on 20K hours of human video, mid-train with only 4 hours (!) of robot play data with Sharpa hands. 54% gains over training from scratch across 5 highly dexterous tasks. - Most surprising result: a *single* teleop demo is sufficient to learn a never-before-seen task. Our recipe enables extreme data efficiency. - Although we pre-train in 22-DoF hand joint space, the policy transfers to a Unitree G1 with 7-DoF tri-finger hands. 30%+ gains over training on G1 data alone. The scalable path to robot dexterity was never more robots. It was always us. Deep dives in thread:

Jim Fan

293,961 views โ€ข 5 months ago

What would a World Model look like if we start from a real embodied agent acting in the real world? It has to have: 1) A real, physically grounded and complex action spaceโ€”not just abstract control signals. 2) Diverse, real-life scenarios and activities. Or in short: It has to be annoyingly complexโ€”in both the action and vision spaceโ€”to even get close to real life. We did an initial attempt: Whole-Body Conditioned Egocentric Video Prediction. In collaboration with @dans_t123 , Amir Bar, Yann LeCun , trevordarrell and Jitendra MALIK. (For more details, check: What we did is very simple: Predict Egocentric Video from human Actions (PEVA) - Given the past video and a future action represented by relative 3D body pose, PEVA predicts how the world looks nextโ€”from the first-person view. By conditioning on kinematic pose trajectories, structured by the joint hierarchy of the body, it learns how physical actions shape perception.
3:12

What would a World Model look like if we start from a real embodied agent acting in the real world? It has to have: 1) A real, physically grounded and complex action spaceโ€”not just abstract control signals. 2) Diverse, real-life scenarios and activities. Or in short: It has to be annoyingly complexโ€”in both the action and vision spaceโ€”to even get close to real life. We did an initial attempt: Whole-Body Conditioned Egocentric Video Prediction. In collaboration with @dans_t123 , Amir Bar, Yann LeCun , trevordarrell and Jitendra MALIK. (For more details, check: What we did is very simple: Predict Egocentric Video from human Actions (PEVA) - Given the past video and a future action represented by relative 3D body pose, PEVA predicts how the world looks nextโ€”from the first-person view. By conditioning on kinematic pose trajectories, structured by the joint hierarchy of the body, it learns how physical actions shape perception.

Yutong Bai

179,017 views โ€ข 1 year ago

Huge opportunity for creators right now Why Perle Labs got my attention ๐Ÿ‘€ ๐Ÿ”ธPerleLabs just dropped a campaign with $55K up for grabs across 400 winners. ๐Ÿ”ธI spent hours poured into this piece ๐ŸŽจ creating art with my feet ๐Ÿพ to show that no matter the challenge, creativity finds a way. This isn't just a post; it's my contribution to a future where humans and AI grow together. ๐Ÿ”ธWeโ€™re entering a world where machines generate knowledge, but rarely understand its source. ๐Ÿ”ธPerle Labs is building a system where AI learns from real human knowledge instead of random or low quality data. ๐Ÿ”ธYou complete simple tasks like labeling reviewing or correcting data, and every action is recorded onchain as proof of your work. ๐Ÿ”ธYour effort builds a visible reputation, which helps you access better tasks and higher rewards over time. ๐Ÿ”ธOver 50,000 people are already contributing with more than 1.7 million tasks completed across the platform. ๐Ÿ”ธThis creates a model where humans stay important in AI and your input becomes a real digital asset. ๐Ÿ”ธPerle Labs anchors AI back to human insight through verified data and onchain reputation. Making contribution not just useful, but permanent. How to Participate ๐Ÿ‘‡ 1. Join Perleโ€™s discord via this link 2. Post an original tweet on X (text, meme, image, or video or anything ) I have made a art with my leg ๐Ÿฆต art to express my gratitude ) 3. Include the hashtags: #PerleAl + #ToPerle 4. End your tweet with: "- participating in Perle Labs community campaign" 5. Submit your entry โ†’ I wish my best luck to all the participants let's support each other and win big . I will be sharing how you can be easily be in top 50 position ๐Ÿค‘in my next post Glad to mention I am "- participating in Perle Labs community campaign "
0:23

Huge opportunity for creators right now Why Perle Labs got my attention ๐Ÿ‘€ ๐Ÿ”ธPerleLabs just dropped a campaign with $55K up for grabs across 400 winners. ๐Ÿ”ธI spent hours poured into this piece ๐ŸŽจ creating art with my feet ๐Ÿพ to show that no matter the challenge, creativity finds a way. This isn't just a post; it's my contribution to a future where humans and AI grow together. ๐Ÿ”ธWeโ€™re entering a world where machines generate knowledge, but rarely understand its source. ๐Ÿ”ธPerle Labs is building a system where AI learns from real human knowledge instead of random or low quality data. ๐Ÿ”ธYou complete simple tasks like labeling reviewing or correcting data, and every action is recorded onchain as proof of your work. ๐Ÿ”ธYour effort builds a visible reputation, which helps you access better tasks and higher rewards over time. ๐Ÿ”ธOver 50,000 people are already contributing with more than 1.7 million tasks completed across the platform. ๐Ÿ”ธThis creates a model where humans stay important in AI and your input becomes a real digital asset. ๐Ÿ”ธPerle Labs anchors AI back to human insight through verified data and onchain reputation. Making contribution not just useful, but permanent. How to Participate ๐Ÿ‘‡ 1. Join Perleโ€™s discord via this link 2. Post an original tweet on X (text, meme, image, or video or anything ) I have made a art with my leg ๐Ÿฆต art to express my gratitude ) 3. Include the hashtags: #PerleAl + #ToPerle 4. End your tweet with: "- participating in Perle Labs community campaign" 5. Submit your entry โ†’ I wish my best luck to all the participants let's support each other and win big . I will be sharing how you can be easily be in top 50 position ๐Ÿค‘in my next post Glad to mention I am "- participating in Perle Labs community campaign "

dhiraj

41,009 views โ€ข 4 months ago

Can vision-language-action (VLA) models generalize to diverse OOD tasks and align with customized objectives? ๐Ÿค” ๐Ÿš€ We introduce GRAPE, a plug-and-play algorithm to generalize robot policies via preference alignment. GRAPE unfolds three benefits to boost the generalizability of VLAs: ๐Ÿ‘‰1. GRAPE aligns VLAs on a trajectory level and endows the model with the ability for global decision-making, instead of merely cloning behavior; ๐Ÿ‘‰2. GRAPE implicitly models reward from both successful and failed trials to boost generalizability to diverse tasks; ๐Ÿ‘‰3. GRAPE adopts a scalable preference synthesis algorithm to rank trajectories with preferences that align with arbitrary objectives. Our experiments on a diverse array of real-world and simulated robotic tasks reveal: 1โƒฃGRAPE enhances the performance of state-of-the-art VLA models, increasing success rates on in-domain and unseen manipulation tasks by 51.79% and 60.36%; 2โƒฃGRAPE is versatile to be aligned with diverse objectives and reduce collision rates by 44.31% or rollout length by 11.15% when aligning towards safer or more efficient manipulation policy, respectively. Check out our full project for more details: ๐Ÿ”ฅ Paper: ๐Ÿ”ฅ Project: ๐Ÿ”ฅ Code:
1:06

Can vision-language-action (VLA) models generalize to diverse OOD tasks and align with customized objectives? ๐Ÿค” ๐Ÿš€ We introduce GRAPE, a plug-and-play algorithm to generalize robot policies via preference alignment. GRAPE unfolds three benefits to boost the generalizability of VLAs: ๐Ÿ‘‰1. GRAPE aligns VLAs on a trajectory level and endows the model with the ability for global decision-making, instead of merely cloning behavior; ๐Ÿ‘‰2. GRAPE implicitly models reward from both successful and failed trials to boost generalizability to diverse tasks; ๐Ÿ‘‰3. GRAPE adopts a scalable preference synthesis algorithm to rank trajectories with preferences that align with arbitrary objectives. Our experiments on a diverse array of real-world and simulated robotic tasks reveal: 1โƒฃGRAPE enhances the performance of state-of-the-art VLA models, increasing success rates on in-domain and unseen manipulation tasks by 51.79% and 60.36%; 2โƒฃGRAPE is versatile to be aligned with diverse objectives and reduce collision rates by 44.31% or rollout length by 11.15% when aligning towards safer or more efficient manipulation policy, respectively. Check out our full project for more details: ๐Ÿ”ฅ Paper: ๐Ÿ”ฅ Project: ๐Ÿ”ฅ Code:

Huaxiu Yao

19,988 views โ€ข 1 year ago

We believe weโ€™re the first robotics company to demonstrate a robot peeling an apple with dual dexterous human-like hands. This breakthrough closes a key gap in robotics, achieving bimanual, contact-rich manipulation and moving far beyond the limits of simple grippers. ๐Ÿงตโ†“ Todayโ€™s AI models (VLMs) are excellent at perception but struggle with action. Controlling high-degree-of-freedom hands for tasks like this is incredibly complex, and precise finger-level teleoperation is nearly impossible for humans. Our first step was a shared-autonomy system: rather than controlling every finger, the operator triggers pre-learned skills like a โ€œrotate apple or tennis ballโ€ primitive via a keyboard press or pedal. This makes scalable data collection and RL training possible. How does the AI manage this? We created "MoDE-VLA" (Mixture of Dexterous Experts). It fuses vision, language, force, and touch data by using a team of specialist "experts," making control in high-dimensional spaces stable and effective. The combination of these two innovations allows for seamless, contact-rich manipulation. The human provides high-level guidance, and the robot executes the complex in-hand coordination required. This work paves the way for robots that can safely handle delicate tasks in human environments. Want the full technical details? ๐Ÿ“„ Read the full research paper: Visit us at NVIDIA GTC Booth #1838, Hall 3 to learn more! #Robotics #AI #DexterousManipulation #VLA #NVIDIAGTC Nancy Villicaรฑa NVIDIA GTC
2:51

We believe weโ€™re the first robotics company to demonstrate a robot peeling an apple with dual dexterous human-like hands. This breakthrough closes a key gap in robotics, achieving bimanual, contact-rich manipulation and moving far beyond the limits of simple grippers. ๐Ÿงตโ†“ Todayโ€™s AI models (VLMs) are excellent at perception but struggle with action. Controlling high-degree-of-freedom hands for tasks like this is incredibly complex, and precise finger-level teleoperation is nearly impossible for humans. Our first step was a shared-autonomy system: rather than controlling every finger, the operator triggers pre-learned skills like a โ€œrotate apple or tennis ballโ€ primitive via a keyboard press or pedal. This makes scalable data collection and RL training possible. How does the AI manage this? We created "MoDE-VLA" (Mixture of Dexterous Experts). It fuses vision, language, force, and touch data by using a team of specialist "experts," making control in high-dimensional spaces stable and effective. The combination of these two innovations allows for seamless, contact-rich manipulation. The human provides high-level guidance, and the robot executes the complex in-hand coordination required. This work paves the way for robots that can safely handle delicate tasks in human environments. Want the full technical details? ๐Ÿ“„ Read the full research paper: Visit us at NVIDIA GTC Booth #1838, Hall 3 to learn more! #Robotics #AI #DexterousManipulation #VLA #NVIDIAGTC Nancy Villicaรฑa NVIDIA GTC

Sharpa

20,429 views โ€ข 4 months ago

Weโ€™re excited to share DiT4DiT, an end-to-end Video-Action Model for robot learning that unifies a video Diffusion Transformer and an action Diffusion Transformer in a single cascaded framework. By leveraging the rich spatiotemporal and physical dynamics learned through video generation, rather than static image-text priors, DiT4DiT achieves state-of-the-art results on LIBERO (98.6%) and RoboCasa GR1 (50.8%) with far less training data, delivering over 10ร— better sample efficiency and up to 7ร— faster convergence. Real-world deployment on a humanoid robot further shows robust generalization. We believe this is a step toward making video generation a powerful backbone for robot policy learning. This work builds upon the brilliant foundations laid by Nvidia's GR00T and Cosmos. Project: Paper: Code: Coming soon. In the meantime, you can ask your coding agent to reproduce the method based on GR00T/Cosmos.
2:32

Weโ€™re excited to share DiT4DiT, an end-to-end Video-Action Model for robot learning that unifies a video Diffusion Transformer and an action Diffusion Transformer in a single cascaded framework. By leveraging the rich spatiotemporal and physical dynamics learned through video generation, rather than static image-text priors, DiT4DiT achieves state-of-the-art results on LIBERO (98.6%) and RoboCasa GR1 (50.8%) with far less training data, delivering over 10ร— better sample efficiency and up to 7ร— faster convergence. Real-world deployment on a humanoid robot further shows robust generalization. We believe this is a step toward making video generation a powerful backbone for robot policy learning. This work builds upon the brilliant foundations laid by Nvidia's GR00T and Cosmos. Project: Paper: Code: Coming soon. In the meantime, you can ask your coding agent to reproduce the method based on GR00T/Cosmos.

Shuo Yang

31,596 views โ€ข 4 months ago

Domain Randomization (DR) is a key component of the data augmentation pipeline at Axis Robotics. By applying DR, we are able to scale verified, high-quality human trajectories by 10x to 100x. During training, we systematically introduce variances in environmental parameters. This prevents the model from relying on spurious visual correlations. The objective is to ensure the policy learns rather than overfitting. To demonstrate the necessity and effectiveness of this approach, we evaluated both DR and No-DR models on Task 74 (pour_water_into_mug). The empirical results show a definitive impact on real-world deployment reliability: integrating DR into the pipeline increased the success rate from 0% to 90% (Fig. 1). This divergence stems from how the respective policies process visual observations (Fig. 2). The baseline (No DR) model overfits to the static visual background. It essentially memorizes the poses from the training dataset but fails to generalize when subjected to the inevitable variances of real-world deployment. Consequently, it cannot execute the correct manipulation on the target object. Conversely, the DR-trained model learns to extract essential geometric features and physical constraints, filtering out superficial visual noise. This leads to significantly higher robustness in dynamic environments. The structural difference in execution is clearly reflected in the end-effector trajectory data: These real-world deployment recordings further illustrate this difference (Videos 1 and 2). Scaling Physical AI requires turning raw trajectory data into robust policies, and a rigorously engineered DR infrastructure is an essential bridge to close the Sim2Real gap.
0:20

Domain Randomization (DR) is a key component of the data augmentation pipeline at Axis Robotics. By applying DR, we are able to scale verified, high-quality human trajectories by 10x to 100x. During training, we systematically introduce variances in environmental parameters. This prevents the model from relying on spurious visual correlations. The objective is to ensure the policy learns rather than overfitting. To demonstrate the necessity and effectiveness of this approach, we evaluated both DR and No-DR models on Task 74 (pour_water_into_mug). The empirical results show a definitive impact on real-world deployment reliability: integrating DR into the pipeline increased the success rate from 0% to 90% (Fig. 1). This divergence stems from how the respective policies process visual observations (Fig. 2). The baseline (No DR) model overfits to the static visual background. It essentially memorizes the poses from the training dataset but fails to generalize when subjected to the inevitable variances of real-world deployment. Consequently, it cannot execute the correct manipulation on the target object. Conversely, the DR-trained model learns to extract essential geometric features and physical constraints, filtering out superficial visual noise. This leads to significantly higher robustness in dynamic environments. The structural difference in execution is clearly reflected in the end-effector trajectory data: These real-world deployment recordings further illustrate this difference (Videos 1 and 2). Scaling Physical AI requires turning raw trajectory data into robust policies, and a rigorously engineered DR infrastructure is an essential bridge to close the Sim2Real gap.

Axis Robotics

27,125 views โ€ข 3 months ago

Robots might learn better from video than from language! ๐Ÿ“ผ Most Vision-Language-Action (VLA) models learn what to do from text, but still struggle with how things move in the real world. That makes them data-hungry and slow to train. mimic video takes a different route. Instead of grounding robot control in text, it grounds it in video, using large pre-trained video models that already capture physical motion and dynamics. The idea is straightforward: let the video model handle โ€œwhat will happen next,โ€ and let a smaller control model focus only on turning that visual plan into robot actions. The result is big gains in practice. Robots trained this way need 10ร— less data, converge twice as fast, and perform better on both simulated benchmarks and real bimanual manipulation tasks. If robots can โ€œimagineโ€ motion using video, control becomes a much simpler problem. Shoutout to Jonas Pai, Liam Achenbach, Oier Mees, Elvis Nava and the rest of the team! Here's the project page: ~~ โ™ป๏ธ Join the weekly robotics newsletter, and never miss any news โ†’
0:47

Robots might learn better from video than from language! ๐Ÿ“ผ Most Vision-Language-Action (VLA) models learn what to do from text, but still struggle with how things move in the real world. That makes them data-hungry and slow to train. mimic video takes a different route. Instead of grounding robot control in text, it grounds it in video, using large pre-trained video models that already capture physical motion and dynamics. The idea is straightforward: let the video model handle โ€œwhat will happen next,โ€ and let a smaller control model focus only on turning that visual plan into robot actions. The result is big gains in practice. Robots trained this way need 10ร— less data, converge twice as fast, and perform better on both simulated benchmarks and real bimanual manipulation tasks. If robots can โ€œimagineโ€ motion using video, control becomes a much simpler problem. Shoutout to Jonas Pai, Liam Achenbach, Oier Mees, Elvis Nava and the rest of the team! Here's the project page: ~~ โ™ป๏ธ Join the weekly robotics newsletter, and never miss any news โ†’

Lukas Ziegler

50,063 views โ€ข 7 months ago

Announcing DreamDojo: our open-source, interactive world model that takes robot motor controls and generates the future in pixels. No engine, no meshes, no hand-authored dynamics. It's Simulation 2.0. Time for robotics to take the bitter lesson pill. Real-world robot learning is bottlenecked by time, wear, safety, and resets. If we want Physical AI to move at pretraining speed, we need a simulator that adapts to pretraining scale with as little human engineering as possible. Our key insights: (1) human egocentric videos are a scalable source of first-person physics; (2) latent actions make them "robot-readable" across different hardware; (3) real-time inference unlocks live teleop, policy eval, and test-time planning *inside* a dream. We pre-train on 44K hours of human videos: cheap, abundant, and collected with zero robot-in-the-loop. Humans have already explored the combinatorics: we grasp, pour, fold, assemble, fail, retryโ€”across cluttered scenes, shifting viewpoints, changing light, and hour-long task chainsโ€”at a scale no robot fleet could match. The missing piece: these videos have no action labels. So we introduce latent actions: a unified representation inferred directly from videos that captures "what changed between world states" without knowing the underlying hardware. This lets us train on any first-person video as if it came with motor commands attached. As a result, DreamDojo generalizes zero-shot to objects and environments never seen in any robot training set, because humans saw them first. Next, we post-train onto each robot to fit its specific hardware. Think of it as separating "how the world looks and behaves" from "how this particular robot actuates." The base model follows the general physical rules, then "snaps onto" the robot's unique mechanics. It's kind of like loading a new character and scene assets into Unreal Engine, but done through gradient descent and generalizes far beyond the post-training dataset. A world simulator is only useful if it runs fast enough to close the loop. We train a real-time version of DreamDojo that runs at 10 FPS, stable for over a minute of continuous rollout. This unlocks exciting possibilities: - Live teleoperation *inside* a dream. Connect a VR controller, stream actions into DreamDojo, and teleop a virtual robot in real time. We demo this on Unitree G1 with a PICO headset and one RTX 5090. - Policy evaluation. You can benchmark a policy checkpoint in DreamDojo instead of the real world. The simulated success rates strongly correlate with real-world results - accurate enough to rank checkpoints without burning a single motor. - Model-based planning. Sample multiple action proposals โ†’ simulate them all in parallel โ†’ pick the best future. Gains +17% real-world success out of the box on a fruit packing task. We open-source everything!! Weights, code, post-training dataset, eval set, and whitepaper with tons of details to reproduce. DreamDojo is based on NVIDIA Cosmos, which is open-weight too. 2026 is the year of World Models for physical AI. We want you to build with us. Happy scaling! Links in thread:
1:17

Announcing DreamDojo: our open-source, interactive world model that takes robot motor controls and generates the future in pixels. No engine, no meshes, no hand-authored dynamics. It's Simulation 2.0. Time for robotics to take the bitter lesson pill. Real-world robot learning is bottlenecked by time, wear, safety, and resets. If we want Physical AI to move at pretraining speed, we need a simulator that adapts to pretraining scale with as little human engineering as possible. Our key insights: (1) human egocentric videos are a scalable source of first-person physics; (2) latent actions make them "robot-readable" across different hardware; (3) real-time inference unlocks live teleop, policy eval, and test-time planning *inside* a dream. We pre-train on 44K hours of human videos: cheap, abundant, and collected with zero robot-in-the-loop. Humans have already explored the combinatorics: we grasp, pour, fold, assemble, fail, retryโ€”across cluttered scenes, shifting viewpoints, changing light, and hour-long task chainsโ€”at a scale no robot fleet could match. The missing piece: these videos have no action labels. So we introduce latent actions: a unified representation inferred directly from videos that captures "what changed between world states" without knowing the underlying hardware. This lets us train on any first-person video as if it came with motor commands attached. As a result, DreamDojo generalizes zero-shot to objects and environments never seen in any robot training set, because humans saw them first. Next, we post-train onto each robot to fit its specific hardware. Think of it as separating "how the world looks and behaves" from "how this particular robot actuates." The base model follows the general physical rules, then "snaps onto" the robot's unique mechanics. It's kind of like loading a new character and scene assets into Unreal Engine, but done through gradient descent and generalizes far beyond the post-training dataset. A world simulator is only useful if it runs fast enough to close the loop. We train a real-time version of DreamDojo that runs at 10 FPS, stable for over a minute of continuous rollout. This unlocks exciting possibilities: - Live teleoperation *inside* a dream. Connect a VR controller, stream actions into DreamDojo, and teleop a virtual robot in real time. We demo this on Unitree G1 with a PICO headset and one RTX 5090. - Policy evaluation. You can benchmark a policy checkpoint in DreamDojo instead of the real world. The simulated success rates strongly correlate with real-world results - accurate enough to rank checkpoints without burning a single motor. - Model-based planning. Sample multiple action proposals โ†’ simulate them all in parallel โ†’ pick the best future. Gains +17% real-world success out of the box on a fruit packing task. We open-source everything!! Weights, code, post-training dataset, eval set, and whitepaper with tons of details to reproduce. DreamDojo is based on NVIDIA Cosmos, which is open-weight too. 2026 is the year of World Models for physical AI. We want you to build with us. Happy scaling! Links in thread:

Jim Fan

226,180 views โ€ข 5 months ago

Introducing FLUX-mimic, a next-generation Video-Action Model for general purpose dexterity, developed in partnership with Black Forest Labs. Late last year we published mimic-video and introduced Video-Action Models (VAM): a new family of robotics foundation models built on top of video generation models. We showed that robot control reduces to visual prediction, and that robot capability is downstream of improvements in video modeling accuracy. The obvious implication was that advances in the video modeling frontier would directly translate to increased capabilities in end-to-end robot learning. FLUX-mimic is that thesis at frontier scale: We've applied our VAM architecture to the strongest video backbone available today, FLUX 3 from Black Forest Labs, and trained it on data from our own robots and wearables. General-purpose dexterity, running on a single GPU on premises. Because the model already understands world dynamics, it needs far fewer demonstrations to learn a new task. This is game-changing for our mission to deploy robots to factory floors, where industrial robot data is scarce and expensive to collect. We're now testing and deploying FLUX-mimic with manufacturing leaders like Audi USA, on complex, multi-step manipulation long considered impossible for conventional automation.
3:49

Introducing FLUX-mimic, a next-generation Video-Action Model for general purpose dexterity, developed in partnership with Black Forest Labs. Late last year we published mimic-video and introduced Video-Action Models (VAM): a new family of robotics foundation models built on top of video generation models. We showed that robot control reduces to visual prediction, and that robot capability is downstream of improvements in video modeling accuracy. The obvious implication was that advances in the video modeling frontier would directly translate to increased capabilities in end-to-end robot learning. FLUX-mimic is that thesis at frontier scale: We've applied our VAM architecture to the strongest video backbone available today, FLUX 3 from Black Forest Labs, and trained it on data from our own robots and wearables. General-purpose dexterity, running on a single GPU on premises. Because the model already understands world dynamics, it needs far fewer demonstrations to learn a new task. This is game-changing for our mission to deploy robots to factory floors, where industrial robot data is scarce and expensive to collect. We're now testing and deploying FLUX-mimic with manufacturing leaders like Audi USA, on complex, multi-step manipulation long considered impossible for conventional automation.

mimic

114,454 views โ€ข 9 days ago

Most video-action robot models are a content-creation video generator with an action module attached. LingBot-VA 2.0 from Robbyant, a video-action foundation model, throws that starting point out and trains the whole stack natively for control. And it runs closed-loop at a peak 225 Hz. It's so important because A robot cannot move responsively when its controller pauses to imagine the next few frames. LingBot-VA 2.0 predicts during execution, then corrects using each real observation. And it carries only about 13B video parameters while activating roughly 1.9B per token. Bigger robot models usually mean slower reactions, creating a direct conflict between intelligence and control. LingBot-VA 2.0 is trained from scratch for robot control rather than adapted from a video generator built for content creation. Robbyant, an embodied AI company under Ant Group, built it to learn how scenes change under actions, predict what should happen next, and turn those predictions into real-time robot movements. Most video-action systems inherit a tokenizer and video backbone trained mainly to reproduce visual appearance. LingBot-VA 2.0 rebuilds both parts around physical control. Its semantic visual-action tokenizer maps observations toward features from a frozen vision foundation model and learns compact latent actions from frame-to-frame changes using self-supervised inverse and forward dynamics. Unlabeled web video can therefore carry action-relevant training signals without robot action labels. The policy is causal from the start, so every prediction can use only past observations. Its sparse Mixture-of-Experts video backbone has about 13B total parameters, while about 1.9B are active per token, keeping the compute lower during each step. A high-level vision-language planner breaks long tasks into smaller instructions, while the low-level video-action policy handles continuous movement. Foresight Reasoning predicts future visual states while the robot is already acting, then replaces imagined states with every new real observation. Combined with few-step distillation and systems acceleration, the paper reports a peak asynchronous execution frequency of 225 Hz. The model adapts from 10โ€“15 demonstrations, transfers across robot embodiments, and handles some new tasks zero-shot. In the paperโ€™s own evaluations, it reaches 93.6 average on RoboTwin 2.0 and reports stronger real-world results than LingBot-VA and ฯ€0.5 across the tested tasks. ๐Ÿงต 1.
1:17

Most video-action robot models are a content-creation video generator with an action module attached. LingBot-VA 2.0 from Robbyant, a video-action foundation model, throws that starting point out and trains the whole stack natively for control. And it runs closed-loop at a peak 225 Hz. It's so important because A robot cannot move responsively when its controller pauses to imagine the next few frames. LingBot-VA 2.0 predicts during execution, then corrects using each real observation. And it carries only about 13B video parameters while activating roughly 1.9B per token. Bigger robot models usually mean slower reactions, creating a direct conflict between intelligence and control. LingBot-VA 2.0 is trained from scratch for robot control rather than adapted from a video generator built for content creation. Robbyant, an embodied AI company under Ant Group, built it to learn how scenes change under actions, predict what should happen next, and turn those predictions into real-time robot movements. Most video-action systems inherit a tokenizer and video backbone trained mainly to reproduce visual appearance. LingBot-VA 2.0 rebuilds both parts around physical control. Its semantic visual-action tokenizer maps observations toward features from a frozen vision foundation model and learns compact latent actions from frame-to-frame changes using self-supervised inverse and forward dynamics. Unlabeled web video can therefore carry action-relevant training signals without robot action labels. The policy is causal from the start, so every prediction can use only past observations. Its sparse Mixture-of-Experts video backbone has about 13B total parameters, while about 1.9B are active per token, keeping the compute lower during each step. A high-level vision-language planner breaks long tasks into smaller instructions, while the low-level video-action policy handles continuous movement. Foresight Reasoning predicts future visual states while the robot is already acting, then replaces imagined states with every new real observation. Combined with few-step distillation and systems acceleration, the paper reports a peak asynchronous execution frequency of 225 Hz. The model adapts from 10โ€“15 demonstrations, transfers across robot embodiments, and handles some new tasks zero-shot. In the paperโ€™s own evaluations, it reaches 93.6 average on RoboTwin 2.0 and reports stronger real-world results than LingBot-VA and ฯ€0.5 across the tested tasks. ๐Ÿงต 1.

Rohan Paul

11,253 views โ€ข 19 days ago

NEWS: Humanoid robotics company Figure has released Helix 02, what they claim in their most capable humanoid model yet. "A single neural system that controls the full body directly from pixels, enabling dexterous, long horizon autonomy across an entire room: โ€ข Autonomous, longโ€‘horizon loco-manipulation: Helix 02 unloads and reloads a dishwasher across a full-sized kitchen - a four-minute, end-to-end autonomous task that integrates walking, manipulation, and balance with no resets and no human intervention. We believe this is the longest horizon, most complex task completed autonomously by a humanoid robot to date. โ€ข All sensors in. All actuators out: Helix 02 connects every onboard sensor - vision, touch, and proprioception - directly to every actuator through a single unified visuomotor neural network. โ€ข Human-like whole body control from human data: All results are enabled by System 0, a learned wholeโ€‘body controller trained on over 1,000 hours of human motion data and simโ€‘toโ€‘real reinforcement learning. System 0 replaces 109,504 lines of handโ€‘engineered C++ with a single neural prior for stable, natural motion. โ€ข New classes of dexterity: With Figure 03โ€™s embedded tactile sensing and palm cameras, Helix 02 performs manipulation that was previously out of reach: extracting individual pills, dispensing precise syringe volumes, and singulating small, irregular objects from clutter despite selfโ€‘occlusion. Helix 02 is trained on over 1,000 hours of human motion data and integrates vision, touch, and proprioception."
3:25

NEWS: Humanoid robotics company Figure has released Helix 02, what they claim in their most capable humanoid model yet. "A single neural system that controls the full body directly from pixels, enabling dexterous, long horizon autonomy across an entire room: โ€ข Autonomous, longโ€‘horizon loco-manipulation: Helix 02 unloads and reloads a dishwasher across a full-sized kitchen - a four-minute, end-to-end autonomous task that integrates walking, manipulation, and balance with no resets and no human intervention. We believe this is the longest horizon, most complex task completed autonomously by a humanoid robot to date. โ€ข All sensors in. All actuators out: Helix 02 connects every onboard sensor - vision, touch, and proprioception - directly to every actuator through a single unified visuomotor neural network. โ€ข Human-like whole body control from human data: All results are enabled by System 0, a learned wholeโ€‘body controller trained on over 1,000 hours of human motion data and simโ€‘toโ€‘real reinforcement learning. System 0 replaces 109,504 lines of handโ€‘engineered C++ with a single neural prior for stable, natural motion. โ€ข New classes of dexterity: With Figure 03โ€™s embedded tactile sensing and palm cameras, Helix 02 performs manipulation that was previously out of reach: extracting individual pills, dispensing precise syringe volumes, and singulating small, irregular objects from clutter despite selfโ€‘occlusion. Helix 02 is trained on over 1,000 hours of human motion data and integrates vision, touch, and proprioception."

Sawyer Merritt

624,689 views โ€ข 6 months ago

Excited to announce GR00T N1, the worldโ€™s first open foundation model for humanoid robots! We are on a mission to democratize Physical AI. The power of general robot brain, in the palm of your hand - with only 2B parameters, N1 learns from the most diverse physical action dataset ever compiled and punches above its weight: - Real humanoid teleoperation data. - Large-scale simulation data: we are open-sourcing 300K+ trajectories! - Neural trajectories: we apply SOTA video generation models to โ€œhallucinateโ€ new synthetic data that features accurate physics in pixels. Using Jensenโ€™s words, โ€œsystematically infinite dataโ€! - Latent actions: we develop novel algorithms to extract action tokens from in-the-wild human videos and neural generated videos. GR00T N1 is a single end-to-end neural net, from photons to actions: - Vision-Language Model (System 2) that interprets the physical world through vision and language instructions, enabling robots to reason about their environment and instructions, and plan the right actions. - Diffusion Transformer (System 1) that โ€œrendersโ€ smooth and precise motor actions at 120 Hz, executing the latent plan made by System 2. We deploy N1 on GR1 robot, 1X Neo robot, and a large collection of simulation benchmarks. N1 achieves up to +30% boost in diverse manipulation tasks for household and industrial settings. While humanoid robots are the main focus of N1, our model also supports cross-embodiment. We finetune it to work on the $110 HuggingFace LeRobot SO100 robot arm! Open robot brain runs on open hardware. Sounds just right. Letโ€™s solve robotics, together, one token at a time. Links to our Whitepaper, Github repo, HuggingFace model, and open dataset page in the thread: ๐Ÿงต
2:21

Excited to announce GR00T N1, the worldโ€™s first open foundation model for humanoid robots! We are on a mission to democratize Physical AI. The power of general robot brain, in the palm of your hand - with only 2B parameters, N1 learns from the most diverse physical action dataset ever compiled and punches above its weight: - Real humanoid teleoperation data. - Large-scale simulation data: we are open-sourcing 300K+ trajectories! - Neural trajectories: we apply SOTA video generation models to โ€œhallucinateโ€ new synthetic data that features accurate physics in pixels. Using Jensenโ€™s words, โ€œsystematically infinite dataโ€! - Latent actions: we develop novel algorithms to extract action tokens from in-the-wild human videos and neural generated videos. GR00T N1 is a single end-to-end neural net, from photons to actions: - Vision-Language Model (System 2) that interprets the physical world through vision and language instructions, enabling robots to reason about their environment and instructions, and plan the right actions. - Diffusion Transformer (System 1) that โ€œrendersโ€ smooth and precise motor actions at 120 Hz, executing the latent plan made by System 2. We deploy N1 on GR1 robot, 1X Neo robot, and a large collection of simulation benchmarks. N1 achieves up to +30% boost in diverse manipulation tasks for household and industrial settings. While humanoid robots are the main focus of N1, our model also supports cross-embodiment. We finetune it to work on the $110 HuggingFace LeRobot SO100 robot arm! Open robot brain runs on open hardware. Sounds just right. Letโ€™s solve robotics, together, one token at a time. Links to our Whitepaper, Github repo, HuggingFace model, and open dataset page in the thread: ๐Ÿงต

Jim Fan

466,333 views โ€ข 1 year ago

๐Ÿ‡ธ๐Ÿ‡พ๐Ÿ‡บ๐Ÿ‡ธ Richard Black, Former Virginia State Senator: Al-Qaeda has always been our proxy force on the ground; they, together with ISIS, have carried out the mission of the United States; we are supporters of al-Qaeda today where theyโ€™re bottled up in Idlib Province; we wanted to starve and we wanted to freeze to death the people of Syria; we wanted to take the wheat away to cause famine among the Syrian people; there was an organized campaign of rape across the nation of Syria; the highest prices went to the youngest children because there were a great number of pedophiles and the pedophiles wanted to possess small children; we cut off the medical supplies so that the women in Syria would die of breast cancer because they could not get the medications; the United States has a strategic policy of using proxies to engage in war; when we fight these wars we have no limits on the cruelty and the inhumanity that weโ€™re prepared to impose on the people; ๐Ÿ”ธ The war [in Syria] began in 2011 when the United States landed Central Intelligence operatives to begin coordinating with Al-Qaeda and other terrorist groups; ๐Ÿ”ธ We have been unwavering supporters of Al-Qaeda since before the war formally began; ๐Ÿ”ธ We are supporters of Al-Qaeda today where theyโ€™re bottled up in Idlib Province; ๐Ÿ”ธ The CIA supplied them under secret operation Timber Sycamore; ๐Ÿ”ธ We gave them all of their anti-tank weapons and all of their anti-air missiles; ๐Ÿ”ธ Al-Qaeda has always been our proxy force on the ground; they together with ISIS have carried out the mission of the United States together with a great number of affiliates that really are kind of interchangeableโ€”you have the Free Syrian Army; soldiers move from ISIS to Al-Qaeda to Free Syrian Army rather fluidly; ๐Ÿ”ธ The United States has a strategic policy of using proxies to engage in war; ๐Ÿ”ธ Our objective was to overthrow the legitimate government of Syria and, in order to do that, we employed proxy soldiers who were the the most vile of all terrorists; ๐Ÿ”ธ Something very similar is happening right now in Ukraine; ๐Ÿ”ธ Plan B was the American seizure of the northern portion of Syria; it is the bread basket for all of the Syrian people; we wanted to take the wheat away to cause famine among the Syrian people; by stealing the oil and the gas we would be able to shut down the transportation system, and, at the same time, during the Syrian winters we could freeze to death the Syrian civilian population which, in many cases, were living in rubble where these terrorist armies with mechanized divisions had attacked and just totally destroyed these cities and left people just living in little pockets of rubble; ๐Ÿ”ธ We wanted to starve and we wanted to freeze to death the people of Syria; ๐Ÿ”ธ The Caesar sanctions were the most brutal sanctions ever imposed on ever any nation; during the Second World War sanctions were not nearly as strict as they were on Syria; ๐Ÿ”ธ We devalued their currency through the SWIFT system for international payments making it impossible for them to purchase medications; ๐Ÿ”ธ We cut off the medical supplies so that the women in Syria would die of breast cancer because they could not get the medications; ๐Ÿ”ธ One of the last things that we did, and the evidence is vague on it, but there was a mysterious explosion in the harbor [of Beirut] in Lebanon; it was a massive explosion of a shipload of ammonium nitrate fertilizer; it killed hundreds of Lebanese people; it wounded thousands and thousands, destroyed the economy of Lebanon, and, most importantly, it destroyed the banking system of Lebanon which was one of the few lifelines remaining to Syria; I donโ€™t think that explosion was accidental; I think it was orchestrated and I suspect that the Central Intelligence Agency was aware of the nation that carried out that action to destroy Beirut Harbor; ๐Ÿ”ธ Throughout you see this Machiavellian approach where we use unlimited force and violence and, at the same time, we control the the global media to where we erase all discussions of whatโ€™s truly happening; so to the man and woman in the street, they think things are fine, that everything is being done for altruistic reasons, but itโ€™s not; ๐Ÿ”ธ One of the things that we did as we as we allied ourselves with Al-Qaeda and on-and-off with ISIS; we fought ISIS in a very serious way but at the same time we often employed them to use against the Syrian government; so itโ€™s kind of a love-hate [relationship]; but we have always worked with the terrorists, they were the core; ๐Ÿ”ธ We facilitated the movement of Islamic terrorists from one hundred countries and they came and they joined ISIS, they joined Al-Qaeda, they joined the Free Syrian Army, all of these different ones; ๐Ÿ”ธ There was an organized campaign of rape across the nation of Syria; the highest prices went to the youngest children because there were a great number of pedophiles and the pedophiles wanted to possess small children; they were permitted to rape these children repeatedly; they were able to rape the widows of the slain civilians and possess them and buy them and sell them among themselves; ๐Ÿ”ธ There were so many tens hundreds of thousands of Syrian women impregnated by these terrorists who were imported into Syria that it was necessary to change the law so that they would have Syrian citizenship and they wouldnโ€™t have to be returned to their ISIS father in Saudi Arabia or in Tunisia; ๐Ÿ”ธ When we fight these wars we have no limits on the cruelty and the inhumanity that weโ€™re prepared to impose on the people, making them suffer, so that somehow that will translate into overthrowing the government and, perhaps, taking their oil, taking their resources;
8:45

๐Ÿ‡ธ๐Ÿ‡พ๐Ÿ‡บ๐Ÿ‡ธ Richard Black, Former Virginia State Senator: Al-Qaeda has always been our proxy force on the ground; they, together with ISIS, have carried out the mission of the United States; we are supporters of al-Qaeda today where theyโ€™re bottled up in Idlib Province; we wanted to starve and we wanted to freeze to death the people of Syria; we wanted to take the wheat away to cause famine among the Syrian people; there was an organized campaign of rape across the nation of Syria; the highest prices went to the youngest children because there were a great number of pedophiles and the pedophiles wanted to possess small children; we cut off the medical supplies so that the women in Syria would die of breast cancer because they could not get the medications; the United States has a strategic policy of using proxies to engage in war; when we fight these wars we have no limits on the cruelty and the inhumanity that weโ€™re prepared to impose on the people; ๐Ÿ”ธ The war [in Syria] began in 2011 when the United States landed Central Intelligence operatives to begin coordinating with Al-Qaeda and other terrorist groups; ๐Ÿ”ธ We have been unwavering supporters of Al-Qaeda since before the war formally began; ๐Ÿ”ธ We are supporters of Al-Qaeda today where theyโ€™re bottled up in Idlib Province; ๐Ÿ”ธ The CIA supplied them under secret operation Timber Sycamore; ๐Ÿ”ธ We gave them all of their anti-tank weapons and all of their anti-air missiles; ๐Ÿ”ธ Al-Qaeda has always been our proxy force on the ground; they together with ISIS have carried out the mission of the United States together with a great number of affiliates that really are kind of interchangeableโ€”you have the Free Syrian Army; soldiers move from ISIS to Al-Qaeda to Free Syrian Army rather fluidly; ๐Ÿ”ธ The United States has a strategic policy of using proxies to engage in war; ๐Ÿ”ธ Our objective was to overthrow the legitimate government of Syria and, in order to do that, we employed proxy soldiers who were the the most vile of all terrorists; ๐Ÿ”ธ Something very similar is happening right now in Ukraine; ๐Ÿ”ธ Plan B was the American seizure of the northern portion of Syria; it is the bread basket for all of the Syrian people; we wanted to take the wheat away to cause famine among the Syrian people; by stealing the oil and the gas we would be able to shut down the transportation system, and, at the same time, during the Syrian winters we could freeze to death the Syrian civilian population which, in many cases, were living in rubble where these terrorist armies with mechanized divisions had attacked and just totally destroyed these cities and left people just living in little pockets of rubble; ๐Ÿ”ธ We wanted to starve and we wanted to freeze to death the people of Syria; ๐Ÿ”ธ The Caesar sanctions were the most brutal sanctions ever imposed on ever any nation; during the Second World War sanctions were not nearly as strict as they were on Syria; ๐Ÿ”ธ We devalued their currency through the SWIFT system for international payments making it impossible for them to purchase medications; ๐Ÿ”ธ We cut off the medical supplies so that the women in Syria would die of breast cancer because they could not get the medications; ๐Ÿ”ธ One of the last things that we did, and the evidence is vague on it, but there was a mysterious explosion in the harbor [of Beirut] in Lebanon; it was a massive explosion of a shipload of ammonium nitrate fertilizer; it killed hundreds of Lebanese people; it wounded thousands and thousands, destroyed the economy of Lebanon, and, most importantly, it destroyed the banking system of Lebanon which was one of the few lifelines remaining to Syria; I donโ€™t think that explosion was accidental; I think it was orchestrated and I suspect that the Central Intelligence Agency was aware of the nation that carried out that action to destroy Beirut Harbor; ๐Ÿ”ธ Throughout you see this Machiavellian approach where we use unlimited force and violence and, at the same time, we control the the global media to where we erase all discussions of whatโ€™s truly happening; so to the man and woman in the street, they think things are fine, that everything is being done for altruistic reasons, but itโ€™s not; ๐Ÿ”ธ One of the things that we did as we as we allied ourselves with Al-Qaeda and on-and-off with ISIS; we fought ISIS in a very serious way but at the same time we often employed them to use against the Syrian government; so itโ€™s kind of a love-hate [relationship]; but we have always worked with the terrorists, they were the core; ๐Ÿ”ธ We facilitated the movement of Islamic terrorists from one hundred countries and they came and they joined ISIS, they joined Al-Qaeda, they joined the Free Syrian Army, all of these different ones; ๐Ÿ”ธ There was an organized campaign of rape across the nation of Syria; the highest prices went to the youngest children because there were a great number of pedophiles and the pedophiles wanted to possess small children; they were permitted to rape these children repeatedly; they were able to rape the widows of the slain civilians and possess them and buy them and sell them among themselves; ๐Ÿ”ธ There were so many tens hundreds of thousands of Syrian women impregnated by these terrorists who were imported into Syria that it was necessary to change the law so that they would have Syrian citizenship and they wouldnโ€™t have to be returned to their ISIS father in Saudi Arabia or in Tunisia; ๐Ÿ”ธ When we fight these wars we have no limits on the cruelty and the inhumanity that weโ€™re prepared to impose on the people, making them suffer, so that somehow that will translate into overthrowing the government and, perhaps, taking their oil, taking their resources;

๐Ÿ…ฐpocalypsis ๐Ÿ…ฐpocalypseos ๐Ÿ‡ท๐Ÿ‡บ ๐Ÿ‡จ๐Ÿ‡ณ ๐Ÿ…‰

225,994 views โ€ข 1 year ago

JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models paper page: Achieving human-like planning and control with multimodal observations in an open world is a key milestone for more functional generalist agents. Existing approaches can handle certain long-horizon tasks in an open world. However, they still struggle when the number of open-world tasks could potentially be infinite and lack the capability to progressively enhance task completion as game time progresses. We introduce JARVIS-1, an open-world agent that can perceive multimodal input (visual observations and human instructions), generate sophisticated plans, and perform embodied control, all within the popular yet challenging open-world Minecraft universe. Specifically, we develop JARVIS-1 on top of pre-trained multimodal language models, which map visual observations and textual instructions to plans. The plans will be ultimately dispatched to the goal-conditioned controllers. We outfit JARVIS-1 with a multimodal memory, which facilitates planning using both pre-trained knowledge and its actual game survival experiences. In our experiments, JARVIS-1 exhibits nearly perfect performances across over 200 varying tasks from the Minecraft Universe Benchmark, ranging from entry to intermediate levels. JARVIS-1 has achieved a completion rate of 12.5% in the long-horizon diamond pickaxe task. This represents a significant increase up to 5 times compared to previous records. Furthermore, we show that JARVIS-1 is able to self-improve following a life-long learning paradigm thanks to multimodal memory, sparking a more general intelligence and improved autonomy.
0:22

JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models paper page: Achieving human-like planning and control with multimodal observations in an open world is a key milestone for more functional generalist agents. Existing approaches can handle certain long-horizon tasks in an open world. However, they still struggle when the number of open-world tasks could potentially be infinite and lack the capability to progressively enhance task completion as game time progresses. We introduce JARVIS-1, an open-world agent that can perceive multimodal input (visual observations and human instructions), generate sophisticated plans, and perform embodied control, all within the popular yet challenging open-world Minecraft universe. Specifically, we develop JARVIS-1 on top of pre-trained multimodal language models, which map visual observations and textual instructions to plans. The plans will be ultimately dispatched to the goal-conditioned controllers. We outfit JARVIS-1 with a multimodal memory, which facilitates planning using both pre-trained knowledge and its actual game survival experiences. In our experiments, JARVIS-1 exhibits nearly perfect performances across over 200 varying tasks from the Minecraft Universe Benchmark, ranging from entry to intermediate levels. JARVIS-1 has achieved a completion rate of 12.5% in the long-horizon diamond pickaxe task. This represents a significant increase up to 5 times compared to previous records. Furthermore, we show that JARVIS-1 is able to self-improve following a life-long learning paradigm thanks to multimodal memory, sparking a more general intelligence and improved autonomy.

AK

141,440 views โ€ข 2 years ago

General Intuition CEO Pim de Witte, who's building foundation models trained on video game controller input data ("action-labeled gameplay clips"), says general intelligence won't "taste like an LLM": "We have a scale of data that's going to allow us to jump to the frontier in one capability โ€” which is any system that can be controlled with a game controller (which is most robots) โ€” and then, you can use that to create a sufficiently general intelligence." "As humans, the decision to talk or type is a very, very small subset of the actions that we can actually take." "So in order to create a sufficiently general intelligence to play 10,000+ video games, the model has to be able to predict across the entire action space of human cognition when they're interacting with these environments. Which are 2D and 3D environments, interfaces, long-horizon tasks, short-horizon tasks, [etc.]." "It has to be a sufficiently general intelligence in order to predict actions. Therefore, the type of model you get out is not going to taste like an LLM. This model is going to be incredibly good at navigating unforeseen environments. It's going to be incredibly good at zero-shotting any task that can be done with a game controller."
1:53

General Intuition CEO Pim de Witte, who's building foundation models trained on video game controller input data ("action-labeled gameplay clips"), says general intelligence won't "taste like an LLM": "We have a scale of data that's going to allow us to jump to the frontier in one capability โ€” which is any system that can be controlled with a game controller (which is most robots) โ€” and then, you can use that to create a sufficiently general intelligence." "As humans, the decision to talk or type is a very, very small subset of the actions that we can actually take." "So in order to create a sufficiently general intelligence to play 10,000+ video games, the model has to be able to predict across the entire action space of human cognition when they're interacting with these environments. Which are 2D and 3D environments, interfaces, long-horizon tasks, short-horizon tasks, [etc.]." "It has to be a sufficiently general intelligence in order to predict actions. Therefore, the type of model you get out is not going to taste like an LLM. This model is going to be incredibly good at navigating unforeseen environments. It's going to be incredibly good at zero-shotting any task that can be done with a game controller."

TBPN

80,199 views โ€ข 1 month ago

๐Ÿšจ BREAKING: Microsoft's first robotics foundation model! ๐Ÿคฏ Microsoft just announced Rho-alpha (ฯฮฑ), their first robotics model derived from the Phi series of vision-language models. Rho-alpha translates natural language commands into control signals for robotic systems performing bimanual manipulation tasks. Commands like "push the green button with the right gripper," "pull out the red wire," "flip the top switch on," or "turn the knob to position 5" get executed directly by dual-arm robots. What makes this different from standard vision-language-action (VLA) models is the additional modalities. Rho-alpha is a VLA+ model that adds tactile sensing to the perceptual mix, with plans to incorporate force feedback. On the learning side, the model is designed to continually improve during deployment by learning from human feedback. The training approach combines trajectories from physical demonstrations and simulated tasks with web-scale visual question answering data. Since teleoperation data is scarce and expensive, Microsoft is using NVIDIA Isaac Sim on Azure to generate physically accurate synthetic datasets via reinforcement learning. These simulated trajectories get combined with commercial and open physical demonstration datasets. The model is currently under evaluation on dual-arm setups and humanoid robots. Microsoft is opening an Early Access Program for organizations interested in evaluating Rho-alpha. Robots that can adapt to dynamic situations and human preferences are more useful in real environments and more trusted by the people operating them. Read more here: ~~ โ™ป๏ธ Join the weekly robotics newsletter, and never miss any news โ†’
0:34

๐Ÿšจ BREAKING: Microsoft's first robotics foundation model! ๐Ÿคฏ Microsoft just announced Rho-alpha (ฯฮฑ), their first robotics model derived from the Phi series of vision-language models. Rho-alpha translates natural language commands into control signals for robotic systems performing bimanual manipulation tasks. Commands like "push the green button with the right gripper," "pull out the red wire," "flip the top switch on," or "turn the knob to position 5" get executed directly by dual-arm robots. What makes this different from standard vision-language-action (VLA) models is the additional modalities. Rho-alpha is a VLA+ model that adds tactile sensing to the perceptual mix, with plans to incorporate force feedback. On the learning side, the model is designed to continually improve during deployment by learning from human feedback. The training approach combines trajectories from physical demonstrations and simulated tasks with web-scale visual question answering data. Since teleoperation data is scarce and expensive, Microsoft is using NVIDIA Isaac Sim on Azure to generate physically accurate synthetic datasets via reinforcement learning. These simulated trajectories get combined with commercial and open physical demonstration datasets. The model is currently under evaluation on dual-arm setups and humanoid robots. Microsoft is opening an Early Access Program for organizations interested in evaluating Rho-alpha. Robots that can adapt to dynamic situations and human preferences are more useful in real environments and more trusted by the people operating them. Read more here: ~~ โ™ป๏ธ Join the weekly robotics newsletter, and never miss any news โ†’

Lukas Ziegler

60,912 views โ€ข 6 months ago

Exciting updates on Project GR00T! We discover a systematic way to scale up robot data, tackling the most painful pain point in robotics. The idea is simple: human collects demonstration on a real robot, and we multiply that data 1000x or more in simulation. Letโ€™s break it down: 1. We use Apple Vision Pro (yes!!) to give the human operator first person control of the humanoid. Vision Pro parses human hand pose and retargets the motion to the robot hand, all in real time. From the humanโ€™s point of view, they are immersed in another body like the Avatar. Teleoperation is slow and time-consuming, but we can afford to collect a small amount of data. 2. We use RoboCasa, a generative simulation framework, to multiply the demonstration data by varying the visual appearance and layout of the environment. In Jensenโ€™s keynote video below, the humanoid is now placing the cup in hundreds of kitchens with a huge diversity of textures, furniture, and object placement. We only have 1 physical kitchen at the GEAR Lab in NVIDIA HQ, but we can conjure up infinite ones in simulation. 3. Finally, we apply MimicGen, a technique to multiply the above data even more by varying the *motion* of the robot. MimicGen generates vast number of new action trajectories based on the original human data, and filters out failed ones (e.g. those that drop the cup) to form a much larger dataset. To sum up, given 1 human trajectory with Vision Pro -> RoboCasa produces N (varying visuals) -> MimicGen further augments to NxM (varying motions). This is the way to trade compute for expensive human data by GPU-accelerated simulation. A while ago, I mentioned that teleoperation is fundamentally not scalable, because we are always limited by 24 hrs/robot/day in the world of atoms. Our new GR00T synthetic data pipeline breaks this barrier in the world of bits. Scaling has been so much fun for LLMs, and it's finally our turn to have fun in robotics! We are building tools to enable everyone in the ecosystem to scale up with us. Links in thread:
1:21

Exciting updates on Project GR00T! We discover a systematic way to scale up robot data, tackling the most painful pain point in robotics. The idea is simple: human collects demonstration on a real robot, and we multiply that data 1000x or more in simulation. Letโ€™s break it down: 1. We use Apple Vision Pro (yes!!) to give the human operator first person control of the humanoid. Vision Pro parses human hand pose and retargets the motion to the robot hand, all in real time. From the humanโ€™s point of view, they are immersed in another body like the Avatar. Teleoperation is slow and time-consuming, but we can afford to collect a small amount of data. 2. We use RoboCasa, a generative simulation framework, to multiply the demonstration data by varying the visual appearance and layout of the environment. In Jensenโ€™s keynote video below, the humanoid is now placing the cup in hundreds of kitchens with a huge diversity of textures, furniture, and object placement. We only have 1 physical kitchen at the GEAR Lab in NVIDIA HQ, but we can conjure up infinite ones in simulation. 3. Finally, we apply MimicGen, a technique to multiply the above data even more by varying the *motion* of the robot. MimicGen generates vast number of new action trajectories based on the original human data, and filters out failed ones (e.g. those that drop the cup) to form a much larger dataset. To sum up, given 1 human trajectory with Vision Pro -> RoboCasa produces N (varying visuals) -> MimicGen further augments to NxM (varying motions). This is the way to trade compute for expensive human data by GPU-accelerated simulation. A while ago, I mentioned that teleoperation is fundamentally not scalable, because we are always limited by 24 hrs/robot/day in the world of atoms. Our new GR00T synthetic data pipeline breaks this barrier in the world of bits. Scaling has been so much fun for LLMs, and it's finally our turn to have fun in robotics! We are building tools to enable everyone in the ecosystem to scale up with us. Links in thread:

Jim Fan

364,380 views โ€ข 2 years ago

Excited to share our new work on building a multimodal atlas of human skin in health and inflammatory disease โ€” a project Iโ€™m especially proud of, bringing together AI, high-throughput genomics, and clinical science to accelerate discovery. Over the past decade, single-cell genomics has transformed how we map cells in human tissues. But a major challenge remains: can we systematically decode how cells organize into functional niches in situ โ€” including those invisible to standard histopathology? To address this, we integrated large-scale scRNA-seq, spatial transcriptomics, histopathology, and AI-driven modeling frameworks to build an in situ atlas of human skin across health and disease. Led by Lloyd Steele, an MD/PhD student working between Haniffa Lab and my lab at Wellcome Sanger Institute and Cambridge University . Another amazing collaboration with Muzz Haniffa, the mastermind behind the work as part of Human Cell Atlas. A key part of this study is that we didnโ€™t build everything from scratch โ€” we leveraged and combined AI methods that actually work! and showed how they can be used together to extract biological insight at scale. We used: โ€ข scArches to build and map into a reference scRNA-seq atlas of human skin: โ€ข NicheCompass to identify and characterize spatial niches: โ€ข MINT-Flow to extract microenvironment-induced cell states and gene programs: Together, these enabled an end-to-end workflow from atlas construction to spatial mapping, niche discovery, and cell state decoding. At scale, we integrated ~5 million cells and 100+ spatial sections, enabling a systematic view of tissue organization. Using this framework, we identified 26 niches in skin, including known histopathologic structures as well as hidden disease-associated niches not visible on H&E. Among the most striking findings were a resident memory T cell-rich sebaceous gland niche and a plasma cell-rich sweat gland niche, suggesting that appendageal structures act as active immunological microenvironments and may contribute to inflammatory memory and disease persistence. Importantly, this atlas is not just descriptive โ€” it is usable. It can support mapping of new datasets, resolve finer cell types and niches, extract microenvironment-driven programs, and enable predictive analyses at scale. More broadly, this work shows what becomes possible when AI, spatial genomics, and atlas-scale data are integrated end-to-end: not just mapping tissues, but systematically decoding them. This was a massive collaboration, and Iโ€™m very grateful to the amazing scientists April Foster, Kenny Roberts, and Chloe Admane. Lloyd is an amazing scientist, and Iโ€™m especially excited for the community to see more of his work soon โ€” stay tuned. The data and pre-trained models will be released soon. Preprint:
1:09

Excited to share our new work on building a multimodal atlas of human skin in health and inflammatory disease โ€” a project Iโ€™m especially proud of, bringing together AI, high-throughput genomics, and clinical science to accelerate discovery. Over the past decade, single-cell genomics has transformed how we map cells in human tissues. But a major challenge remains: can we systematically decode how cells organize into functional niches in situ โ€” including those invisible to standard histopathology? To address this, we integrated large-scale scRNA-seq, spatial transcriptomics, histopathology, and AI-driven modeling frameworks to build an in situ atlas of human skin across health and disease. Led by Lloyd Steele, an MD/PhD student working between Haniffa Lab and my lab at Wellcome Sanger Institute and Cambridge University . Another amazing collaboration with Muzz Haniffa, the mastermind behind the work as part of Human Cell Atlas. A key part of this study is that we didnโ€™t build everything from scratch โ€” we leveraged and combined AI methods that actually work! and showed how they can be used together to extract biological insight at scale. We used: โ€ข scArches to build and map into a reference scRNA-seq atlas of human skin: โ€ข NicheCompass to identify and characterize spatial niches: โ€ข MINT-Flow to extract microenvironment-induced cell states and gene programs: Together, these enabled an end-to-end workflow from atlas construction to spatial mapping, niche discovery, and cell state decoding. At scale, we integrated ~5 million cells and 100+ spatial sections, enabling a systematic view of tissue organization. Using this framework, we identified 26 niches in skin, including known histopathologic structures as well as hidden disease-associated niches not visible on H&E. Among the most striking findings were a resident memory T cell-rich sebaceous gland niche and a plasma cell-rich sweat gland niche, suggesting that appendageal structures act as active immunological microenvironments and may contribute to inflammatory memory and disease persistence. Importantly, this atlas is not just descriptive โ€” it is usable. It can support mapping of new datasets, resolve finer cell types and niches, extract microenvironment-driven programs, and enable predictive analyses at scale. More broadly, this work shows what becomes possible when AI, spatial genomics, and atlas-scale data are integrated end-to-end: not just mapping tissues, but systematically decoding them. This was a massive collaboration, and Iโ€™m very grateful to the amazing scientists April Foster, Kenny Roberts, and Chloe Admane. Lloyd is an amazing scientist, and Iโ€™m especially excited for the community to see more of his work soon โ€” stay tuned. The data and pre-trained models will be released soon. Preprint:

Mo Lotfollahi

11,759 views โ€ข 4 months ago