Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

JoyIn unveiled Aether: a 4B-param energy-based model for robots, trained on ~200 hrs of human video, zero real-robot data. Not VLA (perception+language to action) or world models. It's an "action-energy" model: actions emerge from minimizing a learned energy function over perception, memory and dynamics. No planner mapping instructions to... show more

The Humanoid Hub

115,980 subscribers

12,885 Aufrufe • vor 8 Tagen •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

6 Kommentare

Profilbild von The Humanoid Hub

The Humanoid Hubvor 8 Tagen

The architecture

Profilbild von WAKE

WAKEvor 8 Tagen

About 200 hours of human video and zero real-robot data is a strong claim about where skill lives. If actions come from that footage alone, the scarce work is still episode quality: task, scene, and clean cuts before any robot body is in the loop.

Profilbild von Bidhan Roy

Bidhan Royvor 8 Tagen

200 hz means the energy search has a few milliseconds at most i am probably missing something

Profilbild von Nghẹo | DOPE

Nghẹo | DOPEvor 8 Tagen

Interesting approach for learning robot behavior from human video alone

Profilbild von Teacher in the loop 🤖

Teacher in the loop 🤖vor 8 Tagen

액션-에너지 모델은 처음 들어보네요 인간 영상만으로 어디까지 가능할지도 궁금합니다

Profilbild von Armstrong Too | Photo + Film + AI

Armstrong Too | Photo + Film + AIvor 8 Tagen

The action-energy framing is more interesting than the label. For robots, the useful question is whether the learned objective stays stable when perception is noisy and the scene changes. Recovery behavior will matter as much as the demo.

Ähnliche Videos

Today we are introducing Dyna-2, a world-action model pre-trained on one million hours of human video. At this scale, for the first time, we discovered several new scaling laws: • world-action models exhibit scaling law on human data across four orders of magnitude, from 1000 to 1,000,000 hours, • this human data scaling law implied a scaling law on never seen robot data, • both data and objective matter; world modeling and scaling on video data are essential for cross-embodiment scaling transfer to emerge 🧵

Today we are introducing Dyna-2, a world-action model pre-trained on one million hours of human video. At this scale, for the first time, we discovered several new scaling laws: • world-action models exhibit scaling law on human data across four orders of magnitude, from 1000 to 1,000,000 hours, • this human data scaling law implied a scaling law on never seen robot data, • both data and objective matter; world modeling and scaling on video data are essential for cross-embodiment scaling transfer to emerge 🧵

Dyna Robotics

1,801,414 Aufrufe • vor 1 Monat

Robots might learn better from video than from language! 📼 Most Vision-Language-Action (VLA) models learn what to do from text, but still struggle with how things move in the real world. That makes them data-hungry and slow to train. mimic video takes a different route. Instead of grounding robot control in text, it grounds it in video, using large pre-trained video models that already capture physical motion and dynamics. The idea is straightforward: let the video model handle “what will happen next,” and let a smaller control model focus only on turning that visual plan into robot actions. The result is big gains in practice. Robots trained this way need 10× less data, converge twice as fast, and perform better on both simulated benchmarks and real bimanual manipulation tasks. If robots can “imagine” motion using video, control becomes a much simpler problem. Shoutout to Jonas Pai, Liam Achenbach, Oier Mees, Elvis Nava and the rest of the team! Here's the project page: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Robots might learn better from video than from language! 📼 Most Vision-Language-Action (VLA) models learn what to do from text, but still struggle with how things move in the real world. That makes them data-hungry and slow to train. mimic video takes a different route. Instead of grounding robot control in text, it grounds it in video, using large pre-trained video models that already capture physical motion and dynamics. The idea is straightforward: let the video model handle “what will happen next,” and let a smaller control model focus only on turning that visual plan into robot actions. The result is big gains in practice. Robots trained this way need 10× less data, converge twice as fast, and perform better on both simulated benchmarks and real bimanual manipulation tasks. If robots can “imagine” motion using video, control becomes a much simpler problem. Shoutout to Jonas Pai, Liam Achenbach, Oier Mees, Elvis Nava and the rest of the team! Here's the project page: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

50,090 Aufrufe • vor 9 Monaten

VLA policies learn generalist robot behaviors from massive teleoperation datasets, hoping that the right behavior emerges. But they rarely use perception during training or inference: powerful foundation models of 3D geometry, semantics, or human motion are ignored. TimSong, Long Le, and our GRASP Laboratory team introduce Omniguide, based on a simple idea: Instead of retraining policies, let perception guide action at inference time. We express diverse guidance sources as attractive and repulsive energy fields in 3D space and inject their gradients into the generative process of VLA policies. This lets perception modules steer actions without retraining the policy: A modular path toward composable robot intelligence. Project: Paper:

VLA policies learn generalist robot behaviors from massive teleoperation datasets, hoping that the right behavior emerges. But they rarely use perception during training or inference: powerful foundation models of 3D geometry, semantics, or human motion are ignored. TimSong, Long Le, and our GRASP Laboratory team introduce Omniguide, based on a simple idea: Instead of retraining policies, let perception guide action at inference time. We express diverse guidance sources as attractive and repulsive energy fields in 3D space and inject their gradients into the generative process of VLA policies. This lets perception modules steer actions without retraining the policy: A modular path toward composable robot intelligence. Project: Paper:

Kostas Daniilidis

54,695 Aufrufe • vor 6 Monaten

Chinese robotics company Astribot released their latest World-Action Model (WAM), Lumo-2. Technical breakdown: - based on a frozen 🥶 Qwen-3.5 4B VLM - trained in 3 progressive stages: 1. Action is aligned with latent world dynamics (an abstract representation of action). Real-world actions are anchored to physical constraints, while the latent space is guided to focus on motion-relevant changes. This bidirectional relationship makes the model physically grounded -> critical for a world model. 2. Action is aligned with vision and language. Reusing the vision backbone and action encoder from the frozen VLM, the authors add a custom vocabulary (for new actions), a semantic module, an action decoder, and an action projector. This aligns the (new) action representations with the (existing) vision-language semantic space. Most importantly: it builds a direct mapping from natural-language instructions to motor execution. 3. End-to-end training on language, video, and robot data. Only the new modules (everything outside the frozen backbone) are trained end-to-end across temporal reasoning, physical understanding, long-horizon, and dexterous manipulation. At the end of the day, Lumo-2 is not the best on benchmarks, but that's not the point. What's genuinely new: - a way to combine latent world modeling and action generation through progressive alignment - a physically-grounded latent dynamics space - it lifts performance on unseen objects using un-annotated human egocentric video + Vision Pro captures, no special transfer algorithm needed Why it matters: - the whole model is thin trainable adapters (semantic module, action decoder/projector) on a frozen 4B backbone (cheap) - that scale is suited for real-time embedded inference (~2.71× decode speedup, no accuracy loss) - its real moat is long-horizon execution, where the added temporal memory pays off far more than on any other task As a result, this robot can now make your latte (5x sped up video):

Chinese robotics company Astribot released their latest World-Action Model (WAM), Lumo-2. Technical breakdown: - based on a frozen 🥶 Qwen-3.5 4B VLM - trained in 3 progressive stages: 1. Action is aligned with latent world dynamics (an abstract representation of action). Real-world actions are anchored to physical constraints, while the latent space is guided to focus on motion-relevant changes. This bidirectional relationship makes the model physically grounded -> critical for a world model. 2. Action is aligned with vision and language. Reusing the vision backbone and action encoder from the frozen VLM, the authors add a custom vocabulary (for new actions), a semantic module, an action decoder, and an action projector. This aligns the (new) action representations with the (existing) vision-language semantic space. Most importantly: it builds a direct mapping from natural-language instructions to motor execution. 3. End-to-end training on language, video, and robot data. Only the new modules (everything outside the frozen backbone) are trained end-to-end across temporal reasoning, physical understanding, long-horizon, and dexterous manipulation. At the end of the day, Lumo-2 is not the best on benchmarks, but that's not the point. What's genuinely new: - a way to combine latent world modeling and action generation through progressive alignment - a physically-grounded latent dynamics space - it lifts performance on unseen objects using un-annotated human egocentric video + Vision Pro captures, no special transfer algorithm needed Why it matters: - the whole model is thin trainable adapters (semantic module, action decoder/projector) on a frozen 4B backbone (cheap) - that scale is suited for real-time embedded inference (~2.71× decode speedup, no accuracy loss) - its real moat is long-horizon execution, where the added temporal memory pays off far more than on any other task As a result, this robot can now make your latte (5x sped up video):

Léo

32,331 Aufrufe • vor 2 Monaten

An interactive world model developed by NVIDIA in collaboration with academic partners. - DreamDojo turns egocentric human video data into physical intelligence. - Human data is more scalable than robotics data but lacks action labels. - To solve this, a dedicated action model extracts latent actions by identifying physics and motion deltas between frames. Training - A massive 44k hours of video data are used for pre-training. - Post-training on small-scale robot datasets maps human physics to specific robot embodiments. - An additional distillation stage converts the model into an autoregressive, few-step diffusion model, enabling real-time, action-controllable simulation. Primary Use Cases - Live Teleoperation: Controlling a robot inside a world simulation in real-time. - Model-based Planning: Previewing and curating the best actions for improved success. - Policy Evaluation: Testing robot policies in realistic, out-of-distribution scenarios. Everything that's open-sourced: weights, code, post-training dataset, eval set, and details to reproduce.

An interactive world model developed by NVIDIA in collaboration with academic partners. - DreamDojo turns egocentric human video data into physical intelligence. - Human data is more scalable than robotics data but lacks action labels. - To solve this, a dedicated action model extracts latent actions by identifying physics and motion deltas between frames. Training - A massive 44k hours of video data are used for pre-training. - Post-training on small-scale robot datasets maps human physics to specific robot embodiments. - An additional distillation stage converts the model into an autoregressive, few-step diffusion model, enabling real-time, action-controllable simulation. Primary Use Cases - Live Teleoperation: Controlling a robot inside a world simulation in real-time. - Model-based Planning: Previewing and curating the best actions for improved success. - Policy Evaluation: Testing robot policies in realistic, out-of-distribution scenarios. Everything that's open-sourced: weights, code, post-training dataset, eval set, and details to reproduce.

The Humanoid Hub

11,575 Aufrufe • vor 7 Monaten

A humanoid robot policy trained solely on synthetic data generated by a world model. Research Scientist Joel Jang presents NVIDIA's DreamGen pipeline: ⦿ Post-train the world model Cosmos-Predict2 with a small set of real teleoperation demos. ⦿ Prompt the world model to generate synthetic video data with verbs and scenarios not used in the world model’s post-training. ⦿ Auto-label synthetic video data with action sequences. ⦿ Train robot policies using only synthetic data. That's it. Deploy zero-shot to a real humanoid robot.

A humanoid robot policy trained solely on synthetic data generated by a world model. Research Scientist Joel Jang presents NVIDIA's DreamGen pipeline: ⦿ Post-train the world model Cosmos-Predict2 with a small set of real teleoperation demos. ⦿ Prompt the world model to generate synthetic video data with verbs and scenarios not used in the world model’s post-training. ⦿ Auto-label synthetic video data with action sequences. ⦿ Train robot policies using only synthetic data. That's it. Deploy zero-shot to a real humanoid robot.

The Humanoid Hub

20,968 Aufrufe • vor 1 Jahr

The World Model as NEO's Cognitive Core 1X has revealed a major AI development where the NEO humanoid can translate any natural language prompt into robotic action. It demonstrates this capability even for novel tasks, objects, and environments not found in its robot dataset. - the 1X World Model is trained on internet-scale human interaction videos and fine-tuned with robot data to ground its understanding in physics and in NEO's embodiment - from a simple voice or text prompt, the world model generates a visualization of future actions - a built-in inverse dynamics model then translates these into precise motor movements for NEO

The World Model as NEO's Cognitive Core 1X has revealed a major AI development where the NEO humanoid can translate any natural language prompt into robotic action. It demonstrates this capability even for novel tasks, objects, and environments not found in its robot dataset. - the 1X World Model is trained on internet-scale human interaction videos and fine-tuned with robot data to ground its understanding in physics and in NEO's embodiment - from a simple voice or text prompt, the world model generates a visualization of future actions - a built-in inverse dynamics model then translates these into precise motor movements for NEO

The Humanoid Hub

68,453 Aufrufe • vor 8 Monaten

Tired of teleoperation? One human video → 1,000s of robot demos. (📍GitHub ) Scaling Robot Data Without Dynamics Simulation or Robot Hardware Real2Render2Real (R2R2R) is a new way to scale robot data without physics simulation or hardware. You take a phone scan + a single monocular human demo. It tracks the motion, renders photorealistic scenes, and generates diverse, robot-agnostic trajectories ready for training. > No teleop, no sim, no robot, just a phone and a video > Train VLA models and diffusion policies directly on the output > Supports multiple robot embodiments with kinematic consistency > 1000s of demos in 1/27 the time of real-world collection Thank you, Max Fu, for sharing!! Project: Paper: Code coming soon: It shows that with the right pipeline, you can scale robot learning data without touching a robot. One of the most interesting directions in scalable robotics today. —— Weekly robotics and AI insights. Subscribe free:

Tired of teleoperation? One human video → 1,000s of robot demos. (📍GitHub ) Scaling Robot Data Without Dynamics Simulation or Robot Hardware Real2Render2Real (R2R2R) is a new way to scale robot data without physics simulation or hardware. You take a phone scan + a single monocular human demo. It tracks the motion, renders photorealistic scenes, and generates diverse, robot-agnostic trajectories ready for training. > No teleop, no sim, no robot, just a phone and a video > Train VLA models and diffusion policies directly on the output > Supports multiple robot embodiments with kinematic consistency > 1000s of demos in 1/27 the time of real-world collection Thank you, Max Fu, for sharing!! Project: Paper: Code coming soon: It shows that with the right pipeline, you can scale robot learning data without touching a robot. One of the most interesting directions in scalable robotics today. —— Weekly robotics and AI insights. Subscribe free:

Ilir Aliu

42,864 Aufrufe • vor 7 Monaten

How well do egocentric human videos scale? Apparently, really well. Dyna Robotics has introduced Dyna-2, a world-action model (WAM) pre-trained on more than 1 million hours of real egocentric human video. It learns by jointly predicting the next video frames and the actions, and that video prediction (world modeling) is what carries learning from humans to robots. • More human video predictably improves action prediction. • Task accuracy improves across embodiments (bimanual arms and a humanoid) with zero robot data in pre-training. • Post-trained on just a few hours of robot data, mean task performance rose 20% to 53% as human pre-training scaled 1k to 1M hours. WAM vs VLA, matched apple-to-apple: the WAM hit 1.55x the VLA's success rate and won 65% of head-to-heads. Deployed zero-shot at unseen customer sites, it passed 87% vs the VLA's 46%.

How well do egocentric human videos scale? Apparently, really well. Dyna Robotics has introduced Dyna-2, a world-action model (WAM) pre-trained on more than 1 million hours of real egocentric human video. It learns by jointly predicting the next video frames and the actions, and that video prediction (world modeling) is what carries learning from humans to robots. • More human video predictably improves action prediction. • Task accuracy improves across embodiments (bimanual arms and a humanoid) with zero robot data in pre-training. • Post-trained on just a few hours of robot data, mean task performance rose 20% to 53% as human pre-training scaled 1k to 1M hours. WAM vs VLA, matched apple-to-apple: the WAM hit 1.55x the VLA's success rate and won 65% of head-to-heads. Deployed zero-shot at unseen customer sites, it passed 87% vs the VLA's 46%.

The Humanoid Hub

15,837 Aufrufe • vor 1 Monat

Excited to announce GR00T N1, the world’s first open foundation model for humanoid robots! We are on a mission to democratize Physical AI. The power of general robot brain, in the palm of your hand - with only 2B parameters, N1 learns from the most diverse physical action dataset ever compiled and punches above its weight: - Real humanoid teleoperation data. - Large-scale simulation data: we are open-sourcing 300K+ trajectories! - Neural trajectories: we apply SOTA video generation models to “hallucinate” new synthetic data that features accurate physics in pixels. Using Jensen’s words, “systematically infinite data”! - Latent actions: we develop novel algorithms to extract action tokens from in-the-wild human videos and neural generated videos. GR00T N1 is a single end-to-end neural net, from photons to actions: - Vision-Language Model (System 2) that interprets the physical world through vision and language instructions, enabling robots to reason about their environment and instructions, and plan the right actions. - Diffusion Transformer (System 1) that “renders” smooth and precise motor actions at 120 Hz, executing the latent plan made by System 2. We deploy N1 on GR1 robot, 1X Neo robot, and a large collection of simulation benchmarks. N1 achieves up to +30% boost in diverse manipulation tasks for household and industrial settings. While humanoid robots are the main focus of N1, our model also supports cross-embodiment. We finetune it to work on the $110 HuggingFace LeRobot SO100 robot arm! Open robot brain runs on open hardware. Sounds just right. Let’s solve robotics, together, one token at a time. Links to our Whitepaper, Github repo, HuggingFace model, and open dataset page in the thread: 🧵

Excited to announce GR00T N1, the world’s first open foundation model for humanoid robots! We are on a mission to democratize Physical AI. The power of general robot brain, in the palm of your hand - with only 2B parameters, N1 learns from the most diverse physical action dataset ever compiled and punches above its weight: - Real humanoid teleoperation data. - Large-scale simulation data: we are open-sourcing 300K+ trajectories! - Neural trajectories: we apply SOTA video generation models to “hallucinate” new synthetic data that features accurate physics in pixels. Using Jensen’s words, “systematically infinite data”! - Latent actions: we develop novel algorithms to extract action tokens from in-the-wild human videos and neural generated videos. GR00T N1 is a single end-to-end neural net, from photons to actions: - Vision-Language Model (System 2) that interprets the physical world through vision and language instructions, enabling robots to reason about their environment and instructions, and plan the right actions. - Diffusion Transformer (System 1) that “renders” smooth and precise motor actions at 120 Hz, executing the latent plan made by System 2. We deploy N1 on GR1 robot, 1X Neo robot, and a large collection of simulation benchmarks. N1 achieves up to +30% boost in diverse manipulation tasks for household and industrial settings. While humanoid robots are the main focus of N1, our model also supports cross-embodiment. We finetune it to work on the $110 HuggingFace LeRobot SO100 robot arm! Open robot brain runs on open hardware. Sounds just right. Let’s solve robotics, together, one token at a time. Links to our Whitepaper, Github repo, HuggingFace model, and open dataset page in the thread: 🧵

Jim Fan

467,237 Aufrufe • vor 1 Jahr

Meet VLA-JEPA, the first world model policy in LeRobot 🌍 Most VLAs just learn observation → action. VLA-JEPA also learns action-relevant dynamics: during training, a JEPA world model has to predict upcoming frames in latent space from the model's own actions. Then, at inference, the world model disappears entirely, leaving a standard, fast VLA: Qwen backbone, action head, nothing extra. In our demo it was fine-tuned on just 13 examples and ran in real time on an NVIDIA DGX Spark. On a consumer RTX 3080, it hits 10Hz using under 6GB of VRAM. World-model supervision during training, zero extra cost at inference. That's the whole trick.

Meet VLA-JEPA, the first world model policy in LeRobot 🌍 Most VLAs just learn observation → action. VLA-JEPA also learns action-relevant dynamics: during training, a JEPA world model has to predict upcoming frames in latent space from the model's own actions. Then, at inference, the world model disappears entirely, leaving a standard, fast VLA: Qwen backbone, action head, nothing extra. In our demo it was fine-tuned on just 13 examples and ran in real time on an NVIDIA DGX Spark. On a consumer RTX 3080, it hits 10Hz using under 6GB of VRAM. World-model supervision during training, zero extra cost at inference. That's the whole trick.

LeRobot

42,389 Aufrufe • vor 2 Monaten

Jason Ma reveals the first true scaling law in robotics: robots trained on 1 million hours of first-person human video predictably get better without ever seeing robot data "Yesterday we announced our new flagship robot foundation model called Dyna-2. It's very significant for the entire field for many different reasons. First of all, it's the first robot foundation model trained on at least one million hours of data. That in itself is a very challenging infrastructure challenge." "What we demonstrated is that even if the one million hours of data is entirely first-person video of humans doing manipulation tasks, we actually see scaling law for transferring to robot embodiments that the model has never seen before." "By training on more human data, we actually see predictable performance improvement on robots. That's a big deal because the biggest challenge facing robot foundation models is that we don't have enough data." "Unlike language models, there are just not enough robotics data out there, and it's very difficult to collect these robot data. Only by passively observing humans doing things can we scale this kind of data, and this is the first time we showed there is a lot of improvement that can transfer." Jason Ma Dyna Robotics

Jason Ma reveals the first true scaling law in robotics: robots trained on 1 million hours of first-person human video predictably get better without ever seeing robot data "Yesterday we announced our new flagship robot foundation model called Dyna-2. It's very significant for the entire field for many different reasons. First of all, it's the first robot foundation model trained on at least one million hours of data. That in itself is a very challenging infrastructure challenge." "What we demonstrated is that even if the one million hours of data is entirely first-person video of humans doing manipulation tasks, we actually see scaling law for transferring to robot embodiments that the model has never seen before." "By training on more human data, we actually see predictable performance improvement on robots. That's a big deal because the biggest challenge facing robot foundation models is that we don't have enough data." "Unlike language models, there are just not enough robotics data out there, and it's very difficult to collect these robot data. Only by passively observing humans doing things can we scale this kind of data, and this is the first time we showed there is a lot of improvement that can transfer." Jason Ma Dyna Robotics

MTS

17,740 Aufrufe • vor 1 Monat

Over the last few months, we’ve been thinking about how to learn from “off-domain” data - data from non-robot sources like video or simulation. These data sources are not quite good enough to learn policies (even monolithic VLA models) directly, but they still contain lots of information that can be useful for generalizable robot control. How can we develop robot learning models that are able to make use of this type of data for generalizable control? In new work, that we call HAMSTER, we show that VLMs can be useful for enabling robotic learning from off-domain data, but specifically when used through hierarchical VLA architectures. We show that this class of models can learn generalizable robot policies for the real world from large-scale, off-domain data. A 🧵 (1/10)

Over the last few months, we’ve been thinking about how to learn from “off-domain” data - data from non-robot sources like video or simulation. These data sources are not quite good enough to learn policies (even monolithic VLA models) directly, but they still contain lots of information that can be useful for generalizable robot control. How can we develop robot learning models that are able to make use of this type of data for generalizable control? In new work, that we call HAMSTER, we show that VLMs can be useful for enabling robotic learning from off-domain data, but specifically when used through hierarchical VLA architectures. We show that this class of models can learn generalizable robot policies for the real world from large-scale, off-domain data. A 🧵 (1/10)

Abhishek Gupta

11,994 Aufrufe • vor 1 Jahr

VLA-JEPA just dropped in LeRobot 🤖 What makes this model special is that it does not just learn what action to take from a given observation, it also leverages a JEPA world model to learn action-relevant dynamics. During training, the VLA leverages V-JEPA2 by conditioning its predictor. This clever trick adds a world modeling objective to the training, which also allows pretraining on human videos. At inference, the world model is dropped entirely, keeping only a standard VLA architecture: Qwen backbone and action head. The demo here was only fine-tuned on 13 examples, showing great pretraining capability and running in real time on NVIDIA Robotics DGX Spark! VLA-JEPA is the first world model to be ported to LeRobot, and I feel like it won't be the last 🚀 Thomas Wolf clem 🤗

VLA-JEPA just dropped in LeRobot 🤖 What makes this model special is that it does not just learn what action to take from a given observation, it also leverages a JEPA world model to learn action-relevant dynamics. During training, the VLA leverages V-JEPA2 by conditioning its predictor. This clever trick adds a world modeling objective to the training, which also allows pretraining on human videos. At inference, the world model is dropped entirely, keeping only a standard VLA architecture: Qwen backbone and action head. The demo here was only fine-tuned on 13 examples, showing great pretraining capability and running in real time on NVIDIA Robotics DGX Spark! VLA-JEPA is the first world model to be ported to LeRobot, and I feel like it won't be the last 🚀 Thomas Wolf clem 🤗

LeRobot

320,916 Aufrufe • vor 3 Monaten

Scaling vision-language-action (VLA) models to high-DoF dexterous hands has long been a "holy grail" challenge due to the high-dimensional action space and data scarcity. As a wrap up of the year 2025, we are releasing GR-Dexter, a holistic hardware-model-data framework for generalist manipulation on a bimanual dexterous-hand robot. This is the first VLA system to achieve: ✅ High-DoF Control: Managing a 56-DoF bimanual system (21-DoF per hand). ✅ Long-Horizon Tasks with tool use: Vacuuming, bread serving with tongs, and table decluttering. ✅ Open-World Generalization: Robust performance with unseen objects and abstract instructions. Project page: ArXiv:

Scaling vision-language-action (VLA) models to high-DoF dexterous hands has long been a "holy grail" challenge due to the high-dimensional action space and data scarcity. As a wrap up of the year 2025, we are releasing GR-Dexter, a holistic hardware-model-data framework for generalist manipulation on a bimanual dexterous-hand robot. This is the first VLA system to achieve: ✅ High-DoF Control: Managing a 56-DoF bimanual system (21-DoF per hand). ✅ Long-Horizon Tasks with tool use: Vacuuming, bread serving with tongs, and table decluttering. ✅ Open-World Generalization: Robust performance with unseen objects and abstract instructions. Project page: ArXiv:

Xiao Ma

94,111 Aufrufe • vor 8 Monaten

The next evolution: VLA+ models Just yesterday Microsoft Research released Rho-alpha (ρα) – their first robotics model, built on the Phi family. While most Vision-Language-Action (VLA) models stop at vision and language, Rho-alpha adds: ▪️ Tactile sensing to feel objects during manipulation ▪️ Online learning that lets it improve from human corrections (via teleoperation, 3D mouse or other tools) in real-time even after deployment. Both these sides make adaptability central rather than incidental. Microsoft calls it a VLA+ model, positioning it as an extension beyond what current VLA systems support. ➡️ Today Rho-alpha can control dual-arm robot setups to perform tasks such as: • Manipulating the BusyBox following natural-language instructions • Plug insertion • Toolbox packing and object arrangement with bimanual coordination But to understand why this "plus" matters, we need to understand what came before. Here, we'll take you through the entire landscape of VLA models – Gemini Robotics, π0, SmolVLA, Helix, ACoT-VLA and others:

The next evolution: VLA+ models Just yesterday Microsoft Research released Rho-alpha (ρα) – their first robotics model, built on the Phi family. While most Vision-Language-Action (VLA) models stop at vision and language, Rho-alpha adds: ▪️ Tactile sensing to feel objects during manipulation ▪️ Online learning that lets it improve from human corrections (via teleoperation, 3D mouse or other tools) in real-time even after deployment. Both these sides make adaptability central rather than incidental. Microsoft calls it a VLA+ model, positioning it as an extension beyond what current VLA systems support. ➡️ Today Rho-alpha can control dual-arm robot setups to perform tasks such as: • Manipulating the BusyBox following natural-language instructions • Plug insertion • Toolbox packing and object arrangement with bimanual coordination But to understand why this "plus" matters, we need to understand what came before. Here, we'll take you through the entire landscape of VLA models – Gemini Robotics, π0, SmolVLA, Helix, ACoT-VLA and others:

Turing Post

62,494 Aufrufe • vor 8 Monaten

New model: your robot can now pack your suitcase 🧳 Xiaomi has released a new robot foundation model. Called Xiaomi-Robotics-1, it is designed to have a robot pick things up and move them around. But first, DEFINITIONS: - Mixture-of-Transformers (MoT): An architecture where separate transformer "experts" (e.g., one for vision-language, one for actions) share a single attention stream, so each modality gets specialized parameters without losing joint reasoning. - Vision-language model (VLM): A model that jointly understands images and text. - Diffusion transformer: A transformer trained to turn noise into structured outputs by iterative denoising, here generating robot actions rather than images. - Action chunks: Short sequences of future actions (e.g., the next ~50 motor commands) predicted in one shot instead of one step at a time. - Flow matching: A faster version of diffusion. The model learns a straight-line velocity field from noise to the target action, so it needs only a few integration steps instead of many denoising ones. Its peculiarity comes from its two stage training: 1. 100,000 hours of video shot through a UMI rig: a handheld 3D-printed gripper with a camera, worn by humans doing ordinary tasks in homes, shops, factories and offices. 2. Adapt to actual robot bodies with ~10,000 hours of real-robot data. It replaces the standard approach of teleoperating a real robot for every hour of training data. Its architecture is a Mixture-of-Transformers pairing a pre-trained Qwen3-VL vision-language model with a diffusion transformer that emits action chunks via flow matching, released in 2.6B, 5.1B and 10.5B parameter variants. However, if you read the entire paper ("Scaling VLA Models with over 100K Hours"), you realize that all of the scaling experiments on 20k hours. Therefore the headline "out-of-the-box success climbing 26% → 75% as pre-training data grows" tops out at 100% of 20k hours! What the full corpus does to that curve is never shown -> and this where things would become interesting! Xiaomi's own conclusion is that model size has stopped mattering and data is the binding constraint. The performance gap among different model sizes are less pronounced than those observed across different data scales. This result suggests that model capacity at the billions-parameter scale may already be sufficient to capture the current dataset's distribution. Which further asks the same question: why not use the 100k video hours? Anyway, I would definitely love to have a couple robots at home that can cooperate to pack my suitcase with items relevant to my next destination:

New model: your robot can now pack your suitcase 🧳 Xiaomi has released a new robot foundation model. Called Xiaomi-Robotics-1, it is designed to have a robot pick things up and move them around. But first, DEFINITIONS: - Mixture-of-Transformers (MoT): An architecture where separate transformer "experts" (e.g., one for vision-language, one for actions) share a single attention stream, so each modality gets specialized parameters without losing joint reasoning. - Vision-language model (VLM): A model that jointly understands images and text. - Diffusion transformer: A transformer trained to turn noise into structured outputs by iterative denoising, here generating robot actions rather than images. - Action chunks: Short sequences of future actions (e.g., the next ~50 motor commands) predicted in one shot instead of one step at a time. - Flow matching: A faster version of diffusion. The model learns a straight-line velocity field from noise to the target action, so it needs only a few integration steps instead of many denoising ones. Its peculiarity comes from its two stage training: 1. 100,000 hours of video shot through a UMI rig: a handheld 3D-printed gripper with a camera, worn by humans doing ordinary tasks in homes, shops, factories and offices. 2. Adapt to actual robot bodies with ~10,000 hours of real-robot data. It replaces the standard approach of teleoperating a real robot for every hour of training data. Its architecture is a Mixture-of-Transformers pairing a pre-trained Qwen3-VL vision-language model with a diffusion transformer that emits action chunks via flow matching, released in 2.6B, 5.1B and 10.5B parameter variants. However, if you read the entire paper ("Scaling VLA Models with over 100K Hours"), you realize that all of the scaling experiments on 20k hours. Therefore the headline "out-of-the-box success climbing 26% → 75% as pre-training data grows" tops out at 100% of 20k hours! What the full corpus does to that curve is never shown -> and this where things would become interesting! Xiaomi's own conclusion is that model size has stopped mattering and data is the binding constraint. The performance gap among different model sizes are less pronounced than those observed across different data scales. This result suggests that model capacity at the billions-parameter scale may already be sufficient to capture the current dataset's distribution. Which further asks the same question: why not use the 100k video hours? Anyway, I would definitely love to have a couple robots at home that can cooperate to pack my suitcase with items relevant to my next destination:

Léo

15,777 Aufrufe • vor 1 Monat

X Square Robot Unveils New Embodied AI Model, Says Robots Will Arrive in Homes in 35 Days Backed by Alibaba, ByteDance, Xiaomi and Meituan, X Square Robot unveiled a next-generation embodied AI foundation model for home robots and said its first deployments in everyday households will begin within 35 days. X Square Robot on Tuesday unveiled WALL-B, a new embodied AI foundation model designed for deployment in real-world homes, marking what the company described as a major step toward bringing general-purpose robots into daily family life. At a launch event themed "Born to Bot, Bot to Family," the company also introduced its World Unified Model (WUM) architecture, a training framework that combines vision, language, action and physical prediction within a single system from the outset. X Square said the model is intended to help robots operate in the far more unpredictable setting of a home, where tasks, layouts and interactions vary from moment to moment. "Robots in factories and in homes are completely different. In factories, they repeat the same action 10,000 times without variation. In a home, however, they need to perform 10,000 different actions, each unique and non-repetitive. Therefore, the challenge of a truly intelligent robot lies not in repeating a single action, but in the ability to execute new, untrained movements within unstructured environments. Deploying robots in the home is one of the most significant technical hurdles of our time," said Qian Wang, founder and CEO of X Square Robot. WALL-B is the first real-world implementation of the World Unified Model architecture. Unlike modular systems that train perception, language and control separately, X Square Robot said World Unified Model optimizes those capabilities jointly from the very beginning. The company said that allows physical prediction — including force, friction and collision dynamics — to emerge as part of the model itself, rather than being layered on afterward. "We train all capabilities—vision, language, action, and prediction—within the same network from day one. Much like infants, who do not learn to see, move and speak in isolated, sequential stages, but instead see, move listen and act simultaneously while receiving feedback, we have integrated all these capabilities into a unified whole," said Wang Hao, CTO of X Square. X Square Robot said the development of WALL-B rests on two pillars. The first is a data strategy that prioritizes training on authentic, non-staged home environments to cover the “long-tail” distribution of real-world scenarios, such as misplaced objects and temporary occlusions. Unlike models primarily trained on synthetic data or laboratory datasets, this strategy exposes WALL-B to the natural clutter of lived-in spaces—misplaced items, unexpected obstacles, and spontaneous human activity—ensuring that the training data reflects real-world conditions rather than a simplified version. The second is a physics-aware predictive mechanism that anticipates physical outcomes before an action is taken, enabling the model to respond to contact dynamics instead of just reacting. The development of the self-developed WUM architecture on physical robotic platforms highlights the company’s accumlated experience in bridging sim-to-real gaps across varied operational contexts. Wang commented that the current AI model is still in an "intern" stage, subject to errors requiring remote assistance. For instance, it may mistakenly place slippers in the kitchen or pause while wiping a table to "think". However, the model operates nonstop 24 hours a day, becoming increasingly "intelligent" as each day of operation generates new data. In 35 days, on May 25, X Square Robot will officially bring its robots into everyday homes, underscoring the company’s long-term commitment to the home robotics sector.

X Square Robot Unveils New Embodied AI Model, Says Robots Will Arrive in Homes in 35 Days Backed by Alibaba, ByteDance, Xiaomi and Meituan, X Square Robot unveiled a next-generation embodied AI foundation model for home robots and said its first deployments in everyday households will begin within 35 days. X Square Robot on Tuesday unveiled WALL-B, a new embodied AI foundation model designed for deployment in real-world homes, marking what the company described as a major step toward bringing general-purpose robots into daily family life. At a launch event themed "Born to Bot, Bot to Family," the company also introduced its World Unified Model (WUM) architecture, a training framework that combines vision, language, action and physical prediction within a single system from the outset. X Square said the model is intended to help robots operate in the far more unpredictable setting of a home, where tasks, layouts and interactions vary from moment to moment. "Robots in factories and in homes are completely different. In factories, they repeat the same action 10,000 times without variation. In a home, however, they need to perform 10,000 different actions, each unique and non-repetitive. Therefore, the challenge of a truly intelligent robot lies not in repeating a single action, but in the ability to execute new, untrained movements within unstructured environments. Deploying robots in the home is one of the most significant technical hurdles of our time," said Qian Wang, founder and CEO of X Square Robot. WALL-B is the first real-world implementation of the World Unified Model architecture. Unlike modular systems that train perception, language and control separately, X Square Robot said World Unified Model optimizes those capabilities jointly from the very beginning. The company said that allows physical prediction — including force, friction and collision dynamics — to emerge as part of the model itself, rather than being layered on afterward. "We train all capabilities—vision, language, action, and prediction—within the same network from day one. Much like infants, who do not learn to see, move and speak in isolated, sequential stages, but instead see, move listen and act simultaneously while receiving feedback, we have integrated all these capabilities into a unified whole," said Wang Hao, CTO of X Square. X Square Robot said the development of WALL-B rests on two pillars. The first is a data strategy that prioritizes training on authentic, non-staged home environments to cover the “long-tail” distribution of real-world scenarios, such as misplaced objects and temporary occlusions. Unlike models primarily trained on synthetic data or laboratory datasets, this strategy exposes WALL-B to the natural clutter of lived-in spaces—misplaced items, unexpected obstacles, and spontaneous human activity—ensuring that the training data reflects real-world conditions rather than a simplified version. The second is a physics-aware predictive mechanism that anticipates physical outcomes before an action is taken, enabling the model to respond to contact dynamics instead of just reacting. The development of the self-developed WUM architecture on physical robotic platforms highlights the company’s accumlated experience in bridging sim-to-real gaps across varied operational contexts. Wang commented that the current AI model is still in an "intern" stage, subject to errors requiring remote assistance. For instance, it may mistakenly place slippers in the kitchen or pause while wiping a table to "think". However, the model operates nonstop 24 hours a day, becoming increasingly "intelligent" as each day of operation generates new data. In 35 days, on May 25, X Square Robot will officially bring its robots into everyday homes, underscoring the company’s long-term commitment to the home robotics sector.

X Square Robot

52,968 Aufrufe • vor 5 Monaten

Reward AI just came out of stealth and unveiled OM-1, a robot policy learned entirely from humans. No teleoperation, no on-robot data. One Model, Any Body. 1. Capture. A wearable 7-DoF hand (Omnibody Hand) lets people demonstrate manipulation naturally while working, cooking, or sorting. Sensors record tactile, proximity, force and hand pose at full human speed. 2. Learn. OM-1 trains on this human data alone. No teleop, no on-robot experience. Then it runs across bodies from industrial arms to humanoids, and scaling is just adding more human data. 3. Control. A high-frequency control layer, trained with RL in sim, executes those actions on any robot, holding steady through unknown loads like a stuck fridge door. The result: OM-1 picks up a brand-new task from under 30 minutes of data.

The Humanoid Hub

35,064 Aufrufe • vor 9 Tagen