Загрузка видео...

Не удалось загрузить видео

На главную

HumanX: a scalable framework that converts single monocular human videos into agile, generalizable interaction skills for humanoid robots without task-specific rewards. Core parts: - XGen: retargets human motion + synthesizes diverse physically plausible training data (via physics-driven object trajectories and augmentation). - XMimic: unified imitation pipeline for robust generalization....

TheHumanoidHub's profile picture

The Humanoid Hub

93,606 subscribers

20,358 просмотров • 6 месяцев назад •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

HDMI (HumanoiD iMitation for Interaction) is a framework enabling humanoid robots to learn whole-body object interaction skills from monocular RGB human videos. It extracts and retargets human poses and object trajectories using GVHMR and LocoMujoco, building reference datasets with contact annotations. The data is used to train an RL policy via robot-object co-tracking. HDMI achieved 67 consecutive door traversals.
0:45

HDMI (HumanoiD iMitation for Interaction) is a framework enabling humanoid robots to learn whole-body object interaction skills from monocular RGB human videos. It extracts and retargets human poses and object trajectories using GVHMR and LocoMujoco, building reference datasets with contact annotations. The data is used to train an RL policy via robot-object co-tracking. HDMI achieved 67 consecutive door traversals.

The Humanoid Hub

17,395 просмотров • 10 месяцев назад

Remarkably lifelike motion and fluidity. BeyondMimic is a framework for training humanoid whole-body control from large mocap datasets. First, an open-source motion-tracking pipeline to reproduce diverse, highly dynamic human skills on real hardware, then distilling them into a guided state-action diffusion model for zero-shot, task-specific control. Project page:
2:42

Remarkably lifelike motion and fluidity. BeyondMimic is a framework for training humanoid whole-body control from large mocap datasets. First, an open-source motion-tracking pipeline to reproduce diverse, highly dynamic human skills on real hardware, then distilling them into a guided state-action diffusion model for zero-shot, task-specific control. Project page:

The Humanoid Hub

58,081 просмотров • 1 год назад

Tired of teleoperation? One human video → 1,000s of robot demos. (📍GitHub ) Scaling Robot Data Without Dynamics Simulation or Robot Hardware Real2Render2Real (R2R2R) is a new way to scale robot data without physics simulation or hardware. You take a phone scan + a single monocular human demo. It tracks the motion, renders photorealistic scenes, and generates diverse, robot-agnostic trajectories ready for training. > No teleop, no sim, no robot, just a phone and a video > Train VLA models and diffusion policies directly on the output > Supports multiple robot embodiments with kinematic consistency > 1000s of demos in 1/27 the time of real-world collection Thank you, Max Fu, for sharing!! Project: Paper: Code coming soon: It shows that with the right pipeline, you can scale robot learning data without touching a robot. One of the most interesting directions in scalable robotics today. —— Weekly robotics and AI insights. Subscribe free:
0:29

Tired of teleoperation? One human video → 1,000s of robot demos. (📍GitHub ) Scaling Robot Data Without Dynamics Simulation or Robot Hardware Real2Render2Real (R2R2R) is a new way to scale robot data without physics simulation or hardware. You take a phone scan + a single monocular human demo. It tracks the motion, renders photorealistic scenes, and generates diverse, robot-agnostic trajectories ready for training. > No teleop, no sim, no robot, just a phone and a video > Train VLA models and diffusion policies directly on the output > Supports multiple robot embodiments with kinematic consistency > 1000s of demos in 1/27 the time of real-world collection Thank you, Max Fu, for sharing!! Project: Paper: Code coming soon: It shows that with the right pipeline, you can scale robot learning data without touching a robot. One of the most interesting directions in scalable robotics today. —— Weekly robotics and AI insights. Subscribe free:

Ilir Aliu

42,864 просмотров • 6 месяцев назад

Humanoids finally move like humans… and can do more than copy. [Details + demos in thread 👇] A new framework, BeyondMimic, shows how to learn naturalistic whole-body control from human motion. But then goes further by composing those skills into versatile, zero-shot policies. Two key pieces make it work: •Motion tracking pipeline: robustly turns human mocap into dynamic robot motions (jump spins, sprinting, cartwheels) on real hardware •Guided diffusion policy: distills motion primitives and enables zero-shot downstream tasks with simple cost functions The result: ✅ State-of-the-art motion quality from mocap tracking ✅ Versatile task execution; waypoint navigation, joystick teleop, obstacle avoidance ✅ Stable sim-to-real transfer for diverse humanoid skills Thank you for sharing, Qiayuan Liao! 📍 Paper: Video:
0:42

Humanoids finally move like humans… and can do more than copy. [Details + demos in thread 👇] A new framework, BeyondMimic, shows how to learn naturalistic whole-body control from human motion. But then goes further by composing those skills into versatile, zero-shot policies. Two key pieces make it work: •Motion tracking pipeline: robustly turns human mocap into dynamic robot motions (jump spins, sprinting, cartwheels) on real hardware •Guided diffusion policy: distills motion primitives and enables zero-shot downstream tasks with simple cost functions The result: ✅ State-of-the-art motion quality from mocap tracking ✅ Versatile task execution; waypoint navigation, joystick teleop, obstacle avoidance ✅ Stable sim-to-real transfer for diverse humanoid skills Thank you for sharing, Qiayuan Liao! 📍 Paper: Video:

Ilir Aliu - eu/acc

72,522 просмотров • 1 год назад

NVIDIA has published a paper on DREAMGEN – a powerful 4-step pipeline for generating synthetic data for humanoids that enables task and environment generalization. - Step 1: Fine-tune a video generation model using a small number of human teleoperation videos - Step 2: Prompt the fine-tuned model to turn a single real image into new AI-imagined videos - Step 3: Automatically label actions in the generated videos - Step 4: Train a robot AI model with the labeled synthetic dataset This enabled humanoid robots to perform 22 novel behaviors – such as pouring, opening/closing articulated objects, and manipulating a variety of tools. The original teleoperation dataset only included pick-and-place tasks. This takes task extensibility to another level without requiring human teleoperation for every single task. The pipeline will be made open-source soon. Project page:
0:37

NVIDIA has published a paper on DREAMGEN – a powerful 4-step pipeline for generating synthetic data for humanoids that enables task and environment generalization. - Step 1: Fine-tune a video generation model using a small number of human teleoperation videos - Step 2: Prompt the fine-tuned model to turn a single real image into new AI-imagined videos - Step 3: Automatically label actions in the generated videos - Step 4: Train a robot AI model with the labeled synthetic dataset This enabled humanoid robots to perform 22 novel behaviors – such as pouring, opening/closing articulated objects, and manipulating a variety of tools. The original teleoperation dataset only included pick-and-place tasks. This takes task extensibility to another level without requiring human teleoperation for every single task. The pipeline will be made open-source soon. Project page:

The Humanoid Hub

12,074 просмотров • 1 год назад

NEWS: Humanoid robotics company Figure has released Helix 02, what they claim in their most capable humanoid model yet. "A single neural system that controls the full body directly from pixels, enabling dexterous, long horizon autonomy across an entire room: • Autonomous, long‑horizon loco-manipulation: Helix 02 unloads and reloads a dishwasher across a full-sized kitchen - a four-minute, end-to-end autonomous task that integrates walking, manipulation, and balance with no resets and no human intervention. We believe this is the longest horizon, most complex task completed autonomously by a humanoid robot to date. • All sensors in. All actuators out: Helix 02 connects every onboard sensor - vision, touch, and proprioception - directly to every actuator through a single unified visuomotor neural network. • Human-like whole body control from human data: All results are enabled by System 0, a learned whole‑body controller trained on over 1,000 hours of human motion data and sim‑to‑real reinforcement learning. System 0 replaces 109,504 lines of hand‑engineered C++ with a single neural prior for stable, natural motion. • New classes of dexterity: With Figure 03’s embedded tactile sensing and palm cameras, Helix 02 performs manipulation that was previously out of reach: extracting individual pills, dispensing precise syringe volumes, and singulating small, irregular objects from clutter despite self‑occlusion. Helix 02 is trained on over 1,000 hours of human motion data and integrates vision, touch, and proprioception."
3:25

NEWS: Humanoid robotics company Figure has released Helix 02, what they claim in their most capable humanoid model yet. "A single neural system that controls the full body directly from pixels, enabling dexterous, long horizon autonomy across an entire room: • Autonomous, long‑horizon loco-manipulation: Helix 02 unloads and reloads a dishwasher across a full-sized kitchen - a four-minute, end-to-end autonomous task that integrates walking, manipulation, and balance with no resets and no human intervention. We believe this is the longest horizon, most complex task completed autonomously by a humanoid robot to date. • All sensors in. All actuators out: Helix 02 connects every onboard sensor - vision, touch, and proprioception - directly to every actuator through a single unified visuomotor neural network. • Human-like whole body control from human data: All results are enabled by System 0, a learned whole‑body controller trained on over 1,000 hours of human motion data and sim‑to‑real reinforcement learning. System 0 replaces 109,504 lines of hand‑engineered C++ with a single neural prior for stable, natural motion. • New classes of dexterity: With Figure 03’s embedded tactile sensing and palm cameras, Helix 02 performs manipulation that was previously out of reach: extracting individual pills, dispensing precise syringe volumes, and singulating small, irregular objects from clutter despite self‑occlusion. Helix 02 is trained on over 1,000 hours of human motion data and integrates vision, touch, and proprioception."

Sawyer Merritt

624,770 просмотров • 6 месяцев назад

Physics-based Motion Retargeting from Sparse Inputs paper page: Avatars are important to create interactive and immersive experiences in virtual worlds. One challenge in animating these characters to mimic a user's motion is that commercial AR/VR products consist only of a headset and controllers, providing very limited sensor data of the user's pose. Another challenge is that an avatar might have a different skeleton structure than a human and the mapping between them is unclear. In this work we address both of these challenges. We introduce a method to retarget motions in real-time from sparse human sensor data to characters of various morphologies. Our method uses reinforcement learning to train a policy to control characters in a physics simulator. We only require human motion capture data for training, without relying on artist-generated animations for each avatar. This allows us to use large motion capture datasets to train general policies that can track unseen users from real and sparse data in real-time. We demonstrate the feasibility of our approach on three characters with different skeleton structure: a dinosaur, a mouse-like creature and a human. We show that the avatar poses often match the user surprisingly well, despite having no sensor information of the lower body available. We discuss and ablate the important components in our framework, specifically the kinematic retargeting step, the imitation, contact and action reward as well as our asymmetric actor-critic observations. We further explore the robustness of our method in a variety of settings including unbalancing, dancing and sports motions.
7:12

Physics-based Motion Retargeting from Sparse Inputs paper page: Avatars are important to create interactive and immersive experiences in virtual worlds. One challenge in animating these characters to mimic a user's motion is that commercial AR/VR products consist only of a headset and controllers, providing very limited sensor data of the user's pose. Another challenge is that an avatar might have a different skeleton structure than a human and the mapping between them is unclear. In this work we address both of these challenges. We introduce a method to retarget motions in real-time from sparse human sensor data to characters of various morphologies. Our method uses reinforcement learning to train a policy to control characters in a physics simulator. We only require human motion capture data for training, without relying on artist-generated animations for each avatar. This allows us to use large motion capture datasets to train general policies that can track unseen users from real and sparse data in real-time. We demonstrate the feasibility of our approach on three characters with different skeleton structure: a dinosaur, a mouse-like creature and a human. We show that the avatar poses often match the user surprisingly well, despite having no sensor information of the lower body available. We discuss and ablate the important components in our framework, specifically the kinematic retargeting step, the imitation, contact and action reward as well as our asymmetric actor-critic observations. We further explore the robustness of our method in a variety of settings including unbalancing, dancing and sports motions.

AK

106,527 просмотров • 3 лет назад

This is how ALOHA's "teleoperation" system works - a fancy word for "remote control". Training robots will be more and more like playing games in the physical world. A human operates a "joystick++" to perform tasks and collect data, or intervene if there's any safety concern. There's actually a learning curve to master the controller, much like practicing gaming skills. Teleoperation can be done in many different ways. ALOHA is an impressive custom-built system with very low cost. Here're a few alternatives: (1) Motion Capture (MoCap): apply the MoCap systems used for Hollywood movies to capture the fine-grained motions of hand joints. There would be no "embodiment gap" if the robot hand has 5 fingers. For instance, a demonstrator can wear a CyberGlove ( and manipulate the objects. CyberGlove will capture the motion signals & haptic feedback in real-time, which can be re-targeted onto the humanoid. (2) Wearing gloves & markers can be clumsy. An alternative way to do MoCap is through computer vision. DexPilot from NVIDIA enables marker-less and glove-free data collection. The human operator simply uses their bare hands to perform the tasks. 4 Intel RealSense depth cameras and 2 NVIDIA Titan XP GPUs (yeah, 2019 work) translate the pixels to precise motion signals for robot learning. (3) VR Headset: turn the training room into a VR game and "role play" the robot. This has the advantage of scalable remote data collection - annotators from around the world can contribute without coming onsite. VR demonstration technique appeared in research projects like the iGibson home robot simulator, an initiative that I participated in at Stanford: Behind-the-scene video by Litian Liang
0:52

This is how ALOHA's "teleoperation" system works - a fancy word for "remote control". Training robots will be more and more like playing games in the physical world. A human operates a "joystick++" to perform tasks and collect data, or intervene if there's any safety concern. There's actually a learning curve to master the controller, much like practicing gaming skills. Teleoperation can be done in many different ways. ALOHA is an impressive custom-built system with very low cost. Here're a few alternatives: (1) Motion Capture (MoCap): apply the MoCap systems used for Hollywood movies to capture the fine-grained motions of hand joints. There would be no "embodiment gap" if the robot hand has 5 fingers. For instance, a demonstrator can wear a CyberGlove ( and manipulate the objects. CyberGlove will capture the motion signals & haptic feedback in real-time, which can be re-targeted onto the humanoid. (2) Wearing gloves & markers can be clumsy. An alternative way to do MoCap is through computer vision. DexPilot from NVIDIA enables marker-less and glove-free data collection. The human operator simply uses their bare hands to perform the tasks. 4 Intel RealSense depth cameras and 2 NVIDIA Titan XP GPUs (yeah, 2019 work) translate the pixels to precise motion signals for robot learning. (3) VR Headset: turn the training room into a VR game and "role play" the robot. This has the advantage of scalable remote data collection - annotators from around the world can contribute without coming onsite. VR demonstration technique appeared in research projects like the iGibson home robot simulator, an initiative that I participated in at Stanford: Behind-the-scene video by Litian Liang

Jim Fan

124,588 просмотров • 2 лет назад

I don’t know if we live in a Matrix, but I know for sure that robots will spend most of their lives in simulation. Let machines train machines. I’m excited to introduce DexMimicGen, a massive-scale synthetic data generator that enables a humanoid robot to learn complex skills from only a handful of human demonstrations. Yes, as few as 5! DexMimicGen addresses the biggest pain point in robotics: where do we get data? Unlike with LLMs, where vast amounts of texts are readily available, you cannot simply download motor control signals from the internet. So researchers teleoperate the robots to collect motion data via XR headsets. They have to repeat the same skill over and over and over again, because neural nets are data hungry. This is a very slow and uncomfortable process. At NVIDIA, we believe the majority of high-quality tokens for robot foundation models will come from simulation. What DexMimicGen does is to trade GPU compute time for human time. It takes one motion trajectory from human, and multiplies into 1000s of new trajectories. A robot brain trained on this augmented dataset will generalize far better in the real world. Think of DexMimicGen as a learning signal amplifier. It maps a small dataset to a large (de facto infinite) dataset, using physics simulation in the loop. In this way, we free humans from babysitting the bots all day. The future of robot data is generative. The future of the entire robot learning pipeline will also be generative. 🧵
0:58

I don’t know if we live in a Matrix, but I know for sure that robots will spend most of their lives in simulation. Let machines train machines. I’m excited to introduce DexMimicGen, a massive-scale synthetic data generator that enables a humanoid robot to learn complex skills from only a handful of human demonstrations. Yes, as few as 5! DexMimicGen addresses the biggest pain point in robotics: where do we get data? Unlike with LLMs, where vast amounts of texts are readily available, you cannot simply download motor control signals from the internet. So researchers teleoperate the robots to collect motion data via XR headsets. They have to repeat the same skill over and over and over again, because neural nets are data hungry. This is a very slow and uncomfortable process. At NVIDIA, we believe the majority of high-quality tokens for robot foundation models will come from simulation. What DexMimicGen does is to trade GPU compute time for human time. It takes one motion trajectory from human, and multiplies into 1000s of new trajectories. A robot brain trained on this augmented dataset will generalize far better in the real world. Think of DexMimicGen as a learning signal amplifier. It maps a small dataset to a large (de facto infinite) dataset, using physics simulation in the loop. In this way, we free humans from babysitting the bots all day. The future of robot data is generative. The future of the entire robot learning pipeline will also be generative. 🧵

Jim Fan

165,246 просмотров • 1 год назад

New framework: Kick down your robot, it will get back up every time 🥋 Chinese startup RoboParty is a Beijing startup founded April 2025 by Huang Yi, originally shipping ROBOTO ORIGIN, the world's first full-stack open-source bipedal humanoid. They released UFO: Unsupervised Reinforcement Learning Framework for Humanoid Control. DEFINITIONS -> what differs is where the learning signal comes from: - SUPERVISED: humans supply the right answers (labels), the model imitates them. - UNSUPERVISED: no answer key, the model finds structure in raw data on its own. - REINFORCEMENT LEARNING: no answer key either, the model tries things and a reward scores each attempt. → UNSUPERVISED RL: trial and error where the agent invents its own rewards, instead of engineers hand-writing one per task. REPRESENTATION LEARNING: compress raw states into a useful internal map. TEMPORAL DISTANCE: distance on that map is "how many steps from A to B." CONTRASTIVE: trained by pulling together what's close in time, pushing apart what isn't. -> CONTRASTIVE TEMPORAL-DISTANCE REPRESENTATION LEARNING: the model builds an internal map of body states where distance means how many steps it takes to get from one to another. It is trained by contrast: states that occur close together in a movement get pulled together in the map, randomly paired states get pushed apart. UFO is an open-source training framework that teaches humanoid robots skills, like getting up, walking, goal-reaching, teleoperation, without reference motions -> no motion-capture or human-video demonstrations to imitate. Its core is TeCH, a contrastive temporal-distance representation-learning algorithm: the robot explores, builds pseudo-goals by temporal rolling, and learns goal-conditioned policies from a single unified progress reward. One framework trains five different robots (Unitree G1/H1, RoboParty RP0/RP1, AgiBot X2) with automatic config conversion in ~2–3 hours per robot! The real novelty here "no demonstrations at all". No data-collection arms race,the dominant humanoid-locomotion recipe is tracking: imitate mocap/retargeted-human reference trajectories. The robot self-generates goals from its own exploration and learns from a progress reward, needing zero reference motion data. Everybody else is fighting over data acquisition, while this team just teleports out of the race entirely (inb4 "competition is for losers 💀 ). This strategy reminds me of the DeepSeek playbook applied to robots: open-source the whole stack to become the global default and commoditize everyone else. RoboParty is giving away hardware and now control software (UFO) to be the Android of humanoids. Yet another reason for the US to ban Chinese open models perhaps 🥶 ? What I also really like about this approach is the cross-embodiment infrastructure, one framework trains Unitree G1/H1, RoboParty RP0/RP1, and AgiBot X2 with automatic configuration conversion. Just like Physical Intelligence, RoboParty seems to place itself as a neutral hardware agnostic middle man. Also woth mentioning: their ability ot perform stable skill injection, e.g. adding a cartwheel without forgetting how to walk. A common failure of RL humanoid policies is that teaching a new agile skill destabilizes the existing ones (catastrophic forgetting). UFO claims you can inject rare motions (cartwheel) without collapsing learned behavior. If it holds, that's a significant incremental/continual skill-learning! But again, I have to underline it: no arXiv, no external validation, no success-rate numbers. -> robotics badely needs an independent unbiased evaluator imho. Still, look at that cool demo: robot is getting kicked and pushed around (serious disturbance) during teleoperation (controlled the person at the back wearing the VR headset), and still managed to always get back up. This is some serious demonstration of stability and robustness!
0:49

New framework: Kick down your robot, it will get back up every time 🥋 Chinese startup RoboParty is a Beijing startup founded April 2025 by Huang Yi, originally shipping ROBOTO ORIGIN, the world's first full-stack open-source bipedal humanoid. They released UFO: Unsupervised Reinforcement Learning Framework for Humanoid Control. DEFINITIONS -> what differs is where the learning signal comes from: - SUPERVISED: humans supply the right answers (labels), the model imitates them. - UNSUPERVISED: no answer key, the model finds structure in raw data on its own. - REINFORCEMENT LEARNING: no answer key either, the model tries things and a reward scores each attempt. → UNSUPERVISED RL: trial and error where the agent invents its own rewards, instead of engineers hand-writing one per task. REPRESENTATION LEARNING: compress raw states into a useful internal map. TEMPORAL DISTANCE: distance on that map is "how many steps from A to B." CONTRASTIVE: trained by pulling together what's close in time, pushing apart what isn't. -> CONTRASTIVE TEMPORAL-DISTANCE REPRESENTATION LEARNING: the model builds an internal map of body states where distance means how many steps it takes to get from one to another. It is trained by contrast: states that occur close together in a movement get pulled together in the map, randomly paired states get pushed apart. UFO is an open-source training framework that teaches humanoid robots skills, like getting up, walking, goal-reaching, teleoperation, without reference motions -> no motion-capture or human-video demonstrations to imitate. Its core is TeCH, a contrastive temporal-distance representation-learning algorithm: the robot explores, builds pseudo-goals by temporal rolling, and learns goal-conditioned policies from a single unified progress reward. One framework trains five different robots (Unitree G1/H1, RoboParty RP0/RP1, AgiBot X2) with automatic config conversion in ~2–3 hours per robot! The real novelty here "no demonstrations at all". No data-collection arms race,the dominant humanoid-locomotion recipe is tracking: imitate mocap/retargeted-human reference trajectories. The robot self-generates goals from its own exploration and learns from a progress reward, needing zero reference motion data. Everybody else is fighting over data acquisition, while this team just teleports out of the race entirely (inb4 "competition is for losers 💀 ). This strategy reminds me of the DeepSeek playbook applied to robots: open-source the whole stack to become the global default and commoditize everyone else. RoboParty is giving away hardware and now control software (UFO) to be the Android of humanoids. Yet another reason for the US to ban Chinese open models perhaps 🥶 ? What I also really like about this approach is the cross-embodiment infrastructure, one framework trains Unitree G1/H1, RoboParty RP0/RP1, and AgiBot X2 with automatic configuration conversion. Just like Physical Intelligence, RoboParty seems to place itself as a neutral hardware agnostic middle man. Also woth mentioning: their ability ot perform stable skill injection, e.g. adding a cartwheel without forgetting how to walk. A common failure of RL humanoid policies is that teaching a new agile skill destabilizes the existing ones (catastrophic forgetting). UFO claims you can inject rare motions (cartwheel) without collapsing learned behavior. If it holds, that's a significant incremental/continual skill-learning! But again, I have to underline it: no arXiv, no external validation, no success-rate numbers. -> robotics badely needs an independent unbiased evaluator imho. Still, look at that cool demo: robot is getting kicked and pushed around (serious disturbance) during teleoperation (controlled the person at the back wearing the VR headset), and still managed to always get back up. This is some serious demonstration of stability and robustness!

Léo

35,705 просмотров • 12 дней назад

Let's reverse engineer this demo. You need 3 things: (1) robust hardware and motor designs that treat simulation as first-class citizen; (2) a human motion capture ("mocap") dataset, such as those for film and gaming characters; (3) massively parallel RL training in GPU-accelerated simulation. Last October, our team trained a 1.5M parameter foundation model called HOVER for such agile motor control. It follows this recipe, roughly speaking (details in thread): (1) Simulation used to be an after-thought. Now, it has to be part of the hardware design process. If your robot doesn't simulate well, you can kiss RL goodbye. Hardware-simulation co-design is a very interesting emergent topic that only becomes meaningful with today's compute capability. (2) Human mocap dataset to produce natural-looking walking and running gaits. That's one huge advantage of using humanoid robot - you get to imitate from tons of human motions that were originally captured for movies or AAA games. At least 3 ways to use the data: - For initialization: pre-train the neural net to imitate human, and then finetune it into the robot form factor with physics turned on; - For reward function: penalize any deviations from the target pose; - For representation learning: treat the human poses as a "motion prior" to constrain the space of robot behaviors. (3) Shove the above into Isaac sim, add a lot of randomization, pump it through PPO, throw in a bunch of GPUs, and then watch Netflix till loss converges. If you have an urge to comment this is CGI, let me save you a few keystrokes — many academic labs now own the G1 robot in the flesh. See our team's HOVER work in the thread: 🧵
1:04

Let's reverse engineer this demo. You need 3 things: (1) robust hardware and motor designs that treat simulation as first-class citizen; (2) a human motion capture ("mocap") dataset, such as those for film and gaming characters; (3) massively parallel RL training in GPU-accelerated simulation. Last October, our team trained a 1.5M parameter foundation model called HOVER for such agile motor control. It follows this recipe, roughly speaking (details in thread): (1) Simulation used to be an after-thought. Now, it has to be part of the hardware design process. If your robot doesn't simulate well, you can kiss RL goodbye. Hardware-simulation co-design is a very interesting emergent topic that only becomes meaningful with today's compute capability. (2) Human mocap dataset to produce natural-looking walking and running gaits. That's one huge advantage of using humanoid robot - you get to imitate from tons of human motions that were originally captured for movies or AAA games. At least 3 ways to use the data: - For initialization: pre-train the neural net to imitate human, and then finetune it into the robot form factor with physics turned on; - For reward function: penalize any deviations from the target pose; - For representation learning: treat the human poses as a "motion prior" to constrain the space of robot behaviors. (3) Shove the above into Isaac sim, add a lot of randomization, pump it through PPO, throw in a bunch of GPUs, and then watch Netflix till loss converges. If you have an urge to comment this is CGI, let me save you a few keystrokes — many academic labs now own the G1 robot in the flesh. See our team's HOVER work in the thread: 🧵

Jim Fan

216,139 просмотров • 1 год назад

Exciting updates on Project GR00T! We discover a systematic way to scale up robot data, tackling the most painful pain point in robotics. The idea is simple: human collects demonstration on a real robot, and we multiply that data 1000x or more in simulation. Let’s break it down: 1. We use Apple Vision Pro (yes!!) to give the human operator first person control of the humanoid. Vision Pro parses human hand pose and retargets the motion to the robot hand, all in real time. From the human’s point of view, they are immersed in another body like the Avatar. Teleoperation is slow and time-consuming, but we can afford to collect a small amount of data. 2. We use RoboCasa, a generative simulation framework, to multiply the demonstration data by varying the visual appearance and layout of the environment. In Jensen’s keynote video below, the humanoid is now placing the cup in hundreds of kitchens with a huge diversity of textures, furniture, and object placement. We only have 1 physical kitchen at the GEAR Lab in NVIDIA HQ, but we can conjure up infinite ones in simulation. 3. Finally, we apply MimicGen, a technique to multiply the above data even more by varying the *motion* of the robot. MimicGen generates vast number of new action trajectories based on the original human data, and filters out failed ones (e.g. those that drop the cup) to form a much larger dataset. To sum up, given 1 human trajectory with Vision Pro -> RoboCasa produces N (varying visuals) -> MimicGen further augments to NxM (varying motions). This is the way to trade compute for expensive human data by GPU-accelerated simulation. A while ago, I mentioned that teleoperation is fundamentally not scalable, because we are always limited by 24 hrs/robot/day in the world of atoms. Our new GR00T synthetic data pipeline breaks this barrier in the world of bits. Scaling has been so much fun for LLMs, and it's finally our turn to have fun in robotics! We are building tools to enable everyone in the ecosystem to scale up with us. Links in thread:
1:21

Exciting updates on Project GR00T! We discover a systematic way to scale up robot data, tackling the most painful pain point in robotics. The idea is simple: human collects demonstration on a real robot, and we multiply that data 1000x or more in simulation. Let’s break it down: 1. We use Apple Vision Pro (yes!!) to give the human operator first person control of the humanoid. Vision Pro parses human hand pose and retargets the motion to the robot hand, all in real time. From the human’s point of view, they are immersed in another body like the Avatar. Teleoperation is slow and time-consuming, but we can afford to collect a small amount of data. 2. We use RoboCasa, a generative simulation framework, to multiply the demonstration data by varying the visual appearance and layout of the environment. In Jensen’s keynote video below, the humanoid is now placing the cup in hundreds of kitchens with a huge diversity of textures, furniture, and object placement. We only have 1 physical kitchen at the GEAR Lab in NVIDIA HQ, but we can conjure up infinite ones in simulation. 3. Finally, we apply MimicGen, a technique to multiply the above data even more by varying the *motion* of the robot. MimicGen generates vast number of new action trajectories based on the original human data, and filters out failed ones (e.g. those that drop the cup) to form a much larger dataset. To sum up, given 1 human trajectory with Vision Pro -> RoboCasa produces N (varying visuals) -> MimicGen further augments to NxM (varying motions). This is the way to trade compute for expensive human data by GPU-accelerated simulation. A while ago, I mentioned that teleoperation is fundamentally not scalable, because we are always limited by 24 hrs/robot/day in the world of atoms. Our new GR00T synthetic data pipeline breaks this barrier in the world of bits. Scaling has been so much fun for LLMs, and it's finally our turn to have fun in robotics! We are building tools to enable everyone in the ecosystem to scale up with us. Links in thread:

Jim Fan

364,514 просмотров • 2 лет назад

🚀 𝐏𝐫𝐞𝐬𝐞𝐧𝐭𝐢𝐧𝐠 𝐚 𝐩𝐚𝐩𝐞𝐫 𝐢𝐬 𝐚𝐧 𝐚𝐫𝐭!🎤 🤔 Ever felt that most presentation tools lack flexibility and creativity? 𝘔𝘦𝘳𝘦𝘭𝘺 𝘦𝘹𝘵𝘳𝘢𝘤𝘵𝘪𝘯𝘨 𝘤𝘰𝘯𝘵𝘦𝘯𝘵, 𝘧𝘰𝘳𝘤𝘪𝘯𝘨 𝘳𝘪𝘨𝘪𝘥 𝘥𝘦𝘴𝘪𝘨𝘯𝘴, 𝘢𝘯𝘥 𝘥𝘦𝘮𝘢𝘯𝘥𝘪𝘯𝘨 𝘮𝘢𝘯𝘶𝘢𝘭 𝘵𝘸𝘦𝘢𝘬𝘴. 𝐄𝐯𝐨𝐏𝐫𝐞𝐬𝐞𝐧𝐭 changes all of that! ✨ EvoPresent is a self-optimizing framework that unites storytelling, design, and feedback to create effortless, engaging presentation videos. 🎥 💡 𝐊𝐞𝐲 𝐇𝐢𝐠𝐡𝐥𝐢𝐠𝐡𝐭𝐬: 💠 𝐏𝐫𝐞𝐬𝐀𝐞𝐬𝐭𝐡, the core multi-task RL model, continuously refines both content and design — ensuring slides that are impactful and visually captivating. 📊 𝐄𝐯𝐨𝐏𝐫𝐞𝐬𝐞𝐧𝐭 𝐁𝐞𝐧𝐜𝐡𝐦𝐚𝐫𝐤 is a comprehensive evaluation suite: 650+ top AI papers & diverse formats to assess content and design, and 2000+ slide pairs for aesthetic scoring, defect correction, and design comparison. 🎯 🧠 𝘏𝘪𝘨𝘩-𝘲𝘶𝘢𝘭𝘪𝘵𝘺 𝘧𝘦𝘦𝘥𝘣𝘢𝘤𝘬 𝘱𝘰𝘸𝘦𝘳𝘴 𝘤𝘰𝘯𝘵𝘪𝘯𝘶𝘰𝘶𝘴 𝘴𝘦𝘭𝘧-𝘪𝘮𝘱𝘳𝘰𝘷𝘦𝘮𝘦𝘯𝘵. ⚖️ 𝘉𝘢𝘭𝘢𝘯𝘤𝘦 𝘣𝘦𝘵𝘸𝘦𝘦𝘯 𝘤𝘰𝘯𝘵𝘦𝘯𝘵 & 𝘥𝘦𝘴𝘪𝘨𝘯 is the secret to presentation excellence. 🔁 𝐌𝐮𝐥𝐭𝐢-𝐭𝐚𝐬𝐤 𝐑𝐋 training boosts generalization in aesthetic awareness tasks. 𝐃𝐢𝐬𝐜𝐥𝐚𝐢𝐦𝐞𝐫: the demo video was completely generated by EvoPresent, no human refinement.
9:18

🚀 𝐏𝐫𝐞𝐬𝐞𝐧𝐭𝐢𝐧𝐠 𝐚 𝐩𝐚𝐩𝐞𝐫 𝐢𝐬 𝐚𝐧 𝐚𝐫𝐭!🎤 🤔 Ever felt that most presentation tools lack flexibility and creativity? 𝘔𝘦𝘳𝘦𝘭𝘺 𝘦𝘹𝘵𝘳𝘢𝘤𝘵𝘪𝘯𝘨 𝘤𝘰𝘯𝘵𝘦𝘯𝘵, 𝘧𝘰𝘳𝘤𝘪𝘯𝘨 𝘳𝘪𝘨𝘪𝘥 𝘥𝘦𝘴𝘪𝘨𝘯𝘴, 𝘢𝘯𝘥 𝘥𝘦𝘮𝘢𝘯𝘥𝘪𝘯𝘨 𝘮𝘢𝘯𝘶𝘢𝘭 𝘵𝘸𝘦𝘢𝘬𝘴. 𝐄𝐯𝐨𝐏𝐫𝐞𝐬𝐞𝐧𝐭 changes all of that! ✨ EvoPresent is a self-optimizing framework that unites storytelling, design, and feedback to create effortless, engaging presentation videos. 🎥 💡 𝐊𝐞𝐲 𝐇𝐢𝐠𝐡𝐥𝐢𝐠𝐡𝐭𝐬: 💠 𝐏𝐫𝐞𝐬𝐀𝐞𝐬𝐭𝐡, the core multi-task RL model, continuously refines both content and design — ensuring slides that are impactful and visually captivating. 📊 𝐄𝐯𝐨𝐏𝐫𝐞𝐬𝐞𝐧𝐭 𝐁𝐞𝐧𝐜𝐡𝐦𝐚𝐫𝐤 is a comprehensive evaluation suite: 650+ top AI papers & diverse formats to assess content and design, and 2000+ slide pairs for aesthetic scoring, defect correction, and design comparison. 🎯 🧠 𝘏𝘪𝘨𝘩-𝘲𝘶𝘢𝘭𝘪𝘵𝘺 𝘧𝘦𝘦𝘥𝘣𝘢𝘤𝘬 𝘱𝘰𝘸𝘦𝘳𝘴 𝘤𝘰𝘯𝘵𝘪𝘯𝘶𝘰𝘶𝘴 𝘴𝘦𝘭𝘧-𝘪𝘮𝘱𝘳𝘰𝘷𝘦𝘮𝘦𝘯𝘵. ⚖️ 𝘉𝘢𝘭𝘢𝘯𝘤𝘦 𝘣𝘦𝘵𝘸𝘦𝘦𝘯 𝘤𝘰𝘯𝘵𝘦𝘯𝘵 & 𝘥𝘦𝘴𝘪𝘨𝘯 is the secret to presentation excellence. 🔁 𝐌𝐮𝐥𝐭𝐢-𝐭𝐚𝐬𝐤 𝐑𝐋 training boosts generalization in aesthetic awareness tasks. 𝐃𝐢𝐬𝐜𝐥𝐚𝐢𝐦𝐞𝐫: the demo video was completely generated by EvoPresent, no human refinement.

Xin Eric Wang

18,226 просмотров • 10 месяцев назад

Alibaba presents MIMO Controllable Character Video Synthesis with Spatial Decomposed Modeling Character video synthesis aims to produce realistic videos of animatable characters within lifelike scenes. As a fundamental problem in the computer vision and graphics community, 3D works typically require multi-view captures for per-case training, which severely limits their applicability of modeling arbitrary characters in a short time. Recent 2D methods break this limitation via pre-trained diffusion models, but they struggle for pose generality and scene interaction. To this end, we propose MIMO, a novel framework which can not only synthesize character videos with controllable attributes (i.e., character, motion and scene) provided by simple user inputs, but also simultaneously achieve advanced scalability to arbitrary characters, generality to novel 3D motions, and applicability to interactive real-world scenes in a unified framework. The core idea is to encode the 2D video to compact spatial codes, considering the inherent 3D nature of video occurrence. Concretely, we lift the 2D frame pixels into 3D using monocular depth estimators, and decompose the video clip to three spatial components (i.e., main human, underlying scene, and floating occlusion) in hierarchical layers based on the 3D depth. These components are further encoded to canonical identity code, structured motion code and full scene code, which are utilized as control signals of synthesis process. The design of spatial decomposed modeling enables flexible user control, complex motion expression, as well as 3D-aware synthesis for scene interactions. Experimental results demonstrate effectiveness and robustness of the proposed method.
1:29

Alibaba presents MIMO Controllable Character Video Synthesis with Spatial Decomposed Modeling Character video synthesis aims to produce realistic videos of animatable characters within lifelike scenes. As a fundamental problem in the computer vision and graphics community, 3D works typically require multi-view captures for per-case training, which severely limits their applicability of modeling arbitrary characters in a short time. Recent 2D methods break this limitation via pre-trained diffusion models, but they struggle for pose generality and scene interaction. To this end, we propose MIMO, a novel framework which can not only synthesize character videos with controllable attributes (i.e., character, motion and scene) provided by simple user inputs, but also simultaneously achieve advanced scalability to arbitrary characters, generality to novel 3D motions, and applicability to interactive real-world scenes in a unified framework. The core idea is to encode the 2D video to compact spatial codes, considering the inherent 3D nature of video occurrence. Concretely, we lift the 2D frame pixels into 3D using monocular depth estimators, and decompose the video clip to three spatial components (i.e., main human, underlying scene, and floating occlusion) in hierarchical layers based on the 3D depth. These components are further encoded to canonical identity code, structured motion code and full scene code, which are utilized as control signals of synthesis process. The design of spatial decomposed modeling enables flexible user control, complex motion expression, as well as 3D-aware synthesis for scene interactions. Experimental results demonstrate effectiveness and robustness of the proposed method.

AK

149,079 просмотров • 1 год назад

Domain Randomization (DR) is a key component of the data augmentation pipeline at Axis Robotics. By applying DR, we are able to scale verified, high-quality human trajectories by 10x to 100x. During training, we systematically introduce variances in environmental parameters. This prevents the model from relying on spurious visual correlations. The objective is to ensure the policy learns rather than overfitting. To demonstrate the necessity and effectiveness of this approach, we evaluated both DR and No-DR models on Task 74 (pour_water_into_mug). The empirical results show a definitive impact on real-world deployment reliability: integrating DR into the pipeline increased the success rate from 0% to 90% (Fig. 1). This divergence stems from how the respective policies process visual observations (Fig. 2). The baseline (No DR) model overfits to the static visual background. It essentially memorizes the poses from the training dataset but fails to generalize when subjected to the inevitable variances of real-world deployment. Consequently, it cannot execute the correct manipulation on the target object. Conversely, the DR-trained model learns to extract essential geometric features and physical constraints, filtering out superficial visual noise. This leads to significantly higher robustness in dynamic environments. The structural difference in execution is clearly reflected in the end-effector trajectory data: These real-world deployment recordings further illustrate this difference (Videos 1 and 2). Scaling Physical AI requires turning raw trajectory data into robust policies, and a rigorously engineered DR infrastructure is an essential bridge to close the Sim2Real gap.
0:20

Domain Randomization (DR) is a key component of the data augmentation pipeline at Axis Robotics. By applying DR, we are able to scale verified, high-quality human trajectories by 10x to 100x. During training, we systematically introduce variances in environmental parameters. This prevents the model from relying on spurious visual correlations. The objective is to ensure the policy learns rather than overfitting. To demonstrate the necessity and effectiveness of this approach, we evaluated both DR and No-DR models on Task 74 (pour_water_into_mug). The empirical results show a definitive impact on real-world deployment reliability: integrating DR into the pipeline increased the success rate from 0% to 90% (Fig. 1). This divergence stems from how the respective policies process visual observations (Fig. 2). The baseline (No DR) model overfits to the static visual background. It essentially memorizes the poses from the training dataset but fails to generalize when subjected to the inevitable variances of real-world deployment. Consequently, it cannot execute the correct manipulation on the target object. Conversely, the DR-trained model learns to extract essential geometric features and physical constraints, filtering out superficial visual noise. This leads to significantly higher robustness in dynamic environments. The structural difference in execution is clearly reflected in the end-effector trajectory data: These real-world deployment recordings further illustrate this difference (Videos 1 and 2). Scaling Physical AI requires turning raw trajectory data into robust policies, and a rigorously engineered DR infrastructure is an essential bridge to close the Sim2Real gap.

Axis Robotics

27,125 просмотров • 4 месяцев назад

AgiBot has formally unveiled its G2 humanoid robot, a system designed to transition into various industries and liberate humans from repetitive labor. G2 features high-performance joints, precision torque sensors, and an advanced spatial perception system, supporting quick deployment and multi-modal voice interaction. ► Factory Floor Performance: The G2 is engineered to industrial standards. In a safety belt lock production line, robots collaborate with human workers, performing tasks like pressing lock cores. The G2 collects production data to continuously train and iterate models (local server deployment ensures data privacy), steadily improving its operational ability. ► Mobility & Safety: The G2 navigates narrow factory aisles using dual LiDAR and full-panorama vision for environment sensing and collision detection. Its chassis is designed to overcome common obstacles (speed bumps, elevator gaps). It supports 24/7 continuous operation via autonomous return-to-charge and battery swapping. ► Humanoid Design Advantage: The G2's design includes a three-degree-of-freedom flexible waist, allowing it to mimic natural human movements like bending and side-leaning. This dramatically expands its operational workspace and enables seamless integration into existing human-centric production lines without costly modifications. ► Advanced Dexterity & Learning (Lab): The new G02 arm is the world's first cross-moment arm, featuring high-precision joint torque sensors that allow it to precisely sense external forces and adjust stiffness, mimicking human hand compliance. Using Real-Machine Reinforcement Learning (RL), the G2 can learn complex, delicate tasks like memory stick insertion in about one hour with minimal human intervention. ► Logistics & Grasping: In logistics sorting, the G2 uses a 19-degree-of-freedom mechanical dexterous hand (20N maximum fingertip force; 35kg capacity for hard objects) equipped with 3D tactile sensors to ensure it grasps securely without damaging items. Its full-body articulation (waist and legs) aids grasping and posture adjustment. ► Model & Data: G2's intelligence is powered by the Go-One Large Embodied Model (VLA architecture: Vision-Language-Latent Action) and the GE-One World Model (vision-centric predictive modeling), trained using the AgiBot Word true-machine dataset (over 500k downloads). ► Service & Interaction: The G2 is deployed as a guide/receptionist in settings like art museums. It uses its high-DOF head, arms, and waist to point to exhibits, maintains eye contact while navigating difficult spaces (chassis walks forward, body faces backward), handles specialized and random queries, and uses proactive safety features (stops movement, issues warnings) when people get too close.
2:49

AgiBot has formally unveiled its G2 humanoid robot, a system designed to transition into various industries and liberate humans from repetitive labor. G2 features high-performance joints, precision torque sensors, and an advanced spatial perception system, supporting quick deployment and multi-modal voice interaction. ► Factory Floor Performance: The G2 is engineered to industrial standards. In a safety belt lock production line, robots collaborate with human workers, performing tasks like pressing lock cores. The G2 collects production data to continuously train and iterate models (local server deployment ensures data privacy), steadily improving its operational ability. ► Mobility & Safety: The G2 navigates narrow factory aisles using dual LiDAR and full-panorama vision for environment sensing and collision detection. Its chassis is designed to overcome common obstacles (speed bumps, elevator gaps). It supports 24/7 continuous operation via autonomous return-to-charge and battery swapping. ► Humanoid Design Advantage: The G2's design includes a three-degree-of-freedom flexible waist, allowing it to mimic natural human movements like bending and side-leaning. This dramatically expands its operational workspace and enables seamless integration into existing human-centric production lines without costly modifications. ► Advanced Dexterity & Learning (Lab): The new G02 arm is the world's first cross-moment arm, featuring high-precision joint torque sensors that allow it to precisely sense external forces and adjust stiffness, mimicking human hand compliance. Using Real-Machine Reinforcement Learning (RL), the G2 can learn complex, delicate tasks like memory stick insertion in about one hour with minimal human intervention. ► Logistics & Grasping: In logistics sorting, the G2 uses a 19-degree-of-freedom mechanical dexterous hand (20N maximum fingertip force; 35kg capacity for hard objects) equipped with 3D tactile sensors to ensure it grasps securely without damaging items. Its full-body articulation (waist and legs) aids grasping and posture adjustment. ► Model & Data: G2's intelligence is powered by the Go-One Large Embodied Model (VLA architecture: Vision-Language-Latent Action) and the GE-One World Model (vision-centric predictive modeling), trained using the AgiBot Word true-machine dataset (over 500k downloads). ► Service & Interaction: The G2 is deployed as a guide/receptionist in settings like art museums. It uses its high-DOF head, arms, and waist to point to exhibits, maintains eye contact while navigating difficult spaces (chassis walks forward, body faces backward), handles specialized and random queries, and uses proactive safety features (stops movement, issues warnings) when people get too close.

RoboHub🤖

46,733 просмотров • 10 месяцев назад

Video: World’s first humanoid robot labor that swaps its own batteries to work endlessly | Jijo Malayil, Interesting Engineering Walker S2 uses dual-battery balancing and standardized modules to boost efficiency and ensure uninterrupted, optimized performance. In a leap for robotics, China’s UBTech has unveiled the Walker S2, the world’s first humanoid robot capable of fully autonomous battery swapping. Designed for non-stop industrial operations, the Walker S2 can replace its own power pack in just three minutes—no human intervention required. Equipped with advanced anthropomorphic bipedal locomotion and a hot-swappable battery system, Walker S2 is built to operate 24/7 across dynamic industrial environments. According to UBTech, the next-generation humanoid robot marks a major milestone in automation, bringing continuous, hands-free performance to the factory floor. In May 2025, UBTech Robotics and Huawei Technologies inked a significant partnership to accelerate the adoption of humanoid robots across China’s factories and households. Uninterrupted robot operations A video posted by the robotics firm opens with the sleek UBTech Walker S2 humanoid robot working in an industrial setting. The highlight, however, is its autonomous battery swap. Walker S2 approaches the charging station, carefully detaches its depleted power pack, and seamlessly installs a fresh one—all within about three minutes—without any human assistance, according to CGTN. The camera captures close-ups of the robot’s articulated limbs and the intelligent battery-handling mechanism, conveying precision and reliability. As the swap completes, Walker S2 resumes its duties, reinforcing the promise of uninterrupted, 24/7 operations in dynamic factory environments. UBTech’s Walker S2 humanoid robot is equipped with advanced dual-battery power balancing technology and uses standardized battery modules to optimize performance, reports CNEVPOST. This dual-battery system allows the robot to automatically switch to a backup battery in case of a main battery failure, ensuring that critical tasks are carried out without interruption. In addition to battery swapping, the robot can intelligently choose between charging and swapping based on task urgency, allowing it to manage energy dynamically and adapt to real-time operational demands. UBTech highlights these features as a step forward in deploying humanoid robots for industrial and domestic applications, combining flexibility, reliability, and autonomy in one intelligent platform. Factory intelligence upgrade Earlier in the year, UBTech unveiled a major advancement in humanoid robot collaboration, claiming the world’s first deployment of multiple humanoids working together across varied industrial tasks. Demonstrated at Zeekr’s 5G-enabled smart factory, the breakthrough centers on UBTech’s “BrainNet” framework, which orchestrates cooperative behavior through a cloud-device intelligence system. BrainNet integrates a “super brain” for high-level decision-making with an “intelligent sub-brain” for distributed multi-robot control. The super brain, powered by a proprietary large-scale multimodal reasoning model, handles complex production-line scheduling and decision-making. Meanwhile, the sub-brain coordinates real-time tasks using cross-field perception and Transformer-based control for dynamic adaptability. Together, they enable the Walker S1 humanoid robots to move beyond isolated operations and perform coordinated tasks with high precision and speed. The system is built on DeepSeek-R1 reasoning technology and trained on real-world data from automotive factory settings. Leveraging Retrieval-Augmented Generation (RAG), the model adapts to specific job functions and improves scalability across workstations. At Zeekr’s facility, dozens of Walker S1s now collaborate on tasks like assembly, inspection, and part handling. Using semantic VSLAM and shared mapping, they coordinate seamlessly via vision-based navigation and agile manipulation. UBTech says this marks a transition to “Practical Training 2.0,” where humanoid robots operate as a swarm, maximizing efficiency and setting the stage for next-generation intelligent manufacturing.
1:04

Video: World’s first humanoid robot labor that swaps its own batteries to work endlessly | Jijo Malayil, Interesting Engineering Walker S2 uses dual-battery balancing and standardized modules to boost efficiency and ensure uninterrupted, optimized performance. In a leap for robotics, China’s UBTech has unveiled the Walker S2, the world’s first humanoid robot capable of fully autonomous battery swapping. Designed for non-stop industrial operations, the Walker S2 can replace its own power pack in just three minutes—no human intervention required. Equipped with advanced anthropomorphic bipedal locomotion and a hot-swappable battery system, Walker S2 is built to operate 24/7 across dynamic industrial environments. According to UBTech, the next-generation humanoid robot marks a major milestone in automation, bringing continuous, hands-free performance to the factory floor. In May 2025, UBTech Robotics and Huawei Technologies inked a significant partnership to accelerate the adoption of humanoid robots across China’s factories and households. Uninterrupted robot operations A video posted by the robotics firm opens with the sleek UBTech Walker S2 humanoid robot working in an industrial setting. The highlight, however, is its autonomous battery swap. Walker S2 approaches the charging station, carefully detaches its depleted power pack, and seamlessly installs a fresh one—all within about three minutes—without any human assistance, according to CGTN. The camera captures close-ups of the robot’s articulated limbs and the intelligent battery-handling mechanism, conveying precision and reliability. As the swap completes, Walker S2 resumes its duties, reinforcing the promise of uninterrupted, 24/7 operations in dynamic factory environments. UBTech’s Walker S2 humanoid robot is equipped with advanced dual-battery power balancing technology and uses standardized battery modules to optimize performance, reports CNEVPOST. This dual-battery system allows the robot to automatically switch to a backup battery in case of a main battery failure, ensuring that critical tasks are carried out without interruption. In addition to battery swapping, the robot can intelligently choose between charging and swapping based on task urgency, allowing it to manage energy dynamically and adapt to real-time operational demands. UBTech highlights these features as a step forward in deploying humanoid robots for industrial and domestic applications, combining flexibility, reliability, and autonomy in one intelligent platform. Factory intelligence upgrade Earlier in the year, UBTech unveiled a major advancement in humanoid robot collaboration, claiming the world’s first deployment of multiple humanoids working together across varied industrial tasks. Demonstrated at Zeekr’s 5G-enabled smart factory, the breakthrough centers on UBTech’s “BrainNet” framework, which orchestrates cooperative behavior through a cloud-device intelligence system. BrainNet integrates a “super brain” for high-level decision-making with an “intelligent sub-brain” for distributed multi-robot control. The super brain, powered by a proprietary large-scale multimodal reasoning model, handles complex production-line scheduling and decision-making. Meanwhile, the sub-brain coordinates real-time tasks using cross-field perception and Transformer-based control for dynamic adaptability. Together, they enable the Walker S1 humanoid robots to move beyond isolated operations and perform coordinated tasks with high precision and speed. The system is built on DeepSeek-R1 reasoning technology and trained on real-world data from automotive factory settings. Leveraging Retrieval-Augmented Generation (RAG), the model adapts to specific job functions and improves scalability across workstations. At Zeekr’s facility, dozens of Walker S1s now collaborate on tasks like assembly, inspection, and part handling. Using semantic VSLAM and shared mapping, they coordinate seamlessly via vision-based navigation and agile manipulation. UBTech says this marks a transition to “Practical Training 2.0,” where humanoid robots operate as a swarm, maximizing efficiency and setting the stage for next-generation intelligent manufacturing.

Owen Gregorian

35,637 просмотров • 1 год назад

China unveils humanoid robot worker with brain that runs 275 trillion ops/sec | Jijo Malayil, Interesting Engineering In tests, SUYUAN used vision and joint control to sort and move crates of various sizes, greatly improving warehouse productivity. Chinese manufacturing firm Shanghai Electric has unveiled its first self-developed industrial humanoid robot, “SUYUAN,” marking a major milestone in its robotics journey. Debuting at the World Artificial Intelligence Conference (WAIC 2025) on July 26 in Shanghai, SUYUAN boasts 38 degrees of freedom and 275 TOPS of on-device computing power, enabling precise operations and fluid movements. According to the firm, designed for diverse industrial use, the robot showcases Shanghai Electric’s end-to-end capabilities—from core tech to integrated solutions—and reinforces its commitment to next-gen industrial automation through a full industry chain strategy. At WAIC 2025, Shanghai Electric also unveiled a new joint venture with Johnson Electric for next-gen humanoid robotics and showcased its “LINGKE” dual-arm robot. Recently, Hangzhou-based Unitree Robotics launched the R1 humanoid with 26 joints for $5,900, showcasing athletic feats like cartwheels, running, and quick recovery. Smart factory assistant Shanghai Electric claims SUYUAN, equipped with 38 degrees of freedom (DoF) and a powerful 275 TOPS on-device computing processor, delivers fluid, human-like movements and high-precision operations across various industrial scenarios. Its advanced articulation and real-time processing capabilities make it highly adaptable, enabling smooth execution of complex tasks in dynamic work environments. SUYUAN, who weighs 110 pounds (50 kilograms) and is 5 feet 6 inches (167 cm) tall, was designed to have human-like proportions. Its 38-DoF articulation offers dexterity, allowing for both wide-range motion and sensitive manipulation. With a single arm, the robot can lift objects up to 4.4 pounds (2 kilograms) in weight and carry a total payload of up to 22 pounds (10 kilograms). With a walking pace of 3.1 miles per hour (5 km/h), SUYUAN is ideal for environments including assembly lines, warehousing, and logistics, according to a statement. To navigate complex industrial settings, SUYUAN combines LiDAR and binocular vision for self-guided mobility. Its 275-TOPS AI processor enables rapid data analysis and integration with large language models, allowing it to understand tasks in natural language and handle objects adaptively, reports Fox 44 News. In pilot demonstrations, the robot successfully identified, picked, and relocated crates of varying sizes using advanced computer vision and coordinated joint control—delivering measurable gains in warehouse efficiency. The company claims that SUYUAN’s launch represents a major turning point in Shanghai Electric’s foray into humanoid robotics and strengthens its vertically integrated approach to industrial automation solutions. Intelligent task handling Shanghai Electric also demonstrated its most recent developments in intelligent manufacturing at WAIC 2025, introducing a new joint venture with Johnson Electric centered on next-generation humanoid robotics and showcasing the “LINGKE” dual-arm robot. With its high-precision operations, adaptive teamwork, and closed-loop data capabilities, the LINGKE robot demonstrated live talents in handling complicated production jobs. LINGKE is made to do more than just replace human labor; it uses compliant force control and bimanual coordination to relieve workers of high-intensity, repetitive jobs. According to the company, the robot enhances operational efficiency by up to five times. Its core strength lies in a Data-Model-Deployment closed-loop system that starts with operational data, followed by data cleansing, model training, live deployment, and feedback-driven optimization—enabling autonomous learning and workflow improvement. Also at the event, Shanghai Electric and Johnson Electric introduced advanced hardware modules for humanoid robots, including rotary joints, linear joints, and dexterous finger joints. These components are designed to support smooth, precise, and quiet motion performance across robotics systems, reports Stock Titan. The joint venture announced two strategic agreements: a first-unit supply deal with the National and Local Co-Built Humanoid Robotics Innovation Center (Qinglong Project) and a cooperation memorandum with Fourier Robotics. Read more:
0:58

China unveils humanoid robot worker with brain that runs 275 trillion ops/sec | Jijo Malayil, Interesting Engineering In tests, SUYUAN used vision and joint control to sort and move crates of various sizes, greatly improving warehouse productivity. Chinese manufacturing firm Shanghai Electric has unveiled its first self-developed industrial humanoid robot, “SUYUAN,” marking a major milestone in its robotics journey. Debuting at the World Artificial Intelligence Conference (WAIC 2025) on July 26 in Shanghai, SUYUAN boasts 38 degrees of freedom and 275 TOPS of on-device computing power, enabling precise operations and fluid movements. According to the firm, designed for diverse industrial use, the robot showcases Shanghai Electric’s end-to-end capabilities—from core tech to integrated solutions—and reinforces its commitment to next-gen industrial automation through a full industry chain strategy. At WAIC 2025, Shanghai Electric also unveiled a new joint venture with Johnson Electric for next-gen humanoid robotics and showcased its “LINGKE” dual-arm robot. Recently, Hangzhou-based Unitree Robotics launched the R1 humanoid with 26 joints for $5,900, showcasing athletic feats like cartwheels, running, and quick recovery. Smart factory assistant Shanghai Electric claims SUYUAN, equipped with 38 degrees of freedom (DoF) and a powerful 275 TOPS on-device computing processor, delivers fluid, human-like movements and high-precision operations across various industrial scenarios. Its advanced articulation and real-time processing capabilities make it highly adaptable, enabling smooth execution of complex tasks in dynamic work environments. SUYUAN, who weighs 110 pounds (50 kilograms) and is 5 feet 6 inches (167 cm) tall, was designed to have human-like proportions. Its 38-DoF articulation offers dexterity, allowing for both wide-range motion and sensitive manipulation. With a single arm, the robot can lift objects up to 4.4 pounds (2 kilograms) in weight and carry a total payload of up to 22 pounds (10 kilograms). With a walking pace of 3.1 miles per hour (5 km/h), SUYUAN is ideal for environments including assembly lines, warehousing, and logistics, according to a statement. To navigate complex industrial settings, SUYUAN combines LiDAR and binocular vision for self-guided mobility. Its 275-TOPS AI processor enables rapid data analysis and integration with large language models, allowing it to understand tasks in natural language and handle objects adaptively, reports Fox 44 News. In pilot demonstrations, the robot successfully identified, picked, and relocated crates of varying sizes using advanced computer vision and coordinated joint control—delivering measurable gains in warehouse efficiency. The company claims that SUYUAN’s launch represents a major turning point in Shanghai Electric’s foray into humanoid robotics and strengthens its vertically integrated approach to industrial automation solutions. Intelligent task handling Shanghai Electric also demonstrated its most recent developments in intelligent manufacturing at WAIC 2025, introducing a new joint venture with Johnson Electric centered on next-generation humanoid robotics and showcasing the “LINGKE” dual-arm robot. With its high-precision operations, adaptive teamwork, and closed-loop data capabilities, the LINGKE robot demonstrated live talents in handling complicated production jobs. LINGKE is made to do more than just replace human labor; it uses compliant force control and bimanual coordination to relieve workers of high-intensity, repetitive jobs. According to the company, the robot enhances operational efficiency by up to five times. Its core strength lies in a Data-Model-Deployment closed-loop system that starts with operational data, followed by data cleansing, model training, live deployment, and feedback-driven optimization—enabling autonomous learning and workflow improvement. Also at the event, Shanghai Electric and Johnson Electric introduced advanced hardware modules for humanoid robots, including rotary joints, linear joints, and dexterous finger joints. These components are designed to support smooth, precise, and quiet motion performance across robotics systems, reports Stock Titan. The joint venture announced two strategic agreements: a first-unit supply deal with the National and Local Co-Built Humanoid Robotics Innovation Center (Qinglong Project) and a cooperation memorandum with Fourier Robotics. Read more:

Owen Gregorian

51,638 просмотров • 1 год назад

China’s pretty humanoid robot stuns by opening a car door in a ‘world’s first’ | Jijo Malayil, Interesting Engineering Mornine used onboard sensors and full-body control to locate the handle, adjust posture, and open a car door—no human input needed. AiMOGA Robotics has claimed to have reached a significant milestone in embodied AI with its humanoid robot, Mornine, autonomously opening a car door inside a functioning Chery dealership in China. Relying solely on onboard sensors, full-body motion control, and end-to-end reinforcement learning, Mornine performed the task without any human input. Unlike scripted or teleoperated robots, Mornie identified the door handle, adjusted its posture, and used coordinated force across its limbs and torso to complete the action—demonstrating advanced autonomy in a real-world setting. “The deployment marks one of the first instances of a service robot executing such a high-friction, physical interaction in a live commercial setting,” said the firm in a statement. In April, at the Shanghai Auto Show, automotive brands Omoda and Jaecoo, subsidiaries of Chery Automobile, introduced Mornine, designed for use in car dealerships. From sim to service Opening a car door may seem like a simple task, but AiMOGA Robotics views it as a pivotal moment in robotics—signaling a shift from simulation to real-world service, and from basic command execution to autonomous capability. Using only onboard sensors and full-body motion control, Mornine identified the door handle, adjusted her posture, and applied coordinated force across her limbs to open the door—entirely without human intervention. Mornine’s advanced sensor suite includes 3D LiDAR, depth and wide-angle cameras, and a visual-language model (VLM), enabling real-time perception of door position and opening status. Uniquely, Mornine wasn’t explicitly programmed to recognize door handles. Instead, she learned through reinforcement learning, undergoing millions of simulated cycles to focus on the right region and perform the task independently. “We never explicitly told the robot what a door handle is. It learned to focus on that region by itself,” said the engineering team at AiMOGA Robotics in a statement. The learned model was transferred to the real world using Sim2Real methods. Mornine continuously gathers live sensor data during operation, which feeds into a cloud-based training loop, allowing her to improve through continuous learning in real-world settings, reports Robotics Tomorrow. Now active in multiple Chery 4S dealerships in China, Mornine not only opens car doors but also assists with customer greetings, vehicle introductions, and item delivery—marking a step forward in humanoid robotics for commercial retail environments. AI meets retail Originally introduced as the AiMOGA Robot, Mornine was developed to support dealership sales by performing tasks such as explaining vehicle specifications, leading showroom tours, serving refreshments, and engaging with customers in multiple languages. First conceived by Chery as a virtual character to appeal to Generation Z using metaverse and virtual human technologies, Mornine gradually evolved into a real-world interactive humanoid. After multiple iterations of character and model design, Mornine debuted as a digital persona in animations, livestreams, and promotional content, gaining brand recognition. Chery later expanded the concept beyond the virtual space, resulting in the creation of the AiMOGA humanoid robot. Leveraging Chery’s expertise in autonomous driving, environmental sensing, and control systems, AiMOGA features full-stack capabilities in perception, cognition, decision-making, and execution. It uses multimodal sensing—combining speech, vision, and environmental data—to interpret user gestures, commands, and showroom dynamics. A bionic motion system and automotive-grade hardware enable dexterous movement and upright mobility, while multi-robot collaboration allows for coordinated tasks like guided tours. At the decision-making layer, Deepseek’s large language models enable natural language understanding and personalized interaction. In April 2025, Mornine officially began commercial service as an “Intelligent Sales Consultant” at the OMODA C5 JOYSTAR 4S dealership in Kuala Lumpur, Malaysia—marking her full transition from a virtual concept to a real-world humanoid sales assistant.
0:48

China’s pretty humanoid robot stuns by opening a car door in a ‘world’s first’ | Jijo Malayil, Interesting Engineering Mornine used onboard sensors and full-body control to locate the handle, adjust posture, and open a car door—no human input needed. AiMOGA Robotics has claimed to have reached a significant milestone in embodied AI with its humanoid robot, Mornine, autonomously opening a car door inside a functioning Chery dealership in China. Relying solely on onboard sensors, full-body motion control, and end-to-end reinforcement learning, Mornine performed the task without any human input. Unlike scripted or teleoperated robots, Mornie identified the door handle, adjusted its posture, and used coordinated force across its limbs and torso to complete the action—demonstrating advanced autonomy in a real-world setting. “The deployment marks one of the first instances of a service robot executing such a high-friction, physical interaction in a live commercial setting,” said the firm in a statement. In April, at the Shanghai Auto Show, automotive brands Omoda and Jaecoo, subsidiaries of Chery Automobile, introduced Mornine, designed for use in car dealerships. From sim to service Opening a car door may seem like a simple task, but AiMOGA Robotics views it as a pivotal moment in robotics—signaling a shift from simulation to real-world service, and from basic command execution to autonomous capability. Using only onboard sensors and full-body motion control, Mornine identified the door handle, adjusted her posture, and applied coordinated force across her limbs to open the door—entirely without human intervention. Mornine’s advanced sensor suite includes 3D LiDAR, depth and wide-angle cameras, and a visual-language model (VLM), enabling real-time perception of door position and opening status. Uniquely, Mornine wasn’t explicitly programmed to recognize door handles. Instead, she learned through reinforcement learning, undergoing millions of simulated cycles to focus on the right region and perform the task independently. “We never explicitly told the robot what a door handle is. It learned to focus on that region by itself,” said the engineering team at AiMOGA Robotics in a statement. The learned model was transferred to the real world using Sim2Real methods. Mornine continuously gathers live sensor data during operation, which feeds into a cloud-based training loop, allowing her to improve through continuous learning in real-world settings, reports Robotics Tomorrow. Now active in multiple Chery 4S dealerships in China, Mornine not only opens car doors but also assists with customer greetings, vehicle introductions, and item delivery—marking a step forward in humanoid robotics for commercial retail environments. AI meets retail Originally introduced as the AiMOGA Robot, Mornine was developed to support dealership sales by performing tasks such as explaining vehicle specifications, leading showroom tours, serving refreshments, and engaging with customers in multiple languages. First conceived by Chery as a virtual character to appeal to Generation Z using metaverse and virtual human technologies, Mornine gradually evolved into a real-world interactive humanoid. After multiple iterations of character and model design, Mornine debuted as a digital persona in animations, livestreams, and promotional content, gaining brand recognition. Chery later expanded the concept beyond the virtual space, resulting in the creation of the AiMOGA humanoid robot. Leveraging Chery’s expertise in autonomous driving, environmental sensing, and control systems, AiMOGA features full-stack capabilities in perception, cognition, decision-making, and execution. It uses multimodal sensing—combining speech, vision, and environmental data—to interpret user gestures, commands, and showroom dynamics. A bionic motion system and automotive-grade hardware enable dexterous movement and upright mobility, while multi-robot collaboration allows for coordinated tasks like guided tours. At the decision-making layer, Deepseek’s large language models enable natural language understanding and personalized interaction. In April 2025, Mornine officially began commercial service as an “Intelligent Sales Consultant” at the OMODA C5 JOYSTAR 4S dealership in Kuala Lumpur, Malaysia—marking her full transition from a virtual concept to a real-world humanoid sales assistant.

Owen Gregorian

67,975 просмотров • 1 год назад

NEW RESEARCH: You can now create a new robot optimized for any given task! I love this new project by Huy Ha, Shuran Song, and others. Called "Transformer Transformer: A Unified Model for Motion-Conditioned Robot Co-design", it generates a robot's physical design and its controller together from a task spec. DEFINITIONS: - Reward function: A scoring rule that assigns a number to how well a behavior achieves the task. Here, it is the objective the generated design is pushed to maximize (e.g., track the target motion with low error). - Tokenizing: dividing continuous or structured data (a robot's links, joints, motor specs, states, actions) into a discrete vocabulary of symbols a transformer can process, the same step that turned pixels and audio into "language" for these models. - Diffusion transformer (DiT): A transformer trained to turn random noise into structured output through iterative denoising. Here, it generates robot bodies and trajectories instead of images. - MuJoCo: The standard fast physics simulator for robotics research (DeepMind-maintained). The Menagerie is its curated zoo of ready-to-use robot models. - CMA-ES: Covariance Matrix Adaptation Evolution Strategy, the workhorse black-box optimizer: it evolves a population of candidate designs, keeps the best, and needs thousands of simulator rollouts. - Bimanual multi-trajectory optimization: Finding one design/controller that performs well across several target motions for a two-armed robot at once, harder than optimizing for a single arm and a single motion. - BERT/MAE masked-modeling trick: Train one model to fill in whatever parts of the input you hide (words for BERT, image patches for MAE); at inference, choosing what to mask chooses the task, so masking the body makes it a designer and masking the actions makes it a controller. In practice, you give it a target end-effector motion and a reward function, and it outputs a complete embodiment (link, joint, motor, and inertial property), as well as a controller to drive it. It works by tokenizing both the body (links/joints/motors) and the dynamics (states/actions) into a compact scheme called RoboTokens, training a diffusion transformer (DiT) over them. The same model predicts dynamics using those predictions ("Dynamics Self-Guidance") to push generated designs toward higher reward at inference time. Masking different token types (using the BERT/MAE masked-modeling trick) lets the one model do three jobs: generate an embodiment, control an arbitrary embodiment, or design one conditioned on a motion. It is trained on 11 robots from the MuJoCo Menagerie (0.65 kg hand to 67.5 kg quadruped, 6–35 joints), and validated in sim and on a physical ALOHA doing cloth flinging. I like the fact that this approach inverts the entire recent robotics ideas: designing a policy for a fixed robot -> designing the robot for a fixed task. Every other approach assumes the body is given and learns a controller. Transformer Transformer takes the task (target motion + reward), then generates the body and controller jointly. In practice, it is a ~180× speedup over the standard optimizer at equal-or-better quality. It reaches "CMA-ES-level quality in seconds" and finishes bimanual multi-trajectory optimization in that is worth underlining nowadays! Also worth mentioning: this is the lab behind UMI and Handroid, that I mentioned here previously! The team seems extremely creative, i love these out-of-the-box approaches. Enjoy watching the demo of robot optimization in 3D, data acquisition, then real-life testing:
0:28

NEW RESEARCH: You can now create a new robot optimized for any given task! I love this new project by Huy Ha, Shuran Song, and others. Called "Transformer Transformer: A Unified Model for Motion-Conditioned Robot Co-design", it generates a robot's physical design and its controller together from a task spec. DEFINITIONS: - Reward function: A scoring rule that assigns a number to how well a behavior achieves the task. Here, it is the objective the generated design is pushed to maximize (e.g., track the target motion with low error). - Tokenizing: dividing continuous or structured data (a robot's links, joints, motor specs, states, actions) into a discrete vocabulary of symbols a transformer can process, the same step that turned pixels and audio into "language" for these models. - Diffusion transformer (DiT): A transformer trained to turn random noise into structured output through iterative denoising. Here, it generates robot bodies and trajectories instead of images. - MuJoCo: The standard fast physics simulator for robotics research (DeepMind-maintained). The Menagerie is its curated zoo of ready-to-use robot models. - CMA-ES: Covariance Matrix Adaptation Evolution Strategy, the workhorse black-box optimizer: it evolves a population of candidate designs, keeps the best, and needs thousands of simulator rollouts. - Bimanual multi-trajectory optimization: Finding one design/controller that performs well across several target motions for a two-armed robot at once, harder than optimizing for a single arm and a single motion. - BERT/MAE masked-modeling trick: Train one model to fill in whatever parts of the input you hide (words for BERT, image patches for MAE); at inference, choosing what to mask chooses the task, so masking the body makes it a designer and masking the actions makes it a controller. In practice, you give it a target end-effector motion and a reward function, and it outputs a complete embodiment (link, joint, motor, and inertial property), as well as a controller to drive it. It works by tokenizing both the body (links/joints/motors) and the dynamics (states/actions) into a compact scheme called RoboTokens, training a diffusion transformer (DiT) over them. The same model predicts dynamics using those predictions ("Dynamics Self-Guidance") to push generated designs toward higher reward at inference time. Masking different token types (using the BERT/MAE masked-modeling trick) lets the one model do three jobs: generate an embodiment, control an arbitrary embodiment, or design one conditioned on a motion. It is trained on 11 robots from the MuJoCo Menagerie (0.65 kg hand to 67.5 kg quadruped, 6–35 joints), and validated in sim and on a physical ALOHA doing cloth flinging. I like the fact that this approach inverts the entire recent robotics ideas: designing a policy for a fixed robot -> designing the robot for a fixed task. Every other approach assumes the body is given and learns a controller. Transformer Transformer takes the task (target motion + reward), then generates the body and controller jointly. In practice, it is a ~180× speedup over the standard optimizer at equal-or-better quality. It reaches "CMA-ES-level quality in seconds" and finishes bimanual multi-trajectory optimization in that is worth underlining nowadays! Also worth mentioning: this is the lab behind UMI and Handroid, that I mentioned here previously! The team seems extremely creative, i love these out-of-the-box approaches. Enjoy watching the demo of robot optimization in 3D, data acquisition, then real-life testing:

Léo

25,735 просмотров • 8 дней назад