Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

HumanX: a scalable framework that converts single monocular human videos into agile, generalizable interaction skills for humanoid robots without task-specific rewards. Core parts: - XGen: retargets human motion + synthesizes diverse physically plausible training data (via physics-driven object trajectories and augmentation). - XMimic: unified imitation pipeline for robust generalization.... show more

The Humanoid Hub

93,606 subscribers

20,358 views • 5 months ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

HDMI (HumanoiD iMitation for Interaction) is a framework enabling humanoid robots to learn whole-body object interaction skills from monocular RGB human videos. It extracts and retargets human poses and object trajectories using GVHMR and LocoMujoco, building reference datasets with contact annotations. The data is used to train an RL policy via robot-object co-tracking. HDMI achieved 67 consecutive door traversals.

HDMI (HumanoiD iMitation for Interaction) is a framework enabling humanoid robots to learn whole-body object interaction skills from monocular RGB human videos. It extracts and retargets human poses and object trajectories using GVHMR and LocoMujoco, building reference datasets with contact annotations. The data is used to train an RL policy via robot-object co-tracking. HDMI achieved 67 consecutive door traversals.

The Humanoid Hub

17,395 views • 9 months ago

Remarkably lifelike motion and fluidity. BeyondMimic is a framework for training humanoid whole-body control from large mocap datasets. First, an open-source motion-tracking pipeline to reproduce diverse, highly dynamic human skills on real hardware, then distilling them into a guided state-action diffusion model for zero-shot, task-specific control. Project page:

Remarkably lifelike motion and fluidity. BeyondMimic is a framework for training humanoid whole-body control from large mocap datasets. First, an open-source motion-tracking pipeline to reproduce diverse, highly dynamic human skills on real hardware, then distilling them into a guided state-action diffusion model for zero-shot, task-specific control. Project page:

The Humanoid Hub

58,081 views • 11 months ago

Tired of teleoperation? One human video → 1,000s of robot demos. (📍GitHub ) Scaling Robot Data Without Dynamics Simulation or Robot Hardware Real2Render2Real (R2R2R) is a new way to scale robot data without physics simulation or hardware. You take a phone scan + a single monocular human demo. It tracks the motion, renders photorealistic scenes, and generates diverse, robot-agnostic trajectories ready for training. > No teleop, no sim, no robot, just a phone and a video > Train VLA models and diffusion policies directly on the output > Supports multiple robot embodiments with kinematic consistency > 1000s of demos in 1/27 the time of real-world collection Thank you, Max Fu, for sharing!! Project: Paper: Code coming soon: It shows that with the right pipeline, you can scale robot learning data without touching a robot. One of the most interesting directions in scalable robotics today. —— Weekly robotics and AI insights. Subscribe free:

Tired of teleoperation? One human video → 1,000s of robot demos. (📍GitHub ) Scaling Robot Data Without Dynamics Simulation or Robot Hardware Real2Render2Real (R2R2R) is a new way to scale robot data without physics simulation or hardware. You take a phone scan + a single monocular human demo. It tracks the motion, renders photorealistic scenes, and generates diverse, robot-agnostic trajectories ready for training. > No teleop, no sim, no robot, just a phone and a video > Train VLA models and diffusion policies directly on the output > Supports multiple robot embodiments with kinematic consistency > 1000s of demos in 1/27 the time of real-world collection Thank you, Max Fu, for sharing!! Project: Paper: Code coming soon: It shows that with the right pipeline, you can scale robot learning data without touching a robot. One of the most interesting directions in scalable robotics today. —— Weekly robotics and AI insights. Subscribe free:

Ilir Aliu

42,804 views • 5 months ago

Humanoids finally move like humans… and can do more than copy. [Details + demos in thread 👇] A new framework, BeyondMimic, shows how to learn naturalistic whole-body control from human motion. But then goes further by composing those skills into versatile, zero-shot policies. Two key pieces make it work: •Motion tracking pipeline: robustly turns human mocap into dynamic robot motions (jump spins, sprinting, cartwheels) on real hardware •Guided diffusion policy: distills motion primitives and enables zero-shot downstream tasks with simple cost functions The result: ✅ State-of-the-art motion quality from mocap tracking ✅ Versatile task execution; waypoint navigation, joystick teleop, obstacle avoidance ✅ Stable sim-to-real transfer for diverse humanoid skills Thank you for sharing, Qiayuan Liao! 📍 Paper: Video:

Humanoids finally move like humans… and can do more than copy. [Details + demos in thread 👇] A new framework, BeyondMimic, shows how to learn naturalistic whole-body control from human motion. But then goes further by composing those skills into versatile, zero-shot policies. Two key pieces make it work: •Motion tracking pipeline: robustly turns human mocap into dynamic robot motions (jump spins, sprinting, cartwheels) on real hardware •Guided diffusion policy: distills motion primitives and enables zero-shot downstream tasks with simple cost functions The result: ✅ State-of-the-art motion quality from mocap tracking ✅ Versatile task execution; waypoint navigation, joystick teleop, obstacle avoidance ✅ Stable sim-to-real transfer for diverse humanoid skills Thank you for sharing, Qiayuan Liao! 📍 Paper: Video:

Ilir Aliu - eu/acc

72,522 views • 11 months ago

NVIDIA has published a paper on DREAMGEN – a powerful 4-step pipeline for generating synthetic data for humanoids that enables task and environment generalization. - Step 1: Fine-tune a video generation model using a small number of human teleoperation videos - Step 2: Prompt the fine-tuned model to turn a single real image into new AI-imagined videos - Step 3: Automatically label actions in the generated videos - Step 4: Train a robot AI model with the labeled synthetic dataset This enabled humanoid robots to perform 22 novel behaviors – such as pouring, opening/closing articulated objects, and manipulating a variety of tools. The original teleoperation dataset only included pick-and-place tasks. This takes task extensibility to another level without requiring human teleoperation for every single task. The pipeline will be made open-source soon. Project page:

NVIDIA has published a paper on DREAMGEN – a powerful 4-step pipeline for generating synthetic data for humanoids that enables task and environment generalization. - Step 1: Fine-tune a video generation model using a small number of human teleoperation videos - Step 2: Prompt the fine-tuned model to turn a single real image into new AI-imagined videos - Step 3: Automatically label actions in the generated videos - Step 4: Train a robot AI model with the labeled synthetic dataset This enabled humanoid robots to perform 22 novel behaviors – such as pouring, opening/closing articulated objects, and manipulating a variety of tools. The original teleoperation dataset only included pick-and-place tasks. This takes task extensibility to another level without requiring human teleoperation for every single task. The pipeline will be made open-source soon. Project page:

The Humanoid Hub

12,074 views • 1 year ago

NEWS: Humanoid robotics company Figure has released Helix 02, what they claim in their most capable humanoid model yet. "A single neural system that controls the full body directly from pixels, enabling dexterous, long horizon autonomy across an entire room: • Autonomous, long‑horizon loco-manipulation: Helix 02 unloads and reloads a dishwasher across a full-sized kitchen - a four-minute, end-to-end autonomous task that integrates walking, manipulation, and balance with no resets and no human intervention. We believe this is the longest horizon, most complex task completed autonomously by a humanoid robot to date. • All sensors in. All actuators out: Helix 02 connects every onboard sensor - vision, touch, and proprioception - directly to every actuator through a single unified visuomotor neural network. • Human-like whole body control from human data: All results are enabled by System 0, a learned whole‑body controller trained on over 1,000 hours of human motion data and sim‑to‑real reinforcement learning. System 0 replaces 109,504 lines of hand‑engineered C++ with a single neural prior for stable, natural motion. • New classes of dexterity: With Figure 03’s embedded tactile sensing and palm cameras, Helix 02 performs manipulation that was previously out of reach: extracting individual pills, dispensing precise syringe volumes, and singulating small, irregular objects from clutter despite self‑occlusion. Helix 02 is trained on over 1,000 hours of human motion data and integrates vision, touch, and proprioception."

NEWS: Humanoid robotics company Figure has released Helix 02, what they claim in their most capable humanoid model yet. "A single neural system that controls the full body directly from pixels, enabling dexterous, long horizon autonomy across an entire room: • Autonomous, long‑horizon loco-manipulation: Helix 02 unloads and reloads a dishwasher across a full-sized kitchen - a four-minute, end-to-end autonomous task that integrates walking, manipulation, and balance with no resets and no human intervention. We believe this is the longest horizon, most complex task completed autonomously by a humanoid robot to date. • All sensors in. All actuators out: Helix 02 connects every onboard sensor - vision, touch, and proprioception - directly to every actuator through a single unified visuomotor neural network. • Human-like whole body control from human data: All results are enabled by System 0, a learned whole‑body controller trained on over 1,000 hours of human motion data and sim‑to‑real reinforcement learning. System 0 replaces 109,504 lines of hand‑engineered C++ with a single neural prior for stable, natural motion. • New classes of dexterity: With Figure 03’s embedded tactile sensing and palm cameras, Helix 02 performs manipulation that was previously out of reach: extracting individual pills, dispensing precise syringe volumes, and singulating small, irregular objects from clutter despite self‑occlusion. Helix 02 is trained on over 1,000 hours of human motion data and integrates vision, touch, and proprioception."

Sawyer Merritt

624,689 views • 5 months ago

Physics-based Motion Retargeting from Sparse Inputs paper page: Avatars are important to create interactive and immersive experiences in virtual worlds. One challenge in animating these characters to mimic a user's motion is that commercial AR/VR products consist only of a headset and controllers, providing very limited sensor data of the user's pose. Another challenge is that an avatar might have a different skeleton structure than a human and the mapping between them is unclear. In this work we address both of these challenges. We introduce a method to retarget motions in real-time from sparse human sensor data to characters of various morphologies. Our method uses reinforcement learning to train a policy to control characters in a physics simulator. We only require human motion capture data for training, without relying on artist-generated animations for each avatar. This allows us to use large motion capture datasets to train general policies that can track unseen users from real and sparse data in real-time. We demonstrate the feasibility of our approach on three characters with different skeleton structure: a dinosaur, a mouse-like creature and a human. We show that the avatar poses often match the user surprisingly well, despite having no sensor information of the lower body available. We discuss and ablate the important components in our framework, specifically the kinematic retargeting step, the imitation, contact and action reward as well as our asymmetric actor-critic observations. We further explore the robustness of our method in a variety of settings including unbalancing, dancing and sports motions.

AK

106,527 views • 3 years ago

This is how ALOHA's "teleoperation" system works - a fancy word for "remote control". Training robots will be more and more like playing games in the physical world. A human operates a "joystick++" to perform tasks and collect data, or intervene if there's any safety concern. There's actually a learning curve to master the controller, much like practicing gaming skills. Teleoperation can be done in many different ways. ALOHA is an impressive custom-built system with very low cost. Here're a few alternatives: (1) Motion Capture (MoCap): apply the MoCap systems used for Hollywood movies to capture the fine-grained motions of hand joints. There would be no "embodiment gap" if the robot hand has 5 fingers. For instance, a demonstrator can wear a CyberGlove ( and manipulate the objects. CyberGlove will capture the motion signals & haptic feedback in real-time, which can be re-targeted onto the humanoid. (2) Wearing gloves & markers can be clumsy. An alternative way to do MoCap is through computer vision. DexPilot from NVIDIA enables marker-less and glove-free data collection. The human operator simply uses their bare hands to perform the tasks. 4 Intel RealSense depth cameras and 2 NVIDIA Titan XP GPUs (yeah, 2019 work) translate the pixels to precise motion signals for robot learning. (3) VR Headset: turn the training room into a VR game and "role play" the robot. This has the advantage of scalable remote data collection - annotators from around the world can contribute without coming onsite. VR demonstration technique appeared in research projects like the iGibson home robot simulator, an initiative that I participated in at Stanford: Behind-the-scene video by Litian Liang

This is how ALOHA's "teleoperation" system works - a fancy word for "remote control". Training robots will be more and more like playing games in the physical world. A human operates a "joystick++" to perform tasks and collect data, or intervene if there's any safety concern. There's actually a learning curve to master the controller, much like practicing gaming skills. Teleoperation can be done in many different ways. ALOHA is an impressive custom-built system with very low cost. Here're a few alternatives: (1) Motion Capture (MoCap): apply the MoCap systems used for Hollywood movies to capture the fine-grained motions of hand joints. There would be no "embodiment gap" if the robot hand has 5 fingers. For instance, a demonstrator can wear a CyberGlove ( and manipulate the objects. CyberGlove will capture the motion signals & haptic feedback in real-time, which can be re-targeted onto the humanoid. (2) Wearing gloves & markers can be clumsy. An alternative way to do MoCap is through computer vision. DexPilot from NVIDIA enables marker-less and glove-free data collection. The human operator simply uses their bare hands to perform the tasks. 4 Intel RealSense depth cameras and 2 NVIDIA Titan XP GPUs (yeah, 2019 work) translate the pixels to precise motion signals for robot learning. (3) VR Headset: turn the training room into a VR game and "role play" the robot. This has the advantage of scalable remote data collection - annotators from around the world can contribute without coming onsite. VR demonstration technique appeared in research projects like the iGibson home robot simulator, an initiative that I participated in at Stanford: Behind-the-scene video by Litian Liang

Jim Fan

124,588 views • 2 years ago

I don’t know if we live in a Matrix, but I know for sure that robots will spend most of their lives in simulation. Let machines train machines. I’m excited to introduce DexMimicGen, a massive-scale synthetic data generator that enables a humanoid robot to learn complex skills from only a handful of human demonstrations. Yes, as few as 5! DexMimicGen addresses the biggest pain point in robotics: where do we get data? Unlike with LLMs, where vast amounts of texts are readily available, you cannot simply download motor control signals from the internet. So researchers teleoperate the robots to collect motion data via XR headsets. They have to repeat the same skill over and over and over again, because neural nets are data hungry. This is a very slow and uncomfortable process. At NVIDIA, we believe the majority of high-quality tokens for robot foundation models will come from simulation. What DexMimicGen does is to trade GPU compute time for human time. It takes one motion trajectory from human, and multiplies into 1000s of new trajectories. A robot brain trained on this augmented dataset will generalize far better in the real world. Think of DexMimicGen as a learning signal amplifier. It maps a small dataset to a large (de facto infinite) dataset, using physics simulation in the loop. In this way, we free humans from babysitting the bots all day. The future of robot data is generative. The future of the entire robot learning pipeline will also be generative. 🧵

I don’t know if we live in a Matrix, but I know for sure that robots will spend most of their lives in simulation. Let machines train machines. I’m excited to introduce DexMimicGen, a massive-scale synthetic data generator that enables a humanoid robot to learn complex skills from only a handful of human demonstrations. Yes, as few as 5! DexMimicGen addresses the biggest pain point in robotics: where do we get data? Unlike with LLMs, where vast amounts of texts are readily available, you cannot simply download motor control signals from the internet. So researchers teleoperate the robots to collect motion data via XR headsets. They have to repeat the same skill over and over and over again, because neural nets are data hungry. This is a very slow and uncomfortable process. At NVIDIA, we believe the majority of high-quality tokens for robot foundation models will come from simulation. What DexMimicGen does is to trade GPU compute time for human time. It takes one motion trajectory from human, and multiplies into 1000s of new trajectories. A robot brain trained on this augmented dataset will generalize far better in the real world. Think of DexMimicGen as a learning signal amplifier. It maps a small dataset to a large (de facto infinite) dataset, using physics simulation in the loop. In this way, we free humans from babysitting the bots all day. The future of robot data is generative. The future of the entire robot learning pipeline will also be generative. 🧵

Jim Fan

165,246 views • 1 year ago

Let's reverse engineer this demo. You need 3 things: (1) robust hardware and motor designs that treat simulation as first-class citizen; (2) a human motion capture ("mocap") dataset, such as those for film and gaming characters; (3) massively parallel RL training in GPU-accelerated simulation. Last October, our team trained a 1.5M parameter foundation model called HOVER for such agile motor control. It follows this recipe, roughly speaking (details in thread): (1) Simulation used to be an after-thought. Now, it has to be part of the hardware design process. If your robot doesn't simulate well, you can kiss RL goodbye. Hardware-simulation co-design is a very interesting emergent topic that only becomes meaningful with today's compute capability. (2) Human mocap dataset to produce natural-looking walking and running gaits. That's one huge advantage of using humanoid robot - you get to imitate from tons of human motions that were originally captured for movies or AAA games. At least 3 ways to use the data: - For initialization: pre-train the neural net to imitate human, and then finetune it into the robot form factor with physics turned on; - For reward function: penalize any deviations from the target pose; - For representation learning: treat the human poses as a "motion prior" to constrain the space of robot behaviors. (3) Shove the above into Isaac sim, add a lot of randomization, pump it through PPO, throw in a bunch of GPUs, and then watch Netflix till loss converges. If you have an urge to comment this is CGI, let me save you a few keystrokes — many academic labs now own the G1 robot in the flesh. See our team's HOVER work in the thread: 🧵

Let's reverse engineer this demo. You need 3 things: (1) robust hardware and motor designs that treat simulation as first-class citizen; (2) a human motion capture ("mocap") dataset, such as those for film and gaming characters; (3) massively parallel RL training in GPU-accelerated simulation. Last October, our team trained a 1.5M parameter foundation model called HOVER for such agile motor control. It follows this recipe, roughly speaking (details in thread): (1) Simulation used to be an after-thought. Now, it has to be part of the hardware design process. If your robot doesn't simulate well, you can kiss RL goodbye. Hardware-simulation co-design is a very interesting emergent topic that only becomes meaningful with today's compute capability. (2) Human mocap dataset to produce natural-looking walking and running gaits. That's one huge advantage of using humanoid robot - you get to imitate from tons of human motions that were originally captured for movies or AAA games. At least 3 ways to use the data: - For initialization: pre-train the neural net to imitate human, and then finetune it into the robot form factor with physics turned on; - For reward function: penalize any deviations from the target pose; - For representation learning: treat the human poses as a "motion prior" to constrain the space of robot behaviors. (3) Shove the above into Isaac sim, add a lot of randomization, pump it through PPO, throw in a bunch of GPUs, and then watch Netflix till loss converges. If you have an urge to comment this is CGI, let me save you a few keystrokes — many academic labs now own the G1 robot in the flesh. See our team's HOVER work in the thread: 🧵

Jim Fan

216,139 views • 1 year ago

Exciting updates on Project GR00T! We discover a systematic way to scale up robot data, tackling the most painful pain point in robotics. The idea is simple: human collects demonstration on a real robot, and we multiply that data 1000x or more in simulation. Let’s break it down: 1. We use Apple Vision Pro (yes!!) to give the human operator first person control of the humanoid. Vision Pro parses human hand pose and retargets the motion to the robot hand, all in real time. From the human’s point of view, they are immersed in another body like the Avatar. Teleoperation is slow and time-consuming, but we can afford to collect a small amount of data. 2. We use RoboCasa, a generative simulation framework, to multiply the demonstration data by varying the visual appearance and layout of the environment. In Jensen’s keynote video below, the humanoid is now placing the cup in hundreds of kitchens with a huge diversity of textures, furniture, and object placement. We only have 1 physical kitchen at the GEAR Lab in NVIDIA HQ, but we can conjure up infinite ones in simulation. 3. Finally, we apply MimicGen, a technique to multiply the above data even more by varying the motion of the robot. MimicGen generates vast number of new action trajectories based on the original human data, and filters out failed ones (e.g. those that drop the cup) to form a much larger dataset. To sum up, given 1 human trajectory with Vision Pro -> RoboCasa produces N (varying visuals) -> MimicGen further augments to NxM (varying motions). This is the way to trade compute for expensive human data by GPU-accelerated simulation. A while ago, I mentioned that teleoperation is fundamentally not scalable, because we are always limited by 24 hrs/robot/day in the world of atoms. Our new GR00T synthetic data pipeline breaks this barrier in the world of bits. Scaling has been so much fun for LLMs, and it's finally our turn to have fun in robotics! We are building tools to enable everyone in the ecosystem to scale up with us. Links in thread:

Jim Fan

364,380 views • 1 year ago

Alibaba presents MIMO Controllable Character Video Synthesis with Spatial Decomposed Modeling Character video synthesis aims to produce realistic videos of animatable characters within lifelike scenes. As a fundamental problem in the computer vision and graphics community, 3D works typically require multi-view captures for per-case training, which severely limits their applicability of modeling arbitrary characters in a short time. Recent 2D methods break this limitation via pre-trained diffusion models, but they struggle for pose generality and scene interaction. To this end, we propose MIMO, a novel framework which can not only synthesize character videos with controllable attributes (i.e., character, motion and scene) provided by simple user inputs, but also simultaneously achieve advanced scalability to arbitrary characters, generality to novel 3D motions, and applicability to interactive real-world scenes in a unified framework. The core idea is to encode the 2D video to compact spatial codes, considering the inherent 3D nature of video occurrence. Concretely, we lift the 2D frame pixels into 3D using monocular depth estimators, and decompose the video clip to three spatial components (i.e., main human, underlying scene, and floating occlusion) in hierarchical layers based on the 3D depth. These components are further encoded to canonical identity code, structured motion code and full scene code, which are utilized as control signals of synthesis process. The design of spatial decomposed modeling enables flexible user control, complex motion expression, as well as 3D-aware synthesis for scene interactions. Experimental results demonstrate effectiveness and robustness of the proposed method.

Alibaba presents MIMO Controllable Character Video Synthesis with Spatial Decomposed Modeling Character video synthesis aims to produce realistic videos of animatable characters within lifelike scenes. As a fundamental problem in the computer vision and graphics community, 3D works typically require multi-view captures for per-case training, which severely limits their applicability of modeling arbitrary characters in a short time. Recent 2D methods break this limitation via pre-trained diffusion models, but they struggle for pose generality and scene interaction. To this end, we propose MIMO, a novel framework which can not only synthesize character videos with controllable attributes (i.e., character, motion and scene) provided by simple user inputs, but also simultaneously achieve advanced scalability to arbitrary characters, generality to novel 3D motions, and applicability to interactive real-world scenes in a unified framework. The core idea is to encode the 2D video to compact spatial codes, considering the inherent 3D nature of video occurrence. Concretely, we lift the 2D frame pixels into 3D using monocular depth estimators, and decompose the video clip to three spatial components (i.e., main human, underlying scene, and floating occlusion) in hierarchical layers based on the 3D depth. These components are further encoded to canonical identity code, structured motion code and full scene code, which are utilized as control signals of synthesis process. The design of spatial decomposed modeling enables flexible user control, complex motion expression, as well as 3D-aware synthesis for scene interactions. Experimental results demonstrate effectiveness and robustness of the proposed method.

AK

148,998 views • 1 year ago

Domain Randomization (DR) is a key component of the data augmentation pipeline at Axis Robotics. By applying DR, we are able to scale verified, high-quality human trajectories by 10x to 100x. During training, we systematically introduce variances in environmental parameters. This prevents the model from relying on spurious visual correlations. The objective is to ensure the policy learns rather than overfitting. To demonstrate the necessity and effectiveness of this approach, we evaluated both DR and No-DR models on Task 74 (pour_water_into_mug). The empirical results show a definitive impact on real-world deployment reliability: integrating DR into the pipeline increased the success rate from 0% to 90% (Fig. 1). This divergence stems from how the respective policies process visual observations (Fig. 2). The baseline (No DR) model overfits to the static visual background. It essentially memorizes the poses from the training dataset but fails to generalize when subjected to the inevitable variances of real-world deployment. Consequently, it cannot execute the correct manipulation on the target object. Conversely, the DR-trained model learns to extract essential geometric features and physical constraints, filtering out superficial visual noise. This leads to significantly higher robustness in dynamic environments. The structural difference in execution is clearly reflected in the end-effector trajectory data: These real-world deployment recordings further illustrate this difference (Videos 1 and 2). Scaling Physical AI requires turning raw trajectory data into robust policies, and a rigorously engineered DR infrastructure is an essential bridge to close the Sim2Real gap.

Domain Randomization (DR) is a key component of the data augmentation pipeline at Axis Robotics. By applying DR, we are able to scale verified, high-quality human trajectories by 10x to 100x. During training, we systematically introduce variances in environmental parameters. This prevents the model from relying on spurious visual correlations. The objective is to ensure the policy learns rather than overfitting. To demonstrate the necessity and effectiveness of this approach, we evaluated both DR and No-DR models on Task 74 (pour_water_into_mug). The empirical results show a definitive impact on real-world deployment reliability: integrating DR into the pipeline increased the success rate from 0% to 90% (Fig. 1). This divergence stems from how the respective policies process visual observations (Fig. 2). The baseline (No DR) model overfits to the static visual background. It essentially memorizes the poses from the training dataset but fails to generalize when subjected to the inevitable variances of real-world deployment. Consequently, it cannot execute the correct manipulation on the target object. Conversely, the DR-trained model learns to extract essential geometric features and physical constraints, filtering out superficial visual noise. This leads to significantly higher robustness in dynamic environments. The structural difference in execution is clearly reflected in the end-effector trajectory data: These real-world deployment recordings further illustrate this difference (Videos 1 and 2). Scaling Physical AI requires turning raw trajectory data into robust policies, and a rigorously engineered DR infrastructure is an essential bridge to close the Sim2Real gap.

Axis Robotics

27,125 views • 3 months ago

AgiBot has formally unveiled its G2 humanoid robot, a system designed to transition into various industries and liberate humans from repetitive labor. G2 features high-performance joints, precision torque sensors, and an advanced spatial perception system, supporting quick deployment and multi-modal voice interaction. ► Factory Floor Performance: The G2 is engineered to industrial standards. In a safety belt lock production line, robots collaborate with human workers, performing tasks like pressing lock cores. The G2 collects production data to continuously train and iterate models (local server deployment ensures data privacy), steadily improving its operational ability. ► Mobility & Safety: The G2 navigates narrow factory aisles using dual LiDAR and full-panorama vision for environment sensing and collision detection. Its chassis is designed to overcome common obstacles (speed bumps, elevator gaps). It supports 24/7 continuous operation via autonomous return-to-charge and battery swapping. ► Humanoid Design Advantage: The G2's design includes a three-degree-of-freedom flexible waist, allowing it to mimic natural human movements like bending and side-leaning. This dramatically expands its operational workspace and enables seamless integration into existing human-centric production lines without costly modifications. ► Advanced Dexterity & Learning (Lab): The new G02 arm is the world's first cross-moment arm, featuring high-precision joint torque sensors that allow it to precisely sense external forces and adjust stiffness, mimicking human hand compliance. Using Real-Machine Reinforcement Learning (RL), the G2 can learn complex, delicate tasks like memory stick insertion in about one hour with minimal human intervention. ► Logistics & Grasping: In logistics sorting, the G2 uses a 19-degree-of-freedom mechanical dexterous hand (20N maximum fingertip force; 35kg capacity for hard objects) equipped with 3D tactile sensors to ensure it grasps securely without damaging items. Its full-body articulation (waist and legs) aids grasping and posture adjustment. ► Model & Data: G2's intelligence is powered by the Go-One Large Embodied Model (VLA architecture: Vision-Language-Latent Action) and the GE-One World Model (vision-centric predictive modeling), trained using the AgiBot Word true-machine dataset (over 500k downloads). ► Service & Interaction: The G2 is deployed as a guide/receptionist in settings like art museums. It uses its high-DOF head, arms, and waist to point to exhibits, maintains eye contact while navigating difficult spaces (chassis walks forward, body faces backward), handles specialized and random queries, and uses proactive safety features (stops movement, issues warnings) when people get too close.

AgiBot has formally unveiled its G2 humanoid robot, a system designed to transition into various industries and liberate humans from repetitive labor. G2 features high-performance joints, precision torque sensors, and an advanced spatial perception system, supporting quick deployment and multi-modal voice interaction. ► Factory Floor Performance: The G2 is engineered to industrial standards. In a safety belt lock production line, robots collaborate with human workers, performing tasks like pressing lock cores. The G2 collects production data to continuously train and iterate models (local server deployment ensures data privacy), steadily improving its operational ability. ► Mobility & Safety: The G2 navigates narrow factory aisles using dual LiDAR and full-panorama vision for environment sensing and collision detection. Its chassis is designed to overcome common obstacles (speed bumps, elevator gaps). It supports 24/7 continuous operation via autonomous return-to-charge and battery swapping. ► Humanoid Design Advantage: The G2's design includes a three-degree-of-freedom flexible waist, allowing it to mimic natural human movements like bending and side-leaning. This dramatically expands its operational workspace and enables seamless integration into existing human-centric production lines without costly modifications. ► Advanced Dexterity & Learning (Lab): The new G02 arm is the world's first cross-moment arm, featuring high-precision joint torque sensors that allow it to precisely sense external forces and adjust stiffness, mimicking human hand compliance. Using Real-Machine Reinforcement Learning (RL), the G2 can learn complex, delicate tasks like memory stick insertion in about one hour with minimal human intervention. ► Logistics & Grasping: In logistics sorting, the G2 uses a 19-degree-of-freedom mechanical dexterous hand (20N maximum fingertip force; 35kg capacity for hard objects) equipped with 3D tactile sensors to ensure it grasps securely without damaging items. Its full-body articulation (waist and legs) aids grasping and posture adjustment. ► Model & Data: G2's intelligence is powered by the Go-One Large Embodied Model (VLA architecture: Vision-Language-Latent Action) and the GE-One World Model (vision-centric predictive modeling), trained using the AgiBot Word true-machine dataset (over 500k downloads). ► Service & Interaction: The G2 is deployed as a guide/receptionist in settings like art museums. It uses its high-DOF head, arms, and waist to point to exhibits, maintains eye contact while navigating difficult spaces (chassis walks forward, body faces backward), handles specialized and random queries, and uses proactive safety features (stops movement, issues warnings) when people get too close.

RoboHub🤖

46,733 views • 9 months ago

Video: World’s first humanoid robot labor that swaps its own batteries to work endlessly | Jijo Malayil, Interesting Engineering Walker S2 uses dual-battery balancing and standardized modules to boost efficiency and ensure uninterrupted, optimized performance. In a leap for robotics, China’s UBTech has unveiled the Walker S2, the world’s first humanoid robot capable of fully autonomous battery swapping. Designed for non-stop industrial operations, the Walker S2 can replace its own power pack in just three minutes—no human intervention required. Equipped with advanced anthropomorphic bipedal locomotion and a hot-swappable battery system, Walker S2 is built to operate 24/7 across dynamic industrial environments. According to UBTech, the next-generation humanoid robot marks a major milestone in automation, bringing continuous, hands-free performance to the factory floor. In May 2025, UBTech Robotics and Huawei Technologies inked a significant partnership to accelerate the adoption of humanoid robots across China’s factories and households. Uninterrupted robot operations A video posted by the robotics firm opens with the sleek UBTech Walker S2 humanoid robot working in an industrial setting. The highlight, however, is its autonomous battery swap. Walker S2 approaches the charging station, carefully detaches its depleted power pack, and seamlessly installs a fresh one—all within about three minutes—without any human assistance, according to CGTN. The camera captures close-ups of the robot’s articulated limbs and the intelligent battery-handling mechanism, conveying precision and reliability. As the swap completes, Walker S2 resumes its duties, reinforcing the promise of uninterrupted, 24/7 operations in dynamic factory environments. UBTech’s Walker S2 humanoid robot is equipped with advanced dual-battery power balancing technology and uses standardized battery modules to optimize performance, reports CNEVPOST. This dual-battery system allows the robot to automatically switch to a backup battery in case of a main battery failure, ensuring that critical tasks are carried out without interruption. In addition to battery swapping, the robot can intelligently choose between charging and swapping based on task urgency, allowing it to manage energy dynamically and adapt to real-time operational demands. UBTech highlights these features as a step forward in deploying humanoid robots for industrial and domestic applications, combining flexibility, reliability, and autonomy in one intelligent platform. Factory intelligence upgrade Earlier in the year, UBTech unveiled a major advancement in humanoid robot collaboration, claiming the world’s first deployment of multiple humanoids working together across varied industrial tasks. Demonstrated at Zeekr’s 5G-enabled smart factory, the breakthrough centers on UBTech’s “BrainNet” framework, which orchestrates cooperative behavior through a cloud-device intelligence system. BrainNet integrates a “super brain” for high-level decision-making with an “intelligent sub-brain” for distributed multi-robot control. The super brain, powered by a proprietary large-scale multimodal reasoning model, handles complex production-line scheduling and decision-making. Meanwhile, the sub-brain coordinates real-time tasks using cross-field perception and Transformer-based control for dynamic adaptability. Together, they enable the Walker S1 humanoid robots to move beyond isolated operations and perform coordinated tasks with high precision and speed. The system is built on DeepSeek-R1 reasoning technology and trained on real-world data from automotive factory settings. Leveraging Retrieval-Augmented Generation (RAG), the model adapts to specific job functions and improves scalability across workstations. At Zeekr’s facility, dozens of Walker S1s now collaborate on tasks like assembly, inspection, and part handling. Using semantic VSLAM and shared mapping, they coordinate seamlessly via vision-based navigation and agile manipulation. UBTech says this marks a transition to “Practical Training 2.0,” where humanoid robots operate as a swarm, maximizing efficiency and setting the stage for next-generation intelligent manufacturing.

Video: World’s first humanoid robot labor that swaps its own batteries to work endlessly | Jijo Malayil, Interesting Engineering Walker S2 uses dual-battery balancing and standardized modules to boost efficiency and ensure uninterrupted, optimized performance. In a leap for robotics, China’s UBTech has unveiled the Walker S2, the world’s first humanoid robot capable of fully autonomous battery swapping. Designed for non-stop industrial operations, the Walker S2 can replace its own power pack in just three minutes—no human intervention required. Equipped with advanced anthropomorphic bipedal locomotion and a hot-swappable battery system, Walker S2 is built to operate 24/7 across dynamic industrial environments. According to UBTech, the next-generation humanoid robot marks a major milestone in automation, bringing continuous, hands-free performance to the factory floor. In May 2025, UBTech Robotics and Huawei Technologies inked a significant partnership to accelerate the adoption of humanoid robots across China’s factories and households. Uninterrupted robot operations A video posted by the robotics firm opens with the sleek UBTech Walker S2 humanoid robot working in an industrial setting. The highlight, however, is its autonomous battery swap. Walker S2 approaches the charging station, carefully detaches its depleted power pack, and seamlessly installs a fresh one—all within about three minutes—without any human assistance, according to CGTN. The camera captures close-ups of the robot’s articulated limbs and the intelligent battery-handling mechanism, conveying precision and reliability. As the swap completes, Walker S2 resumes its duties, reinforcing the promise of uninterrupted, 24/7 operations in dynamic factory environments. UBTech’s Walker S2 humanoid robot is equipped with advanced dual-battery power balancing technology and uses standardized battery modules to optimize performance, reports CNEVPOST. This dual-battery system allows the robot to automatically switch to a backup battery in case of a main battery failure, ensuring that critical tasks are carried out without interruption. In addition to battery swapping, the robot can intelligently choose between charging and swapping based on task urgency, allowing it to manage energy dynamically and adapt to real-time operational demands. UBTech highlights these features as a step forward in deploying humanoid robots for industrial and domestic applications, combining flexibility, reliability, and autonomy in one intelligent platform. Factory intelligence upgrade Earlier in the year, UBTech unveiled a major advancement in humanoid robot collaboration, claiming the world’s first deployment of multiple humanoids working together across varied industrial tasks. Demonstrated at Zeekr’s 5G-enabled smart factory, the breakthrough centers on UBTech’s “BrainNet” framework, which orchestrates cooperative behavior through a cloud-device intelligence system. BrainNet integrates a “super brain” for high-level decision-making with an “intelligent sub-brain” for distributed multi-robot control. The super brain, powered by a proprietary large-scale multimodal reasoning model, handles complex production-line scheduling and decision-making. Meanwhile, the sub-brain coordinates real-time tasks using cross-field perception and Transformer-based control for dynamic adaptability. Together, they enable the Walker S1 humanoid robots to move beyond isolated operations and perform coordinated tasks with high precision and speed. The system is built on DeepSeek-R1 reasoning technology and trained on real-world data from automotive factory settings. Leveraging Retrieval-Augmented Generation (RAG), the model adapts to specific job functions and improves scalability across workstations. At Zeekr’s facility, dozens of Walker S1s now collaborate on tasks like assembly, inspection, and part handling. Using semantic VSLAM and shared mapping, they coordinate seamlessly via vision-based navigation and agile manipulation. UBTech says this marks a transition to “Practical Training 2.0,” where humanoid robots operate as a swarm, maximizing efficiency and setting the stage for next-generation intelligent manufacturing.

Owen Gregorian

35,637 views • 1 year ago

China unveils humanoid robot worker with brain that runs 275 trillion ops/sec | Jijo Malayil, Interesting Engineering In tests, SUYUAN used vision and joint control to sort and move crates of various sizes, greatly improving warehouse productivity. Chinese manufacturing firm Shanghai Electric has unveiled its first self-developed industrial humanoid robot, “SUYUAN,” marking a major milestone in its robotics journey. Debuting at the World Artificial Intelligence Conference (WAIC 2025) on July 26 in Shanghai, SUYUAN boasts 38 degrees of freedom and 275 TOPS of on-device computing power, enabling precise operations and fluid movements. According to the firm, designed for diverse industrial use, the robot showcases Shanghai Electric’s end-to-end capabilities—from core tech to integrated solutions—and reinforces its commitment to next-gen industrial automation through a full industry chain strategy. At WAIC 2025, Shanghai Electric also unveiled a new joint venture with Johnson Electric for next-gen humanoid robotics and showcased its “LINGKE” dual-arm robot. Recently, Hangzhou-based Unitree Robotics launched the R1 humanoid with 26 joints for $5,900, showcasing athletic feats like cartwheels, running, and quick recovery. Smart factory assistant Shanghai Electric claims SUYUAN, equipped with 38 degrees of freedom (DoF) and a powerful 275 TOPS on-device computing processor, delivers fluid, human-like movements and high-precision operations across various industrial scenarios. Its advanced articulation and real-time processing capabilities make it highly adaptable, enabling smooth execution of complex tasks in dynamic work environments. SUYUAN, who weighs 110 pounds (50 kilograms) and is 5 feet 6 inches (167 cm) tall, was designed to have human-like proportions. Its 38-DoF articulation offers dexterity, allowing for both wide-range motion and sensitive manipulation. With a single arm, the robot can lift objects up to 4.4 pounds (2 kilograms) in weight and carry a total payload of up to 22 pounds (10 kilograms). With a walking pace of 3.1 miles per hour (5 km/h), SUYUAN is ideal for environments including assembly lines, warehousing, and logistics, according to a statement. To navigate complex industrial settings, SUYUAN combines LiDAR and binocular vision for self-guided mobility. Its 275-TOPS AI processor enables rapid data analysis and integration with large language models, allowing it to understand tasks in natural language and handle objects adaptively, reports Fox 44 News. In pilot demonstrations, the robot successfully identified, picked, and relocated crates of varying sizes using advanced computer vision and coordinated joint control—delivering measurable gains in warehouse efficiency. The company claims that SUYUAN’s launch represents a major turning point in Shanghai Electric’s foray into humanoid robotics and strengthens its vertically integrated approach to industrial automation solutions. Intelligent task handling Shanghai Electric also demonstrated its most recent developments in intelligent manufacturing at WAIC 2025, introducing a new joint venture with Johnson Electric centered on next-generation humanoid robotics and showcasing the “LINGKE” dual-arm robot. With its high-precision operations, adaptive teamwork, and closed-loop data capabilities, the LINGKE robot demonstrated live talents in handling complicated production jobs. LINGKE is made to do more than just replace human labor; it uses compliant force control and bimanual coordination to relieve workers of high-intensity, repetitive jobs. According to the company, the robot enhances operational efficiency by up to five times. Its core strength lies in a Data-Model-Deployment closed-loop system that starts with operational data, followed by data cleansing, model training, live deployment, and feedback-driven optimization—enabling autonomous learning and workflow improvement. Also at the event, Shanghai Electric and Johnson Electric introduced advanced hardware modules for humanoid robots, including rotary joints, linear joints, and dexterous finger joints. These components are designed to support smooth, precise, and quiet motion performance across robotics systems, reports Stock Titan. The joint venture announced two strategic agreements: a first-unit supply deal with the National and Local Co-Built Humanoid Robotics Innovation Center (Qinglong Project) and a cooperation memorandum with Fourier Robotics. Read more:

China unveils humanoid robot worker with brain that runs 275 trillion ops/sec | Jijo Malayil, Interesting Engineering In tests, SUYUAN used vision and joint control to sort and move crates of various sizes, greatly improving warehouse productivity. Chinese manufacturing firm Shanghai Electric has unveiled its first self-developed industrial humanoid robot, “SUYUAN,” marking a major milestone in its robotics journey. Debuting at the World Artificial Intelligence Conference (WAIC 2025) on July 26 in Shanghai, SUYUAN boasts 38 degrees of freedom and 275 TOPS of on-device computing power, enabling precise operations and fluid movements. According to the firm, designed for diverse industrial use, the robot showcases Shanghai Electric’s end-to-end capabilities—from core tech to integrated solutions—and reinforces its commitment to next-gen industrial automation through a full industry chain strategy. At WAIC 2025, Shanghai Electric also unveiled a new joint venture with Johnson Electric for next-gen humanoid robotics and showcased its “LINGKE” dual-arm robot. Recently, Hangzhou-based Unitree Robotics launched the R1 humanoid with 26 joints for $5,900, showcasing athletic feats like cartwheels, running, and quick recovery. Smart factory assistant Shanghai Electric claims SUYUAN, equipped with 38 degrees of freedom (DoF) and a powerful 275 TOPS on-device computing processor, delivers fluid, human-like movements and high-precision operations across various industrial scenarios. Its advanced articulation and real-time processing capabilities make it highly adaptable, enabling smooth execution of complex tasks in dynamic work environments. SUYUAN, who weighs 110 pounds (50 kilograms) and is 5 feet 6 inches (167 cm) tall, was designed to have human-like proportions. Its 38-DoF articulation offers dexterity, allowing for both wide-range motion and sensitive manipulation. With a single arm, the robot can lift objects up to 4.4 pounds (2 kilograms) in weight and carry a total payload of up to 22 pounds (10 kilograms). With a walking pace of 3.1 miles per hour (5 km/h), SUYUAN is ideal for environments including assembly lines, warehousing, and logistics, according to a statement. To navigate complex industrial settings, SUYUAN combines LiDAR and binocular vision for self-guided mobility. Its 275-TOPS AI processor enables rapid data analysis and integration with large language models, allowing it to understand tasks in natural language and handle objects adaptively, reports Fox 44 News. In pilot demonstrations, the robot successfully identified, picked, and relocated crates of varying sizes using advanced computer vision and coordinated joint control—delivering measurable gains in warehouse efficiency. The company claims that SUYUAN’s launch represents a major turning point in Shanghai Electric’s foray into humanoid robotics and strengthens its vertically integrated approach to industrial automation solutions. Intelligent task handling Shanghai Electric also demonstrated its most recent developments in intelligent manufacturing at WAIC 2025, introducing a new joint venture with Johnson Electric centered on next-generation humanoid robotics and showcasing the “LINGKE” dual-arm robot. With its high-precision operations, adaptive teamwork, and closed-loop data capabilities, the LINGKE robot demonstrated live talents in handling complicated production jobs. LINGKE is made to do more than just replace human labor; it uses compliant force control and bimanual coordination to relieve workers of high-intensity, repetitive jobs. According to the company, the robot enhances operational efficiency by up to five times. Its core strength lies in a Data-Model-Deployment closed-loop system that starts with operational data, followed by data cleansing, model training, live deployment, and feedback-driven optimization—enabling autonomous learning and workflow improvement. Also at the event, Shanghai Electric and Johnson Electric introduced advanced hardware modules for humanoid robots, including rotary joints, linear joints, and dexterous finger joints. These components are designed to support smooth, precise, and quiet motion performance across robotics systems, reports Stock Titan. The joint venture announced two strategic agreements: a first-unit supply deal with the National and Local Co-Built Humanoid Robotics Innovation Center (Qinglong Project) and a cooperation memorandum with Fourier Robotics. Read more:

Owen Gregorian

51,638 views • 11 months ago

China’s pretty humanoid robot stuns by opening a car door in a ‘world’s first’ | Jijo Malayil, Interesting Engineering Mornine used onboard sensors and full-body control to locate the handle, adjust posture, and open a car door—no human input needed. AiMOGA Robotics has claimed to have reached a significant milestone in embodied AI with its humanoid robot, Mornine, autonomously opening a car door inside a functioning Chery dealership in China. Relying solely on onboard sensors, full-body motion control, and end-to-end reinforcement learning, Mornine performed the task without any human input. Unlike scripted or teleoperated robots, Mornie identified the door handle, adjusted its posture, and used coordinated force across its limbs and torso to complete the action—demonstrating advanced autonomy in a real-world setting. “The deployment marks one of the first instances of a service robot executing such a high-friction, physical interaction in a live commercial setting,” said the firm in a statement. In April, at the Shanghai Auto Show, automotive brands Omoda and Jaecoo, subsidiaries of Chery Automobile, introduced Mornine, designed for use in car dealerships. From sim to service Opening a car door may seem like a simple task, but AiMOGA Robotics views it as a pivotal moment in robotics—signaling a shift from simulation to real-world service, and from basic command execution to autonomous capability. Using only onboard sensors and full-body motion control, Mornine identified the door handle, adjusted her posture, and applied coordinated force across her limbs to open the door—entirely without human intervention. Mornine’s advanced sensor suite includes 3D LiDAR, depth and wide-angle cameras, and a visual-language model (VLM), enabling real-time perception of door position and opening status. Uniquely, Mornine wasn’t explicitly programmed to recognize door handles. Instead, she learned through reinforcement learning, undergoing millions of simulated cycles to focus on the right region and perform the task independently. “We never explicitly told the robot what a door handle is. It learned to focus on that region by itself,” said the engineering team at AiMOGA Robotics in a statement. The learned model was transferred to the real world using Sim2Real methods. Mornine continuously gathers live sensor data during operation, which feeds into a cloud-based training loop, allowing her to improve through continuous learning in real-world settings, reports Robotics Tomorrow. Now active in multiple Chery 4S dealerships in China, Mornine not only opens car doors but also assists with customer greetings, vehicle introductions, and item delivery—marking a step forward in humanoid robotics for commercial retail environments. AI meets retail Originally introduced as the AiMOGA Robot, Mornine was developed to support dealership sales by performing tasks such as explaining vehicle specifications, leading showroom tours, serving refreshments, and engaging with customers in multiple languages. First conceived by Chery as a virtual character to appeal to Generation Z using metaverse and virtual human technologies, Mornine gradually evolved into a real-world interactive humanoid. After multiple iterations of character and model design, Mornine debuted as a digital persona in animations, livestreams, and promotional content, gaining brand recognition. Chery later expanded the concept beyond the virtual space, resulting in the creation of the AiMOGA humanoid robot. Leveraging Chery’s expertise in autonomous driving, environmental sensing, and control systems, AiMOGA features full-stack capabilities in perception, cognition, decision-making, and execution. It uses multimodal sensing—combining speech, vision, and environmental data—to interpret user gestures, commands, and showroom dynamics. A bionic motion system and automotive-grade hardware enable dexterous movement and upright mobility, while multi-robot collaboration allows for coordinated tasks like guided tours. At the decision-making layer, Deepseek’s large language models enable natural language understanding and personalized interaction. In April 2025, Mornine officially began commercial service as an “Intelligent Sales Consultant” at the OMODA C5 JOYSTAR 4S dealership in Kuala Lumpur, Malaysia—marking her full transition from a virtual concept to a real-world humanoid sales assistant.

Owen Gregorian

67,975 views • 11 months ago

Dr Fei-Fei-Li explains with a simple example how everyday household chores are so extremely difficult for Robots. "If you tell a robot to open the top drawer and watch out for the vase, this is actually a really hard task for robots." because the robot must ground language into the real world. Words like "top", "drawer", and "vase" are abstract. The system has to map them to 3D locations, objects, and relations in a noisy scene. This requires robust perception, object recognition, and spatial reasoning under uncertainty. The robot also lacks human commonsense. "Watch out" implies predicting consequences, estimating clearances, and understanding that vases are fragile. Encoding such priors, like how heavy a drawer is or how a vase might tip, is very complex and difficult without rich world knowledge. Learning the behavior from rewards is tough. The success signal is very sparse here, so naive exploration almost never stumbles on a full success sequence. This makes policy learning sample inefficient and brittle, especially when the environment changes between training and deployment. A sparse reward situation is when the agent only gets a success signal at the very end, and gets little or no feedback along the way. If a robot must open a drawer without hitting a vase, it might get reward only if the drawer ends up open and the vase is intact. Every partial try before that looks the same to the learner, reward equals 0. --- From "DSAI by Dr. Osbert Tay" YT channel

Dr Fei-Fei-Li explains with a simple example how everyday household chores are so extremely difficult for Robots. "If you tell a robot to open the top drawer and watch out for the vase, this is actually a really hard task for robots." because the robot must ground language into the real world. Words like "top", "drawer", and "vase" are abstract. The system has to map them to 3D locations, objects, and relations in a noisy scene. This requires robust perception, object recognition, and spatial reasoning under uncertainty. The robot also lacks human commonsense. "Watch out" implies predicting consequences, estimating clearances, and understanding that vases are fragile. Encoding such priors, like how heavy a drawer is or how a vase might tip, is very complex and difficult without rich world knowledge. Learning the behavior from rewards is tough. The success signal is very sparse here, so naive exploration almost never stumbles on a full success sequence. This makes policy learning sample inefficient and brittle, especially when the environment changes between training and deployment. A sparse reward situation is when the agent only gets a success signal at the very end, and gets little or no feedback along the way. If a robot must open a drawer without hitting a vase, it might get reward only if the drawer ends up open and the vase is intact. Every partial try before that looks the same to the learner, reward equals 0. --- From "DSAI by Dr. Osbert Tay" YT channel

Rohan Paul

342,553 views • 8 months ago

This is WILD! MIT just solved one of the hardest unsolved problems in robotics (Save this). For decades, the fundamental problem with soft robots and wearable exoskeletons has not been compute or AI, it has been actuation. The moment you try to give a soft robot meaningful strength, you run into the same wall every engineer has hit since the field began, fluid-driven systems require external pumps, hydraulic reservoirs, and heavy infrastructure that makes the entire thing impractical to wear or embed into fabric. MIT's new Electrofluidic Fiber Muscles solve that problem by eliminating external infrastructure entirely. The key insight is electrohydrodynamic pumping using electric fields to generate pressure directly from electricity, with no moving parts, no motors, and no external fluid reservoir. The fibers are less than 2 millimeters thick, can be woven into fabric like ordinary textile, and operate in complete silence because nothing physically moves inside them, it is just ions propelling fluid through a closed circuit. The performance numbers published in Science Robotics are not conceptual, they are empirical results from actual hardware. These fibers achieve a power density of 50 watts per kilogram, matching skeletal muscle, with a contraction strain of 20% and a response time of 0.3 seconds. A single bundled configuration lifted 4 kilograms, 200 times its own weight while a separate configuration drove a robotic arm through a 40-degree bend compliant enough to safely complete a human handshake. Another configuration launched objects in under 100 milliseconds, which is faster than a human flinch reflex. The design mirrors biological muscle architecture in a way that prior artificial muscle approaches never achieved. The fibers are organized into antagonistic pairs, one contracts while the other extends, exactly like biceps and triceps and because the system runs in a closed loop, the relaxing fiber serves as the fluid reservoir for the contracting one, which is what allows the whole system to operate untethered with no external tank. The applications are not hypothetical but rather are the exact use cases the industry has been waiting years for the hardware to catch up to. Exoskeletons for physical labor, prosthetic limbs that move with the natural compliance of biological tissue, assistive garments for patients with motor disorders, and soft robots capable of safe physical contact with humans are all immediately unlocked by a muscle technology that is silent, lightweight, and weavable into clothing. The deeper significance is what this technology does when it meets the AI robotics wave that is already underway. Every major humanoid robot program, Figure, 1X, Boston Dynamics, Tesla Optimus is currently bottlenecked by the same hardware limitations these fibers address, actuators that are too rigid, too loud, too heavy, or too dependent on infrastructure to operate naturally alongside humans. Electrofluidic fiber muscles do not just solve a materials science problem but rather they remove one of the last physical barriers between robots that live in labs and robots that live in the world.

This is WILD! MIT just solved one of the hardest unsolved problems in robotics (Save this). For decades, the fundamental problem with soft robots and wearable exoskeletons has not been compute or AI, it has been actuation. The moment you try to give a soft robot meaningful strength, you run into the same wall every engineer has hit since the field began, fluid-driven systems require external pumps, hydraulic reservoirs, and heavy infrastructure that makes the entire thing impractical to wear or embed into fabric. MIT's new Electrofluidic Fiber Muscles solve that problem by eliminating external infrastructure entirely. The key insight is electrohydrodynamic pumping using electric fields to generate pressure directly from electricity, with no moving parts, no motors, and no external fluid reservoir. The fibers are less than 2 millimeters thick, can be woven into fabric like ordinary textile, and operate in complete silence because nothing physically moves inside them, it is just ions propelling fluid through a closed circuit. The performance numbers published in Science Robotics are not conceptual, they are empirical results from actual hardware. These fibers achieve a power density of 50 watts per kilogram, matching skeletal muscle, with a contraction strain of 20% and a response time of 0.3 seconds. A single bundled configuration lifted 4 kilograms, 200 times its own weight while a separate configuration drove a robotic arm through a 40-degree bend compliant enough to safely complete a human handshake. Another configuration launched objects in under 100 milliseconds, which is faster than a human flinch reflex. The design mirrors biological muscle architecture in a way that prior artificial muscle approaches never achieved. The fibers are organized into antagonistic pairs, one contracts while the other extends, exactly like biceps and triceps and because the system runs in a closed loop, the relaxing fiber serves as the fluid reservoir for the contracting one, which is what allows the whole system to operate untethered with no external tank. The applications are not hypothetical but rather are the exact use cases the industry has been waiting years for the hardware to catch up to. Exoskeletons for physical labor, prosthetic limbs that move with the natural compliance of biological tissue, assistive garments for patients with motor disorders, and soft robots capable of safe physical contact with humans are all immediately unlocked by a muscle technology that is silent, lightweight, and weavable into clothing. The deeper significance is what this technology does when it meets the AI robotics wave that is already underway. Every major humanoid robot program, Figure, 1X, Boston Dynamics, Tesla Optimus is currently bottlenecked by the same hardware limitations these fibers address, actuators that are too rigid, too loud, too heavy, or too dependent on infrastructure to operate naturally alongside humans. Electrofluidic fiber muscles do not just solve a materials science problem but rather they remove one of the last physical barriers between robots that live in labs and robots that live in the world.

Milk Road AI

1,204,107 views • 2 months ago

The Genie 3 release is a perfect moment to have a discussion about the future of 3D But first it would be nice to make the terminology more clear, specifically: What is a “spatial representation” Implicit vs Explicit Generalization vs Specialization Reconstruction vs Generation Production vs Execution Let’s start: For me, a spatial representation is just a way to describe a thing in the physical world The core property that makes it useful is consistency You can enforce consistency explicitly via rendering equations, geometric constraints, and physics Or implicitly, purely through training data Then, your representation parameters can be explicit, like points, gaussians, triangles, voxels, etc. Or implicit, weights or latent vectors Parameters alone are not the representation. It’s a combination of the parameters, the process that produces them, and the way you materialize them through a function (physics-based rendering, simulation, neural network, etc.) Generalization means you take data from multiple scene observations, and then produce a map from desired input to representation parameters Specialization means you take single-scene observations and directly fit a function parameters to describe thar scene Many representations can serve both of the approaches, as long as you keep them differentiable Both of the above can be used for reconstruction, where the main goal is to explain observations through a lens of physics (hard constraint) On the other hand, generation needs generalization, and its task is to produce statistically plausible results that could be conditioned on observations (soft constraint) Both tasks are not solved yet and they can complement each other in various ways Yet another important aspect is the difference between production and execution Production = process of going from inputs to parameters Execution = process of going from parameters to result It’s important to separate these, because most usecases require fast execution to be viable which is severely constrained by the hardware So, are *world models* like Genie an important step forward? Yes Do they make other representations obsolete? Maybe some of them - but there are tons of economically valuable tasks that won’t be solved by it, at least in any observable future

The Genie 3 release is a perfect moment to have a discussion about the future of 3D But first it would be nice to make the terminology more clear, specifically: What is a “spatial representation” Implicit vs Explicit Generalization vs Specialization Reconstruction vs Generation Production vs Execution Let’s start: For me, a spatial representation is just a way to describe a thing in the physical world The core property that makes it useful is consistency You can enforce consistency explicitly via rendering equations, geometric constraints, and physics Or implicitly, purely through training data Then, your representation parameters can be explicit, like points, gaussians, triangles, voxels, etc. Or implicit, weights or latent vectors Parameters alone are not the representation. It’s a combination of the parameters, the process that produces them, and the way you materialize them through a function (physics-based rendering, simulation, neural network, etc.) Generalization means you take data from multiple scene observations, and then produce a map from desired input to representation parameters Specialization means you take single-scene observations and directly fit a function parameters to describe thar scene Many representations can serve both of the approaches, as long as you keep them differentiable Both of the above can be used for reconstruction, where the main goal is to explain observations through a lens of physics (hard constraint) On the other hand, generation needs generalization, and its task is to produce statistically plausible results that could be conditioned on observations (soft constraint) Both tasks are not solved yet and they can complement each other in various ways Yet another important aspect is the difference between production and execution Production = process of going from inputs to parameters Execution = process of going from parameters to result It’s important to separate these, because most usecases require fast execution to be viable which is severely constrained by the hardware So, are world models like Genie an important step forward? Yes Do they make other representations obsolete? Maybe some of them - but there are tons of economically valuable tasks that won’t be solved by it, at least in any observable future

Lucky Iyinbor

13,959 views • 5 months ago