Video yükleniyor...

Video Yüklenemedi

Bu video yüklenirken bir sorun oluştu. Bu geçici bir ağ sorunundan kaynaklanıyor olabilir veya video kullanılamıyor olabilir.

Ana Sayfaya Dön

1X announces their latest reinforcement learning (RL) controller, which unlocks NEO's full-body mobility for home environments, enabling Redwood AI (1X's in-house AI model) to interact with the physical world more naturally and broadly. The unified controller supports walking in any direction, sitting, standing, kneeling, lying down, getting up, and... climbing stairs using stereo RGB vision - critical for navigating real homes. The controller provides an “action interface” through which teleoperation or Redwood AI can interact in a safe, contact-rich manner with the physical world. Traditional walking controllers often rely on hand-crafted "shaping rewards" to produce human-like gaits, which are time-consuming and don’t scale well across tasks or movement directions. To address this, the team used motion capture data to guide the learning process. The RL system is trained to track kinematic reference trajectories derived from natural human motion while maintaining balance and rhythm in real time. To improve general-purpose utility, the controller goes beyond single-trajectory replay by introducing a two-part design: ⦿ A high-level kinematic planner generates smooth, human-like movement goals from simple input commands (e.g., joystick direction). ⦿ A low-level RL controller tracks these trajectories with dynamic stability. This structure enables smooth transitions between behaviors and resolves the challenge of controlling high-dimensional motion from coarse user input.show more

The Humanoid Hub

63,197 subscribers

113,780 görüntüleme • 1 yıl önce •via X (Twitter)

Sağlık & İyilik Bilim & Teknoloji Eğitim

Anya Rossi• Live Now

Private livecam show

11 Yorum

The Humanoid Hub profil fotoğrafı

The Humanoid Hub1 yıl önce

More context:

Page to Pixel Publishing profil fotoğrafı

Page to Pixel Publishing2 yıl önce

Boost, surf, and weave your way through The Art of Flight, an arcade game about flying multiple ships at the same time. With solo, local co-op, and a leaderboard, there are tons of ways to play. Wishlist on Steam today!

Future mobility profil fotoğrafı

Future mobility1 yıl önce

Cool

MasIp profil fotoğrafı

MasIp1 yıl önce

Slap an A1 logo on there and you have a horror thriller

Arjun Goli 🌎 profil fotoğrafı

Arjun Goli 🌎1 yıl önce

Aesthetics r amazing

cromagnus◀️▶️⏸ profil fotoğrafı

cromagnus◀️▶️⏸1 yıl önce

give this bot a better outfit and it will 10X. their appeal. i get the approach with the drab onesie but they would be getting so much more marketing coverage with more modern custom unique functional fashion sensibilities. its making this amazing bot feel dull.

Brian Bellia profil fotoğrafı

Brian Bellia1 yıl önce

The first 10 seconds of this clip are magical. I can't wait until I can do that with a humanoid. NEO on the sand at the beach ... now, that would really be something.

Don Diego de la Tega profil fotoğrafı

Don Diego de la Tega1 yıl önce

Some of these scenes are CGI, those led on their "ears", are way too bright under daylight to be real.

AdamHumphreys profil fotoğrafı

AdamHumphreys1 yıl önce

I think the mistake made is the lack of padding on the digits of robots. This distributes weight and grip strength leverage. Current models are too rigid but we’ll get there.

vikramvi profil fotoğrafı

vikramvi1 yıl önce

when will you share details like this @Tesla_Optimus $TSLA

Brian Bellia profil fotoğrafı

Brian Bellia1 yıl önce

Given the partnership between 1X and OpenAI, I wonder if there's any role for ChatGPT. I thought it might be able to operate alongside Redwood AI to give NEO a way to verbally interact with humans.

Benzer Videolar

TWIST: a real-time teleoperation system for humanoid robots to mimic whole-body motions. Reference motion data is generated by retargeting human motion-capture data to the robot. Then, the controller is trained in simulation using reinforcement learning and behavior cloning.

TWIST: a real-time teleoperation system for humanoid robots to mimic whole-body motions. Reference motion data is generated by retargeting human motion-capture data to the robot. Then, the controller is trained in simulation using reinforcement learning and behavior cloning.

The Humanoid Hub

46,715 görüntüleme • 1 yıl önce

Humanoid robots playing table tennis fully autonomously. The 'HITTER' system combines a model-based planner with a reinforcement learning (RL) whole-body controller. It is fully autonomous but relies on an external sensing system. A 9-camera OptiTrack motion capture setup tracks the ball, which is covered with reflective markers, to predict its precise trajectory and compute the optimal strike position, velocity, and timing. The RL controller then executes coordinated arm, leg, and waist movements, producing agile, human-like returns.

Humanoid robots playing table tennis fully autonomously. The 'HITTER' system combines a model-based planner with a reinforcement learning (RL) whole-body controller. It is fully autonomous but relies on an external sensing system. A 9-camera OptiTrack motion capture setup tracks the ball, which is covered with reflective markers, to predict its precise trajectory and compute the optimal strike position, velocity, and timing. The RL controller then executes coordinated arm, leg, and waist movements, producing agile, human-like returns.

The Humanoid Hub

91,812 görüntüleme • 9 ay önce

NVIDIA researchers present SONIC, a generalist humanoid controller: It scales motion tracking on a single policy to achieve natural, robust whole-body movement. The scalable foundation avoids manual reward engineering and features a universal token space and kinematic planner to seamlessly integrate diverse inputs, including VR/video teleoperation and VLA foundation models, for real-world tasks.

NVIDIA researchers present SONIC, a generalist humanoid controller: It scales motion tracking on a single policy to achieve natural, robust whole-body movement. The scalable foundation avoids manual reward engineering and features a universal token space and kinematic planner to seamlessly integrate diverse inputs, including VR/video teleoperation and VLA foundation models, for real-world tasks.

The Humanoid Hub

31,445 görüntüleme • 7 ay önce

Full-body joint tracking + smart gloves capture every nuance of human motion, which is retargeted in real-time to a humanoid robot (here, Tesla Optimus) for smooth teleoperation. This is the power of inertial motion capture. More than control, mocap is the gold standard for collecting high-fidelity humanoid training data. Advanced AI algorithms transform raw accelerometer streams into precise 3D joint poses, turning human movement into lifelike robot action.

Full-body joint tracking + smart gloves capture every nuance of human motion, which is retargeted in real-time to a humanoid robot (here, Tesla Optimus) for smooth teleoperation. This is the power of inertial motion capture. More than control, mocap is the gold standard for collecting high-fidelity humanoid training data. Advanced AI algorithms transform raw accelerometer streams into precise 3D joint poses, turning human movement into lifelike robot action.

Reborn

95,416 görüntüleme • 10 ay önce

“Stair mode” in NEO's RL controller engages stereo RGB vision to infer the height of the floor around it, combining this with proprioceptive history to anticipate each step's height and plan precise, stable foot placement.

“Stair mode” in NEO's RL controller engages stereo RGB vision to infer the height of the floor around it, combining this with proprioceptive history to anticipate each step's height and plan precise, stable foot placement.

The Humanoid Hub

51,412 görüntüleme • 1 yıl önce

This is how ALOHA's "teleoperation" system works - a fancy word for "remote control". Training robots will be more and more like playing games in the physical world. A human operates a "joystick++" to perform tasks and collect data, or intervene if there's any safety concern. There's actually a learning curve to master the controller, much like practicing gaming skills. Teleoperation can be done in many different ways. ALOHA is an impressive custom-built system with very low cost. Here're a few alternatives: (1) Motion Capture (MoCap): apply the MoCap systems used for Hollywood movies to capture the fine-grained motions of hand joints. There would be no "embodiment gap" if the robot hand has 5 fingers. For instance, a demonstrator can wear a CyberGlove ( and manipulate the objects. CyberGlove will capture the motion signals & haptic feedback in real-time, which can be re-targeted onto the humanoid. (2) Wearing gloves & markers can be clumsy. An alternative way to do MoCap is through computer vision. DexPilot from NVIDIA enables marker-less and glove-free data collection. The human operator simply uses their bare hands to perform the tasks. 4 Intel RealSense depth cameras and 2 NVIDIA Titan XP GPUs (yeah, 2019 work) translate the pixels to precise motion signals for robot learning. (3) VR Headset: turn the training room into a VR game and "role play" the robot. This has the advantage of scalable remote data collection - annotators from around the world can contribute without coming onsite. VR demonstration technique appeared in research projects like the iGibson home robot simulator, an initiative that I participated in at Stanford: Behind-the-scene video by Litian Liang

This is how ALOHA's "teleoperation" system works - a fancy word for "remote control". Training robots will be more and more like playing games in the physical world. A human operates a "joystick++" to perform tasks and collect data, or intervene if there's any safety concern. There's actually a learning curve to master the controller, much like practicing gaming skills. Teleoperation can be done in many different ways. ALOHA is an impressive custom-built system with very low cost. Here're a few alternatives: (1) Motion Capture (MoCap): apply the MoCap systems used for Hollywood movies to capture the fine-grained motions of hand joints. There would be no "embodiment gap" if the robot hand has 5 fingers. For instance, a demonstrator can wear a CyberGlove ( and manipulate the objects. CyberGlove will capture the motion signals & haptic feedback in real-time, which can be re-targeted onto the humanoid. (2) Wearing gloves & markers can be clumsy. An alternative way to do MoCap is through computer vision. DexPilot from NVIDIA enables marker-less and glove-free data collection. The human operator simply uses their bare hands to perform the tasks. 4 Intel RealSense depth cameras and 2 NVIDIA Titan XP GPUs (yeah, 2019 work) translate the pixels to precise motion signals for robot learning. (3) VR Headset: turn the training room into a VR game and "role play" the robot. This has the advantage of scalable remote data collection - annotators from around the world can contribute without coming onsite. VR demonstration technique appeared in research projects like the iGibson home robot simulator, an initiative that I participated in at Stanford: Behind-the-scene video by Litian Liang

Jim Fan

124,588 görüntüleme • 2 yıl önce

Physics-based Motion Retargeting from Sparse Inputs paper page: Avatars are important to create interactive and immersive experiences in virtual worlds. One challenge in animating these characters to mimic a user's motion is that commercial AR/VR products consist only of a headset and controllers, providing very limited sensor data of the user's pose. Another challenge is that an avatar might have a different skeleton structure than a human and the mapping between them is unclear. In this work we address both of these challenges. We introduce a method to retarget motions in real-time from sparse human sensor data to characters of various morphologies. Our method uses reinforcement learning to train a policy to control characters in a physics simulator. We only require human motion capture data for training, without relying on artist-generated animations for each avatar. This allows us to use large motion capture datasets to train general policies that can track unseen users from real and sparse data in real-time. We demonstrate the feasibility of our approach on three characters with different skeleton structure: a dinosaur, a mouse-like creature and a human. We show that the avatar poses often match the user surprisingly well, despite having no sensor information of the lower body available. We discuss and ablate the important components in our framework, specifically the kinematic retargeting step, the imitation, contact and action reward as well as our asymmetric actor-critic observations. We further explore the robustness of our method in a variety of settings including unbalancing, dancing and sports motions.

AK

106,519 görüntüleme • 3 yıl önce

Excited to share a new humanoid robot platform we’ve been working on. Berkeley Humanoid is a reliable and low-cost mid-scale research platform for learning-based control. We demonstrate the robot walking on various terrains and dynamic hopping with a simple RL controller.

Excited to share a new humanoid robot platform we’ve been working on. Berkeley Humanoid is a reliable and low-cost mid-scale research platform for learning-based control. We demonstrate the robot walking on various terrains and dynamic hopping with a simple RL controller.

Qiayuan Liao

80,294 görüntüleme • 1 yıl önce

The power of generative models — now embodied in humanoids. Announcing DreamControl –– After a year-long research effort at General Robotics — we present a scalable framework for whole-body humanoid control that fuses diffusion priors with reinforcement learning to unlock real-world scene interaction. Diffusion + RL → natural whole-body skills on real robots. DreamControl enables humanoids to move beyond locomotion demos → performing natural, human-like skills such as –– Picking & lifting objects, Opening drawers & doors, Precise punching, kicking, and jumping, Bimanual manipulation tasks Our key innovation: a diffusion prior over human motion that guides RL, eliminating the need for massive teleoperation datasets, and producing motions that look human while transferring to real hardware. Trained purely in simulation, deployed on the Unitree G1 humanoid, DreamControl policies run in real time, bridging sim-to-real with unprecedented naturalness. We leverage a novel hybrid edge + cloud infrastructure that runs RL-trained policies on the edge backed by powerful AI models running in the cloud This is the next step in General Robotics’ journey toward general-purpose humanoid assistants that interact, adapt, and assist autonomously. Paper: Blog: 1/n

The power of generative models — now embodied in humanoids. Announcing DreamControl –– After a year-long research effort at General Robotics — we present a scalable framework for whole-body humanoid control that fuses diffusion priors with reinforcement learning to unlock real-world scene interaction. Diffusion + RL → natural whole-body skills on real robots. DreamControl enables humanoids to move beyond locomotion demos → performing natural, human-like skills such as –– Picking & lifting objects, Opening drawers & doors, Precise punching, kicking, and jumping, Bimanual manipulation tasks Our key innovation: a diffusion prior over human motion that guides RL, eliminating the need for massive teleoperation datasets, and producing motions that look human while transferring to real hardware. Trained purely in simulation, deployed on the Unitree G1 humanoid, DreamControl policies run in real time, bridging sim-to-real with unprecedented naturalness. We leverage a novel hybrid edge + cloud infrastructure that runs RL-trained policies on the edge backed by powerful AI models running in the cloud This is the next step in General Robotics’ journey toward general-purpose humanoid assistants that interact, adapt, and assist autonomously. Paper: Blog: 1/n

Ashish Kapoor

118,084 görüntüleme • 9 ay önce

🤖 Introducing H2O (Human2HumanOid): - 🧠 An RL-based human-to-humanoid real-time whole-body teleoperation framework - 💃 Scalable retargeting and training using large human motion dataset - 🎥 With just an RGB camera, everyone can teleoperate a full-sized humanoid to perform actions like pick and place, walking, kicking, boxing, etc - 💡Unleash the potential of humanoids with human cognitive skills and adaptability 🔗:

🤖 Introducing H2O (Human2HumanOid): - 🧠 An RL-based human-to-humanoid real-time whole-body teleoperation framework - 💃 Scalable retargeting and training using large human motion dataset - 🎥 With just an RGB camera, everyone can teleoperate a full-sized humanoid to perform actions like pick and place, walking, kicking, boxing, etc - 💡Unleash the potential of humanoids with human cognitive skills and adaptability 🔗:

Tairan He

96,714 görüntüleme • 2 yıl önce

This work makes a humanoid robot do simple parkour moves by looking with a depth camera and choosing the right move on the fly. The big deal is that it turns lots of small human moves into long, real-time robot behavior, without hand-coding every transition or retraining for each new course. A humanoid robot is usually good at steady walking, but it often fails when it has to do fast moves like jumping up, vaulting, or rolling, and then keep going to the next obstacle. The hard part is that you cannot easily collect training data for every possible obstacle shape, distance, and mistake, so robots end up learning a few moves that only work in a narrow setup. This work starts from short clips of real human parkour moves, like stepping over, vaulting, climbing, and rolling. It uses motion matching, which is basically a smart “pick the next clip that fits best right now” search, to stitch those short clips into a long, smooth plan that looks like a human doing a whole course. Then it trains a controller with reinforcement learning (RL), which means the robot learns by trial and error to copy that plan while staying balanced and not falling. After training separate expert controllers for different moves, it compresses them into 1 controller that uses only onboard depth sensing and a simple “go this fast in this direction” command. In real tests on a Unitree G1 humanoid, it can clear multiple obstacles in a row, adapt when obstacles get moved, and climb a wall up to 1.25m.

This work makes a humanoid robot do simple parkour moves by looking with a depth camera and choosing the right move on the fly. The big deal is that it turns lots of small human moves into long, real-time robot behavior, without hand-coding every transition or retraining for each new course. A humanoid robot is usually good at steady walking, but it often fails when it has to do fast moves like jumping up, vaulting, or rolling, and then keep going to the next obstacle. The hard part is that you cannot easily collect training data for every possible obstacle shape, distance, and mistake, so robots end up learning a few moves that only work in a narrow setup. This work starts from short clips of real human parkour moves, like stepping over, vaulting, climbing, and rolling. It uses motion matching, which is basically a smart “pick the next clip that fits best right now” search, to stitch those short clips into a long, smooth plan that looks like a human doing a whole course. Then it trains a controller with reinforcement learning (RL), which means the robot learns by trial and error to copy that plan while staying balanced and not falling. After training separate expert controllers for different moves, it compresses them into 1 controller that uses only onboard depth sensing and a simple “go this fast in this direction” command. In real tests on a Unitree G1 humanoid, it can clear multiple obstacles in a row, adapt when obstacles get moved, and climb a wall up to 1.25m.

Rohan Paul

37,121 görüntüleme • 4 ay önce

VLAI Robotics Launches High-Dexterity Dual-Arm Robot Starting at ~$5,500 USD VLAI Robotics, utilizing the open-source design of Japan’s OpenArm team, has introduced a high-dexterity, low-cost bimanual robotic system. The product is priced from 39,900 RMB (approximately 5,500 USD), making high-level research tools accessible to institutions and developers. The robot features a human-scale, arm-hand integrated design with up to 16 DOF (8 DOF per arm, including the gripper). It can handle a dual-arm peak payload of 12 kg while maintaining high precision. A key breakthrough is its ability to replicate human upper limb movement trajectories, which enhances data quality for imitation learning and remote operation. VLAI Robotics handled the domestic engineering, manufacturing, and VLA (Vision-Language-Action) algorithm integration, with quality control organized according to the strict standards of the OpenArm R&D framework. The robot is highly adaptable for research, education, and Physical AI training, supporting easy expansion for features like VR teleoperation.

VLAI Robotics Launches High-Dexterity Dual-Arm Robot Starting at ~$5,500 USD VLAI Robotics, utilizing the open-source design of Japan’s OpenArm team, has introduced a high-dexterity, low-cost bimanual robotic system. The product is priced from 39,900 RMB (approximately 5,500 USD), making high-level research tools accessible to institutions and developers. The robot features a human-scale, arm-hand integrated design with up to 16 DOF (8 DOF per arm, including the gripper). It can handle a dual-arm peak payload of 12 kg while maintaining high precision. A key breakthrough is its ability to replicate human upper limb movement trajectories, which enhances data quality for imitation learning and remote operation. VLAI Robotics handled the domestic engineering, manufacturing, and VLA (Vision-Language-Action) algorithm integration, with quality control organized according to the strict standards of the OpenArm R&D framework. The robot is highly adaptable for research, education, and Physical AI training, supporting easy expansion for features like VR teleoperation.

RoboHub🤖

12,961 görüntüleme • 8 ay önce

Agile Continuous Jumping in Discontinuous Terrains discuss: We focus on agile, continuous, and terrain-adaptive jumping of quadrupedal robots in discontinuous terrains such as stairs and stepping stones. Unlike single-step jumping, continuous jumping requires accurately executing highly dynamic motions over long horizons, which is challenging for existing approaches. To accomplish this task, we design a hierarchical learning and control framework, which consists of a learned heightmap predictor for robust terrain perception, a reinforcement-learning-based centroidal-level motion policy for versatile and terrain-adaptive planning, and a low-level model-based leg controller for accurate motion tracking. In addition, we minimize the sim-to-real gap by accurately modeling the hardware characteristics. Our framework enables a Unitree Go1 robot to perform agile and continuous jumps on human-sized stairs and sparse stepping stones, for the first time to the best of our knowledge. In particular, the robot can cross two stair steps in each jump and completes a 3.5m long, 2.8m high, 14-step staircase in 4.5 seconds. Moreover, the same policy outperforms baselines in various other parkour tasks, such as jumping over single horizontal or vertical discontinuities.

Agile Continuous Jumping in Discontinuous Terrains discuss: We focus on agile, continuous, and terrain-adaptive jumping of quadrupedal robots in discontinuous terrains such as stairs and stepping stones. Unlike single-step jumping, continuous jumping requires accurately executing highly dynamic motions over long horizons, which is challenging for existing approaches. To accomplish this task, we design a hierarchical learning and control framework, which consists of a learned heightmap predictor for robust terrain perception, a reinforcement-learning-based centroidal-level motion policy for versatile and terrain-adaptive planning, and a low-level model-based leg controller for accurate motion tracking. In addition, we minimize the sim-to-real gap by accurately modeling the hardware characteristics. Our framework enables a Unitree Go1 robot to perform agile and continuous jumps on human-sized stairs and sparse stepping stones, for the first time to the best of our knowledge. In particular, the robot can cross two stair steps in each jump and completes a 3.5m long, 2.8m high, 14-step staircase in 4.5 seconds. Moreover, the same policy outperforms baselines in various other parkour tasks, such as jumping over single horizontal or vertical discontinuities.

AK

35,785 görüntüleme • 1 yıl önce

The World Model as NEO's Cognitive Core 1X has revealed a major AI development where the NEO humanoid can translate any natural language prompt into robotic action. It demonstrates this capability even for novel tasks, objects, and environments not found in its robot dataset. - the 1X World Model is trained on internet-scale human interaction videos and fine-tuned with robot data to ground its understanding in physics and in NEO's embodiment - from a simple voice or text prompt, the world model generates a visualization of future actions - a built-in inverse dynamics model then translates these into precise motor movements for NEO

The World Model as NEO's Cognitive Core 1X has revealed a major AI development where the NEO humanoid can translate any natural language prompt into robotic action. It demonstrates this capability even for novel tasks, objects, and environments not found in its robot dataset. - the 1X World Model is trained on internet-scale human interaction videos and fine-tuned with robot data to ground its understanding in physics and in NEO's embodiment - from a simple voice or text prompt, the world model generates a visualization of future actions - a built-in inverse dynamics model then translates these into precise motor movements for NEO

The Humanoid Hub

68,453 görüntüleme • 5 ay önce

With Helix 02, Figure has introduced a new foundational layer for whole-body control that replaces 109k lines of hand-engineered C++ with a single neural prior for stable, natural motion. - Learned from 1,000+ hours of human motion data and sim-to-real RL across 200,000+ parallel environments. - A 10M-parameter neural network that takes full-body joint state and base motion as input and outputs joint-level actuator commands at 1 kHz (1,000 times a second).

With Helix 02, Figure has introduced a new foundational layer for whole-body control that replaces 109k lines of hand-engineered C++ with a single neural prior for stable, natural motion. - Learned from 1,000+ hours of human motion data and sim-to-real RL across 200,000+ parallel environments. - A 10M-parameter neural network that takes full-body joint state and base motion as input and outputs joint-level actuator commands at 1 kHz (1,000 times a second).

The Humanoid Hub

28,004 görüntüleme • 4 ay önce

The best way to get robust, high-quality robot performance is through reinforcement learning; but RL in either the real world or a traditional simulation has lots of limitations. Instead, Jiazhi Yang in RISE does RL in a compositional world model. Learn more ->

The best way to get robust, high-quality robot performance is through reinforcement learning; but RL in either the real world or a traditional simulation has lots of limitations. Instead, Jiazhi Yang in RISE does RL in a compositional world model. Learn more ->

Chris Paxton

33,613 görüntüleme • 12 gün önce

In order for robots to be deployed in the real world, performing tasks of real value, they must be reliable. Unfortunately, even more, most robotic demos work maybe 70-80% of the time at best. The way to get better reliability is to do real-world reinforcement learning: having the robot teach itself how to perform the task up to a high level of success. The key to doing this is to start with a core of expert human data, use that to train a policy then iteratively improve it, until finally finishing with on-policy reinforcement learning. Kun Lei talks through a unified framework for imitation and reinforcement learning based on PPO, which enables this improvement process. In this episode, Kun Lei explains the theory behind his reinforcement learning method and how it allowed his robot to run in a shopping mall juicing oranges for seven hours at a time, among experiments on a wide variety of tasks and embodiments. Watch episode 58 of RoboPapers now, hosted by Michael Cho - Rbt/Acc and Chris Paxton!

In order for robots to be deployed in the real world, performing tasks of real value, they must be reliable. Unfortunately, even more, most robotic demos work maybe 70-80% of the time at best. The way to get better reliability is to do real-world reinforcement learning: having the robot teach itself how to perform the task up to a high level of success. The key to doing this is to start with a core of expert human data, use that to train a policy then iteratively improve it, until finally finishing with on-policy reinforcement learning. Kun Lei talks through a unified framework for imitation and reinforcement learning based on PPO, which enables this improvement process. In this episode, Kun Lei explains the theory behind his reinforcement learning method and how it allowed his robot to run in a shopping mall juicing oranges for seven hours at a time, among experiments on a wide variety of tasks and embodiments. Watch episode 58 of RoboPapers now, hosted by Michael Cho - Rbt/Acc and Chris Paxton!

RoboPapers

18,541 görüntüleme • 5 ay önce

MotionGPT: Human Motion as a Foreign Language paper page: Though the advancement of pre-trained large language models unfolds, the exploration of building a unified model for language and other multi-modal data, such as motion, remains challenging and untouched so far. Fortunately, human motion displays a semantic coupling akin to human language, often perceived as a form of body language. By fusing language data with large-scale motion models, motion-language pre-training that can enhance the performance of motion-related tasks becomes feasible. Driven by this insight, we propose MotionGPT, a unified, versatile, and user-friendly motion-language model to handle multiple motion-relevant tasks. Specifically, we employ the discrete vector quantization for human motion and transfer 3D motion into motion tokens, similar to the generation process of word tokens. Building upon this "motion vocabulary", we perform language modeling on both motion and text in a unified manner, treating human motion as a specific language. Moreover, inspired by prompt learning, we pre-train MotionGPT with a mixture of motion-language data and fine-tune it on prompt-based question-and-answer tasks. Extensive experiments demonstrate that MotionGPT achieves state-of-the-art performances on multiple motion tasks including text-driven motion generation, motion captioning, motion prediction, and motion in-between.

MotionGPT: Human Motion as a Foreign Language paper page: Though the advancement of pre-trained large language models unfolds, the exploration of building a unified model for language and other multi-modal data, such as motion, remains challenging and untouched so far. Fortunately, human motion displays a semantic coupling akin to human language, often perceived as a form of body language. By fusing language data with large-scale motion models, motion-language pre-training that can enhance the performance of motion-related tasks becomes feasible. Driven by this insight, we propose MotionGPT, a unified, versatile, and user-friendly motion-language model to handle multiple motion-relevant tasks. Specifically, we employ the discrete vector quantization for human motion and transfer 3D motion into motion tokens, similar to the generation process of word tokens. Building upon this "motion vocabulary", we perform language modeling on both motion and text in a unified manner, treating human motion as a specific language. Moreover, inspired by prompt learning, we pre-train MotionGPT with a mixture of motion-language data and fine-tune it on prompt-based question-and-answer tasks. Extensive experiments demonstrate that MotionGPT achieves state-of-the-art performances on multiple motion tasks including text-driven motion generation, motion captioning, motion prediction, and motion in-between.

AK

125,319 görüntüleme • 3 yıl önce

🤔 Ever wondered if simulation-based animation/avatar learnings can be applied to real humanoid in real-time? 🤖 Introducing H2O (Human2HumanOid): - 🧠 An RL-based human-to-humanoid real-time whole-body teleoperation framework - 💃 Scalable retargeting and training using large human motion dataset - 🎥 With just an RGB camera, everyone can teleoperate a full-sized humanoid to perform actions like pick and place, walking, kicking, boxing, etc - 💡Unleash the potential of humanoids with human cognitive skills and adaptability 🔗: 📄: 🎬: H2O proposes: - A scalable retargeting framework for obtaining large-scale humanoid motion dataset, intelligently filtering out infeasible motion for the humanoid embodiment. - We train a full-body motion imitator (similar to PHC) and deploy to the real world zero-shot. - Using this framework, we enable real-time teleoperation of a humanoid via a human operator and webcam, performing skills such as pick and place, kicking, walking strollers, etc. Team: Tairan He Zhengyi “Zen” Luo Wenli Xiao @ChongZitaZhang Kris Kitani Changliu Liu Guanya Shi

Zhengyi “Zen” Luo

47,287 görüntüleme • 2 yıl önce

Today, we're announcing the next evolution of Sword Thrive: a fully revamped experience anchored by a new conversational care model, Vision AI, and a more powerful Phoenix. At the center of this evolution is a first-of-its-kind care model: a 3-way chat connecting the member, Phoenix, and their Doctor of Physical Therapy in a single interface, making continuous, 24/7 care possible for the first time, with the clinical oversight only a human specialist can provide. Vision AI, our proprietary technology trained on years of movement data, now delivers precise movement detection in real-world home environments - cluttered, noisy, and everything in between - and runs entirely through the Sword Health app on smartphones and tablets. Phoenix, our AI Care Specialist, is now more capable as well, providing deeper, personalized two-way conversational guidance and real-time form correction, adapting to each member as they move. From proactive care to continuous recovery, this is how healthcare should show up: with reliability, clinical memory, and real-time support that never drops the thread. Watch the video to see it in action:

Today, we're announcing the next evolution of Sword Thrive: a fully revamped experience anchored by a new conversational care model, Vision AI, and a more powerful Phoenix. At the center of this evolution is a first-of-its-kind care model: a 3-way chat connecting the member, Phoenix, and their Doctor of Physical Therapy in a single interface, making continuous, 24/7 care possible for the first time, with the clinical oversight only a human specialist can provide. Vision AI, our proprietary technology trained on years of movement data, now delivers precise movement detection in real-world home environments - cluttered, noisy, and everything in between - and runs entirely through the Sword Health app on smartphones and tablets. Phoenix, our AI Care Specialist, is now more capable as well, providing deeper, personalized two-way conversational guidance and real-time form correction, adapting to each member as they move. From proactive care to continuous recovery, this is how healthcare should show up: with reliability, clinical memory, and real-time support that never drops the thread. Watch the video to see it in action:

Sword Health

27,974 görüntüleme • 4 ay önce