Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Most motion papers tailor one controller to one specific task. This year at SIGGRAPH, our research team asks: can motor control itself be pretrained and reused? Generative Pretrained Controllers, or GPC, turn motor skills into a vocabulary of discrete tokens and train a transformer-based generative controller through next-token prediction.... show more

NVIDIA AI

316,673 subscribers

164,173 views • 2 days ago •via X (Twitter)

Science & Technology

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

A new paper by Disney Research introduces a 2-stage technique for motion tracking in physics-based character animation, using a pretrained latent space and RL to control diverse and unseen motions on both virtual and real-world humanoid robots. Paper:

A new paper by Disney Research introduces a 2-stage technique for motion tracking in physics-based character animation, using a pretrained latent space and RL to control diverse and unseen motions on both virtual and real-world humanoid robots. Paper:

The Humanoid Hub

20,051 views • 1 year ago

What is missing to bring real-time motion research into AAA games and real-world robotics? We present MotionBricks, a step toward bridging this gap with two key components: - a single generative latent motion backbone covering 350,000+ motion skills, running at 15,000 FPS with 2 ms latency and substantially improved quality and reliability. - a unified smart primitive interface for locomotion, object / scene interaction, with fine-grained control over generated behaviors. Webpage: Code: Paper: (ACM TOG / SIGGRAPH 2026)

What is missing to bring real-time motion research into AAA games and real-world robotics? We present MotionBricks, a step toward bridging this gap with two key components: - a single generative latent motion backbone covering 350,000+ motion skills, running at 15,000 FPS with 2 ms latency and substantially improved quality and reliability. - a unified smart primitive interface for locomotion, object / scene interaction, with fine-grained control over generated behaviors. Webpage: Code: Paper: (ACM TOG / SIGGRAPH 2026)

tingwu.wang

152,688 views • 2 months ago

Cosmos Policy turns a pretrained video diffusion model into a robot controller. Instead of redesigning the architecture, it injects robot state, actions, and values directly as latent frames inside the video model

Cosmos Policy turns a pretrained video diffusion model into a robot controller. Instead of redesigning the architecture, it injects robot state, actions, and values directly as latent frames inside the video model

Robots Digest 🤖

22,933 views • 5 months ago

Rock'em Sock'em Robots, but with Unitree G1! “Unitree will be livestreaming robot combat in about a month — stay tuned! The robot is actively learning even more new skills. You can control it via motion capture, a controller, or voice commands.”

Rock'em Sock'em Robots, but with Unitree G1! “Unitree will be livestreaming robot combat in about a month — stay tuned! The robot is actively learning even more new skills. You can control it via motion capture, a controller, or voice commands.”

The Humanoid Hub

208,841 views • 1 year ago

MotionGPT: Human Motion as a Foreign Language paper page: Though the advancement of pre-trained large language models unfolds, the exploration of building a unified model for language and other multi-modal data, such as motion, remains challenging and untouched so far. Fortunately, human motion displays a semantic coupling akin to human language, often perceived as a form of body language. By fusing language data with large-scale motion models, motion-language pre-training that can enhance the performance of motion-related tasks becomes feasible. Driven by this insight, we propose MotionGPT, a unified, versatile, and user-friendly motion-language model to handle multiple motion-relevant tasks. Specifically, we employ the discrete vector quantization for human motion and transfer 3D motion into motion tokens, similar to the generation process of word tokens. Building upon this "motion vocabulary", we perform language modeling on both motion and text in a unified manner, treating human motion as a specific language. Moreover, inspired by prompt learning, we pre-train MotionGPT with a mixture of motion-language data and fine-tune it on prompt-based question-and-answer tasks. Extensive experiments demonstrate that MotionGPT achieves state-of-the-art performances on multiple motion tasks including text-driven motion generation, motion captioning, motion prediction, and motion in-between.

MotionGPT: Human Motion as a Foreign Language paper page: Though the advancement of pre-trained large language models unfolds, the exploration of building a unified model for language and other multi-modal data, such as motion, remains challenging and untouched so far. Fortunately, human motion displays a semantic coupling akin to human language, often perceived as a form of body language. By fusing language data with large-scale motion models, motion-language pre-training that can enhance the performance of motion-related tasks becomes feasible. Driven by this insight, we propose MotionGPT, a unified, versatile, and user-friendly motion-language model to handle multiple motion-relevant tasks. Specifically, we employ the discrete vector quantization for human motion and transfer 3D motion into motion tokens, similar to the generation process of word tokens. Building upon this "motion vocabulary", we perform language modeling on both motion and text in a unified manner, treating human motion as a specific language. Moreover, inspired by prompt learning, we pre-train MotionGPT with a mixture of motion-language data and fine-tune it on prompt-based question-and-answer tasks. Extensive experiments demonstrate that MotionGPT achieves state-of-the-art performances on multiple motion tasks including text-driven motion generation, motion captioning, motion prediction, and motion in-between.

AK

125,319 views • 3 years ago

TWIST: a real-time teleoperation system for humanoid robots to mimic whole-body motions. Reference motion data is generated by retargeting human motion-capture data to the robot. Then, the controller is trained in simulation using reinforcement learning and behavior cloning.

TWIST: a real-time teleoperation system for humanoid robots to mimic whole-body motions. Reference motion data is generated by retargeting human motion-capture data to the robot. Then, the controller is trained in simulation using reinforcement learning and behavior cloning.

The Humanoid Hub

46,715 views • 1 year ago

Super clean and efficient meshes by an AI? YES! The typical 3D Generative AI solutions produce lots of artifacts and usually way to many polygons due to volumetric approaches. In comparison “MeshGPT creates triangle meshes by autoregressively sampling from a transformer model that has been trained to produce tokens from a learned geometric vocabulary. These tokens can then be decoded into the faces of a triangle mesh. This method generates clean, coherent, and compact meshes, characterized by sharp edges and high fidelity.” Surely it is limited by the trained vocabulary but various versions can be trained for specific sets to create generative model libraries for certain object groups. Very promising approach with the high quality.

Super clean and efficient meshes by an AI? YES! The typical 3D Generative AI solutions produce lots of artifacts and usually way to many polygons due to volumetric approaches. In comparison “MeshGPT creates triangle meshes by autoregressively sampling from a transformer model that has been trained to produce tokens from a learned geometric vocabulary. These tokens can then be decoded into the faces of a triangle mesh. This method generates clean, coherent, and compact meshes, characterized by sharp edges and high fidelity.” Surely it is limited by the trained vocabulary but various versions can be trained for specific sets to create generative model libraries for certain object groups. Very promising approach with the high quality.

René Schulte

20,772 views • 2 years ago

PaLM + AudioLM = AudioPaLM ! We start from PaLM pretrained on text and extend its vocab w/ audio tokens. This model can then be finetuned on a mix of any (speech, text) task e.g. ASR, TTS, MT and speech2speech translation in one's voice! 🧵1/4

PaLM + AudioLM = AudioPaLM ! We start from PaLM pretrained on text and extend its vocab w/ audio tokens. This model can then be finetuned on a mix of any (speech, text) task e.g. ASR, TTS, MT and speech2speech translation in one's voice! 🧵1/4

Neil Zeghidour

41,240 views • 3 years ago

[1/n] Happy to share our new work "MuPT: A Generative Symbolic Music Pretrained Transformer", encompassing a series of music generation models ranging from 190 million to 4.2 billion parameters, all based on the ABC Notation. According to human preference evaluations, our models create music that is more melodious than what GPT-4 can produce. Enjoy the selected music collection generated by MuPT!

[1/n] Happy to share our new work "MuPT: A Generative Symbolic Music Pretrained Transformer", encompassing a series of music generation models ranging from 190 million to 4.2 billion parameters, all based on the ABC Notation. According to human preference evaluations, our models create music that is more melodious than what GPT-4 can produce. Enjoy the selected music collection generated by MuPT!

Ge Zhang

13,404 views • 2 years ago

Why do generalist robotic models fail when a cup is moved just two inches to the left? It’s not a lack of motor skill, it’s an alignment problem. Today, we introduce VLS: Vision-Language Steering of Pretrained Robot Policies, a training-free framework that guides robot behavior in real time. Check out the project: 👇🧵 (Watch till the end: VLS runs uncut, steering pretrained policies across long-horizon tasks.)

Why do generalist robotic models fail when a cup is moved just two inches to the left? It’s not a lack of motor skill, it’s an alignment problem. Today, we introduce VLS: Vision-Language Steering of Pretrained Robot Policies, a training-free framework that guides robot behavior in real time. Check out the project: 👇🧵 (Watch till the end: VLS runs uncut, steering pretrained policies across long-horizon tasks.)

Jiafei Duan

72,319 views • 4 months ago

1X announces their latest reinforcement learning (RL) controller, which unlocks NEO's full-body mobility for home environments, enabling Redwood AI (1X's in-house AI model) to interact with the physical world more naturally and broadly. The unified controller supports walking in any direction, sitting, standing, kneeling, lying down, getting up, and climbing stairs using stereo RGB vision - critical for navigating real homes. The controller provides an “action interface” through which teleoperation or Redwood AI can interact in a safe, contact-rich manner with the physical world. Traditional walking controllers often rely on hand-crafted "shaping rewards" to produce human-like gaits, which are time-consuming and don’t scale well across tasks or movement directions. To address this, the team used motion capture data to guide the learning process. The RL system is trained to track kinematic reference trajectories derived from natural human motion while maintaining balance and rhythm in real time. To improve general-purpose utility, the controller goes beyond single-trajectory replay by introducing a two-part design: ⦿ A high-level kinematic planner generates smooth, human-like movement goals from simple input commands (e.g., joystick direction). ⦿ A low-level RL controller tracks these trajectories with dynamic stability. This structure enables smooth transitions between behaviors and resolves the challenge of controlling high-dimensional motion from coarse user input.

1X announces their latest reinforcement learning (RL) controller, which unlocks NEO's full-body mobility for home environments, enabling Redwood AI (1X's in-house AI model) to interact with the physical world more naturally and broadly. The unified controller supports walking in any direction, sitting, standing, kneeling, lying down, getting up, and climbing stairs using stereo RGB vision - critical for navigating real homes. The controller provides an “action interface” through which teleoperation or Redwood AI can interact in a safe, contact-rich manner with the physical world. Traditional walking controllers often rely on hand-crafted "shaping rewards" to produce human-like gaits, which are time-consuming and don’t scale well across tasks or movement directions. To address this, the team used motion capture data to guide the learning process. The RL system is trained to track kinematic reference trajectories derived from natural human motion while maintaining balance and rhythm in real time. To improve general-purpose utility, the controller goes beyond single-trajectory replay by introducing a two-part design: ⦿ A high-level kinematic planner generates smooth, human-like movement goals from simple input commands (e.g., joystick direction). ⦿ A low-level RL controller tracks these trajectories with dynamic stability. This structure enables smooth transitions between behaviors and resolves the challenge of controlling high-dimensional motion from coarse user input.

The Humanoid Hub

113,780 views • 1 year ago

GameFactory Creating New Games with Generative Interactive Videos present GameFactory, a generalizable world model that learns from a small-scale dataset of Minecraft game videos. By leveraging the prior knowledge of a pretrained video diffusion model, it can create new games in an open domain.

GameFactory Creating New Games with Generative Interactive Videos present GameFactory, a generalizable world model that learns from a small-scale dataset of Minecraft game videos. By leveraging the prior knowledge of a pretrained video diffusion model, it can create new games in an open domain.

AK

70,029 views • 1 year ago

$Example-based Motion Synthesis via Generative Motion Matching paper page: present GenMM, a generative model that "mines" as many diverse motions as possible from a single or few example sequences. In stark contrast to existing data-driven methods, which typically require long offline training time, are prone to visual artifacts, and tend to fail on large and complex skeletons, GenMM inherits the training-free nature and the superior quality of the well-known Motion Matching method. GenMM can synthesize a high-quality motion within a fraction of a second, even with highly complex and large skeletal structures. At the heart of our generative framework lies the generative motion matching module, which utilizes the bidirectional visual similarity as a generative cost function to motion matching, and operates in a multi-stage framework to progressively refine a random guess using exemplar motion matches. In addition to diverse motion generation, we show the versatility of our generative framework by extending it to a number of scenarios that are not possible with motion matching alone, including motion completion, key frame-guided generation, infinite looping, and motion reassembly.$

Example-based Motion Synthesis via Generative Motion Matching paper page: present GenMM, a generative model that "mines" as many diverse motions as possible from a single or few example sequences. In stark contrast to existing data-driven methods, which typically require long offline training time, are prone to visual artifacts, and tend to fail on large and complex skeletons, GenMM inherits the training-free nature and the superior quality of the well-known Motion Matching method. GenMM can synthesize a high-quality motion within a fraction of a second, even with highly complex and large skeletal structures. At the heart of our generative framework lies the generative motion matching module, which utilizes the bidirectional visual similarity as a generative cost function to motion matching, and operates in a multi-stage framework to progressively refine a random guess using exemplar motion matches. In addition to diverse motion generation, we show the versatility of our generative framework by extending it to a number of scenarios that are not possible with motion matching alone, including motion completion, key frame-guided generation, infinite looping, and motion reassembly.

AK

93,694 views • 3 years ago

People say you need a controller to be competitive in racing games. Then Chinese keyboard players show up and completely break that narrative. The level of precision, consistency, and control they're achieving on a keyboard is something most players still can't match with a controller. ⌨️🔥 #ForzaHorizon６ #FH6 #ForzaHorizon6

People say you need a controller to be competitive in racing games. Then Chinese keyboard players show up and completely break that narrative. The level of precision, consistency, and control they're achieving on a keyboard is something most players still can't match with a controller. ⌨️🔥 #ForzaHorizon６ #FH6 #ForzaHorizon6

PookieRaju ✭

206,828 views • 22 days ago

The fact that you can create a video like this in just a few minutes is pretty insane and magical. Anyone can place themselves into any world they want, real or imagined, instantly and soon in real time. The Moore's law of generative AI was right: the quality, resolution, and control of models doubles about every year.

The fact that you can create a video like this in just a few minutes is pretty insane and magical. Anyone can place themselves into any world they want, real or imagined, instantly and soon in real time. The Moore's law of generative AI was right: the quality, resolution, and control of models doubles about every year.

Cristóbal Valenzuela

20,713 views • 2 months ago

Can instruction tuning be used in robotics?🚨Excited to release LLARVA, our new Vision-Action Instruction-tuned LMM for robotics! Key aspects: • Perform vision-action instruction tuning • Align vision and action modalities • Pretrained on robotic instruction data Berkeley AI Research

Can instruction tuning be used in robotics?🚨Excited to release LLARVA, our new Vision-Action Instruction-tuned LMM for robotics! Key aspects: • Perform vision-action instruction tuning • Align vision and action modalities • Pretrained on robotic instruction data Berkeley AI Research

Roei Herzig ✈️ CVPR

10,502 views • 2 years ago

We are excited to share a research preview of our generative agent. The agent is being trained to solve the hardest tasks in 3D and beyond, using only keyboard and mouse actions. Join the waitlist: Our agent app runs on Windows or Mac, either locally or with one-click setup for a Windows VM. It’s still early days, but this paves the way for production-level workflows for the first time ever. Blog:

We are excited to share a research preview of our generative agent. The agent is being trained to solve the hardest tasks in 3D and beyond, using only keyboard and mouse actions. Join the waitlist: Our agent app runs on Windows or Mac, either locally or with one-click setup for a Windows VM. It’s still early days, but this paves the way for production-level workflows for the first time ever. Blog:

Common Sense Machines

155,199 views • 1 year ago

Transformer Explainer Interactive Learning of Text-Generative Models discuss: Transformers have revolutionized machine learning, yet their inner workings remain opaque to many. We present Transformer Explainer, an interactive visualization tool designed for non-experts to learn about Transformers through the GPT-2 model. Our tool helps users understand complex Transformer concepts by integrating a model overview and enabling smooth transitions across abstraction levels of mathematical operations and model structures. It runs a live GPT-2 instance locally in the user's browser, empowering users to experiment with their own input and observe in real-time how the internal components and parameters of the Transformer work together to predict the next tokens. Our tool requires no installation or special hardware, broadening the public's education access to modern generative AI techniques.

Transformer Explainer Interactive Learning of Text-Generative Models discuss: Transformers have revolutionized machine learning, yet their inner workings remain opaque to many. We present Transformer Explainer, an interactive visualization tool designed for non-experts to learn about Transformers through the GPT-2 model. Our tool helps users understand complex Transformer concepts by integrating a model overview and enabling smooth transitions across abstraction levels of mathematical operations and model structures. It runs a live GPT-2 instance locally in the user's browser, empowering users to experiment with their own input and observe in real-time how the internal components and parameters of the Transformer work together to predict the next tokens. Our tool requires no installation or special hardware, broadening the public's education access to modern generative AI techniques.

AK

90,771 views • 1 year ago

There is now a nice video tutorial that explains how to reproduce our SIGGRAPH 2022 DeepPhase research for learning a quadruped motion controller with the AI4Animation framework. Thanks to Paul Starke! Video: GitHub:

There is now a nice video tutorial that explains how to reproduce our SIGGRAPH 2022 DeepPhase research for learning a quadruped motion controller with the AI4Animation framework. Thanks to Paul Starke! Video: GitHub:

Sebastian Starke

12,451 views • 3 years ago

Air traffic control audio from LaGuardia Airport is raising questions about controller workload and staffing after a deadly runway collision that killed two pilots on Sunday, Kris Van Cleave reports. In the recording, a single controller appears to be managing multiple emergencies at once before an Air Canada plane collided with a firetruck. At one point, the controller can be heard telling a pilot, "We were dealing with an emergency earlier. I messed up."

Air traffic control audio from LaGuardia Airport is raising questions about controller workload and staffing after a deadly runway collision that killed two pilots on Sunday, Kris Van Cleave reports. In the recording, a single controller appears to be managing multiple emergencies at once before an Air Canada plane collided with a firetruck. At one point, the controller can be heard telling a pilot, "We were dealing with an emergency earlier. I messed up."

CBS News

200,051 views • 3 months ago