Most motion papers tailor one controller to one specific task. This year at SIGGRAPH, our research team asks: can motor control itself be pretrained and reused? Generative Pretrained Controllers, or GPC, turn motor skills into a vocabulary of discrete tokens and train a transformer-based generative controller through next-token prediction.... Just like GPT, the same pretrained controller can then be fine-tuned to solve new tasks. Trained on 600+ hours of motion, GPC runs in real-time inside a physics simulation, producing natural and physically grounded behaviors for interactive control.show more

Read the paper 👇

If you're interested in SIGGRAPH, check out this article from our very own @liu_mingyu. Plus, learn more about papers and sessions here:

I smell a good zombie game coming.

@HeroOfKVcache Can’t wait for all video games to implement yet another technology that makes them look the same!

Looks like we're gonna have some nimble robots before long 🤯

Turning motor skills into tokens and pretraining like a language model is such a clever crossover, curious how well it generalizes to totally new movements.

That's a cool training methodology! Identify “skill tokens,” then adapt them to specific tasks. I am wondering where this decomposition/recomposition pattern could produce interesting results🤔

Walking in Silicon Valley I see nothing but robotics startups…. This is exciting for all of them.

This is a fascinating direction. Treating motor control like language and reusing pretrained skills could unlock much more general, adaptable robotics. Excited to see where GPC goes. 👏

tokenize anything, then control anything

so is the environment fed in separately, or are the motion tokens supposed to capture that too?

That’s kind of insane

Now imagine a robot moving like this

It is the best i have ever seen

In this case, tokens are actions + position or position in and actions out

Is this basically like modular token packages for things like start, stop, mantle, jump, recenter balance, etc. ?

Pretrained controllers feel like the next step beyond task specific models

Pretraining is expanding far beyond language. Applying the same paradigm to motor control could be a major step toward more capable robotics.

Wow.

flips are just for fun?

Pattern matching. Turns out there's lots of patterns in nature.

Mesmerizing

Tokens define motion, Transformer predicts physics; motor control finally scales like language

Applying the GPT framework to motor skills is brilliant! Turning physical movements into discrete tokens is a massive paradigm shift for robotics and animation. Can't wait to see the full SIGGRAPH presentation!

600+ hours of motion is no joke thats impressive

Is that red laser money, power, sex, or love?

Wild

if this was real the robot would have thrown its hands in the air and stopped chasing after you did that slow shuffle of the target for the fifth or so time tokenize the laziness guys

so it's like a motion GPT but what's the token vocabulary size? feels like discretizing continuous movement is the hard part

The result is a reusable "generative controller" that can be fine-tuned for new tasks, just like how GPT models are pretrained then adapted.

fascinating approach using transformer models for motor control transferability sounds promising especially with 600+ hours of data in real-time physics simulations

cool move

wow

Imagine your pose data being input. You could put your hand out and the ai might intuit that you want a hand shake