Evolving dexterity with GPT-6 Astra 🖐️ Been trying Astra recently. Its zero-shot dexterous manipulation is already quite surprising. More interesting is seeing it learn and improve through simulation training, from pen spinning and Rubik's Cube to hammer use. The real goal would be to evolve this dexterity in the... real world.show more

nice results. How was Astra used here exactly?

For example, with Rubik’s cube, Astra either scripts its motion from one prompt, or builds the simulation/training and composes learned turn-and-roll skills. I mainly check the final rendering.

nice! always have the most aesthetic renders

The simulation progress looks promising. How much of the pen spinning or tool use transfers to a real hand when friction and object weight change?

I assume this simulation progress is just a small step toward complete sim2real transfer. Will try a real hand later. Should be fun!

Amazing result! Beautiful rendering! astra’s sense of aesthetics is much better than mine…

Hey, I went through the repo. I don't see any interface/harness for zero-shot control with Astra. I am curious how Astra exactly works with the pen-spinning task in zero-shot :) Could you provide more details on it?

Yep, this repo is only the final artifact, not a harness. For pen spinning, I validated this prompt: "There's a pen simulator here. A robot hand lies palm-up with a pen on its fingers. Spin it one full turn without dropping it, ending steady. No PPO." When I tried it before, Astra could also set up the sim environment itself from scratch, so providing the sim env is not essential.

That's pretty cool! Any idea how it did it (underlying mechanism)? Like, I am guessing it probably used CEM-style planning

Oh yeah, I could see the CEM implemented in the repo. Nice result :)

How exactly did it approach this? It feels like there are some shortcuts in there, with parts borrowed from previous work.

I assume the agent reused standard PPO impl and may have referred to open-source work for skills like pen spinning or cube manipulation. I didn't restrict that. The main work might be training skills for the specific embodiments/sim setups, then composing them.

A dropped pen is cheap feedback; a misplaced hammer strike isn't. Moving this dexterity from simulation onto hardware will make force limits and safe exploration just as important as the task objective.

what is actually being trained here?

The trained parts are the low-level primitive skills: pen spinning, three cube-rotation primitives, grasping/reorientation, etc. These primitives are not predefined either: the agent decides the skill decomposition itself, then trains the corresponding policies in simulation.

👏

i'm intrigued by the simulation training aspect, wondering how it'll translate to real-world dexterity applications

Cool results!