GPT6 as policy, directly outputting ee-frame actions from two rgbd and instructions. Amazing perf. Should be ~100% succ rate given inf token and time for retrials. 1) "find a wooden box (can be obscured) and put it on the yellow sign." 2) "stack the wooden boxes on each other"

No base/world frame observation, no gripper feedback. Just plain visual understanding from two cams on the same wrist. Video speed 4x and 12x

Takeaway: 1. Closed-loop spatial understanding is off-the-shelf 2. Reasoning and actionability can be coupled without symbolising

No harness btw. Additional prompt generated by Claude to describe two cameras' mounting and action format.

I want to be completely honest, I don't get the big hype with using GPT6 as a control policy. Gemini Robotics was way, way faster and more fluid last year.

GPT 6 generally understand scenes better. Action quality also higher. The point is not to control the robot, is to investigate if it understands the scene/space.

@chris_j_paxton This is both amazing and very limited at the same time. There is no way this loop would be able to handle anything dynamic, i.e not quasi-static. I wonder what would be needed to have the gpt reasoning capabilities while still being able to handle something dynamic and fast.

How much did this cost?

40k tokens for active search one. 160k for stacking one given the accuracy needed.

@chris_j_paxton Very cool. Out of curiosity, have you tried older/non-astra models? My guess is they too would be able to do quasi static manip like this, as they know roughly what direction off center a block is in wrist cam image

Is it markovian? did you try more dexterous tasks like folding a box? we have been at 100% for these tasks for a while for these simple tasks but i would be surprised if it can fold boxes at >90%

No, but I think after some prompting/with some harness, yes you can do that zero-shot. Cost (tokens) can be very high though.

It can deliver individual skills, and it can chain them based on prompting. This might take 30min to execute though, but it's there.

Looks so great! Do you think GPT-based robot control requires a reliable low-level controller? For example, on a low-cost arm like ARX, a command like “move forward 1 cm” might actually move the end-effector 0.8–1.2 cm with some randomness. Do you think GPT can be robust to this kind of control noise?

I think in the middle. for non accurate movement it should tolerate. for more violent system errors no. But it also generalises well enough to recovery from any state, and can learn lessons from history

That’s impressive. Did it work first try?

Tuned a bit of configurations. For example, we find it doesn't know where gravity is pointing based on cam only, so we had to init from vertical pose. We also found 2 cams on the same wrist help a lot.

no harness. that surprises me

GPT-6’s end-to-end approach cuts the middleman. Fewer translation errors mean tighter action loops from RGBD inputs. That ~100% success rate feels inevitable with retry budgets eating up latency quirks. Makes sense, efficiency in motion beats polish on paper

@grok 更具体的解释一下这个，详细说明一下