You can build gui tools that will send messages back to codex (debugging visual issues in my math game)

finally the screenshot can explain itself. gui state back into codex is such an obvious missing loop.

yes! i love making custom ui's that talk to codex server!

How does sending messages to codex work? I tried this with Claude code but had to use a loop to essentially poll the app and pick up waiting messages

I'm told it uses the queue command?

It’s so good. I did this with a 3d rendering task. Just fly through the model of our property and shoot a gun to annotate where the model isn’t right and then codex fixes it

Yea thats how my visual debugger in makepad works as well

Lovely. Have you got a tutorial on how to do this?

is this publically availiable?

Love this. How does it work?

pinning the comment to one animation frame is huge. otherwise 'fix the arm' can somehow turn into a redesign of the entire character lol

this is great! doesn't it bother you to use a terminal for an increasingly gui world tho? it really grinds my gears that we're actively trying to reimplement modern gui in a 60-year-old environment lol

As long as I need to have the code open (90% for CSS and copywriting) i might as well use the terminal. I like the apps too but they're hogs

yeah i rebuilt a harness for myself to get rid of said hogs to get out of the terminal, so i just use zed for looking at code these days... i just find it wild that claude code has hover effects and dividers and other complex ui, while giving you janky text selection etc

The underrated detail: the tool talks back in Codex's format - element, frame, coordinates - not prose. Screenshots explain to humans; structured state makes the fix reproducible for the agent. Closing that loop both ways is what agent GUI tooling should standardize on.

Circling the broken pixel instead of typing a paragraph describing it is such an obvious win I am annoyed I did not think of it.

gui tools that talk back to the agent. debugging just got fun

Describing a UI bug to an agent can take longer than spotting it. I'd rather click the broken element and say "this should line up with that." Steve Ruiz's Codex demo makes me want interfaces that help you give the instruction.

Excuse my ignorance. Is this Tldraw offline or something separately built on top of the Tldraw SDK?

Something separate, just an ad hoc part of a different project