You can build gui tools that will send messages back to codex (debugging visual issues in my math…
@steveruizok61K followers
17,262 views
19 Comments
Comments from the original post on X (Twitter)

@aaapinen
finally the screenshot can explain itself. gui state back into codex is such an obvious missing loop.
1 like

@JungleSilicon
yes! i love making custom ui's that talk to codex server!
1 like

@hot_town
How does sending messages to codex work? I tried this with Claude code but had to use a loop to essentially poll the app and pick up waiting messages
1 like

@steveruizok
I'm told it uses the queue command?
2 likes

@jonas
It’s so good. I did this with a 3d rendering task. Just fly through the model of our property and shoot a gun to annotate where the model isn’t right and then codex fixes it
1 like

@rikarends
Yea thats how my visual debugger in makepad works as well
1 like

@lh________hl
Lovely. Have you got a tutorial on how to do this?

@nilukulasingham
is this publically availiable?

@a10s_code
Love this. How does it work?

@benmodev
pinning the comment to one animation frame is huge. otherwise 'fix the arm' can somehow turn into a redesign of the entire character lol

@blixt
this is great! doesn't it bother you to use a terminal for an increasingly gui world tho? it really grinds my gears that we're actively trying to reimplement modern gui in a 60-year-old environment lol

@steveruizok
As long as I need to have the code open (90% for CSS and copywriting) i might as well use the terminal. I like the apps too but they're hogs

@blixt
yeah i rebuilt a harness for myself to get rid of said hogs to get out of the terminal, so i just use zed for looking at code these days... i just find it wild that claude code has hover effects and dividers and other complex ui, while giving you janky text selection etc

@fgomezserna
The underrated detail: the tool talks back in Codex's format - element, frame, coordinates - not prose. Screenshots explain to humans; structured state makes the fix reproducible for the agent. Closing that loop both ways is what agent GUI tooling should standardize on.

@harshmanwanii
Circling the broken pixel instead of typing a paragraph describing it is such an obvious win I am annoyed I did not think of it.

@neel_sh_
gui tools that talk back to the agent. debugging just got fun

@beeendus
Describing a UI bug to an agent can take longer than spotting it. I'd rather click the broken element and say "this should line up with that." Steve Ruiz's Codex demo makes me want interfaces that help you give the instruction.

@Tenderbase_kw
Excuse my ignorance. Is this Tldraw offline or something separately built on top of the Tldraw SDK?

@steveruizok
Something separate, just an ad hoc part of a different project
1 like