New preprint alert 🚨 Can LLM agents develop video games? We release GameDevBench, the first benchmark evaluating agentic game development in a game engine, Godot. We also present two simple multimodal feedback mechanisms that lead to immediate performance gains. /🧵

Game development tasks are challenging. Gold patch solutions require triple the amount of lines of code and files changed compared to SWE-Bench. Assets are diverse and multimodal: GameDevBench includes 27 file types including shaders, sprites, and more, averaging 6.4 per task.

We find Gemini 3 flash in the Gemini CLI to be by far the most cost-effective model with second best performances and low costs. Claude Sonnet 4.5 and ChatGPT Codex 5.1 actually perform better in OpenHands than their own native agentic frameworks (claude code / codex). This performance, however, comes at increased costs.

To improve multimodal capabilities, we created an MCP server that allows the agent to view screenshots of the game editor or prompted the agent to record video of the game scene. These simple feedback mechanisms almost always improve performance. Different agents prefer screenshot vs video, but you can use both methods at the same time and agents are smart enough to pick their preferred method!

Agents struggle the most with multimodal tasks. Performance drops when tackling 2D graphics tasks (e.g., sprite animation creation) rather than gameplay oriented tasks (e.g., player controls).

Paper: Code: Of course, this is only possible thanks to my wonderful collaborators: Yixiong Fang, Arnav Yayavaram, Siddharth Yayavaram, @sethkarten, @qiuhongannawei, Runkun Chen, Alexander Wang, @valeriechen_, @chrisdonahuey @atalwalkar

@xeophon @stalkermustang @scaling01 you might want to check this out!

@xeophon @stalkermustang @scaling01 Yes. Definitely check it out!

@xeophon @stalkermustang @scaling01 Congrats!! It's an awesome bench and I hope you keep it alive for a long time. Love it 🫶

@xeophon @stalkermustang @scaling01 Thanks! We actually already have more data, environments, and tasks planned for this. We'll be rolling them out in the coming months. I plan to work on game dev and AI for a long time :)

awesome, we need more of this

thanks! There will indeed be more of this

good idea

AI + games = fun surprises!

game dev might be the final boss for “AGI”

Wow this is the benchmark that I care about. Great work.

thanks! please let me know if you have any questions

Have you run the benchmark on opus 4.5 / 4.6 or codex 5.3 yet? I know they just came out, but I'd love to see how they compare to Gemini 3 flash and the other models

Great work guys

thank you!

what is the most limiting factor for agents to develop games atm from your pov?

1. Agents don't understand / have out dated knowledge for godot (and I speculate other game engines). This will come with more training and focus from labs. 2. Agents are clearly still weaker on multimodal reasoning.

Fantastic work! I've been using opus 4.5 with godot for some time, but just with a simple MCP that allows it to start the game to see if it compiles. Does having video input of the screen actually work well? How long are the videos? I would expect that having a video input would blow up the context window?

It definitely increases cost so there's a tradeoff. Most of the time the model turns it into frames via python and just ingests a few images rather than the entire video.

Interesting. I think LLMs could become incredible at game development given the ability to ingest realtime video input whole providing realtime tool call outputs. That way they can "play" the game same as a human, but the current models seem far from being able to do that...

gamedevbench is sick. multimodal feedback for godot agents — what's the lag between agent action and visual state update? that'll be the bottleneck at scale

Today I created a game in gadot using GLM 5. Game similar to beltmatic. It aced it.

Why don't you come over and visit us over @GDevelopApp? We're open source, and we have already started integrating AI into our engine. Our documentation is being agent-optimized. Our built-in agent can already perform many tasks. LMK if you encounter blockers.

GameDevBench with actual Godot integration is wild. most agent benchmarks are synthetic tasks — this is shipping in a real engine. what's the success rate on agents handling GDScript vs visual scripting?

gamedevbench in godot is smart — engine constraints force you to actually solve problems vs infinite llm context window bloat. what's the success rate on playable builds?