Mechanize's banner

Mechanize

@MechanizeWork • 14,243 subscribers

We build environments and evals for training and evaluating frontier coding agents.

Videos

Anya Rossi

sweetdream.ai

SweetDream.ai•Sponsored•Livecam

Watch Anya Live

Anya is streaming live right now! Join her private show and enjoy exclusive content.

Exclusive private shows

1.2k viewers online

Private Show

Join now for exclusive access

Free preview available • Premium content

We gave frontier AI coding agents 24 hours to write a complete Game Boy Advance emulator from scratch. GPT-5.5's emulator runs games best, with Claude Sonnet 4.6 and Opus 4.7 close behind. Gemini 3.1 Pro failed to produce a working emulator.

We gave frontier AI coding agents 24 hours to write a complete Game Boy Advance emulator from scratch. GPT-5.5's emulator runs games best, with Claude Sonnet 4.6 and Opus 4.7 close behind. Gemini 3.1 Pro failed to produce a working emulator.

96,192 次观看 • 2 个月前

Our new podcast on evals, with Max Niederman, Ege Erdil, and Stephen Yang. 0:00:00 – What's an eval, and how's it different from an RL environment? 0:19:33 – Why are models bad at building an emulator when the task is fully verifiable? 0:42:00 – How does training on bad data teach models to write terrible code? 1:04:00 – Why is continual learning still so bad? 2:25:24 – Why haven't software engineers been replaced when coding is basically solved? Listen to the Mechanize Podcast on YouTube, Spotify, etc. Enjoy!

Our new podcast on evals, with Max Niederman, Ege Erdil, and Stephen Yang. 0:00:00 – What's an eval, and how's it different from an RL environment? 0:19:33 – Why are models bad at building an emulator when the task is fully verifiable? 0:42:00 – How does training on bad data teach models to write terrible code? 1:04:00 – Why is continual learning still so bad? 2:25:24 – Why haven't software engineers been replaced when coding is basically solved? Listen to the Mechanize Podcast on YouTube, Spotify, etc. Enjoy!

39,106 次观看 • 1 个月前

We evaluated Gemini 3.5 Flash on GBA Eval. It could not build a working GBA emulator. On Piugba, the game just flashes on screen, unplayable and with no sound. Overall, it achieves a score of 6.7%.

We evaluated Gemini 3.5 Flash on GBA Eval. It could not build a working GBA emulator. On Piugba, the game just flashes on screen, unplayable and with no sound. Overall, it achieves a score of 6.7%.

52,659 次观看 • 2 个月前

If someone from the future gave us a trillion tokens of agents competently planning, executing, and adapting, we'd likely turn today's models into highly capable agents overnight. But we don't currently have that data. We at Mechanize are working to solve that.

If someone from the future gave us a trillion tokens of agents competently planning, executing, and adapting, we'd likely turn today's models into highly capable agents overnight. But we don't currently have that data. We at Mechanize are working to solve that.

27,890 次观看 • 11 个月前

没有更多内容可加载