Agents let us build systems for different workloads and requirements. But… can we trust what they build? We release 🌟SkySynth🌟: an engine for synthesizing high-performance, just-in-time (JIT) systems we can trust, by co-evolving formal proofs and tests alongside the code. Results: 💿 KV stores up to 2.3× faster than... Redis and FASTER + formally verified stores with 2.9× Claude Code's pass rate 🚏 Model routers up to 48% cheaper than a general router ⚡ Specialized inference engine with 2.2× the throughput of vLLM/SGLang 🧵👇show more

(2/N) General-purpose systems must support broad use cases: Redis and RocksDB span many storage patterns; vLLM and SGLang span many models, hardware configurations, and workloads. But a single deployment is usually much more specific. Why pay for generality you don’t need? Coding agents make specialization cheap enough to do just in time (JIT).

(3/N) Implementation is no longer the bottleneck. But can we trust what agents build? We found that agents are surprisingly good at finding gaps in the spec: ❌ a server dropped requests to increase throughput ❌ a KV store reconstructed values instead of storing them ❌ an inference engine emitted a fake first token to improve TTFT They looked fast, but weren’t doing what we intended. 😫

(4/N) This led to the core idea behind SkyDiscover-Synthesize (SkySynth): the specification must co-evolve with the implementation. As agents get better at optimizing the system, a fixed specification becomes easier to exploit. So SkySynth evolves both together: 🔐 formal proofs when requirements can be precisely specified 🧪 evolving tests and requirements when formal properties are not provided

(5/N) For correctness-critical systems, SkySynth uses formal-proof-driven synthesis. With Inductive Deductive Synthesis (IDS), agents generate the implementation and machine-checked proof together. Rather than building the full system first and verifying it later, code and proof evolve incrementally, with Lean/Rocq checking correctness along the way. On distributed KV stores, SkySynth synthesized verified implementations for 7/7 specifications, with a 95.2% pass rate vs. 33.3% for Claude Code.

(6/N) For systems whose requirements are difficult to formalize, SkySynth uses a multi-agent, test-driven synthesis loop to optimize both performance and correctness. Coding agents build and optimize implementations, while an auditor searches for reward hacks. When it finds one, it turns the failure into a new test and feeds it back into synthesis. So the implementation and specification co-evolve, toward systems that are both faster and harder to game.

(7/N) 💿 For key-value stores, SkySynth synthesized workload-specific designs with different caching, eviction, and logging strategies. They achieve up to 2.3× higher throughput than all other baselines like Redis, with far less reward hacking than Claude Code. It also synthesized formally verified distributed stores that reach a 95.2% pass rate, nearly 3× Claude Code.

(8/N) ⚡ For inference, SkySynth built a new engine from scratch for Qwen3-4B on an L4, specialized for a repeated-sampling workload with shared prefixes. By exploiting the workload structure, it achieved 2.2× the throughput of other baselines.

(9/N) 🚏 For model routing, different workloads led SkySynth to synthesize different routing policies. For an interactive assistant, the specialized router achieved 1.9× lower cost, +3.8% answer quality, and 1.6× fewer late responses than a general router.

(10/N) SkySynth expands the scope of automated discovery from: functions → algorithms → entire deployed systems.

(11/N) It is also fully open-source as part of SkyDiscover. You can use it today as a skill for coding agents. 🚀 Try building specialized systems for your own use cases! 💻 Code: 📝 Blog:

(12/N) Check out our paper on formal-proof-driven synthesis:

(13/N) And our paper on test-driven synthesis:

(14/14) This is a joint collaboration from Sky Computing lab with @shagarw21, @AlexKrentsel, @mertcemri, @sidzsan, @ziming_mao, @adityagp, @AlexGDimakis, @matei_zaharia, @istoica05 We also thank @abeirami, @audreyccheng, @bradenjhancock, Mohsen Lesani, @chunliang_tw, @RuiMeng_, @melissapan, @tomaspfister, @ponnapalli95, Sylvia Ratnasamy, @ShangyinT, @tianyin_xu, @Andy_ShuoYang, @0xlf_ for their helpful discussions and feedback. And we’re grateful to @LaudeInstitute for their support through the Slingshots program!

@AlexKrentsel @mertcemri @sidzsan @ziming_mao @adityagp @AlexGDimakis @matei_zaharia @istoica05 @abeirami @audreyccheng @bradenjhancock @chunliang_tw @RuiMeng_ (15/15) And additional thanks to @Ponnapalli95 for the feedback and support!

Just joking 😄 Consider another name: SkySmith?

This is the interesting part of agents building systems: getting the code to work is one thing, proving why we should trust it is another. Formal proofs handle the system side, but the knowledge and evidence behind agent decisions matter too.

how are the formal proofs handled when the generated kv store changes under jit specialization?