When systems see too much load, they sometimes never fully recover. These metastable failure states aren't esoteric or rare, and can be triggered on even very simple systems. Here's an example with single-node Postgres, and a very common query pattern:

Vertical scaling (like Aurora Serverless) and horizontal scaling (like Aurora DSQL) can help completely avoid these states, by providing extra capacity just when its needed.

Common benchmarks don't see them, because they tend to present a constant, closed load (i.e. a load that slows down when the server slows down). But the real world isn't much like that at all! Most of the systems we build are open (people keep arriving, even when service slows).

I'm just having fun animating stuff here. It's so cool how easy stuff like this is to visualize now. This example is an AI animation (thanks @kirodotdev) of a real example captured from real infrastructure.

I'd make recovery time an explicit benchmark result: inject a short overload, restore the previous arrival rate, and measure until latency and queue depth return to baseline. 'Handles N requests/sec' doesn't capture whether it can get back there after a spike.

I like those ideas. None of the common benchmarks do anything like that today. We need harder (more representative) benchmarks!

I would love to see this and other examples on an interactive site. Where we can change the input values and see how it all falls apart.

hysteresis!

Yeah this works a lot better when admission control is wired through the entire system. There is a tendency amongst database folks to just implement at that layer and call it done.

TIL that servers can burnout!

You can also just fix those bugs.