Sequential Monte Carlo speculative decoding from Makora keeps multiple draft tokens alive in parallel instead of rewinding failed matches.

@makora_ai LLM inference is so memory-bound that drafting and scoring parallel particles is basically free compute-wise. Since it's on SGLang, it must be using tree KV caches to keep memory overhead sane.

@makora_ai @no_stp_on_snek

@makora_ai Nice gain on tokens per GPU. But in the field the ceiling is not the decoding trick, it is power and cooling per rack. You can double efficiency and still wait 3 years for interconnect. Algorithms scale faster than substations.