Full F16 precision 34B Code Llama at >20 t/s on M2 Ultra

"Wait, Georgi, how is this even possible?" you might ask. After all, the M2 Ultra only has 800GB/s bandwidth. Other people normally need 4 high-end GPUs to do this The answer is: Speculative Sampling

In this example we demonstrate unbiased F16 34B sampling with the help of a Q4 7B quantum "draft" model (Code Llama 7B) Individually, the speed of these models are: - F16 34B: ~10 t/s - Q4 7B: ~80 t/s However, in combination with speculative sampling we achieve ~20 t/s

The speed of course can vary depending on the content that is generated. But the approach seems to work quite well for code generation as most of the tokens are correctly guessed by the draft model Use cases with grammar sampling might also benefit significantly from this

Here is what a classic F16 sampling looks like without the speculative help

Here are a couple of more examples of speculative sampling

Meta should have release a couple of (1B and 3B) drafter models with the Code Llama release. Is it too late for them to train them or we have to wait for v2 🤔

Well done, Georgi! Is that the GPT-5 source code on your other tab? 😂

it's top secret 😉

This is incredible, things are happening so fast! I wonder if this is at all usable on an M1 🤔 Will mention on @thursdai_pod 👏

@dylan522p look at the GPU-Poor go!