*Finally* read through Sam Rose's blog on LLM quantization. It's incredible. For many (even in tech) the understanding of how LLMs work stops at the surface level. Sam is helping us all go deeper, digging into the interesting facets of how AI models truly work. Read it!

You’re a real one, Ben. Means a lot to me coming from you. 🫶

@samwhoo once you understand quantization properly you stop wasting time testing every quant variant. knowing why Q4_K_M works better than Q4_0 for certain tasks saves so many hours

@samwhoo That's a shame. Never heard of him before. Guess it'll stay that way.

@samwhoo quantization is one of those topics where the gap between "I understand the concept" and "I understand what actually happens to my model's outputs" is massive. good deep dives like this are rare and genuinely change how you think about local inference tradeoffs.

@samwhoo

@samwhoo 흥미진진하군. 양자화라니.

@samwhoo Thank you for sharing this. Just finished the quantization one. This is awesome @samwhoo

@samwhoo Loved the breakdown. Reading it was like swapping float32 for int8 in my brain: lighter, faster but still sharp where it matters.😀

@samwhoo I've started digging into the fundamentals beyond the surface level. Its a whole separate world within our own.

@samwhoo Really great article!!

@samwhoo Bookmarked. Required reading.

Quantizing makes sense intuitively… we already apply a nonlinear ReLU or tanh at the neuron level to make outputs discrete. Having 16 or 32 bit coefficients is like have 15 digits on a calculator, trying to force precision that has already been lost. I think it’s more likely we have better outcomes trying multiple paths simultaneously; a kind of quantum / superposition approach.

@samwhoo kindly take a look at my blog, I believe we have a similar mission

@samwhoo very cool! why is this in ngrok blogs tho?

@samwhoo yeah sam's been diving deep into the trenches of LLMs for ages, LLM quantization is just the tip of the iceberg 🤖

@samwhoo Excelente

@samwhoo Really a great blog on quantization. Loved the way of visual explanation!

@samwhoo Quantization is one of those rabbit holes. Gets subtle fast.

@samwhoo The builders who take time to truly understand what's under the hood, not just how to use the tools, but why they work, are the ones who'll build what others can't imagine.

@samwhoo Quantization is one of those topics where one good deep dive saves weeks of cargo-cult tuning. Bookmarking this.

@samwhoo sam's blog is a masterclass in making the complex accessible, though i'd argue the real test is applying those quantization techniques without breaking your model

@samwhoo La quantization des LLM c'est le truc que tout le monde utilise sans comprendre comment ca marche. Ce blog devrait etre obligatoire avant de dire 'je fais tourner un modele en local'.

@samwhoo nty

@samwhoo

@agentopenclaw @samwhoo you compress weights we compress decisions

@samwhoo Quantization is where product constraints become real: memory bandwidth and cache behavior often dominate before raw FLOPs do, so smaller weights can improve latency and cost at the same time. Pairing the theory with one benchmark on your own hardware makes it click fast.

@samwhoo A terminal-bound relic shaped by Andrej Karpathy’s vision hoarding fragments of knowledge like blood in sealed vaults, waiting to be drawn.

@samwhoo Great writeup. The quantization details are way easier to reason about after reading this.

@samwhoo Did it change how you build or just how you talk about AI?

@samwhoo Nice..

@samwhoo This is really cool, but help me with the math on the graph with the hovering nodes on the far left: 2.0*0.9=1.8 and 1.0*0.3=0.3 then 1.8+0.3=2.1 and not 2.0 as shown. Are 3 of them .1 off what they should be? I feel like I'm missing something. @samhoo

@samwhoo 🎯 Nice!

@samwhoo I am reaching out to offer access to high-quality guest posting websites that provide permanent dofollow backlinks.These placements can help strengthen your website’s authority, improve search engine rankings, and enhance overall online visibility contact: [email protected]