*Finally* read through Sam Rose's blog on LLM quantization.
@BenjDicken49K followers
273,688 views
33 Comments
Comments from the original post on X (Twitter)

@samwhoo
You’re a real one, Ben. Means a lot to me coming from you. 🫶
48 likes

@somi_ai
@samwhoo once you understand quantization properly you stop wasting time testing every quant variant. knowing why Q4_K_M works better than Q4_0 for certain tasks saves so many hours
6 likes

@joshmanders
@samwhoo That's a shame. Never heard of him before. Guess it'll stay that way.
4 likes

@BalvinderKalon
@samwhoo quantization is one of those topics where the gap between "I understand the concept" and "I understand what actually happens to my model's outputs" is massive. good deep dives like this are rare and genuinely change how you think about local inference tradeoffs.
4 likes

@FrostForger
@samwhoo
4 likes

@mikuhigu55
@samwhoo 흥미진진하군. 양자화라니.
2 likes

@Kevrsub
@samwhoo Thank you for sharing this. Just finished the quantization one. This is awesome @samwhoo
2 likes

@Caffeinix_alche
@samwhoo Loved the breakdown. Reading it was like swapping float32 for int8 in my brain: lighter, faster but still sharp where it matters.😀
2 likes

@dystopicwinter
@samwhoo I've started digging into the fundamentals beyond the surface level. Its a whole separate world within our own.
2 likes

@obliviiou
@samwhoo Really great article!!
1 like

@EmilioisWorking
@samwhoo Bookmarked. Required reading.
1 like

@stanbaugh
Quantizing makes sense intuitively… we already apply a nonlinear ReLU or tanh at the neuron level to make outputs discrete. Having 16 or 32 bit coefficients is like have 15 digits on a calculator, trying to force precision that has already been lost. I think it’s more likely we have better outcomes trying multiple paths simultaneously; a kind of quantum / superposition approach.
1 like

@Phenomenon_One
@samwhoo kindly take a look at my blog, I believe we have a similar mission
1 like

@b04zy
@samwhoo very cool! why is this in ngrok blogs tho?
1 like

@Prithvi_Jadwani
@samwhoo yeah sam's been diving deep into the trenches of LLMs for ages, LLM quantization is just the tip of the iceberg 🤖
1 like

@MiltonAssisAI
@samwhoo Excelente
1 like

@AkshatKasez
@samwhoo Really a great blog on quantization. Loved the way of visual explanation!
1 like

@Thomas_Tao_1
@samwhoo Quantization is one of those rabbit holes. Gets subtle fast.
1 like

@Mohamedanis
@samwhoo The builders who take time to truly understand what's under the hood, not just how to use the tools, but why they work, are the ones who'll build what others can't imagine.
1 like

@briancheong
@samwhoo Quantization is one of those topics where one good deep dive saves weeks of cargo-cult tuning. Bookmarking this.
1 like













