Skip navigation
*Finally* read through Sam Rose's blog on LLM quantization.
0:00 / 0:23

*Finally* read through Sam Rose's blog on LLM quantization.

Ben Dicken

@BenjDicken49K followers

Visit on X
3,716 likesDownloadOriginal post

273,688 views

*Finally* read through Sam Rose's blog on LLM quantization. It's incredible. For many (even in tech) the understanding of how LLMs work stops at the surface level. Sam is helping us all go deeper, digging into the interesting facets of how AI models truly work. Read it!

33 Comments

Comments from the original post on X (Twitter)

  • Sam Rose's profile picture

    @samwhoo

    You’re a real one, Ben. Means a lot to me coming from you. 🫶

    48 likes

  • Somi's profile picture

    @somi_ai

    @samwhoo once you understand quantization properly you stop wasting time testing every quant variant. knowing why Q4_K_M works better than Q4_0 for certain tasks saves so many hours

    6 likes

  • Josh's profile picture

    @joshmanders

    @samwhoo That's a shame. Never heard of him before. Guess it'll stay that way.

    4 likes

  • Balvinder Kalon's profile picture

    @BalvinderKalon

    @samwhoo quantization is one of those topics where the gap between "I understand the concept" and "I understand what actually happens to my model's outputs" is massive. good deep dives like this are rare and genuinely change how you think about local inference tradeoffs.

    4 likes

  • Frosty40's profile picture

    @FrostForger

    @samwhoo

    4 likes

  • Mike J. | Future, Culture & HCD Insight's profile picture

    @mikuhigu55

    @samwhoo 흥미진진하군. 양자화라니.

    2 likes

  • Kev's profile picture

    @Kevrsub

    @samwhoo Thank you for sharing this. Just finished the quantization one. This is awesome @samwhoo

    2 likes

  • Imama's profile picture

    @Caffeinix_alche

    @samwhoo Loved the breakdown. Reading it was like swapping float32 for int8 in my brain: lighter, faster but still sharp where it matters.😀

    2 likes

  • Dystopic Winter's profile picture

    @dystopicwinter

    @samwhoo I've started digging into the fundamentals beyond the surface level. Its a whole separate world within our own.

    2 likes

  • Aditya's profile picture

    @obliviiou

    @samwhoo Really great article!!

    1 like

  • Emile Joseph's profile picture

    @EmilioisWorking

    @samwhoo Bookmarked. Required reading.

    1 like

  • Rich Stanbaugh's profile picture

    @stanbaugh

    Quantizing makes sense intuitively… we already apply a nonlinear ReLU or tanh at the neuron level to make outputs discrete. Having 16 or 32 bit coefficients is like have 15 digits on a calculator, trying to force precision that has already been lost. I think it’s more likely we have better outcomes trying multiple paths simultaneously; a kind of quantum / superposition approach.

    1 like

  • Omen's profile picture

    @Phenomenon_One

    @samwhoo kindly take a look at my blog, I believe we have a similar mission

    1 like

  • b04z's profile picture

    @b04zy

    @samwhoo very cool! why is this in ngrok blogs tho?

    1 like

  • Prithvi Jadwani | AI SEO | GEO | REDDIT SEO | GMB's profile picture

    @Prithvi_Jadwani

    @samwhoo yeah sam's been diving deep into the trenches of LLMs for ages, LLM quantization is just the tip of the iceberg 🤖

    1 like

  • Milton Assis's profile picture

    @MiltonAssisAI

    @samwhoo Excelente

    1 like

  • Akshat Kasera's profile picture

    @AkshatKasez

    @samwhoo Really a great blog on quantization. Loved the way of visual explanation!

    1 like

  • Thomas Tao's profile picture

    @Thomas_Tao_1

    @samwhoo Quantization is one of those rabbit holes. Gets subtle fast.

    1 like

  • Mohamed Anis's profile picture

    @Mohamedanis

    @samwhoo The builders who take time to truly understand what's under the hood, not just how to use the tools, but why they work, are the ones who'll build what others can't imagine.

    1 like

  • Brian Cheong's profile picture

    @briancheong

    @samwhoo Quantization is one of those topics where one good deep dive saves weeks of cargo-cult tuning. Bookmarking this.

    1 like