low quant weights make the embedding model lose all discriminative power. I plotted the cosine correlation matrix of jina-v5, and one can see that low quant makes the model really blind. The off-diagonal similarities are pretty high on Q1/2/3, meaning everything looks similar in the semantic space. Q4 is... a sweet spot where model quality becomes acceptable.show more

once high order nonlinearity and discontinuous is truly understood, quantization is dead end.

Why not use a form of "WebP" Like compression instead of "JPEG"-like?

damn u r right!

Q8 is gonna become the new FLAC. It’s not gonna benefit most and yet some will swear by it.

the best visualization of quantization that i've seen

My guess is the results you are seeing has more to do with the specific quantization method. The type of quant you can run on GGUF are "toy" quantization, they are not sota, and do not reflect non-linear quant frameworks, there is an ocean out there

exactly why i stopped trusting q2 for anything that isn't a toy, just switched to llama.cpp with -q4_0 and the retrieval latency dropped while accuracy stayed solid honestly if your vectors look like soup you're either quanting too hard or using a model that was never meant to

Nice viz. Q4's the floor for embeddings. Below that, you're just burning compute on semantic noise.

I’m writing a paper about this right now. The effects of floating point non determinism on quantization

Love this visualization!

yeah low quant kills the separation, those matrices look rough!!

Would love to see this coloration on a logarithmic scale. In my personal experience the Q8_0 is a significantly more stable and generally higher quality experience.

Interesting. For me I see it like topographic map. 4-bit gives you 16 elevations.. that's where you can really start to contrast outliers from the near zeros etc.

very cool!

For me it would be very informative how the situation looks between q3_k_xl and something like iq4_k_xs.

Correlations between what?

Same 1k sentences -> diff quants weights -> 1k embeddings -> Cos(1k,1k)

A very intuitive demonstration, fantastic!

why would higher weight similarity imply 'blindness'? obviously at Q1 weights are either fully or zero correlated, and quant necessarily shifts toward higher correlation. Doesn't mean multiple non-linear stacked layers with high correlation don't retain useful patterns.