Paving the way for unlimited context windows. Introducing ZML /attnd: sparse logarithmic attention, on CPU, over UDP, faster than GPU.

@zml_ai How do you avoid computing the attention values that end up ~0 without computing the dot products between each query-key pair? Are you using some sort of approximation?

@zml_ai we model it as a graph

@zml_ai How does modeling it as a graph let you bypass the N × N for sequence length of N, unless you're making some kind of approximation?

@zml_ai Yes there is approximation in the sense that we don't compute values very close to zero. From the talk:

@zml_ai How? How do you know which attention values *will be* very close to zero without first computing the dot-products?

@zml_ai Maybe what you mean to say is that you compute the full N x N attention matrix, but aggregate the value vectors sparsely by skipping entries where the attention is ~0? That would turn one dense-dense matmul into a sparse-dense one, but wouldn’t make the whole thing log(N), right?

@zml_ai Loved the talk & demo 🤩 @zml_ai rocks 🔥🚀❤️

@zml_ai @TheAhmadOsman looks like a great local llama unlock

@zml_ai Great stuff!! Btw love the T-shirt 🤣

@zml_ai Democratizing AI!!! 👌🏼

@zml_ai Good work man, holy shit

@zml_ai Very excited to share this. Attention is fascinating. I've been toying with this algorithm idea for some time but it took some faith and crunch to actually make it work.

@zml_ai any performance degradation?

@zml_ai no

@zml_ai How to run it in my m2 laptop??

@zml_ai hell yeah, based optimization for cpu guys! will help democratize AI adoption to the immense pool of non gpu socs let's launch/spotlight you on whenever it helps guys

@zml_ai This is wild. Can't wait to see the practical applications of this!

@zml_ai Fascinating approach to solving the context window limitation. CPU-based sparse attention over UDP opening new possibilities for distributed AI processing. Looking forward to seeing how this scales in production environments.

@zml_ai very cool

@zml_ai This breakthrough could really democratize AI. Less hardware, more access. Excited to see how it develops!

@zml_ai 🔥🔥🔥

@zml_ai cool stuff - however you need benchmarks on basic evals on running with CPU vs GPU, to really get this across the crowd/finish line

@zml_ai Outstanding will try this out on my old macbook pro 2017

@zml_ai Exciting innovation in attention technology!

@RemiCadene @zml_ai Amazing demo! Kudos!

@ylecun @zml_ai @grok eli5

@zml_ai Cool

@zml_ai congrats!

@zml_ai Where can we read about sparse logarithm attention?

@zml_ai Loved your CRdo writeup/thread......Do you think this eventually hurts the opical guys like CIEN and GLW? Thanks

@zml_ai 💥

@zml_ai @steeve Cool breakthrough on Attention D sparse log attention on CPU via UDP outpacing GPUs? Game changer for AI scaling. Echoes Megalodon style infinite contexts without heavy hardware. Excited for the code!

@zml_ai What model size will I be able to run with 16gb ram cpu without any gpu with good speed?

@zml_ai Exciting innovation in AI! Great step forward.

@zml_ai Super cool, congrats

@zml_ai Very cool, bold to use zig 👍 But bazel 🤣 please no!

@zml_ai

@zml_ai 👀

@zml_ai Boua very cool ! 🔥