Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Introducing SubQ - a major breakthrough in LLM intelligence. It is the first model built on a fully sub-quadratic sparse-attention architecture (SSA), And the first frontier model with a 12 million token context window which is: - 52x faster than FlashAttention at 1MM tokens - Less than 5% the... show more

Alexander Whedon

26,210 subscribers

13,496,370 views • 5 months ago •via X (Twitter)

Education News & Politics Science & Technology

Anya Rossi• Live Now

Private livecam show

70 Comments

Alexander Whedon5 months ago

SubQ is available for early access today, alongside our coding agent, SubQ Code Get access today ↓

Alexander Whedon5 months ago

We were a little slow on this, but we just got a technical blog post up with more details. Please take a look! We have a model card coming next week, and we are happy to take requests for any specific details there. I am happy to answer any questions here!

Martin Shkreli5 months ago

congrats!! why only 150tok/sec if 52x faster, etc.?

Alexander Whedon5 months ago

52x is for prefill speed. Decoding speedups are coming too! This is the first research announcement, with many to follow.

Chris B. Ward (e/bored) 🇬🇧🇮🇪🤐4 months ago

@MartinShkreli absolute grift, didn't deliver

Alexander Whedon4 months ago

@MartinShkreli Model card shared!

Vincent5 months ago

Any papers? Seems too good to be true

Alexander Whedon5 months ago

The model card is coming next week! We are releasing a technical blog post with more details later today.

Vincent5 months ago

Looking forward to reading it

shirish5 months ago

i still can't believe this.. 12M context window with 98% accuracy 💀 @alex_whedon @subquadratic

Alexander Whedon5 months ago

@subquadratic And at a fraction of the cost...

• nanou •5 months ago

If this works as described, it basically changes how we build with LLMs. A lot of current pipelines exist just to work around context limits.

Alexander Whedon5 months ago

And now that constraint goes away, what people choose to build is going to be very different.

Linus ✦ Ekenstam5 months ago

I remember when Gemini first made the claim of 1M in lab situations. We've come quite a bit from there. I'm extremely intrigued to see the real-world implications of this in bigger and more complex AI powered software. There is the zero shot environment, but there is also the qualitative repetitive work that is prone to drift and degradation over time. I really hope you will share case-studies, since I think this will be one of the most impressive ways to highlight the capabilities. Amazing work Alex and team. Kudos.

Alexander Whedon5 months ago

This is the framing we'd love to test against. If you've got a workload that historically drifts, would genuinely want you to throw it at SubQ and see what happens.

LADI ⟡⭐5 months ago

If SubQ only attends to a small subset of tokens, how does it guarantee recall of critical long range dependencies in worst case inputs (e.g adversarial prompts or tasks where the important tokens aren’t locally obvious)

Alexander Whedon5 months ago

We dynamically select the token relationships that matter and train it against retrieval problems with distractor docs, etc.

LADI ⟡⭐5 months ago

What happens when the selector misses the one token that matters? Any fallback

LADI ⟡⭐5 months ago

I agree, it’s about frequency, not perfection.

Veselin Stoyanov5 months ago

A few weeks ago I was in a discussion under an X post about LLMs, their adoption and cost. Most people argued it doesn't scale and brought up the usual economic concerns. My take was that it's only a matter of time before the next breakthrough. It's always been this way with tech. Congrats on this one 👏

Alexander Whedon5 months ago

This is only the first breakthrough we have announced! More coming.

Rohan Paul5 months ago

What 🤯 So Opus 4.7 is at $15 per million tokens and you guyes are giving it for under $1.50.

Cynical Optimist5 months ago

I hope you consider releasing some variants as open weights. This would change the game for private on-premise work.

Alexander Whedon5 months ago

This first model won't be open-source, but we do want to contribute to the open-source community!

Comrade Banana (ML/UT)5 months ago

@ChemPhysMajor Atleast sharing how you did it?

Cynical Optimist5 months ago

@CommiePat1776 @alex_whedon A white paper would go a long way.

anpaure5 months ago

close enough, welcome back brampton

Carl Vellotti 🥞5 months ago

The transformer was the first workable answer to long context. Everyone scaled it so hard nobody wanted to admit it was a local maximum. You finally found a way out.

Alexander Whedon5 months ago

Thanks @carlvellotti. Try it out at the team will appreciate your insights.

Poonam Soni5 months ago

5% of Anthropic’s price at 98% accuracy at 12M tokens. The math on every agent product being built right now just changed completely.

Alexander Whedon5 months ago

The unit economics of agentic products were quietly upside down. Well, not anymore.

Zephyr5 months ago

spent 2 years engineering around AI that couldn't read long docs. that limit just got removed.

Alexander Whedon5 months ago

Throw your hardest doc workload at it, and let me know how it performs. Check out

AshutoshShrivastava5 months ago

Long-context at under 10% of Anthropic's price damn…..

Alexander Whedon5 months ago

I mean, someone had to do it

Muratcan Koylan5 months ago

Release the API or it never happened

Allie K. Miller5 months ago

As someone who has worked with you for years, I am excited to see you finally releasing this. ✨ may your waitlist be buzzing and your tokens be cheap ✨

Alexander Whedon5 months ago

Thank you!

Danny Limanseta5 months ago

Is this real? Wow. Is there any evals where we see it with other forontier models?

Rezoan Ferdose5 months ago

Congrats to the @subquadratic team! It’s awesome to see someone finally breaking out of the standard transformer box

🅿️5 months ago

Being a SaaS investor in 2026 sounds like Dante’s Inferno

vas5 months ago

Wtf

Vincent Koc5 months ago

I love not wasting tokens, superb

Alexander Whedon5 months ago

Much of standard attention's compute is models talking to themselves about words that don't matter.

kanav5 months ago

- Less than 5% the cost of Opus TAKE MY MONEY

Morgan5 months ago

Super interesting Alexander, and 12m token context window, holy moly 🤯

Alexander Whedon5 months ago

52x faster than FlashAttention, <5% of Opus pricing

Morgan5 months ago

Totally insane.

Burny - Effective Curiosity5 months ago

"It is the first model built on a fully sub-quadratic sparse-attention architecture (SSA)" This doesn't seem right? That already exists?

Leon Lin5 months ago

this is literally epic wtf. ik another lab would make it

Ray Fernando5 months ago

Congrats on the announcement and I'm looking forward to giving this a try.

vxnuaj5 months ago

I’m sorry, wtf. Paper?

Alexander Whedon5 months ago

Next week!

vxnuaj5 months ago

paper?

Alexander Whedon4 months ago

This week!

Mahy4 months ago

@vxnuaj This paper is never gonna get released lol.

vxnuaj4 months ago

@alex_whedon patience young padawan.

vxnuaj4 months ago

@alex_whedon learn to have patience, you must

Dan McAteer5 months ago

ummm...this is a "BIG F*CKING DEAL"?

Alexander Whedon5 months ago

and it's available for early access at

Dan McAteer5 months ago

just submitted a request! would love to try it out and write a post about it.

Subah Wadhwani5 months ago

Was a pleasure collaborating w you guys over the last few weeks!

Devansh Tripathi5 months ago

congratulations it'll be interesting to see how it manages to avoid the quality cliff on a long context window

Alexander Whedon5 months ago

@NotTheCh05en1 Thanks Devansh, would love for you to try it out.

Paul Couvert5 months ago

Congrats on the launch. Can't believe we have a model with a context window this large with this accuracy!

Alexander Whedon5 months ago

Thanks @itsPaulAi! Would love for you try out the product & share any feedback

Csaba Kissi5 months ago

Finally, an LLM thats fast, powerful, and cheap at the same time. Opus is extremely expensive.

Alexander Whedon5 months ago

The tradeoff people learned to accept was "pick two of fast/cheap/smart." We changed that.

Nikhil N5 months ago

If someone said that enterprises are burning cash with AI subscriptions, that ends today. Great work @alex_whedon

Alexander Whedon5 months ago

No subquadratic tax!

Related Videos

Introducing SubQ Snap, the first long-context decision model for real enterprise work. Snap is the only decision model or LLM where compute per token stays constant as context grows. • 384K tokens of multimodal context, 12x larger than Jev. • 61.7 on Decision Index v0.2.1 vs Jev’s 57.9, outperforming on 13 of 16 enterprise benchmarks. •Processes 100K tokens and returns a decision in under one second. •$0.05/MM input tokens. Output is free. Available today to anyone. The first 10,000 sign-ups get 100M free input tokens. We’ll likely hit capacity fast and will add users as we add compute. Compute offers welcome 🙂. Latency will be variable as we tune the platform to this new model and category of users. Model speeds are 80ms for minimal context, 250ms for 32k, 660ms for 100k, and 2.5 seconds for 384k. Visit:

Introducing SubQ Snap, the first long-context decision model for real enterprise work. Snap is the only decision model or LLM where compute per token stays constant as context grows. • 384K tokens of multimodal context, 12x larger than Jev. • 61.7 on Decision Index v0.2.1 vs Jev’s 57.9, outperforming on 13 of 16 enterprise benchmarks. •Processes 100K tokens and returns a decision in under one second. •$0.05/MM input tokens. Output is free. Available today to anyone. The first 10,000 sign-ups get 100M free input tokens. We’ll likely hit capacity fast and will add users as we add compute. Compute offers welcome 🙂. Latency will be variable as we tune the platform to this new model and category of users. Model speeds are 80ms for minimal context, 250ms for 32k, 660ms for 100k, and 2.5 seconds for 384k. Visit:

Alexander Whedon

4,511,016 views • 16 days ago

Met a guy making $1.6 million/year as an LLM engineer. I asked him how he learned LLMs from scratch. He sent me the exact video that got him in. A 1 hour course on how LLMs actually work. He shows how transformers inside LLMs like ChatGPT & Claude are actually built. I watched it last night. Halfway through, I realized LLM architecture is way simpler than they make it look. Bookmark this and read the article below. • 00:00 - LLM foundations • 04:21 - LLM tokenization • 05:43 - LLMs vector embeddings • 22:16 - attention mechanism of LLM • 43:42 - LLM multi head attention

Met a guy making $1.6 million/year as an LLM engineer. I asked him how he learned LLMs from scratch. He sent me the exact video that got him in. A 1 hour course on how LLMs actually work. He shows how transformers inside LLMs like ChatGPT & Claude are actually built. I watched it last night. Halfway through, I realized LLM architecture is way simpler than they make it look. Bookmark this and read the article below. • 00:00 - LLM foundations • 04:21 - LLM tokenization • 05:43 - LLMs vector embeddings • 22:16 - attention mechanism of LLM • 43:42 - LLM multi head attention

Roan

81,846 views • 3 months ago

Over the past month, we completely switched to frontier LLMs to control a robot in a customer deployment. By doing this, we have a classic System 1/System 2 architecture, but on steroids. For the high-level reasoning layer, we use an agent based on Opus and Astra, which are many generations more capable than any VLA. For the fast execution layer, we use computer vision code written by the model, which is fast, editable, and reliable. As a bonus, we got in-context learning, which is just a basic feature of LLMs, and continual learning, since whenever a new situation arises, the large model steps in and adapts the pipeline. And it just feels better to be on the right side of the Bitter Lesson, benefiting from the billions being invested in frontier models around the world.

Over the past month, we completely switched to frontier LLMs to control a robot in a customer deployment. By doing this, we have a classic System 1/System 2 architecture, but on steroids. For the high-level reasoning layer, we use an agent based on Opus and Astra, which are many generations more capable than any VLA. For the fast execution layer, we use computer vision code written by the model, which is fast, editable, and reliable. As a bonus, we got in-context learning, which is just a basic feature of LLMs, and continual learning, since whenever a new situation arises, the large model steps in and adapts the pipeline. And it just feels better to be on the right side of the Bitter Lesson, benefiting from the billions being invested in frontier models around the world.

Igor Kulakov (MicroFactory)

92,956 views • 12 days ago

Andrej Karpathy just exposed how LLMs actually thinks: "LLMs don't want to succeed. They want to imitate." It has 80 transformer layers and spends the SAME compute on every single token as your brain In a 45-minute talk at Microsoft Build, Karpathy reveals the full psychology of LLMs. Worth more than any $500 prompting course you've seen on your timeline.

Andrej Karpathy just exposed how LLMs actually thinks: "LLMs don't want to succeed. They want to imitate." It has 80 transformer layers and spends the SAME compute on every single token as your brain In a 45-minute talk at Microsoft Build, Karpathy reveals the full psychology of LLMs. Worth more than any $500 prompting course you've seen on your timeline.

Morty

140,240 views • 3 months ago

Andrew Feldman, co-founder and CEO of Cerebras gives the best explanation of why Cerebras' wafer-scale architecture is 2,500X faster than a GPU during LLM inference. During inference, there are 2 stages: - pre-fill, where the model first processes the user's prompt, and - decode, where it generates the answer 1 token at a time in sequence. During that sequencial Decode phase, before each token is calculated the model weights have to be moved from memory into compute. On a GPU those weights are moved from HBM, while Cerebras keeps them in much faster SRAM spread across its very large wafer-scale processor, so the memory-to-compute movement that must happen for every token is about 2,500× faster ---- From The MAD Podcast with Matt Turck and Cerebras YouTube channel, (full video link in comment)

Andrew Feldman, co-founder and CEO of Cerebras gives the best explanation of why Cerebras' wafer-scale architecture is 2,500X faster than a GPU during LLM inference. During inference, there are 2 stages: - pre-fill, where the model first processes the user's prompt, and - decode, where it generates the answer 1 token at a time in sequence. During that sequencial Decode phase, before each token is calculated the model weights have to be moved from memory into compute. On a GPU those weights are moved from HBM, while Cerebras keeps them in much faster SRAM spread across its very large wafer-scale processor, so the memory-to-compute movement that must happen for every token is about 2,500× faster ---- From The MAD Podcast with Matt Turck and Cerebras YouTube channel, (full video link in comment)

Rohan Paul

39,999 views • 2 months ago

There’s a good chance your open source model is costing more than the frontier. Cheap tokens ≠ cheap tasks. Here, we introduce Intelligence Density, and Density Aware Training, our post-training technique to achieve less wasted compute, better learning, all with no knobs to tune. Enabled by default in every Trajectory model.

There’s a good chance your open source model is costing more than the frontier. Cheap tokens ≠ cheap tasks. Here, we introduce Intelligence Density, and Density Aware Training, our post-training technique to achieve less wasted compute, better learning, all with no knobs to tune. Enabled by default in every Trajectory model.

Trajectory

57,417 views • 1 month ago

Introducing Claude Opus 5.5, the first model in our new Claude 5.5 family. It performs at the level of Claude Fable 5.1 for most tasks, and costs 40% less to run than Opus 5.

Introducing Claude Opus 5.5, the first model in our new Claude 5.5 family. It performs at the level of Claude Fable 5.1 for most tasks, and costs 40% less to run than Opus 5.

Claude

29,367,781 views • 1 month ago

Announcing How Transformer LLMs Work, created with Jay Alammar and Maarten Grootendorst, co-authors of the beautifully illustrated book, “Hands-On Large Language Models.” This course offers a deep dive into the inner workings of the transformer architecture that powers large language models (LLMs). The transformer architecture revolutionized generative AI; in fact, the "GPT" in ChatGPT stands for "Generative Pre-Trained Transformer." Originally introduced in the Google Brain team's groundbreaking 2017 paper "Attention Is All You Need," by Vaswani and others, transformers were a highly scalable model for machine translation tasks. Variants of this architecture now power today’s LLMs such as those from OpenAI, Google, Meta, Cohere, Anthropic and DeepSeek. In this course, you’ll learn in detail how LLMs process text. You'll also work through code examples that illustrate that transformer's individual components. In details, you’ll learn: - How the representation of language has evolved, from Bag-of-Words to Word2Vec embeddings to the transformer architecture that captures a word's meanings taking into account the context of other words in the input. - How inputs are broken down into tokens before they are sent to the language model. - The details of a transformer's main stages: Tokenization and embedding, the stack of transformer blocks, and the language model head. - The inner workings of the transformer block, including attention, which calculates relevance scores, and the feedforward layer, which incorporates stored information learned in training. - How cached calculations make transformers faster. - Some of the most recent ideas in the latest models such as Mixture-of-Experts (MoE) which uses multiple sub-models and a router on each layer to improve the quality of LLMs. By the end of this course, you’ll have a deep understanding of how LLMs actually process text and be able to read through papers describing the latest models and understand the details. Gaining this intuition will improve your approach to building LLM applications. Please sign up here:

Announcing How Transformer LLMs Work, created with Jay Alammar and Maarten Grootendorst, co-authors of the beautifully illustrated book, “Hands-On Large Language Models.” This course offers a deep dive into the inner workings of the transformer architecture that powers large language models (LLMs). The transformer architecture revolutionized generative AI; in fact, the "GPT" in ChatGPT stands for "Generative Pre-Trained Transformer." Originally introduced in the Google Brain team's groundbreaking 2017 paper "Attention Is All You Need," by Vaswani and others, transformers were a highly scalable model for machine translation tasks. Variants of this architecture now power today’s LLMs such as those from OpenAI, Google, Meta, Cohere, Anthropic and DeepSeek. In this course, you’ll learn in detail how LLMs process text. You'll also work through code examples that illustrate that transformer's individual components. In details, you’ll learn: - How the representation of language has evolved, from Bag-of-Words to Word2Vec embeddings to the transformer architecture that captures a word's meanings taking into account the context of other words in the input. - How inputs are broken down into tokens before they are sent to the language model. - The details of a transformer's main stages: Tokenization and embedding, the stack of transformer blocks, and the language model head. - The inner workings of the transformer block, including attention, which calculates relevance scores, and the feedforward layer, which incorporates stored information learned in training. - How cached calculations make transformers faster. - Some of the most recent ideas in the latest models such as Mixture-of-Experts (MoE) which uses multiple sub-models and a router on each layer to improve the quality of LLMs. By the end of this course, you’ll have a deep understanding of how LLMs actually process text and be able to read through papers describing the latest models and understand the details. Gaining this intuition will improve your approach to building LLM applications. Please sign up here:

Andrew Ng

268,459 views • 1 year ago

You already gave the LLM the answer. So why is it writing it out from scratch? Introducing a new AI model architecture: Constrained Decoding Models (CDMs) LLMs were built to write. Most production workloads just need an answer: fast, consistent, and affordable. CDMs are purpose-built for 1 task each. They read your input once and skip the token-by-token generative loop that makes LLMs slow and expensive. Not just a decision model, but a family of 3 CDMs live today: • Classify: define your labels and get back a calibrated probability + reason for each. Used for routing, tagging, and intent detection. • Extract: describe the entities in natural language and get back well-formed JSON with confidence scores + citations. Used for web scraping, document processing, and PII redaction. • Summarize: pass in long text, images, or PDFs and get back a consistent summary under 1,000 tokens. Used for context compaction, metadata generation, and transcript digests. Built for production: • 1M tokens of multimodal (text + image) input, 31x larger than Jev • 77.2% on DecisionBench vs Jev's 72% • 200x faster, 467x cheaper than frontier LLM APIs • $0.05/M input tokens. Output is free (yes, even for summarize) Live on your favorite platforms today: OpenRouter Vercel Developers LiteLLM (YC W23) LangChain TrustedRouter Arthur RouterPlus (YC F26) Get 50M input tokens for free (for the first 50 hours) at link below 👀

You already gave the LLM the answer. So why is it writing it out from scratch? Introducing a new AI model architecture: Constrained Decoding Models (CDMs) LLMs were built to write. Most production workloads just need an answer: fast, consistent, and affordable. CDMs are purpose-built for 1 task each. They read your input once and skip the token-by-token generative loop that makes LLMs slow and expensive. Not just a decision model, but a family of 3 CDMs live today: • Classify: define your labels and get back a calibrated probability + reason for each. Used for routing, tagging, and intent detection. • Extract: describe the entities in natural language and get back well-formed JSON with confidence scores + citations. Used for web scraping, document processing, and PII redaction. • Summarize: pass in long text, images, or PDFs and get back a consistent summary under 1,000 tokens. Used for context compaction, metadata generation, and transcript digests. Built for production: • 1M tokens of multimodal (text + image) input, 31x larger than Jev • 77.2% on DecisionBench vs Jev's 72% • 200x faster, 467x cheaper than frontier LLM APIs • $0.05/M input tokens. Output is free (yes, even for summarize) Live on your favorite platforms today: OpenRouter Vercel Developers LiteLLM (YC W23) LangChain TrustedRouter Arthur RouterPlus (YC F26) Get 50M input tokens for free (for the first 50 hours) at link below 👀

Neal Patel

1,785,183 views • 15 days ago

New course: Transformers in Practice. You'll get a practical view of how transformer-based LLMs work, so you can reason about their behavior, diagnose problems like slow inference, and make smarter decisions about deployment. This course is built in partnership with AMD and taught by Sharon Zhou. You'll see how transformers generate text one token at a time, how the model decides which earlier words matter most when predicting the next one, and how techniques like quantization speed up inference on GPUs. This is not a video-only course; interactive visualizations throughout let you play with these concepts and build intuition that sticks. Skills you'll gain: - Understand why LLMs hallucinate, and RAG and chain-of-thought shape what they generate - Look inside the model to see how attention and layers combine to predict the next token - Diagnose inference bottlenecks and learn the techniques that speed up transformers on GPUs Join and understand what's really happening inside your LLMs:

New course: Transformers in Practice. You'll get a practical view of how transformer-based LLMs work, so you can reason about their behavior, diagnose problems like slow inference, and make smarter decisions about deployment. This course is built in partnership with AMD and taught by Sharon Zhou. You'll see how transformers generate text one token at a time, how the model decides which earlier words matter most when predicting the next one, and how techniques like quantization speed up inference on GPUs. This is not a video-only course; interactive visualizations throughout let you play with these concepts and build intuition that sticks. Skills you'll gain: - Understand why LLMs hallucinate, and RAG and chain-of-thought shape what they generate - Look inside the model to see how attention and layers combine to predict the next token - Diagnose inference bottlenecks and learn the techniques that speed up transformers on GPUs Join and understand what's really happening inside your LLMs:

Andrew Ng

126,461 views • 5 months ago

Introducing Cognee v1.0: a major breakthrough in agentic intelligence. It is 145% better than Opus 4.8 and GPT 5.5 at long context memory retrieval. Cognee allows a 100 BILLION token context window 100,000x more than Claude. It's: - 6.9x cheaper than GPT 5.5 and Opus 4.8 - Cold starts in 350ms & searches in 260ms Why this matters: Today agents forget important context, redo tasks, waste tokens, and slow down as workflows get more complex. Cognee solves this. It’s not a place to build agents. It connects to the agents you’ve already built, across any platform, and makes them significantly cheaper, faster, and more accurate. Here's how it works:

Introducing Cognee v1.0: a major breakthrough in agentic intelligence. It is 145% better than Opus 4.8 and GPT 5.5 at long context memory retrieval. Cognee allows a 100 BILLION token context window 100,000x more than Claude. It's: - 6.9x cheaper than GPT 5.5 and Opus 4.8 - Cold starts in 350ms & searches in 260ms Why this matters: Today agents forget important context, redo tasks, waste tokens, and slow down as workflows get more complex. Cognee solves this. It’s not a place to build agents. It connects to the agents you’ve already built, across any platform, and makes them significantly cheaper, faster, and more accurate. Here's how it works:

Vasilije

849,451 views • 4 months ago

Not every AI interaction is best served with a large frontier model. There is a long tail of trillion-token use cases, from tagging to search, best served by a sub-10b-parameter model that runs in milliseconds and costs many orders of magnitude less than the frontier. We built Freesolo Flash to make training loops like SFT and RL easy and end-to-end completable through your coding agent.

Not every AI interaction is best served with a large frontier model. There is a long tail of trillion-token use cases, from tagging to search, best served by a sub-10b-parameter model that runs in milliseconds and costs many orders of magnitude less than the frontier. We built Freesolo Flash to make training loops like SFT and RL easy and end-to-end completable through your coding agent.

Freesolo

29,435 views • 3 months ago

$A crypto project actually trained a 72B parameter AI model from scratch using decentralized GPU compute. Not fine-tuned, not a wrapper: trained from zero. The model benchmarks competitively against Meta's LLaMA 3 on reasoning tasks, and the entire training run cost a fraction of what centralized labs spend. If decentralized compute can produce frontier-class models, the moat around OpenAI and Anthropic is thinner than people think.$

A crypto project actually trained a 72B parameter AI model from scratch using decentralized GPU compute. Not fine-tuned, not a wrapper: trained from zero. The model benchmarks competitively against Meta's LLaMA 3 on reasoning tasks, and the entire training run cost a fraction of what centralized labs spend. If decentralized compute can produce frontier-class models, the moat around OpenAI and Anthropic is thinner than people think.

VirtualBacon

40,739 views • 6 months ago

A tricky LLM interview question: You're serving a reasoning model on vLLM, and it keeps running out of GPU memory on long traces. So you add KV cache compression and evict 90% of the cached tokens. VRAM usage stays as is and GPU still runs out of memory. Why? (answer below) Evicting 90% of the KV cache can free almost none of the memory it was using. This sounds counterintuitive, but it follows directly from how production servers store the cache today. The KV cache grows with every token a model generates. Each token appends its key and value vectors across every layer, and nothing is freed while generation continues. This is the dominant memory cost for reasoning models. If a 32K-token CoT caches ~32K tokens of KV vectors, a Qwen3-32B with 4-bit weights will run out-of-memory around 24K tokens on a 24GB GPU. One obvious solution is to keep the important tokens and drop the rest, since attention is sparse enough to allow it. But this does not solve the memory problem yet. The reason is paged attention, which is the memory manager behind vLLM and most production servers. Under the hood, it splits GPU memory into fixed physical blocks, each one holds the KV for about 16 tokens. This block returns to the allocator only when every slot inside it is empty. Since the eviction logic selects tokens by importance, and such tokens are scattered across blocks... ...so despite eviction, almost every block is left with at least some survivor tokens. For instance, if the logic evicts 14k of 16k tokens across 1,000 blocks, most likely every block will still have a token. This means the allocator frees almost nothing. Placing the new tokens into those freed slots is not ideal because it breaks the cache's layout. Say token 16,001 arrives, and it's placed in the slot the 40th token used to hold. The cache now reads position 38, then 16,001, then 41, so the cache is no longer in token order. Attention can still compute the right answer from that, but only if every slot now carries a separate note recording which position it actually holds. This introduces another bookkeeping cost that an in-order layout inherently avoids. So the cache is logically 90% smaller and still physically the same size. Many compression results miss this because they measure on pre-allocated contiguous tensors rather than a paged server. There's another problem. Eviction methods pick which tokens to keep by looking at the attention scores themselves (as expected). But fast attention kernels used in production, like FlashAttention, never save those scores. They compute attention in small pieces and throw the full score grid away as they go, which is also why they're fast. So the exact signal eviction methods need isn't available in memory. The workaround is to fall back to eager attention and build the full matrix, which gives up the speed FlashAttention was there to provide. NVIDIA published a method called TriAttention to solve both these problems. It never needs attention scores. Instead, it scores tokens from the geometry of the model's key and query vectors before RoPE is applied, where those vectors sit in stable clusters. For the memory problem, it runs a compaction pass every 128 decoded tokens. The surviving tokens slide forward to close the holes eviction creates, so whole blocks empty out and return to the allocator while the cache stays in token order. On long reasoning traces, the approach matches full-attention accuracy while decoding 2.5x faster and using 10.7x less KV memory. KV cache compression is a big infrastructure problem. The number that decides whether it works is the count of freed blocks, not the count of evicted tokens. You can find the NVIDIA write-up here: I wrote a first-principles breakdown of how the KV cache works. It walks through why the model stores keys and values at all, why the cache grows with every token, and a comparison of LLM generation speed with and without KV caching. Read it below.

A tricky LLM interview question: You're serving a reasoning model on vLLM, and it keeps running out of GPU memory on long traces. So you add KV cache compression and evict 90% of the cached tokens. VRAM usage stays as is and GPU still runs out of memory. Why? (answer below) Evicting 90% of the KV cache can free almost none of the memory it was using. This sounds counterintuitive, but it follows directly from how production servers store the cache today. The KV cache grows with every token a model generates. Each token appends its key and value vectors across every layer, and nothing is freed while generation continues. This is the dominant memory cost for reasoning models. If a 32K-token CoT caches ~32K tokens of KV vectors, a Qwen3-32B with 4-bit weights will run out-of-memory around 24K tokens on a 24GB GPU. One obvious solution is to keep the important tokens and drop the rest, since attention is sparse enough to allow it. But this does not solve the memory problem yet. The reason is paged attention, which is the memory manager behind vLLM and most production servers. Under the hood, it splits GPU memory into fixed physical blocks, each one holds the KV for about 16 tokens. This block returns to the allocator only when every slot inside it is empty. Since the eviction logic selects tokens by importance, and such tokens are scattered across blocks... ...so despite eviction, almost every block is left with at least some survivor tokens. For instance, if the logic evicts 14k of 16k tokens across 1,000 blocks, most likely every block will still have a token. This means the allocator frees almost nothing. Placing the new tokens into those freed slots is not ideal because it breaks the cache's layout. Say token 16,001 arrives, and it's placed in the slot the 40th token used to hold. The cache now reads position 38, then 16,001, then 41, so the cache is no longer in token order. Attention can still compute the right answer from that, but only if every slot now carries a separate note recording which position it actually holds. This introduces another bookkeeping cost that an in-order layout inherently avoids. So the cache is logically 90% smaller and still physically the same size. Many compression results miss this because they measure on pre-allocated contiguous tensors rather than a paged server. There's another problem. Eviction methods pick which tokens to keep by looking at the attention scores themselves (as expected). But fast attention kernels used in production, like FlashAttention, never save those scores. They compute attention in small pieces and throw the full score grid away as they go, which is also why they're fast. So the exact signal eviction methods need isn't available in memory. The workaround is to fall back to eager attention and build the full matrix, which gives up the speed FlashAttention was there to provide. NVIDIA published a method called TriAttention to solve both these problems. It never needs attention scores. Instead, it scores tokens from the geometry of the model's key and query vectors before RoPE is applied, where those vectors sit in stable clusters. For the memory problem, it runs a compaction pass every 128 decoded tokens. The surviving tokens slide forward to close the holes eviction creates, so whole blocks empty out and return to the allocator while the cache stays in token order. On long reasoning traces, the approach matches full-attention accuracy while decoding 2.5x faster and using 10.7x less KV memory. KV cache compression is a big infrastructure problem. The number that decides whether it works is the count of freed blocks, not the count of evicted tokens. You can find the NVIDIA write-up here: I wrote a first-principles breakdown of how the KV cache works. It walks through why the model stores keys and values at all, why the cache grows with every token, and a comparison of LLM generation speed with and without KV caching. Read it below.

Avi Chawla

273,899 views • 4 months ago

Kimi CEO Zhilin Yang: "a smart agent isn't enough - it has to run long Kimi 3 new attention makes the context 10x faster, so the agent never has to stop" here's how built it: step 1 → chase token efficiency, not compute - Muon optimizer, 2x more learning per token step 2 → make context cheap, not just long - new attention runs 10x faster, so the agent never stalls step 3 → let it run 200-300 steps alone - K2 already solves problems its own builders can't step 4 → give it a strong prior - the agent searches less and finds the answer faster step 5 → stop chasing a smarter model - chase one that never has to stop in 26-min talk is him revealing the exact architecture behind K3 and how to properly build agents on it watch & bookmark - start with the article below ↓

Kimi CEO Zhilin Yang: "a smart agent isn't enough - it has to run long Kimi 3 new attention makes the context 10x faster, so the agent never has to stop" here's how built it: step 1 → chase token efficiency, not compute - Muon optimizer, 2x more learning per token step 2 → make context cheap, not just long - new attention runs 10x faster, so the agent never stalls step 3 → let it run 200-300 steps alone - K2 already solves problems its own builders can't step 4 → give it a strong prior - the agent searches less and finds the answer faster step 5 → stop chasing a smarter model - chase one that never has to stop in 26-min talk is him revealing the exact architecture behind K3 and how to properly build agents on it watch & bookmark - start with the article below ↓

codila

12,037 views • 3 months ago

🚨 2.8 trillion parameters. Under 4GB VRAM. No quantization. An open source tool called AirLLM has changed how large language models run on consumer hardware. Instead of loading the entire model at once, it processes one layer at a time. This means your VRAM needs depend only on the size of a single layer rather than the full model. No quantization. No distillation. No pruning. → DeepSeek V3 (671B) runs on just 12GB → Llama 3.1 405B runs on 8GB → Kimi K3 (2.8 trillion parameters) runs on under 4GB → Works with nearly every major open source model The largest model often needs the least VRAM. Kimi K3 is a sparse mixture of experts model, so it only loads the specific experts activated by each token instead of an entire dense layer. This is why a 2.8 trillion parameter model can run in less memory than a 70B model. AirLLM makes frontier scale AI practical on everyday hardware without any of the usual tradeoffs. (Link in the comments)

🚨 2.8 trillion parameters. Under 4GB VRAM. No quantization. An open source tool called AirLLM has changed how large language models run on consumer hardware. Instead of loading the entire model at once, it processes one layer at a time. This means your VRAM needs depend only on the size of a single layer rather than the full model. No quantization. No distillation. No pruning. → DeepSeek V3 (671B) runs on just 12GB → Llama 3.1 405B runs on 8GB → Kimi K3 (2.8 trillion parameters) runs on under 4GB → Works with nearly every major open source model The largest model often needs the least VRAM. Kimi K3 is a sparse mixture of experts model, so it only loads the specific experts activated by each token instead of an entire dense layer. This is why a 2.8 trillion parameter model can run in less memory than a 70B model. AirLLM makes frontier scale AI practical on everyday hardware without any of the usual tradeoffs. (Link in the comments)

Brooks Whale X 🐋

40,529 views • 12 days ago

New course on serving LLMs efficiently -- how do you serve models to many concurrent users at low latency and reasonable cost? This short course is built with Red Hat and taught by Cedric Clyburn. Efficient LLM serving requires efficient memory management. A 70B-parameter model takes ~140 GB just to load the weights. On top of that, every active request needs its own chunk of GPU memory, the KV cache, to store the token context it has built up so far. In this course, you'll learn to reduce a model's memory footprint with quantization and serve it using vLLM, which handles many concurrent requests efficiently through smart memory management. Skills you'll gain: - Quantize a model and measure the accuracy tradeoff - Serve a model with vLLM and watch it handle concurrent requests efficiently - Benchmark your deployment and make informed tradeoffs between speed, cost, and accuracy Join and learn to serve LLMs efficiently:

New course on serving LLMs efficiently -- how do you serve models to many concurrent users at low latency and reasonable cost? This short course is built with Red Hat and taught by Cedric Clyburn. Efficient LLM serving requires efficient memory management. A 70B-parameter model takes ~140 GB just to load the weights. On top of that, every active request needs its own chunk of GPU memory, the KV cache, to store the token context it has built up so far. In this course, you'll learn to reduce a model's memory footprint with quantization and serve it using vLLM, which handles many concurrent requests efficiently through smart memory management. Skills you'll gain: - Quantize a model and measure the accuracy tradeoff - Serve a model with vLLM and watch it handle concurrent requests efficiently - Benchmark your deployment and make informed tradeoffs between speed, cost, and accuracy Join and learn to serve LLMs efficiently:

Andrew Ng

136,970 views • 4 months ago

Researchers found a way to make LLMs 8.5x faster! (without compromising accuracy) Speculative decoding is quite an effective way to address the single-token bottleneck in traditional LLM inference. A small "draft" model first generates the next several tokens, then the large model verifies all of them at once in a single forward pass. If a token at any position is wrong, you keep everything before it and restart from there. This never does worse than normal decoding. But current drafters in Speculative decoding still guess one token at a time. That makes the drafting step itself a bottleneck, capping real-world speedups at 2-3x. DFlash is a new technique that swaps the autoregressive drafter with a lightweight block diffusion model that guesses all tokens in one parallel shot. Drafting cost stays flat no matter how many tokens you speculate. On top of that, the drafter is conditioned on hidden features pulled from multiple layers of the target model and injected into every draft layer, so it makes significantly better guesses than a drafter working from scratch. In the side-by-side demo below, vanilla decoding runs at 48.5 tokens/sec. DFlash hits 415 tokens/sec on the same model, with zero quality loss. It's already integrated with vLLM, SGLang, and Transformers, with draft models on HuggingFace for several models like Qwen3, Qwen3.5, Llama 3.1, Kimi-K2.5, gpt-oss, and many more. I have shared the GitHub repo in the replies! KV caching is another must-know technique to boost LLM inference. I recently wrote an article about it. Read it below. 👉 Over to you: What use case are you working on that can benefit from this new technique?

Researchers found a way to make LLMs 8.5x faster! (without compromising accuracy) Speculative decoding is quite an effective way to address the single-token bottleneck in traditional LLM inference. A small "draft" model first generates the next several tokens, then the large model verifies all of them at once in a single forward pass. If a token at any position is wrong, you keep everything before it and restart from there. This never does worse than normal decoding. But current drafters in Speculative decoding still guess one token at a time. That makes the drafting step itself a bottleneck, capping real-world speedups at 2-3x. DFlash is a new technique that swaps the autoregressive drafter with a lightweight block diffusion model that guesses all tokens in one parallel shot. Drafting cost stays flat no matter how many tokens you speculate. On top of that, the drafter is conditioned on hidden features pulled from multiple layers of the target model and injected into every draft layer, so it makes significantly better guesses than a drafter working from scratch. In the side-by-side demo below, vanilla decoding runs at 48.5 tokens/sec. DFlash hits 415 tokens/sec on the same model, with zero quality loss. It's already integrated with vLLM, SGLang, and Transformers, with draft models on HuggingFace for several models like Qwen3, Qwen3.5, Llama 3.1, Kimi-K2.5, gpt-oss, and many more. I have shared the GitHub repo in the replies! KV caching is another must-know technique to boost LLM inference. I recently wrote an article about it. Read it below. 👉 Over to you: What use case are you working on that can benefit from this new technique?

Avi Chawla

158,352 views • 5 months ago

Sam Altman says the AI infrastructure race is not about bigger chatbots. It is about shifting the planet's intellectual output from human brains to AI brains. "If people knew what we could do with compute, they would want way, way more." "The thing I am personally most excited about is to use AI and lots of compute to discover new science." "A recent cool example here is we built the Sora Android app using Codex... in less than a month." "You can imagine that way much further, where entire companies can build their products using lots of compute." Then he gives the scale: "Let's say that an AI company today might be generating something on the order of 10 trillion tokens a day out of frontier models." "We're going to have these models at a company be outputting more tokens per day than all of humanity put together, and then 10 times that, and then 100 times that." The bottleneck is moving from software talent to compute supply, inference speed, product integration, and where all that machine intelligence gets pointed.

Sam Altman says the AI infrastructure race is not about bigger chatbots. It is about shifting the planet's intellectual output from human brains to AI brains. "If people knew what we could do with compute, they would want way, way more." "The thing I am personally most excited about is to use AI and lots of compute to discover new science." "A recent cool example here is we built the Sora Android app using Codex... in less than a month." "You can imagine that way much further, where entire companies can build their products using lots of compute." Then he gives the scale: "Let's say that an AI company today might be generating something on the order of 10 trillion tokens a day out of frontier models." "We're going to have these models at a company be outputting more tokens per day than all of humanity put together, and then 10 times that, and then 100 times that." The bottleneck is moving from software talent to compute supply, inference speed, product integration, and where all that machine intelligence gets pointed.

Karl Mehta

16,619 views • 4 months ago