Загрузка видео...

Не удалось загрузить видео

На главную

During an agentic session, a quick “hi” after a coffee break can cost a full dollar. But it’s not because of the “hi.” It’s the million tokens of context you’re repaying for the moment your cache goes cold. Here's the caching math from Evan: 0:00 Why you're billed for...

arena's profile picture

Arena.ai

217,610 subscribers

51,372 просмотров • 4 дней назад •via X (Twitter)

Наука и технологии
LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

Interesting approach to long context/continual learning here from Baseten at Cursor Compact long trajectories by compressing a prefix of the KV cache using an MLP/autoencoder. You can train this "compactor" MLP by learning to reconstruct activations that the original KV cache would produce on subsequent tokens. This maximally reconstructs information from long context that's useful for subsequent outputs. If you run an agent over extremely long context and run this compaction recursively, the activations of this compressed KV cache become like trained weights. Similar to task-specific LoRA or "cartridges". Would not be surprised if OpenAI is running a similar algorithm for their blackbox compaction. Clear benefits here if you can avoid busting the cache, as e.g. compacting via writing to a text file would. Seeing many emerging approaches to fixed size latent states for LLMs that seem promising. If building a task-specific KV cache compression ends up being more sample efficient than running backprop, getting this to "work" feels like one of the 2-3 remaining breakthroughs on the path to true AGI
13:13

Interesting approach to long context/continual learning here from Baseten at Cursor Compact long trajectories by compressing a prefix of the KV cache using an MLP/autoencoder. You can train this "compactor" MLP by learning to reconstruct activations that the original KV cache would produce on subsequent tokens. This maximally reconstructs information from long context that's useful for subsequent outputs. If you run an agent over extremely long context and run this compaction recursively, the activations of this compressed KV cache become like trained weights. Similar to task-specific LoRA or "cartridges". Would not be surprised if OpenAI is running a similar algorithm for their blackbox compaction. Clear benefits here if you can avoid busting the cache, as e.g. compacting via writing to a text file would. Seeing many emerging approaches to fixed size latent states for LLMs that seem promising. If building a task-specific KV cache compression ends up being more sample efficient than running backprop, getting this to "work" feels like one of the 2-3 remaining breakthroughs on the path to true AGI

Jay Hack

29,792 просмотров • 1 месяц назад

THIS GUY AUDITED 926 CLAUDE CODE SESSIONS AND FOUND MOST OF THE TOKEN WASTE WAS ON HIS SIDE everyone is blaming anthropic for the limits, so he decided to actually look at the data 858 sessions, 18,903 turns, and $1,619 estimated spend across 33 days here's what he found: 1\ one default setting was burning 14,000 tokens per turn Claude Code loads the full JSON schema for every tool into context at session start. whether you use them or not. 20,000 tokens of tool definitions sitting there on every single turn. the fix: one line in your settings.json "ENABLE_TOOL_SEARCH": "true" context dropped from 45K to 20K instantly. across 858 sessions that one setting was wasting an estimated 264 million tokens 2\ cache expiry is the single biggest waste 54% of his turns came after a 5+ minute idle gap. every one of those turns re-processed the entire conversation at full price which caused a 10x cost jump you go grab coffee. come back 5 minutes later. type your next message. everything rebuilds from scratch. the context didn't change. you didn't change. the cache just expired. 12.3 million tokens wasted on idle gaps alone 3\ 42 skills loaded. 19 of them used twice or less across 858 sessions. every one of those skill schemas sat in context on every turn eating tokens for nothing. 4\ 1,122 redundant file reads where the same file was read 3+ times one session read the same file 33 times. he ALSO built a full token auditor dashboard that shows you exactly where your waste is coming from 19 charts, opens in your browser, free AND open source
0:20

THIS GUY AUDITED 926 CLAUDE CODE SESSIONS AND FOUND MOST OF THE TOKEN WASTE WAS ON HIS SIDE everyone is blaming anthropic for the limits, so he decided to actually look at the data 858 sessions, 18,903 turns, and $1,619 estimated spend across 33 days here's what he found: 1\ one default setting was burning 14,000 tokens per turn Claude Code loads the full JSON schema for every tool into context at session start. whether you use them or not. 20,000 tokens of tool definitions sitting there on every single turn. the fix: one line in your settings.json "ENABLE_TOOL_SEARCH": "true" context dropped from 45K to 20K instantly. across 858 sessions that one setting was wasting an estimated 264 million tokens 2\ cache expiry is the single biggest waste 54% of his turns came after a 5+ minute idle gap. every one of those turns re-processed the entire conversation at full price which caused a 10x cost jump you go grab coffee. come back 5 minutes later. type your next message. everything rebuilds from scratch. the context didn't change. you didn't change. the cache just expired. 12.3 million tokens wasted on idle gaps alone 3\ 42 skills loaded. 19 of them used twice or less across 858 sessions. every one of those skill schemas sat in context on every turn eating tokens for nothing. 4\ 1,122 redundant file reads where the same file was read 3+ times one session read the same file 33 times. he ALSO built a full token auditor dashboard that shows you exactly where your waste is coming from 19 charts, opens in your browser, free AND open source

Om Patel

299,611 просмотров • 4 месяцев назад

someone just rebuilt claude code in pure python and made it 230x cheaper to run it's called clawcodex. a full from-scratch port of the claude code agent loop. 230k+ lines of python, MIT licensed it keeps your request prefix byte-stable, so DeepSeek's prompt cache covers your entire system + tools + history on every turn cache-hit input then bills at ~$0.0435 per 1M tokens the same 1M tokens on Claude Fable 5 runs you $10 so the longer you code, the more the cache pays off what you actually get: → 25 LLM providers (Claude, GPT, DeepSeek, GLM, local ollama) → 1M token context window → real agent loop: tools, skills, REPL, session history → one-line install, mac/linux/wsl and it's sitting at 677 stars. basically nobody is looking yet bookmark this before it gone
0:26

someone just rebuilt claude code in pure python and made it 230x cheaper to run it's called clawcodex. a full from-scratch port of the claude code agent loop. 230k+ lines of python, MIT licensed it keeps your request prefix byte-stable, so DeepSeek's prompt cache covers your entire system + tools + history on every turn cache-hit input then bills at ~$0.0435 per 1M tokens the same 1M tokens on Claude Fable 5 runs you $10 so the longer you code, the more the cache pays off what you actually get: → 25 LLM providers (Claude, GPT, DeepSeek, GLM, local ollama) → 1M token context window → real agent loop: tools, skills, REPL, session history → one-line install, mac/linux/wsl and it's sitting at 677 stars. basically nobody is looking yet bookmark this before it gone

alex

158,729 просмотров • 19 дней назад

someone just rebuilt claude code in pure python and made it ~230x cheaper to run it's called clawcodex. a full from-scratch port of the claude code agent loop. 230k+ lines of python, MIT licensed it keeps your request prefix byte-stable, so DeepSeek's prompt cache covers your entire system + tools + history on every turn cache-hit input then bills at ~$0.0435 per 1M tokens the same 1M tokens on Claude Fable 5 runs you $10 so the longer you code, the more the cache pays off what you actually get: → 25 LLM providers (Claude, GPT, DeepSeek, GLM, local ollama) → 1M token context window → real agent loop: tools, skills, REPL, session history → one-line install, mac/linux/wsl and it's sitting at 677 stars. basically nobody is looking yet bookmark this before it gone
0:26

someone just rebuilt claude code in pure python and made it ~230x cheaper to run it's called clawcodex. a full from-scratch port of the claude code agent loop. 230k+ lines of python, MIT licensed it keeps your request prefix byte-stable, so DeepSeek's prompt cache covers your entire system + tools + history on every turn cache-hit input then bills at ~$0.0435 per 1M tokens the same 1M tokens on Claude Fable 5 runs you $10 so the longer you code, the more the cache pays off what you actually get: → 25 LLM providers (Claude, GPT, DeepSeek, GLM, local ollama) → 1M token context window → real agent loop: tools, skills, REPL, session history → one-line install, mac/linux/wsl and it's sitting at 677 stars. basically nobody is looking yet bookmark this before it gone

Argona

51,369 просмотров • 1 месяц назад

Anthropic and Andrew Ng built an agent that uses 90% fewer tokens from scratch: they dropped the entire book of Frankenstein into a prompt - 108,000 tokens asked one question the input dropped from 108,000 tokens to 11 here's how: step 1 → order matters: tools first, then system prompt, then your docs. anything that changes goes last step 2 → drop the breakpoint after the last stable block. everything above it gets stored step 3 → the match has to be byte-perfect. one extra space and you're back to full price step 4 → 5 minutes of idle and it's gone. but every hit restarts the timer step 5 → cache reads don't touch your rate limit. that's throughput you're not paying for 99% people never configure this - one afternoon and it pays for itself save this - the full 1-hour course is right below ↓
1:06:45

Anthropic and Andrew Ng built an agent that uses 90% fewer tokens from scratch: they dropped the entire book of Frankenstein into a prompt - 108,000 tokens asked one question the input dropped from 108,000 tokens to 11 here's how: step 1 → order matters: tools first, then system prompt, then your docs. anything that changes goes last step 2 → drop the breakpoint after the last stable block. everything above it gets stored step 3 → the match has to be byte-perfect. one extra space and you're back to full price step 4 → 5 minutes of idle and it's gone. but every hit restarts the timer step 5 → cache reads don't touch your rate limit. that's throughput you're not paying for 99% people never configure this - one afternoon and it pays for itself save this - the full 1-hour course is right below ↓

codila

85,671 просмотров • 17 дней назад

New blackboard lecture w Reiner Pope How do chips actually work – starting with basic logic gates, and working up to why GPUs, TPUs, FPGAs, and the human brain each look the way they do. 0:00:00 – Building a multiply-accumulate from logic gates 0:16:20 – Muxes and the cost of data movement 0:25:59 – How systolic arrays work 0:39:00 – Clock cycles and pipeline registers 0:51:40 – FPGAs vs ASICs 1:03:14 – Cache vs scratchpad 1:07:16 – Why CPU cores are much bigger than GPU cores 1:11:49 – Brains vs chips 1:15:22 – A GPU is just a bunch of tiny TPUs Look up Dwarkesh Podcast on YouTube/Spotify/etc to watch. Enjoy!
1:20:19

New blackboard lecture w Reiner Pope How do chips actually work – starting with basic logic gates, and working up to why GPUs, TPUs, FPGAs, and the human brain each look the way they do. 0:00:00 – Building a multiply-accumulate from logic gates 0:16:20 – Muxes and the cost of data movement 0:25:59 – How systolic arrays work 0:39:00 – Clock cycles and pipeline registers 0:51:40 – FPGAs vs ASICs 1:03:14 – Cache vs scratchpad 1:07:16 – Why CPU cores are much bigger than GPU cores 1:11:49 – Brains vs chips 1:15:22 – A GPU is just a bunch of tiny TPUs Look up Dwarkesh Podcast on YouTube/Spotify/etc to watch. Enjoy!

Dwarkesh Patel

936,901 просмотров • 3 месяцев назад

Your OpenClaw🦞 agent works perfectly for 20 minutes. Then it silently forgets your instructions and goes rogue. The fix isn't better prompts. It's understanding how memory actually works. I'm an OpenClaw maintainer. After 2+ months of daily use, I ended up with a memory system where my agent remembers decisions from weeks ago, checks its notes before acting, and picks up exactly where it left off after every restart: 0:00 The real reason your agent forgets 1:59 Quick start: 3 things that matter most 3:01 The 4 "memories" people confuse 4:52 3 failure modes: why your agent forgets 6:42 Compaction vs pruning 8:05 Prove what's loaded: /context list 10:06 What compaction actually does 13:42 Layer 1: pre-compaction memory flush 16:27 Layer 2: manual saves + /compact trick 19:12 Layer 3: the file architecture 23:22 The memory protocol for AGENTS.md 26:06 Retrieval: Track A, A+, and QMD by tobi lutke 31:59 Cost and cache 33:33 The complete config (copy and go) 36:35 5 things to remember
37:33

Your OpenClaw🦞 agent works perfectly for 20 minutes. Then it silently forgets your instructions and goes rogue. The fix isn't better prompts. It's understanding how memory actually works. I'm an OpenClaw maintainer. After 2+ months of daily use, I ended up with a memory system where my agent remembers decisions from weeks ago, checks its notes before acting, and picks up exactly where it left off after every restart: 0:00 The real reason your agent forgets 1:59 Quick start: 3 things that matter most 3:01 The 4 "memories" people confuse 4:52 3 failure modes: why your agent forgets 6:42 Compaction vs pruning 8:05 Prove what's loaded: /context list 10:06 What compaction actually does 13:42 Layer 1: pre-compaction memory flush 16:27 Layer 2: manual saves + /compact trick 19:12 Layer 3: the file architecture 23:22 The memory protocol for AGENTS.md 26:06 Retrieval: Track A, A+, and QMD by tobi lutke 31:59 Cost and cache 33:33 The complete config (copy and go) 36:35 5 things to remember

Radek Sienkiewicz

94,359 просмотров • 5 месяцев назад

I just crammed the updated Gemma 4 26B A4B QAT (MoE) with 180k context into an 8GB RTX 4060 (8 GB VRAM + 16 GB RAM only!!) and optimized the batch size. 23 tokens/sec decode, 300 tokens/sec prefill Yesterday I showed you a Gemma 4 31B dense model running flawlessly on an RTX 4090. Today, we're breaking the VRAM bank on a budget card using Unsloth’s new Gemma 4 26B (A4B) QAT quants. Following Google’s chat template update that boosted agentic benchmarks by +10%, I pushed this model to its absolute limits. Here is how you squeeze 250k context out of 8GB of VRAM. # The Setup & The Optimization - Hardware: Nvidia RTX 4060 (8GB VRAM) + 16GB System RAM - Environment: CUDA 13.0 build of llama.cpp - Model: gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf - Prompt: 28,000 tokens of prompt for each run If you read my L2 cache breakdown (attached in replies), you know the 4060’s 24MB cache maxes out at `-b 1024 -ub 1024`. Push past that, and prefill crashes. I locked those flags in for every test below to ensure maximum GEMM throughput. # 1. The Raw Context Push (Unquantized KV Cache) First, I wanted to see how far pure 8GB VRAM + 16GB RAM could stretch without touching the KV cache: - 80k Context: Prefill 385 t/s | Decode 25.5 t/s - 120k Context: Prefill 270 t/s | Decode 24 t/s llama.cpp flags: .\llama-server -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 120000 --port 8080 -ub 1024 -b 1024 Without KV quantization, 120k is your hard ceiling. push past that prefill throughput drops off a cliff, making the model practically unusable for large agentic workloads. # 2. The Q8 KV Cache Lifeline To survive 250k context on a budget card, you have to quantize the KV cache. I enabled 8 bit KV cache (`-ctk q8_0 -ctv q8_0`) and re ran: - 180k Context: Prefill 280 t/s | Decode 22.8 t/s - 250k Context: Prefill 115 t/s | Decode 20 t/s llama.cpp flags: .\llama-server -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 180000 --port 8080 -b 1024 -ub 1024 -ctk q8_0 -ctv q8_0 Result: Q8 KV cache brings 250k context back from the dead. Decode speed stabilizes at a highly usable 20 t/s. You are trading a very small bit amount of reasoning precision for an extra 130,000 tokens of context window. if you own a single rtx 3050, 3060, 3070, 4050, 4060, 5050 or 5060, you must try this model and optimize your batch size for higher prefill. Hugging Face links to the updated Unsloth's QAT quants and performance graph are in the replies below. What model are you running on your 6GB, 8GB or 12GB cards right now? Let's see your setups.
0:20

I just crammed the updated Gemma 4 26B A4B QAT (MoE) with 180k context into an 8GB RTX 4060 (8 GB VRAM + 16 GB RAM only!!) and optimized the batch size. 23 tokens/sec decode, 300 tokens/sec prefill Yesterday I showed you a Gemma 4 31B dense model running flawlessly on an RTX 4090. Today, we're breaking the VRAM bank on a budget card using Unsloth’s new Gemma 4 26B (A4B) QAT quants. Following Google’s chat template update that boosted agentic benchmarks by +10%, I pushed this model to its absolute limits. Here is how you squeeze 250k context out of 8GB of VRAM. # The Setup & The Optimization - Hardware: Nvidia RTX 4060 (8GB VRAM) + 16GB System RAM - Environment: CUDA 13.0 build of llama.cpp - Model: gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf - Prompt: 28,000 tokens of prompt for each run If you read my L2 cache breakdown (attached in replies), you know the 4060’s 24MB cache maxes out at `-b 1024 -ub 1024`. Push past that, and prefill crashes. I locked those flags in for every test below to ensure maximum GEMM throughput. # 1. The Raw Context Push (Unquantized KV Cache) First, I wanted to see how far pure 8GB VRAM + 16GB RAM could stretch without touching the KV cache: - 80k Context: Prefill 385 t/s | Decode 25.5 t/s - 120k Context: Prefill 270 t/s | Decode 24 t/s llama.cpp flags: .\llama-server -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 120000 --port 8080 -ub 1024 -b 1024 Without KV quantization, 120k is your hard ceiling. push past that prefill throughput drops off a cliff, making the model practically unusable for large agentic workloads. # 2. The Q8 KV Cache Lifeline To survive 250k context on a budget card, you have to quantize the KV cache. I enabled 8 bit KV cache (`-ctk q8_0 -ctv q8_0`) and re ran: - 180k Context: Prefill 280 t/s | Decode 22.8 t/s - 250k Context: Prefill 115 t/s | Decode 20 t/s llama.cpp flags: .\llama-server -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 180000 --port 8080 -b 1024 -ub 1024 -ctk q8_0 -ctv q8_0 Result: Q8 KV cache brings 250k context back from the dead. Decode speed stabilizes at a highly usable 20 t/s. You are trading a very small bit amount of reasoning precision for an extra 130,000 tokens of context window. if you own a single rtx 3050, 3060, 3070, 4050, 4060, 5050 or 5060, you must try this model and optimize your batch size for higher prefill. Hugging Face links to the updated Unsloth's QAT quants and performance graph are in the replies below. What model are you running on your 6GB, 8GB or 12GB cards right now? Let's see your setups.

Alok

36,617 просмотров • 1 месяц назад

🎉 Congrats to MiniMax (official) on releasing MiniMax M3! Frontier coding and agentic capabilities, native image and video input, computer use, and a 1M-token context window, all in a single open model. At the heart of M3 is MSA, a new sparse attention architecture: instead of attending densely over the full KV cache, each query scores 128-token KV blocks and runs attention only over the top blocks. That is what makes 1M-token context practical to serve. M3 runs in vLLM with day-0 support, verified on NVIDIA and AMD hardware: ✨ MSA sparse attention with dedicated prefill and decode kernels ✨ 1M-token context serving with prefix caching and chunked prefill ✨ BF16 and MXFP8 checkpoints, with MoE backends for both Hopper and Blackwell ✨ Native multimodal input (image + video) ✨ Tool calling, reasoning parsing, and thinking-mode control for agent workloads Day-0 support like this is a true team effort. Grateful to the teams at MiniMax (official), NVIDIA AI, AI at AMD, and Inferact, and to the vLLM community for making it happen. 🙏 Deep dive into the implementation, kernel work, and deployment recipes: 🔗
0:21

🎉 Congrats to MiniMax (official) on releasing MiniMax M3! Frontier coding and agentic capabilities, native image and video input, computer use, and a 1M-token context window, all in a single open model. At the heart of M3 is MSA, a new sparse attention architecture: instead of attending densely over the full KV cache, each query scores 128-token KV blocks and runs attention only over the top blocks. That is what makes 1M-token context practical to serve. M3 runs in vLLM with day-0 support, verified on NVIDIA and AMD hardware: ✨ MSA sparse attention with dedicated prefill and decode kernels ✨ 1M-token context serving with prefix caching and chunked prefill ✨ BF16 and MXFP8 checkpoints, with MoE backends for both Hopper and Blackwell ✨ Native multimodal input (image + video) ✨ Tool calling, reasoning parsing, and thinking-mode control for agent workloads Day-0 support like this is a true team effort. Grateful to the teams at MiniMax (official), NVIDIA AI, AI at AMD, and Inferact, and to the vLLM community for making it happen. 🙏 Deep dive into the implementation, kernel work, and deployment recipes: 🔗

vLLM

40,306 просмотров • 2 месяцев назад

The phone threshold is even harder than the storage number suggests. A phone exposes only part of its memory to an application, and the model must share that budget with its KV cache, activations, runtime, and the rest of the product. At 3.9 GB, 1-bit Bonsai 27B clears that threshold with room to work, making 27B-class local AI possible on a phone for the first time. (Demo Mode: cached & prefilled image context)
1:19

The phone threshold is even harder than the storage number suggests. A phone exposes only part of its memory to an application, and the model must share that budget with its KV cache, activations, runtime, and the rest of the product. At 3.9 GB, 1-bit Bonsai 27B clears that threshold with room to work, making 27B-class local AI possible on a phone for the first time. (Demo Mode: cached & prefilled image context)

PrismML

33,358 просмотров • 1 месяц назад

When Mudith Jayasekara and I met Gabe Pereyra, we were expecting just another vanilla intro call and instead had the best yarn about research, the state of LLMs, and where intelligence is actually heading. It's rare to meet a founder this deep in the weeds who's also building for one of the most important verticals in this new age of intelligence So it was awesome to sit down with Gabe for an extended discussion on what it take to build agents that can reliably complete work over hours, days, or even longer? We talked about why agents today struggle with search and long context windows and how techniques like KV-cache compaction, synthetic data, and continual learning could help. 0:00 Introduction 0:36 Getting legal agents to review the whole data room 2:08 Data rooms larger than any context window 5:28 How far open-source models can go 7:58 Where specialist models fit in legal AI 10:59 Training legal models when client data is off-limits 13:06 Teaching a model how a law firm works 13:59 What belongs in context vs. model weights 15:36 From firm-wide AI to a model for every lawyer 18:37 What training adds beyond retrieving the right cases 20:26 Why context windows have plateaued 24:01 How models could learn continuously on the job 26:12 Can AI recursively improve AI research? 27:07 Research agents can run experiments but not choose them 30:00 Why open-ended research is hard to train 33:47 Why deployment, not intelligence, is the bottleneck 35:08 The cost of frontier intelligence 36:59 Different neolabs, different paths to intelligence 39:26 Using open datasets to compare research methods 41:13 Conclusion
41:50

When Mudith Jayasekara and I met Gabe Pereyra, we were expecting just another vanilla intro call and instead had the best yarn about research, the state of LLMs, and where intelligence is actually heading. It's rare to meet a founder this deep in the weeds who's also building for one of the most important verticals in this new age of intelligence So it was awesome to sit down with Gabe for an extended discussion on what it take to build agents that can reliably complete work over hours, days, or even longer? We talked about why agents today struggle with search and long context windows and how techniques like KV-cache compaction, synthetic data, and continual learning could help. 0:00 Introduction 0:36 Getting legal agents to review the whole data room 2:08 Data rooms larger than any context window 5:28 How far open-source models can go 7:58 Where specialist models fit in legal AI 10:59 Training legal models when client data is off-limits 13:06 Teaching a model how a law firm works 13:59 What belongs in context vs. model weights 15:36 From firm-wide AI to a model for every lawyer 18:37 What training adds beyond retrieving the right cases 20:26 Why context windows have plateaued 24:01 How models could learn continuously on the job 26:12 Can AI recursively improve AI research? 27:07 Research agents can run experiments but not choose them 30:00 Why open-ended research is hard to train 33:47 Why deployment, not intelligence, is the bottleneck 35:08 The cost of frontier intelligence 36:59 Different neolabs, different paths to intelligence 39:26 Using open datasets to compare research methods 41:13 Conclusion

Charlie O'Neill

91,088 просмотров • 1 месяц назад

ANTHROPIC'S LEAD ENGINEER SHOWED HOW A LOOP AT $3.10/DAY DOES THE WORK OF A TEAM OF 4 a team of 4 cost $50,000/month - the loop costs $93 and does the same work even better Caching → Routing → Judge → Graph prompt caching cuts input costs by 90% - same prefix hits cache - you pay 10% of the price right model for the right step cuts 60% - easy step goes to Haiku at $0.01, hard step to Fable 5 at $0.15 Opus 5 grades every output at $0.02 per call - one caught mistake pays for a month of judging graph replaces re-asking - known fact goes to graph at $0 - the model never re-infers what the graph already knows save and read below - in 20 minutes your first loop is already running instead of your team
0:45

ANTHROPIC'S LEAD ENGINEER SHOWED HOW A LOOP AT $3.10/DAY DOES THE WORK OF A TEAM OF 4 a team of 4 cost $50,000/month - the loop costs $93 and does the same work even better Caching → Routing → Judge → Graph prompt caching cuts input costs by 90% - same prefix hits cache - you pay 10% of the price right model for the right step cuts 60% - easy step goes to Haiku at $0.01, hard step to Fable 5 at $0.15 Opus 5 grades every output at $0.02 per call - one caught mistake pays for a month of judging graph replaces re-asking - known fact goes to graph at $0 - the model never re-infers what the graph already knows save and read below - in 20 minutes your first loop is already running instead of your team

Noisy

22,454 просмотров • 26 дней назад

A tricky LLM interview question: You're serving a reasoning model on vLLM, and it keeps running out of GPU memory on long traces. So you add KV cache compression and evict 90% of the cached tokens. VRAM usage stays as is and GPU still runs out of memory. Why? (answer below) Evicting 90% of the KV cache can free almost none of the memory it was using. This sounds counterintuitive, but it follows directly from how production servers store the cache today. The KV cache grows with every token a model generates. Each token appends its key and value vectors across every layer, and nothing is freed while generation continues. This is the dominant memory cost for reasoning models. If a 32K-token CoT caches ~32K tokens of KV vectors, a Qwen3-32B with 4-bit weights will run out-of-memory around 24K tokens on a 24GB GPU. One obvious solution is to keep the important tokens and drop the rest, since attention is sparse enough to allow it. But this does not solve the memory problem yet. The reason is paged attention, which is the memory manager behind vLLM and most production servers. Under the hood, it splits GPU memory into fixed physical blocks, each one holds the KV for about 16 tokens. This block returns to the allocator only when every slot inside it is empty. Since the eviction logic selects tokens by importance, and such tokens are scattered across blocks... ...so despite eviction, almost every block is left with at least some survivor tokens. For instance, if the logic evicts 14k of 16k tokens across 1,000 blocks, most likely every block will still have a token. This means the allocator frees almost nothing. Placing the new tokens into those freed slots is not ideal because it breaks the cache's layout. Say token 16,001 arrives, and it's placed in the slot the 40th token used to hold. The cache now reads position 38, then 16,001, then 41, so the cache is no longer in token order. Attention can still compute the right answer from that, but only if every slot now carries a separate note recording which position it actually holds. This introduces another bookkeeping cost that an in-order layout inherently avoids. So the cache is logically 90% smaller and still physically the same size. Many compression results miss this because they measure on pre-allocated contiguous tensors rather than a paged server. There's another problem. Eviction methods pick which tokens to keep by looking at the attention scores themselves (as expected). But fast attention kernels used in production, like FlashAttention, never save those scores. They compute attention in small pieces and throw the full score grid away as they go, which is also why they're fast. So the exact signal eviction methods need isn't available in memory. The workaround is to fall back to eager attention and build the full matrix, which gives up the speed FlashAttention was there to provide. NVIDIA published a method called TriAttention to solve both these problems. It never needs attention scores. Instead, it scores tokens from the geometry of the model's key and query vectors before RoPE is applied, where those vectors sit in stable clusters. For the memory problem, it runs a compaction pass every 128 decoded tokens. The surviving tokens slide forward to close the holes eviction creates, so whole blocks empty out and return to the allocator while the cache stays in token order. On long reasoning traces, the approach matches full-attention accuracy while decoding 2.5x faster and using 10.7x less KV memory. KV cache compression is a big infrastructure problem. The number that decides whether it works is the count of freed blocks, not the count of evicted tokens. You can find the NVIDIA write-up here: I wrote a first-principles breakdown of how the KV cache works. It walks through why the model stores keys and values at all, why the cache grows with every token, and a comparison of LLM generation speed with and without KV caching. Read it below.
0:23

A tricky LLM interview question: You're serving a reasoning model on vLLM, and it keeps running out of GPU memory on long traces. So you add KV cache compression and evict 90% of the cached tokens. VRAM usage stays as is and GPU still runs out of memory. Why? (answer below) Evicting 90% of the KV cache can free almost none of the memory it was using. This sounds counterintuitive, but it follows directly from how production servers store the cache today. The KV cache grows with every token a model generates. Each token appends its key and value vectors across every layer, and nothing is freed while generation continues. This is the dominant memory cost for reasoning models. If a 32K-token CoT caches ~32K tokens of KV vectors, a Qwen3-32B with 4-bit weights will run out-of-memory around 24K tokens on a 24GB GPU. One obvious solution is to keep the important tokens and drop the rest, since attention is sparse enough to allow it. But this does not solve the memory problem yet. The reason is paged attention, which is the memory manager behind vLLM and most production servers. Under the hood, it splits GPU memory into fixed physical blocks, each one holds the KV for about 16 tokens. This block returns to the allocator only when every slot inside it is empty. Since the eviction logic selects tokens by importance, and such tokens are scattered across blocks... ...so despite eviction, almost every block is left with at least some survivor tokens. For instance, if the logic evicts 14k of 16k tokens across 1,000 blocks, most likely every block will still have a token. This means the allocator frees almost nothing. Placing the new tokens into those freed slots is not ideal because it breaks the cache's layout. Say token 16,001 arrives, and it's placed in the slot the 40th token used to hold. The cache now reads position 38, then 16,001, then 41, so the cache is no longer in token order. Attention can still compute the right answer from that, but only if every slot now carries a separate note recording which position it actually holds. This introduces another bookkeeping cost that an in-order layout inherently avoids. So the cache is logically 90% smaller and still physically the same size. Many compression results miss this because they measure on pre-allocated contiguous tensors rather than a paged server. There's another problem. Eviction methods pick which tokens to keep by looking at the attention scores themselves (as expected). But fast attention kernels used in production, like FlashAttention, never save those scores. They compute attention in small pieces and throw the full score grid away as they go, which is also why they're fast. So the exact signal eviction methods need isn't available in memory. The workaround is to fall back to eager attention and build the full matrix, which gives up the speed FlashAttention was there to provide. NVIDIA published a method called TriAttention to solve both these problems. It never needs attention scores. Instead, it scores tokens from the geometry of the model's key and query vectors before RoPE is applied, where those vectors sit in stable clusters. For the memory problem, it runs a compaction pass every 128 decoded tokens. The surviving tokens slide forward to close the holes eviction creates, so whole blocks empty out and return to the allocator while the cache stays in token order. On long reasoning traces, the approach matches full-attention accuracy while decoding 2.5x faster and using 10.7x less KV memory. KV cache compression is a big infrastructure problem. The number that decides whether it works is the count of freed blocks, not the count of evicted tokens. You can find the NVIDIA write-up here: I wrote a first-principles breakdown of how the KV cache works. It walks through why the model stores keys and values at all, why the cache grows with every token, and a comparison of LLM generation speed with and without KV caching. Read it below.

Avi Chawla

271,839 просмотров • 1 месяц назад

Here's my conversation with the founding team of Cursor, a popular code editor (based on VSCode) that specializes in AI-assisted programming. This is a super technical conversation that is bigger than just about one code editor. It's about the future of programming and, in general, the future of human-AI collaboration. It's here on X in full, and is up on everywhere else. Timestamps: 0:00 - Introduction 0:59 - Code editor basics 3:09 - GitHub Copilot 10:27 - Cursor 16:54 - Cursor Tab 23:08 - Code diff 31:20 - ML details 36:54 - GPT vs Claude 43:28 - Prompt engineering 50:54 - AI agents 1:04:51 - Running code in background 1:09:31 - Debugging 1:14:58 - Dangerous code 1:26:09 - Branching file systems 1:29:20 - Scaling challenges 1:43:32 - Context 1:48:39 - OpenAI o1 2:00:01 - Synthetic data 2:03:48 - RLHF vs RLAIF 2:05:34 - Fields Medal for AI 2:08:17 - Scaling laws 2:17:06 - The future of programming
2:28:55

Here's my conversation with the founding team of Cursor, a popular code editor (based on VSCode) that specializes in AI-assisted programming. This is a super technical conversation that is bigger than just about one code editor. It's about the future of programming and, in general, the future of human-AI collaboration. It's here on X in full, and is up on everywhere else. Timestamps: 0:00 - Introduction 0:59 - Code editor basics 3:09 - GitHub Copilot 10:27 - Cursor 16:54 - Cursor Tab 23:08 - Code diff 31:20 - ML details 36:54 - GPT vs Claude 43:28 - Prompt engineering 50:54 - AI agents 1:04:51 - Running code in background 1:09:31 - Debugging 1:14:58 - Dangerous code 1:26:09 - Branching file systems 1:29:20 - Scaling challenges 1:43:32 - Context 1:48:39 - OpenAI o1 2:00:01 - Synthetic data 2:03:48 - RLHF vs RLAIF 2:05:34 - Fields Medal for AI 2:08:17 - Scaling laws 2:17:06 - The future of programming

Lex Fridman

1,261,340 просмотров • 1 год назад

Microsoft ran its graph system against vector RAG on 8k, 120k, and a full million token context window, and the million token giant lost Jonathan Larson from Microsoft Research put up the numbers → LazyGraphRAG won 92, 90, and 91 percent of data-local questions, the exact place plain RAG was supposed to be strong → and did it at a tenth of the cost of the million token run the lesson buried in the benchmark → a bigger context window is not memory, structured memory is throwing more tokens at the problem was just the expensive way to be wrong open source on GitHub, the lesson a $1500 course would charge you for
14:48

Microsoft ran its graph system against vector RAG on 8k, 120k, and a full million token context window, and the million token giant lost Jonathan Larson from Microsoft Research put up the numbers → LazyGraphRAG won 92, 90, and 91 percent of data-local questions, the exact place plain RAG was supposed to be strong → and did it at a tenth of the cost of the million token run the lesson buried in the benchmark → a bigger context window is not memory, structured memory is throwing more tokens at the problem was just the expensive way to be wrong open source on GitHub, the lesson a $1500 course would charge you for

NO1ennn

29,184 просмотров • 20 дней назад

Anthropic engineer: "Harness engineering is a really rare skill right now - and it's hard to know if you're even good at it." how he actually thinks about building the full system: • step 1 → the model. already smarter than we use it. "freeze development - you'd find new capabilities for another 6-12 months" • step 2 → the harness. delete the steering. "we stopped using plan mode - the model thinks correctly" • step 3 → the agents. workflows let Claude build its own harness per task and spin sub-agents. what didn't fit in one context window now runs across many • step 4 → the context. "RAG is an anti-pattern. use grep." memory via files, clean context window, prompt caching. that's the stack most people optimize step 1. he optimizes all four. Watch the full video, then read the guide below on how to build an agentic system.
46:31

Anthropic engineer: "Harness engineering is a really rare skill right now - and it's hard to know if you're even good at it." how he actually thinks about building the full system: • step 1 → the model. already smarter than we use it. "freeze development - you'd find new capabilities for another 6-12 months" • step 2 → the harness. delete the steering. "we stopped using plan mode - the model thinks correctly" • step 3 → the agents. workflows let Claude build its own harness per task and spin sub-agents. what didn't fit in one context window now runs across many • step 4 → the context. "RAG is an anti-pattern. use grep." memory via files, clean context window, prompt caching. that's the stack most people optimize step 1. he optimizes all four. Watch the full video, then read the guide below on how to build an agentic system.

Codez

43,789 просмотров • 1 месяц назад

ANTHROPIC LEAKED A SETUP WHERE 6 FILES CUT TOKENS BY 84% AND RAISE ACCURACY BY 39% engineers at Google and Microsoft have been running this for a while - they pay $3 where you pay $20. decisions → contracts → dead ends → state → sources → open questions memory moves outside the window - what enters the context is a pointer to a file, not the history. raw history refills the window to 200k every single turn - six pointers take 8k and leave 96% free. same work, same five turns: 1,000,000 tokens against 40,000. decisions written to a file kill the endless "wait, why did we do it this way". written dead ends kill the second lap around the same circle - the agent never walks in there twice. a stable prefix keeps the cache warm - a cache read costs 0.1x of normal input. that's why 20 turns cost 3.15x instead of 20x - one write and nineteen reads. save this and paste it into Claude Code - and your agent stops paying for what it already knew ↓
0:51

ANTHROPIC LEAKED A SETUP WHERE 6 FILES CUT TOKENS BY 84% AND RAISE ACCURACY BY 39% engineers at Google and Microsoft have been running this for a while - they pay $3 where you pay $20. decisions → contracts → dead ends → state → sources → open questions memory moves outside the window - what enters the context is a pointer to a file, not the history. raw history refills the window to 200k every single turn - six pointers take 8k and leave 96% free. same work, same five turns: 1,000,000 tokens against 40,000. decisions written to a file kill the endless "wait, why did we do it this way". written dead ends kill the second lap around the same circle - the agent never walks in there twice. a stable prefix keeps the cache warm - a cache read costs 0.1x of normal input. that's why 20 turns cost 3.15x instead of 20x - one write and nineteen reads. save this and paste it into Claude Code - and your agent stops paying for what it already knew ↓

Noisy

137,288 просмотров • 11 дней назад

A single RTX 4090 (24 GB VRAM) can run the updated gemma 4 31B (dense) model with a 190,000 context window at 33 tokens/second. The VRAM barrier is dying. Google quietly updated Gemma 4, and Unsloth immediately compiled the new quants. I built llama.cpp from source on Ubuntu 22 to benchmark it. Google's stealth update 2 days ago enabled uniform Flash Attention 4 on Hopper to boost prefill and patched the chat template to improve tool calling. The agentic reasoning gains on the benchmark charts are massive: TB2 (Agents): +4.5% (to 25.8%) Tau2 (Telecom): +10.1% (to 62.7%) Running on Ubuntu 22, CUDA 13.0 with a single NVIDIA GeForce RTX 4090. Here is the exact step by step benchmarking process with a massive 28k tokens prompt and the commands I used to squeeze out maximum context without killing my throughput: # 1. The Baseline (Unquantized KV Cache) I started with full GPU offload (-ngl 99) and pushed the context to 40k. llama.cpp flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -ngl 99 -c 40000 -fa on --port 8080 -v VRAM: 23.8 GB (maxed out on card) Throughput: Prefill: 2198.81 t/s | Decode: 35.77 t/s (with 28k tokens prompt) # 2. The CPU Split Trap I tried stretching to 80k context by offloading layers to the CPU (-ngl 52). llama.cpp flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -c 80000 -ngl 52 -fa on --port 8080 -v Throughput: Prefill: 1212.73 t/s | Decode: 5 t/s (with 28k tokens prompt) # 3. The KV Quantization Breakthrough Instead of spilling layers to the CPU, I kept the model fully on card (-ngl 99) but enabled 8-bit KV cache quantization to free up VRAM. flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -c 100000 --cache-type-k q8_0 --cache-type-v q8_0 -ngl 99 --port 8080 -v VRAM: 23.9 GB Throughput: Prefill: 2139.68 t/s | Decode: 32 t/s (with 28k tokens prompt) Result: 100k tokens of context on a single GPU with practically zero speed loss (and minimal intelligence loss). # 4. The Limit Test (Q4 KV Cache) To find the absolute breaking point, I dropped the KV cache to 4 bit (q4_0) and set -c 190000. flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -c 190000 --cache-type-k q4_0 --cache-type-v q4_0 -ngl 99 --port 8080 -v VRAM: 23.8 GB Throughput: Prefill: 2206.66 t/s | Decode: 33 t/s (with 28k tokens prompt) (Note: Pushing it to 220k required dropping to -ngl 58 again, which immediately penalized decode down to 17 t/s). # The Tradeoff: For Max Reasoning: Keep your KV cache unquantized (f16). You get pristine reasoning but hit a strict 40k context ceiling. For Massive Document Retrieval: If you need to feed the model giant codebases, use --cache-type-k q4_0. Getting 190k context at 33 tokens/second on a consumer desktop with a 31b dense model is a cheat code. If you’re rocking a single 3090 or 4090 and slept on Gemma 4 earlier, this update is your cue to dust off the terminal. Hugging Face links to the Unsloth QAT quants are in the replies below.
0:20

A single RTX 4090 (24 GB VRAM) can run the updated gemma 4 31B (dense) model with a 190,000 context window at 33 tokens/second. The VRAM barrier is dying. Google quietly updated Gemma 4, and Unsloth immediately compiled the new quants. I built llama.cpp from source on Ubuntu 22 to benchmark it. Google's stealth update 2 days ago enabled uniform Flash Attention 4 on Hopper to boost prefill and patched the chat template to improve tool calling. The agentic reasoning gains on the benchmark charts are massive: TB2 (Agents): +4.5% (to 25.8%) Tau2 (Telecom): +10.1% (to 62.7%) Running on Ubuntu 22, CUDA 13.0 with a single NVIDIA GeForce RTX 4090. Here is the exact step by step benchmarking process with a massive 28k tokens prompt and the commands I used to squeeze out maximum context without killing my throughput: # 1. The Baseline (Unquantized KV Cache) I started with full GPU offload (-ngl 99) and pushed the context to 40k. llama.cpp flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -ngl 99 -c 40000 -fa on --port 8080 -v VRAM: 23.8 GB (maxed out on card) Throughput: Prefill: 2198.81 t/s | Decode: 35.77 t/s (with 28k tokens prompt) # 2. The CPU Split Trap I tried stretching to 80k context by offloading layers to the CPU (-ngl 52). llama.cpp flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -c 80000 -ngl 52 -fa on --port 8080 -v Throughput: Prefill: 1212.73 t/s | Decode: 5 t/s (with 28k tokens prompt) # 3. The KV Quantization Breakthrough Instead of spilling layers to the CPU, I kept the model fully on card (-ngl 99) but enabled 8-bit KV cache quantization to free up VRAM. flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -c 100000 --cache-type-k q8_0 --cache-type-v q8_0 -ngl 99 --port 8080 -v VRAM: 23.9 GB Throughput: Prefill: 2139.68 t/s | Decode: 32 t/s (with 28k tokens prompt) Result: 100k tokens of context on a single GPU with practically zero speed loss (and minimal intelligence loss). # 4. The Limit Test (Q4 KV Cache) To find the absolute breaking point, I dropped the KV cache to 4 bit (q4_0) and set -c 190000. flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -c 190000 --cache-type-k q4_0 --cache-type-v q4_0 -ngl 99 --port 8080 -v VRAM: 23.8 GB Throughput: Prefill: 2206.66 t/s | Decode: 33 t/s (with 28k tokens prompt) (Note: Pushing it to 220k required dropping to -ngl 58 again, which immediately penalized decode down to 17 t/s). # The Tradeoff: For Max Reasoning: Keep your KV cache unquantized (f16). You get pristine reasoning but hit a strict 40k context ceiling. For Massive Document Retrieval: If you need to feed the model giant codebases, use --cache-type-k q4_0. Getting 190k context at 33 tokens/second on a consumer desktop with a 31b dense model is a cheat code. If you’re rocking a single 3090 or 4090 and slept on Gemma 4 earlier, this update is your cue to dust off the terminal. Hugging Face links to the Unsloth QAT quants are in the replies below.

Alok

76,069 просмотров • 1 месяц назад

10 GitHub repos with ⭐ 450k+ combined stars that stop your agent from wasting tokens 1. caveman 98k⭐ - makes your AI agent talk like a caveman. Same answers, 65% fewer output tokens. 2. rtk 76k⭐ - CLI proxy that filters git/test/docker output before it hits your context. Up to 90% less bash noise. 3. mem0 63k⭐ - memory layer for AI agents. Stops your agent from re-explaining the same context every session. 4. context7 60.7k⭐ - pulls exact, up-to-date docs into context instead of your agent guessing and burning tokens on retries. 5. aider 48k⭐ - builds a map of your repo instead of loading the whole codebase into context. 6. code-review-graph 30k⭐ - graphs your codebase so your agent reads 2-3 files instead of the whole repo. Up to 376x fewer tokens per question. 7. serena 28k⭐ - gives agents symbol-level code access instead of reading full files line by line. 8. repomix 28k⭐ - compresses your entire repo into an AI-friendly format. ~70% token reduction. 9. ccusage 18k⭐ - shows you exactly where your Claude Code sessions are burning tokens. 10. GPTCache 8k⭐ - semantic cache for LLM calls. Stop paying twice for the same question. Save this before your next agent session burns through your subscription 👇
1:03

10 GitHub repos with ⭐ 450k+ combined stars that stop your agent from wasting tokens 1. caveman 98k⭐ - makes your AI agent talk like a caveman. Same answers, 65% fewer output tokens. 2. rtk 76k⭐ - CLI proxy that filters git/test/docker output before it hits your context. Up to 90% less bash noise. 3. mem0 63k⭐ - memory layer for AI agents. Stops your agent from re-explaining the same context every session. 4. context7 60.7k⭐ - pulls exact, up-to-date docs into context instead of your agent guessing and burning tokens on retries. 5. aider 48k⭐ - builds a map of your repo instead of loading the whole codebase into context. 6. code-review-graph 30k⭐ - graphs your codebase so your agent reads 2-3 files instead of the whole repo. Up to 376x fewer tokens per question. 7. serena 28k⭐ - gives agents symbol-level code access instead of reading full files line by line. 8. repomix 28k⭐ - compresses your entire repo into an AI-friendly format. ~70% token reduction. 9. ccusage 18k⭐ - shows you exactly where your Claude Code sessions are burning tokens. 10. GPTCache 8k⭐ - semantic cache for LLM calls. Stop paying twice for the same question. Save this before your next agent session burns through your subscription 👇

unicode

66,024 просмотров • 9 дней назад