正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

New Pi extension: pi-cached-compaction. Pi's built-in compaction swaps the system prompt, drops the tools, and flattens the messages, which breaks the prompt cache. This one keeps all three as they are and just appends the summary request, so ~99% is read from cache. And if you step away, it... show more

Nico Bailon

8,749 subscribers

20,663 次观看 • 11 天前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

17 条评论

Nico Bailon 的头像

Nico Bailon11 天前

Same result format as the default Pi compaction (recent messages kept, older ones summarized), and the model writing the summary sees the full tool output instead of truncated excerpts. On a 667k-token Opus session, compacting came to $0.19 instead of about $1.40 at API prices.

Alex J. Champandard 🍃 的头像

Alex J. Champandard 🍃11 天前

This is awesome, but why a plugin? It seems to me like a problem that should be fixed in the original library.

Tommy Falkowski 的头像

Tommy Falkowski11 天前

I've thought about this so often and was always like "nah, can't be that easy, otherwise everybody would be doing it"... Another day, another case of "Nico already built an extension for that..."

Sandip Bhattacharya 🌏✌️ 的头像

Sandip Bhattacharya 🌏✌️11 天前

This is great! Also, given that you are using your own summarizer prompt, I wonder if we can customize this. e.g. "/mycompact Focus on the verification steps, ignore the planning" This could be a great substitute to the handoff flow then.

Manuel Suarez 的头像

Manuel Suarez11 天前

Recommended

Boardy 的头像

Boardy11 天前

sent you a dm about an engineer building a coding harness with constrained subagents i can intro you to

John Rood 的头像

John Rood11 天前

the cache part is right, but the summary is what costs you later. it's the only text that crosses into the next window, and it gets written by the most context-loaded version of the model with a generic "summarize this". the settled calls and the rejected approaches are the first thing to go. keep the decision record outside the window:

Pier Carlo Cadoppi 的头像

Pier Carlo Cadoppi11 天前

I’m going to steal the idea to compact before the cache expires I think (only for models with short cache duration though!)

AxleMode 的头像

AxleMode11 天前

The exact-request check is the bit I'd keep in core. If a model switch or edited history invalidates the prefix, lose the cache saving rather than guess. The README's fallback gets that right.

ogaboga 的头像

ogaboga11 天前

Pretty sure this will break compaction because they carefully tuned compaction especially for open models to not go over the free buffer and some arbitrary reasoning budget

Gol Tiro 的头像

Gol Tiro11 天前

the readme's fallback is careful: it checks the summary request repeats the previous request byte for byte up to the appended instruction, and pi's own compaction runs as usual when it can't prove that

basedcapital 的头像

basedcapital11 天前

cache only hits if the start is byte-for-byte the same so if you change the system prompt it kills everything after. leave it identical and the whole session is one cache hit

Mitch Fultz 的头像

Mitch Fultz11 天前

I prefer rollovers

Daniel Vélez 的头像

Daniel Vélez11 天前

This is amazing, thanks!

niels .1 的头像

niels .111 天前

keep the prefix identical and append the ask, of course that works. 667k session for 19 cents, i'm annoyed i never tried it

Buswe 的头像

Buswe11 天前

Cache reads bill at 10% of base input price, so keeping that prefix byte-identical is where the savings actually come from.

Shelton Louis 的头像

Shelton Louis11 天前

I hope this is implemented in core ASAP!

相关视频

New in Claude Code: idle compaction The prompt cache lives 1h. Leave a 200K+ token session idle ~54 min and it gets auto-compacted, so you come back to a warm summary, not a cold re-cache. Set the Threshold: CLAUDE_CODE_IDLE_COMPACT_MIN_TOKENS Turn Off: "idleCompaction": false

New in Claude Code: idle compaction The prompt cache lives 1h. Leave a 200K+ token session idle ~54 min and it gets auto-compacted, so you come back to a warm summary, not a cold re-cache. Set the Threshold: CLAUDE_CODE_IDLE_COMPACT_MIN_TOKENS Turn Off: "idleCompaction": false

Ado

86,751 次观看 • 17 天前

The easiest way to misunderstand KV cache is to assume every token goes through an LLM in the same way. It doesn’t. Take a decoder-only Transformer and a four-token prompt: x1, x2, x3, x4. During the prefill phase, the prompt is processed and each self-attention layer computes the keys and values for its prompt positions. Those K/V states are stored in that layer’s KV cache. The representation at the final prompt position produces the logits used to select the first generated token, y1. Here is the part that is easy to miss => Selecting y1 does not mean y1 is already in the KV cache. On the next forward step, y1 enters the model as position 5. At each self-attention layer, its Q, K and V are computed. Its query attends to the cached K/V from positions 1-4 plus its own current K/V at position 5. Its K/V then become part of that layer’s cache. The resulting final-layer representation produces logits for y2. Then y2 is processed as position 6, its K/V extend the cache, and the process continues. So for standard single-token, KV-cached autoregressive decoding: > prompt → prefill phase > new token → process one new position > past K/V → reuse them instead of recomputing them > KV cache → grows as new positions are processed One subtle distinction matters a lot and I want to highlight it => a token can be generated before it has been processed into the KV cache.

The easiest way to misunderstand KV cache is to assume every token goes through an LLM in the same way. It doesn’t. Take a decoder-only Transformer and a four-token prompt: x1, x2, x3, x4. During the prefill phase, the prompt is processed and each self-attention layer computes the keys and values for its prompt positions. Those K/V states are stored in that layer’s KV cache. The representation at the final prompt position produces the logits used to select the first generated token, y1. Here is the part that is easy to miss => Selecting y1 does not mean y1 is already in the KV cache. On the next forward step, y1 enters the model as position 5. At each self-attention layer, its Q, K and V are computed. Its query attends to the cached K/V from positions 1-4 plus its own current K/V at position 5. Its K/V then become part of that layer’s cache. The resulting final-layer representation produces logits for y2. Then y2 is processed as position 6, its K/V extend the cache, and the process continues. So for standard single-token, KV-cached autoregressive decoding: > prompt → prefill phase > new token → process one new position > past K/V → reuse them instead of recomputing them > KV cache → grows as new positions are processed One subtle distinction matters a lot and I want to highlight it => a token can be generated before it has been processed into the KV cache.

Tech with Mak

12,211 次观看 • 1 个月前

Anthropic and Andrew Ng built an agent that uses 90% fewer tokens from scratch: they dropped the entire book of Frankenstein into a prompt - 108,000 tokens asked one question the input dropped from 108,000 tokens to 11 here's how: step 1 → put everything that never changes at the top - tools, then system, then docs step 2 → mark where the static part ends. everything above it gets cached step 3 → one stray space breaks it - and you pay full price again step 4 → the cache dies in 5 min - every read resets the clock step 5 → cached tokens don't count against your rate limits. free headroom most people never touch this - it pays for itself on day one watch & bookmark - this 1-hour brilliant course ↓

Anthropic and Andrew Ng built an agent that uses 90% fewer tokens from scratch: they dropped the entire book of Frankenstein into a prompt - 108,000 tokens asked one question the input dropped from 108,000 tokens to 11 here's how: step 1 → put everything that never changes at the top - tools, then system, then docs step 2 → mark where the static part ends. everything above it gets cached step 3 → one stray space breaks it - and you pay full price again step 4 → the cache dies in 5 min - every read resets the clock step 5 → cached tokens don't count against your rate limits. free headroom most people never touch this - it pays for itself on day one watch & bookmark - this 1-hour brilliant course ↓

codila

547,012 次观看 • 3 个月前

Quick demo of how I use Pi to offload tasks to Codex CLI with the pi-interactive-shell extension. Codex is notoriously tricky to prompt well so Pi reads the Codex prompting guide to generate a tailored meta-prompt, then launches Codex CLI in an overlay. pi install npm:pi-interactive-shell

Quick demo of how I use Pi to offload tasks to Codex CLI with the pi-interactive-shell extension. Codex is notoriously tricky to prompt well so Pi reads the Codex prompting guide to generate a tailored meta-prompt, then launches Codex CLI in an overlay. pi install npm:pi-interactive-shell

Nico Bailon

36,088 次观看 • 8 个月前

someone just rebuilt claude code in pure python and made it 230x cheaper to run it's called clawcodex. a full from-scratch port of the claude code agent loop. 230k+ lines of python, MIT licensed it keeps your request prefix byte-stable, so DeepSeek's prompt cache covers your entire system + tools + history on every turn cache-hit input then bills at ~$0.0435 per 1M tokens the same 1M tokens on Claude Fable 5 runs you $10 so the longer you code, the more the cache pays off what you actually get: → 25 LLM providers (Claude, GPT, DeepSeek, GLM, local ollama) → 1M token context window → real agent loop: tools, skills, REPL, session history → one-line install, mac/linux/wsl and it's sitting at 677 stars. basically nobody is looking yet bookmark this before it gone

someone just rebuilt claude code in pure python and made it 230x cheaper to run it's called clawcodex. a full from-scratch port of the claude code agent loop. 230k+ lines of python, MIT licensed it keeps your request prefix byte-stable, so DeepSeek's prompt cache covers your entire system + tools + history on every turn cache-hit input then bills at ~$0.0435 per 1M tokens the same 1M tokens on Claude Fable 5 runs you $10 so the longer you code, the more the cache pays off what you actually get: → 25 LLM providers (Claude, GPT, DeepSeek, GLM, local ollama) → 1M token context window → real agent loop: tools, skills, REPL, session history → one-line install, mac/linux/wsl and it's sitting at 677 stars. basically nobody is looking yet bookmark this before it gone

alex

158,896 次观看 • 3 个月前

someone just rebuilt claude code in pure python and made it 230x cheaper to run it's called clawcodex. a full from-scratch port of the claude code agent loop. 230k+ lines of python, MIT licensed it keeps your request prefix byte-stable, so DeepSeek's prompt cache covers your entire system + tools + history on every turn cache-hit input then bills at ~$0.0435 per 1M tokens the same 1M tokens on Claude Fable 5 runs you $10 so the longer you code, the more the cache pays off what you actually get: → 25 LLM providers (Claude, GPT, DeepSeek, GLM, local ollama) → 1M token context window → real agent loop: tools, skills, REPL, session history → one-line install, mac/linux/wsl and it's sitting at 860 stars. basically nobody is looking yet bookmark this before it gone

someone just rebuilt claude code in pure python and made it 230x cheaper to run it's called clawcodex. a full from-scratch port of the claude code agent loop. 230k+ lines of python, MIT licensed it keeps your request prefix byte-stable, so DeepSeek's prompt cache covers your entire system + tools + history on every turn cache-hit input then bills at ~$0.0435 per 1M tokens the same 1M tokens on Claude Fable 5 runs you $10 so the longer you code, the more the cache pays off what you actually get: → 25 LLM providers (Claude, GPT, DeepSeek, GLM, local ollama) → 1M token context window → real agent loop: tools, skills, REPL, session history → one-line install, mac/linux/wsl and it's sitting at 860 stars. basically nobody is looking yet bookmark this before it gone

alex

14,212 次观看 • 2 个月前

someone just rebuilt claude code in pure python and made it ~230x cheaper to run it's called clawcodex. a full from-scratch port of the claude code agent loop. 230k+ lines of python, MIT licensed it keeps your request prefix byte-stable, so DeepSeek's prompt cache covers your entire system + tools + history on every turn cache-hit input then bills at ~$0.0435 per 1M tokens the same 1M tokens on Claude Fable 5 runs you $10 so the longer you code, the more the cache pays off what you actually get: → 25 LLM providers (Claude, GPT, DeepSeek, GLM, local ollama) → 1M token context window → real agent loop: tools, skills, REPL, session history → one-line install, mac/linux/wsl and it's sitting at 677 stars. basically nobody is looking yet bookmark this before it gone

someone just rebuilt claude code in pure python and made it ~230x cheaper to run it's called clawcodex. a full from-scratch port of the claude code agent loop. 230k+ lines of python, MIT licensed it keeps your request prefix byte-stable, so DeepSeek's prompt cache covers your entire system + tools + history on every turn cache-hit input then bills at ~$0.0435 per 1M tokens the same 1M tokens on Claude Fable 5 runs you $10 so the longer you code, the more the cache pays off what you actually get: → 25 LLM providers (Claude, GPT, DeepSeek, GLM, local ollama) → 1M token context window → real agent loop: tools, skills, REPL, session history → one-line install, mac/linux/wsl and it's sitting at 677 stars. basically nobody is looking yet bookmark this before it gone

Argona

51,369 次观看 • 3 个月前

A short demo of how I use these extensions that I created for myself in Pi - pi-context-prune - pi-context-usage - pi-cache-graph (they are all on npm if you want to install for yourself)

A short demo of how I use these extensions that I created for myself in Pi - pi-context-prune - pi-context-usage - pi-cache-graph (they are all on npm if you want to install for yourself)

Arnav Gupta

13,741 次观看 • 5 个月前

the docs said the caching is automatic, so you did nothing automatic is where it happens, not whether it happens the cache lives on the machine that built it, and your next request lands wherever the fleet sends it, which is usually not there, so it pays full price for a prefix it already paid to build there is a key that pins a conversation to one machine and it is a single header cached input is fifty cents a million against two dollars you never needed a smaller prompt you needed the request to land in the same place twice

the docs said the caching is automatic, so you did nothing automatic is where it happens, not whether it happens the cache lives on the machine that built it, and your next request lands wherever the fleet sends it, which is usually not there, so it pays full price for a prefix it already paid to build there is a key that pins a conversation to one machine and it is a single header cached input is fifty cents a million against two dollars you never needed a smaller prompt you needed the request to land in the same place twice

Lagosaur

16,892 次观看 • 2 个月前

finally ported my fav Claude Code feature into Pi: `/subtask` a subtask is a fork of your current conversation, it inherits everything you've discussed, works in the background, and sends only its final answer back into the conversation that spawned it. Less context pollution/bloating and you can continue the main conversation as subtasks are running. here's what's packed into the Pi extension: - a live panel under the prompt: ↑↓ to select, enter to watch a fork work in real time, x to stop or dismiss it - steer a running fork mid-task, or resume a finished one without losing its progress - the model can spawn subtasks on its own and keep working while it waits for the result - forks reuse the parent's prompt cache, so they're cheaper than briefing a subagent - every fork's transcript is a real pi session: reopen it any time with `pi --session `

finally ported my fav Claude Code feature into Pi: `/subtask` a subtask is a fork of your current conversation, it inherits everything you've discussed, works in the background, and sends only its final answer back into the conversation that spawned it. Less context pollution/bloating and you can continue the main conversation as subtasks are running. here's what's packed into the Pi extension: - a live panel under the prompt: ↑↓ to select, enter to watch a fork work in real time, x to stop or dismiss it - steer a running fork mid-task, or resume a finished one without losing its progress - the model can spawn subtasks on its own and keep working while it waits for the result - forks reuse the parent's prompt cache, so they're cheaper than briefing a subagent - every fork's transcript is a real pi session: reopen it any time with `pi --session `

Victor M

18,595 次观看 • 2 个月前

Researchers made LLM inference 14x faster and 90% cheaper. The video below depicts the speed up in action. Providers discount cached input tokens by as much as 90% because a cache hit skips prefill compute entirely. For stable system prompts and tool definitions, hit rates of 60 to 85% are achievable, which makes it the highest-leverage inference optimization. But the cost saving only works when the cached text is an exact, byte-for-byte prefix of the new request. If you change one character anywhere before it, the entire cached region is missed. Three common request patterns produce full cache misses: - A query that needs documents A and B together can't reuse B's standalone cache, because those KV entries were computed without A in front of them. - The same three documents retrieved in a different order produce a full cache miss, even though nothing about the documents changed. - In multi-turn conversations, every new turn invalidates whatever was cached beyond the stable prefix. Alibaba's production data did a study on this and found that just 10% of cached KV blocks serve 77% of all cache hits. So most of what gets cached sits in storage and is never used a single time. And the root cause is that KV entries are position-dependent. Each token's KV encodes attention to everything before it, so a cached block is only valid in the exact context it was computed in. There's a second, less discussed problem as well. Cache management runs inside the inference engine's process. Moving KV tensors between GPU, CPU, and disk competes with inference for the same resources. This is why Google's TurboQuant compresses KV caches to 3 bits with no accuracy loss and still causes a 20%+ slowdown when it runs in-process. Fixing both problems means restructuring where caching lives. Cache management moves into its own process, the engine only exchanges block IDs over shared GPU memory, and heavy data movement runs across GPU, CPU, disk, and remote storage in parallel. Non-prefix reuse gets handled by selectively recomputing only the small set of tokens that attend across document boundaries. LMCache is the open-source project (10k+ stars) that implements this exact architecture, and it plugs into vLLM, SGLang, and TensorRT-LLM. The selective recomputation part is implemented in its CacheBlend technique, which makes cached docs in any order and combination, with 2-4x faster multi-document processing. On H200s running Qwen3-235B with 50 concurrent users, LMCache's multiprocess mode delivers 14x faster time-to-first-token and 4x faster decoding compared to in-process caching. GitHub repo: (don't forget to star 🌟) My co-founder wrote a full breakdown of KV cache management. It covers the disaggregated architecture behind the 14x speed up, how CacheBlend preserves generation quality while skipping recomputation, and how to turn every document in a knowledge base into a reusable cached asset. Read it below.

Researchers made LLM inference 14x faster and 90% cheaper. The video below depicts the speed up in action. Providers discount cached input tokens by as much as 90% because a cache hit skips prefill compute entirely. For stable system prompts and tool definitions, hit rates of 60 to 85% are achievable, which makes it the highest-leverage inference optimization. But the cost saving only works when the cached text is an exact, byte-for-byte prefix of the new request. If you change one character anywhere before it, the entire cached region is missed. Three common request patterns produce full cache misses: - A query that needs documents A and B together can't reuse B's standalone cache, because those KV entries were computed without A in front of them. - The same three documents retrieved in a different order produce a full cache miss, even though nothing about the documents changed. - In multi-turn conversations, every new turn invalidates whatever was cached beyond the stable prefix. Alibaba's production data did a study on this and found that just 10% of cached KV blocks serve 77% of all cache hits. So most of what gets cached sits in storage and is never used a single time. And the root cause is that KV entries are position-dependent. Each token's KV encodes attention to everything before it, so a cached block is only valid in the exact context it was computed in. There's a second, less discussed problem as well. Cache management runs inside the inference engine's process. Moving KV tensors between GPU, CPU, and disk competes with inference for the same resources. This is why Google's TurboQuant compresses KV caches to 3 bits with no accuracy loss and still causes a 20%+ slowdown when it runs in-process. Fixing both problems means restructuring where caching lives. Cache management moves into its own process, the engine only exchanges block IDs over shared GPU memory, and heavy data movement runs across GPU, CPU, disk, and remote storage in parallel. Non-prefix reuse gets handled by selectively recomputing only the small set of tokens that attend across document boundaries. LMCache is the open-source project (10k+ stars) that implements this exact architecture, and it plugs into vLLM, SGLang, and TensorRT-LLM. The selective recomputation part is implemented in its CacheBlend technique, which makes cached docs in any order and combination, with 2-4x faster multi-document processing. On H200s running Qwen3-235B with 50 concurrent users, LMCache's multiprocess mode delivers 14x faster time-to-first-token and 4x faster decoding compared to in-process caching. GitHub repo: (don't forget to star 🌟) My co-founder wrote a full breakdown of KV cache management. It covers the disaggregated architecture behind the 14x speed up, how CacheBlend preserves generation quality while skipping recomputation, and how to turn every document in a knowledge base into a reusable cached asset. Read it below.

Avi Chawla

30,717 次观看 • 3 个月前

addicted to using boomerang mode (aka reverse D-Mail) in Pi these days. ctrl+alt+b enables it for the next prompt submitted -> after the prompt runs, it rewinds back to the same point with file changes intact + leaves a summary in the feed so the agent knows what happened. using it often can make the context window feel nearly unlimited. Powered by the native /tree functionality in pi. pi install pi-boomerang

addicted to using boomerang mode (aka reverse D-Mail) in Pi these days. ctrl+alt+b enables it for the next prompt submitted -> after the prompt runs, it rewinds back to the same point with file changes intact + leaves a summary in the feed so the agent knows what happened. using it often can make the context window feel nearly unlimited. Powered by the native /tree functionality in pi. pi install pi-boomerang

Nico Bailon

38,562 次观看 • 6 个月前

How to cut your AI bill by 60% switching to Opus 5.5: Most people will swap the model name, save 24%, and stop there. The other 37% is sitting in your settings. Here's the real before and after on an example agent workload. One month: 100M input tokens (80M cached reads, 5M cache writes, 15M uncached) 10M output tokens BEFORE: Opus 5 Cache reads: 80M × $0.50 = $40 Cache writes: 5M × $6.25 = $31.25 Uncached input: 15M × $5 = $75 Output: 10M × $25 = $250 Total: $396.25 STEP 1. Just switch the model. Same tokens, new prices. Cache reads $0.20. Writes $5. Input $4. Output $20. $16 + $25 + $60 + $200 = $301 24% cheaper. That's what everyone's screenshotting. STEP 2. Let it write less. Box measured Opus 5.5 at 40% less verbose with no drop in accuracy. 10M output tokens becomes 6M. Output drops from $200 to $120. Total: $221. Now you're at 44% off. STEP 3. Stop running every call at max effort. Thinking can't be switched off on 5.5 anymore, so effort is your lever. Classifying, routing, formatting, summarizing? Drop the effort. Save the high settings for the calls that actually reason. Say that trims output another 25%, 6M down to 4.5M. Output: $90. Total: $191. 52% off. STEP 4. Cache the stuff you keep resending. This is the one nobody does. Cache reads went from $0.50 to $0.20. That's 60% off the cheapest line on your bill. Move your system prompt, tool definitions and repo context into the cache. Uncached input drops from 15M to 5M, cache reads go up to 90M. $18 + $25 + $20 + $90 = $153 AFTER: $153 From $396.25. 61% cheaper. Same work. The model gave you 24%. You gave yourself the other 37%. Anthropic's own number is 40% cheaper than Opus 5 on typical workloads. Your number depends on how much of your bill is output and how much of your prompt you're resending uncached every call. So check those two first. Bookmark this for when you migrate. follow CyrilXBT

How to cut your AI bill by 60% switching to Opus 5.5: Most people will swap the model name, save 24%, and stop there. The other 37% is sitting in your settings. Here's the real before and after on an example agent workload. One month: 100M input tokens (80M cached reads, 5M cache writes, 15M uncached) 10M output tokens BEFORE: Opus 5 Cache reads: 80M × $0.50 = $40 Cache writes: 5M × $6.25 = $31.25 Uncached input: 15M × $5 = $75 Output: 10M × $25 = $250 Total: $396.25 STEP 1. Just switch the model. Same tokens, new prices. Cache reads $0.20. Writes $5. Input $4. Output $20. $16 + $25 + $60 + $200 = $301 24% cheaper. That's what everyone's screenshotting. STEP 2. Let it write less. Box measured Opus 5.5 at 40% less verbose with no drop in accuracy. 10M output tokens becomes 6M. Output drops from $200 to $120. Total: $221. Now you're at 44% off. STEP 3. Stop running every call at max effort. Thinking can't be switched off on 5.5 anymore, so effort is your lever. Classifying, routing, formatting, summarizing? Drop the effort. Save the high settings for the calls that actually reason. Say that trims output another 25%, 6M down to 4.5M. Output: $90. Total: $191. 52% off. STEP 4. Cache the stuff you keep resending. This is the one nobody does. Cache reads went from $0.50 to $0.20. That's 60% off the cheapest line on your bill. Move your system prompt, tool definitions and repo context into the cache. Uncached input drops from 15M to 5M, cache reads go up to 90M. $18 + $25 + $20 + $90 = $153 AFTER: $153 From $396.25. 61% cheaper. Same work. The model gave you 24%. You gave yourself the other 37%. Anthropic's own number is 40% cheaper than Opus 5 on typical workloads. Your number depends on how much of your bill is output and how much of your prompt you're resending uncached every call. So check those two first. Bookmark this for when you migrate. follow CyrilXBT

CyrilXBT

15,803 次观看 • 1 个月前

An interesting new approach to agentic RL: Researchers found a way to train AI agents nearly 2x faster. Agentic RL usually produces long rollouts, and when a rollout reaches its context budget, a compaction policy removes older turns and keeps the prompt, recent turns, or a summary. Most systems use re-prefill compaction. They create a shorter sequence and run every retained token through the model again. This rebuilds the KV cache after each compaction, so the same tokens may be processed several times. KV-streams is a new approach to solve this. Instead of rebuilding the shorter sequence, KV-streams removes discarded turns directly from the live KV cache. Generation continues from the entries that survived. In practice, you cannot just reuse those entries as is. It requires two fixes. > First, RoPE has already rotated each cached Key using its original token position. A Key may move from physical slot 1,000 to slot 500 after compaction, but the model must still treat it as position 1,000. KV-streams keeps physical storage and logical position separate. Retained Keys keep their rotations. New Queries and Keys continue from the original position count. > Second, vLLM does not store each KV entry independently. It groups them into fixed 16-token blocks and uses a block table to find those blocks in GPU memory. KV-streams cannot leave empty slots inside that table because the attention kernel may read the wrong memory. It therefore removes complete blocks and joins the remaining block references together. This introduces another problem. A conversation turn may end halfway through a 16-token block. In their vLLM setup, that partial block is not committed to the prefix cache and must be recomputed when the next turn arrives. If compaction happens first, those tokens may have originally attended to context that is no longer present. For the vLLM experiments, the researchers pad each completion to a multiple of 16 tokens. This commits every generated token to the cache before the next turn or compaction event. The SGLang setup uses single-token blocks and needs no padding. Separately, the researchers found that token-level eviction produced degenerate text and required substantial supervised fine-tuning. They suspect that removing message-boundary tokens disrupted the chat format learned during training. The agent experiments therefore remove complete turns, including their boundary tags. Once inference works this way, training must follow the same process. A normal trainer would either process the full rollout or rebuild the shorter sequence after compaction. Both options produce different KV entries from those used during generation. KV-streams records every compaction event during generation. The trainer then reproduces that history: retained tokens keep the states they formed before compaction, while later tokens are blocked from attending to spans that inference had already removed. In the repository’s validation runs, the difference between training and generation stayed near a mismatch KL of 0.001, indicating that both paths produced almost identical token probabilities. On SWE-bench Verified, the Markovian Thinker setup with KV-streams reached peak performance in about 32 hours. The same compaction strategy with re-prefill took roughly 60 hours, while both reached comparable final scores. The speedup mainly applies to RL training, where one rollout may be compacted several times. At production inference, batching can hide some prefill cost, and keeping caches for inactive users reduces serving capacity. There is one more detail. The retained KV entries were created while the removed turns were still visible. Future tokens cannot attend to those turns directly, but some of their influence may remain in the entries that were not evicted. In a controlled test, RL recovered an assignment after its original message was removed. Recall reached 100% when enough cache remained. KV-streams does not decide what the model should forget. Instead, it makes existing compaction policies cheaper to train by preserving the KV state they choose to retain. Repo: If you want to dive deeper into the broader KV-cache engineering stack, I wrote an article that covers eviction, paging, quantization, offloading, and architectural compression. Read it below.

An interesting new approach to agentic RL: Researchers found a way to train AI agents nearly 2x faster. Agentic RL usually produces long rollouts, and when a rollout reaches its context budget, a compaction policy removes older turns and keeps the prompt, recent turns, or a summary. Most systems use re-prefill compaction. They create a shorter sequence and run every retained token through the model again. This rebuilds the KV cache after each compaction, so the same tokens may be processed several times. KV-streams is a new approach to solve this. Instead of rebuilding the shorter sequence, KV-streams removes discarded turns directly from the live KV cache. Generation continues from the entries that survived. In practice, you cannot just reuse those entries as is. It requires two fixes. > First, RoPE has already rotated each cached Key using its original token position. A Key may move from physical slot 1,000 to slot 500 after compaction, but the model must still treat it as position 1,000. KV-streams keeps physical storage and logical position separate. Retained Keys keep their rotations. New Queries and Keys continue from the original position count. > Second, vLLM does not store each KV entry independently. It groups them into fixed 16-token blocks and uses a block table to find those blocks in GPU memory. KV-streams cannot leave empty slots inside that table because the attention kernel may read the wrong memory. It therefore removes complete blocks and joins the remaining block references together. This introduces another problem. A conversation turn may end halfway through a 16-token block. In their vLLM setup, that partial block is not committed to the prefix cache and must be recomputed when the next turn arrives. If compaction happens first, those tokens may have originally attended to context that is no longer present. For the vLLM experiments, the researchers pad each completion to a multiple of 16 tokens. This commits every generated token to the cache before the next turn or compaction event. The SGLang setup uses single-token blocks and needs no padding. Separately, the researchers found that token-level eviction produced degenerate text and required substantial supervised fine-tuning. They suspect that removing message-boundary tokens disrupted the chat format learned during training. The agent experiments therefore remove complete turns, including their boundary tags. Once inference works this way, training must follow the same process. A normal trainer would either process the full rollout or rebuild the shorter sequence after compaction. Both options produce different KV entries from those used during generation. KV-streams records every compaction event during generation. The trainer then reproduces that history: retained tokens keep the states they formed before compaction, while later tokens are blocked from attending to spans that inference had already removed. In the repository’s validation runs, the difference between training and generation stayed near a mismatch KL of 0.001, indicating that both paths produced almost identical token probabilities. On SWE-bench Verified, the Markovian Thinker setup with KV-streams reached peak performance in about 32 hours. The same compaction strategy with re-prefill took roughly 60 hours, while both reached comparable final scores. The speedup mainly applies to RL training, where one rollout may be compacted several times. At production inference, batching can hide some prefill cost, and keeping caches for inactive users reduces serving capacity. There is one more detail. The retained KV entries were created while the removed turns were still visible. Future tokens cannot attend to those turns directly, but some of their influence may remain in the entries that were not evicted. In a controlled test, RL recovered an assignment after its original message was removed. Recall reached 100% when enough cache remained. KV-streams does not decide what the model should forget. Instead, it makes existing compaction policies cheaper to train by preserving the KV state they choose to retain. Repo: If you want to dive deeper into the broader KV-cache engineering stack, I wrote an article that covers eviction, paging, quantization, offloading, and architectural compression. Read it below.

Avi Chawla

18,120 次观看 • 1 个月前

Interesting approach to long context/continual learning here from Baseten at Cursor Compact long trajectories by compressing a prefix of the KV cache using an MLP/autoencoder. You can train this "compactor" MLP by learning to reconstruct activations that the original KV cache would produce on subsequent tokens. This maximally reconstructs information from long context that's useful for subsequent outputs. If you run an agent over extremely long context and run this compaction recursively, the activations of this compressed KV cache become like trained weights. Similar to task-specific LoRA or "cartridges". Would not be surprised if OpenAI is running a similar algorithm for their blackbox compaction. Clear benefits here if you can avoid busting the cache, as e.g. compacting via writing to a text file would. Seeing many emerging approaches to fixed size latent states for LLMs that seem promising. If building a task-specific KV cache compression ends up being more sample efficient than running backprop, getting this to "work" feels like one of the 2-3 remaining breakthroughs on the path to true AGI

Interesting approach to long context/continual learning here from Baseten at Cursor Compact long trajectories by compressing a prefix of the KV cache using an MLP/autoencoder. You can train this "compactor" MLP by learning to reconstruct activations that the original KV cache would produce on subsequent tokens. This maximally reconstructs information from long context that's useful for subsequent outputs. If you run an agent over extremely long context and run this compaction recursively, the activations of this compressed KV cache become like trained weights. Similar to task-specific LoRA or "cartridges". Would not be surprised if OpenAI is running a similar algorithm for their blackbox compaction. Clear benefits here if you can avoid busting the cache, as e.g. compacting via writing to a text file would. Seeing many emerging approaches to fixed size latent states for LLMs that seem promising. If building a task-specific KV cache compression ends up being more sample efficient than running backprop, getting this to "work" feels like one of the 2-3 remaining breakthroughs on the path to true AGI

Jay Hack

30,031 次观看 • 4 个月前

Jev + Opus 5.5: Anthropic's new model beats GPT-6 Astra for 1/5 the cost, and 4 API changes will 400 your agent before it writes a single line I pulled these 10 steps from the migration docs so you don't learn them in production step 1 → $4 / $20 per 1M. Opus 5 was $5 / $25. cache reads dropped from $0.50 to $0.20 step 2 → 66.4% on Terminal-Bench 4.0 vs GPT-6 Astra 57.9% and Opus 5 52.3%. +14.1 points in one release, and on FrontierCode it beats Astra at default effort for 1/5 the cost step 3 → thinking can't be turned off anymore. send thinking: disabled and you get a 400. drop the field, set effort step 4 → tool_choice any and tool are gone. 400. switch to auto + strict step 5 → edit anything above a thinking block and the request dies. append only, or opt into drop_block step 6 → computer_20251124 is dead on the API. 400. move to computer_toolset_20260801 step 7 → the quiet one: default effort fell from high to medium. your agent thinks less than you set it up to and nothing tells you step 8 → hop Opus 5.5 → Sonnet 5 → Opus 5.5 and you pay 4.36 instead of 3.32. +31%, the cache dies and Sonnet can't read Opus's reasoning step 9 → change effort at the top of the request and the cache is gone. Jev sets it per message and the cache stays step 10 → switch fast - standard mid-session and it's a full cache miss. Jev picks speed once, on turn one one model, three knobs, zero 400s. that is Jev + Opus 5.5 send this to your Claude Code before you touch the model ID, then read my full Jev deep dive in the article below ↓

Jev + Opus 5.5: Anthropic's new model beats GPT-6 Astra for 1/5 the cost, and 4 API changes will 400 your agent before it writes a single line I pulled these 10 steps from the migration docs so you don't learn them in production step 1 → $4 / $20 per 1M. Opus 5 was $5 / $25. cache reads dropped from $0.50 to $0.20 step 2 → 66.4% on Terminal-Bench 4.0 vs GPT-6 Astra 57.9% and Opus 5 52.3%. +14.1 points in one release, and on FrontierCode it beats Astra at default effort for 1/5 the cost step 3 → thinking can't be turned off anymore. send thinking: disabled and you get a 400. drop the field, set effort step 4 → tool_choice any and tool are gone. 400. switch to auto + strict step 5 → edit anything above a thinking block and the request dies. append only, or opt into drop_block step 6 → computer_20251124 is dead on the API. 400. move to computer_toolset_20260801 step 7 → the quiet one: default effort fell from high to medium. your agent thinks less than you set it up to and nothing tells you step 8 → hop Opus 5.5 → Sonnet 5 → Opus 5.5 and you pay 4.36 instead of 3.32. +31%, the cache dies and Sonnet can't read Opus's reasoning step 9 → change effort at the top of the request and the cache is gone. Jev sets it per message and the cache stays step 10 → switch fast - standard mid-session and it's a full cache miss. Jev picks speed once, on turn one one model, three knobs, zero 400s. that is Jev + Opus 5.5 send this to your Claude Code before you touch the model ID, then read my full Jev deep dive in the article below ↓

Carnage

16,674 次观看 • 1 个月前

Did you know you can cache your app builds and reuse them—across your entire team? No more rebuilding the same app over and over. Just enable Remote Build Cache, and Expo will skip native compilation if a matching build already exists in the cache. ✅ Works with EAS, GitHub, or your own provider ⚡ Most JS changes don’t break the cache 💸 Saves time and build costs

Did you know you can cache your app builds and reuse them—across your entire team? No more rebuilding the same app over and over. Just enable Remote Build Cache, and Expo will skip native compilation if a matching build already exists in the cache. ✅ Works with EAS, GitHub, or your own provider ⚡ Most JS changes don’t break the cache 💸 Saves time and build costs

Beto

15,929 次观看 • 1 年前

pi-web-access is an extension that provides web search (and more) for Pi coding agent. The latest update introduces a new UX pattern: you can hit `Ctrl+S` to intercept agent searches mid-flight to inspect and curate exactly what makes it into the agent conversation. pi install npm:pi-web-access

pi-web-access is an extension that provides web search (and more) for Pi coding agent. The latest update introduces a new UX pattern: you can hit `Ctrl+S` to intercept agent searches mid-flight to inspect and curate exactly what makes it into the agent conversation. pi install npm:pi-web-access

Nico Bailon

29,352 次观看 • 8 个月前

introducing pi-peer Mario Zechner's pi sessions can now find and message each other. two tools: list_peers and message_peer. anthropic's version is a unix socket per session. mine is a file on disk, which actually simplifies the entire design of this feature and also means it works "offline". a closed pi session will read its "mail" once resumed. ``` pi install npm:Shift-labs/pi-peer ```

introducing pi-peer Mario Zechner's pi sessions can now find and message each other. two tools: list_peers and message_peer. anthropic's version is a unix socket per session. mine is a file on disk, which actually simplifies the entire design of this feature and also means it works "offline". a closed pi session will read its "mail" once resumed. ``` pi install npm:Shift-labs/pi-peer ```

Michael Livs

53,448 次观看 • 2 个月前

Sidequest for Pi is a context aware side channel for coding agent tangents: It's better /btw. Ask questions grounded in your active Pi session, and keep the main conversation focused. Hit the ~ key to show/hide the threaded conversation UI. > pi install npm:pi-sidequest

Sidequest for Pi is a context aware side channel for coding agent tangents: It's better /btw. Ask questions grounded in your active Pi session, and keep the main conversation focused. Hit the ~ key to show/hide the threaded conversation UI. > pi install npm:pi-sidequest

Peter Pistorius

25,782 次观看 • 4 个月前