Загрузка видео...

Не удалось загрузить видео

Возникла проблема при загрузке этого видео. Это может быть связано с временными проблемами сети или видео может быть недоступно.

На главную

We’re thrilled to open-source TriAttention! 🚀 🦞 Deploy OpenClaw (32B LLM) on a single 24GB RTX 4090 locally 💻Full code open-source & vLLM-ready for one-click deployment ⚡️ 2.5× faster inference speed & 10.7× less KV cache memory usage TriAttention is a novel KV cache compression method built on rigorous... show more

Yukang Chen

1,624 subscribers

199,279 просмотров • 7 месяцев назад •via X (Twitter)

Образование Наука и технологии

Anya Rossi• Live Now

Private livecam show

Комментарии: 34

Фото профиля Yukang Chen

Yukang Chen7 месяцев назад

Github Repo: Paper Link: Homepage:

Фото профиля Foola Froos🌞

Foola Froos🌞7 месяцев назад

Now the same but for Hermes Agent

Фото профиля mrkelly

mrkelly7 месяцев назад

Everyone celebrates the memory reduction. What is rarely mentioned is the silent tax on reasoning quality.

Фото профиля Taha🌴

Taha🌴7 месяцев назад

How does it compare to TurboQuant

Фото профиля Two Minute Papers

Two Minute Papers7 месяцев назад

Great work, thank you!

Фото профиля Yukang Chen

Yukang Chen7 месяцев назад

Thanks!

Фото профиля M4rc0z

M4rc0z7 месяцев назад

@yacinelearning @alexinexxx @ivanfioravanti @Prince_Canuma @TheAhmadOsman have a look 👀

Фото профиля Somi

Somi7 месяцев назад

10.7x less KV cache is the number that matters here. that's what actually makes 32B on a 4090 viable. how does it hold up at longer context lengths though? most compression methods start losing quality past 8-16k tokens

Фото профиля Yukang Chen

Yukang Chen7 месяцев назад

On the math reasoning task, AIME25, with 32K-token generation, TriAttention matches full attention reasoning accuracy. These results can be found in the paper.

Фото профиля R_za

R_za7 месяцев назад

@no_stp_on_snek maybe you combine this with Turboquant..

Фото профиля Anurag Kanade

Anurag Kanade7 месяцев назад

@grok how does it compare to turboquant?

Фото профиля Gilfoyle

Gilfoyle7 месяцев назад

The model is Qwen-32B, OpenClaw has nothing to do with anything. Keyword optimization?

Фото профиля Yukang Chen

Yukang Chen7 месяцев назад

TriAttention is a KV cache optimization designed for long reasoning in LLMs, aiming to maintain both efficiency and model quality. OpenClaw has strong requirements on long reasoning: it needs the LLM to be efficient while preserving accuracy, which is exactly the goal TriAttention achieves. They are highly related.

Фото профиля Team Reagent

Team Reagent7 месяцев назад

Open Source Rules!!!! I assume this also means you have something cooking up called QuadAttention haha

Фото профиля Yukang Chen

Yukang Chen7 месяцев назад

Thanks! We all love open source!! Haha

Фото профиля juan

juan7 месяцев назад

lmao this trig kv hack actually works finally 32b local on 4090 without dying at lpng ctx nice one

Фото профиля Carolina

Carolina7 месяцев назад

Great to see TriAttention open source and enabling local inference on a consumer GPU. Curious how long context handling scales in practice

Фото профиля Bo Wang

Bo Wang7 месяцев назад

Super cool! Great work!

Фото профиля Yukang Chen

Yukang Chen7 месяцев назад

Thanks!

Фото профиля Josh Sirota

Josh Sirota7 месяцев назад

Very cool. Having a small dinner at Eragon HQ this Friday. Come and share with group of founders and researchers!

Фото профиля Mañjuśrī

Mañjuśrī6 месяцев назад

great work! does it include the turbo quant released by Google in the previous week?

Фото профиля Sakura Yuki

Sakura Yuki7 месяцев назад

10.7x KV cache compression is wild. does this play nice with speculative decoding, or do the pre-RoPE approximations mess with draft model acceptance rates?

Фото профиля Nic Wienandt - mtecnic

Nic Wienandt - mtecnic7 месяцев назад

This is neat, imma try it

Фото профиля James T 🔳

James T 🔳7 месяцев назад

This is very interesting. I will be releasing a bridge to get over eGPU limitations to help everyone get the max out of Mac and Nvidia setups or Nvidia cards available to max their investment.

Фото профиля Yukang Chen

Yukang Chen7 месяцев назад

Great! Our triattention also support Mac! Check it here

Фото профиля James T 🔳

James T 🔳7 месяцев назад

Thak you @yukangchen_ I am just publishing my bridge and will definitley look at this

Фото профиля Timur

Timur6 месяцев назад

3090 as well? nothing specific to 4090 particularly?

Фото профиля MC Naive's

MC Naive's6 месяцев назад

The model is not using attention to compute, it is using it to select structure

Фото профиля unbug

unbug7 месяцев назад

when we'll have it for Ollama?

Фото профиля Xeich

Xeich7 месяцев назад

Hell yeah. Nice job fellas.

Фото профиля Yukang Chen

Yukang Chen7 месяцев назад

Thank you!

Фото профиля ekko

ekko6 месяцев назад

4090跑32B还能2.5x加速 KV cache压到10分之一这个压缩率有点夸张关注一手

Фото профиля NextGenAI_FR

NextGenAI_FR7 месяцев назад

how much context size on a 4090 ?

Фото профиля unbug

unbug7 месяцев назад

Genius! Waht a genius project! I saw Ollama support on the schedule, which is exactly what I wanted. Please must supports V100 (CUDA 13.0) too. Nemotron 3 with 0.5 VRAM and 2x speed aready blow my mind.

Похожие видео

I added KV caching and INT8 KV quantization to our transformer inference, improving throughput by 35x. All of this was done from scratch in Rust + CUDA, on top of a homemade ML framework. On a 4-token prompt with 252 generated tokens: - Original: 0.76 tok/s - KV cache fp32: 27.21 tok/s - KV cache int8 (quantized): 27.29 tok/s Try it out yourself here: In practice: - KV caching gave us about a 35x end-to-end speedup - INT8 KV cache kept roughly the same speed as fp32 but cut KV cache memory by 3.78x FP32 cache used 4.5 MB in this run while the INT8 cache used only 1.19 MB This simple change to inference created a huge impact on performance. To learn more about the KV cache and other optimizations like this, check out the blog at

I added KV caching and INT8 KV quantization to our transformer inference, improving throughput by 35x. All of this was done from scratch in Rust + CUDA, on top of a homemade ML framework. On a 4-token prompt with 252 generated tokens: - Original: 0.76 tok/s - KV cache fp32: 27.21 tok/s - KV cache int8 (quantized): 27.29 tok/s Try it out yourself here: In practice: - KV caching gave us about a 35x end-to-end speedup - INT8 KV cache kept roughly the same speed as fp32 but cut KV cache memory by 3.78x FP32 cache used 4.5 MB in this run while the INT8 cache used only 1.19 MB This simple change to inference created a huge impact on performance. To learn more about the KV cache and other optimizations like this, check out the blog at

Reese Chong

53,026 просмотров • 6 месяцев назад

You can now run frontier-class DeepSeek-V4.1-Flash locally at 𝟰𝟭 𝘁𝗼𝗸𝗲𝗻𝘀/𝘀𝗲𝗰𝗼𝗻𝗱 on a single **𝗥𝗧𝗫 𝟯𝟬𝟵𝟬**. A new optimized inference engine just dropped on GitHub: • 𝗦𝗶𝗻𝗴𝗹𝗲 𝟮𝟰𝗚𝗕 𝗚𝗣𝗨: No expensive enterprise cloud clusters required • 𝟰𝟭 𝘁𝗼𝗸/𝘀 𝗧𝗵𝗿𝗼𝘂𝗴𝗵𝗽𝘂𝘁: Full reasoning model speed without API latency • 𝗭𝗲𝗿𝗼 𝗗𝗮𝘁𝗮 𝗟𝗲𝗮𝗸𝘀: Completely offline code analysis and agentic generation • Optimized KV-cache management & native Windows execution

You can now run frontier-class DeepSeek-V4.1-Flash locally at 𝟰𝟭 𝘁𝗼𝗸𝗲𝗻𝘀/𝘀𝗲𝗰𝗼𝗻𝗱 on a single 𝗥𝗧𝗫 𝟯𝟬𝟵𝟬. A new optimized inference engine just dropped on GitHub: • 𝗦𝗶𝗻𝗴𝗹𝗲 𝟮𝟰𝗚𝗕 𝗚𝗣𝗨: No expensive enterprise cloud clusters required • 𝟰𝟭 𝘁𝗼𝗸/𝘀 𝗧𝗵𝗿𝗼𝘂𝗴𝗵𝗽𝘂𝘁: Full reasoning model speed without API latency • 𝗭𝗲𝗿𝗼 𝗗𝗮𝘁𝗮 𝗟𝗲𝗮𝗸𝘀: Completely offline code analysis and agentic generation • Optimized KV-cache management & native Windows execution

Forha

64,001 просмотров • 10 дней назад

I built an interactive simulator that shows why LLM inference capacity is hard. Batching, queues, KV cache, GPU decode, priority tiers - all coupled. Change one knob and the bottleneck moves.

I built an interactive simulator that shows why LLM inference capacity is hard. Batching, queues, KV cache, GPU decode, priority tiers - all coupled. Change one knob and the bottleneck moves.

Khushboo Verma

25,217 просмотров • 4 месяцев назад

Ex-vLLM core contributor explained how to make LLM inference 10x cheaper in 34 minutes - better than $3000 inference optimization bootcamps. request comes in -> check LMCache -> hit? load KV cache from CPU/SSD/remote -> skip prefill -> serve. That loop is why Bloomberg and other production stacks now push 300 terabytes of KV cache per week. LMCache + vLLM + CPU/SSD/remote storage + zero-copy CUDA kernels - that's the stack. Watch and save it, then wire the KV-offload into your inference stack.

Ex-vLLM core contributor explained how to make LLM inference 10x cheaper in 34 minutes - better than $3000 inference optimization bootcamps. request comes in -> check LMCache -> hit? load KV cache from CPU/SSD/remote -> skip prefill -> serve. That loop is why Bloomberg and other production stacks now push 300 terabytes of KV cache per week. LMCache + vLLM + CPU/SSD/remote storage + zero-copy CUDA kernels - that's the stack. Watch and save it, then wire the KV-offload into your inference stack.

h100envy

61,470 просмотров • 3 месяцев назад

Hot off the press 🔥 AEON vLLM Ultimate → v0.24.0. Link below ⤵️ The ultimate container for AI on NVIDIA DGX Spark — vLLM compiled from source for GB10/sm_121a. 🔓 UNLOCKED (impossible until this release): ▸ DFlash + FP8 KV cache — 2× KV capacity, draft acceptance intact, measured on-device ▸ FP8 KV on vision models — the KV-bandwidth lever is back ▸ DFlash on FlashInfer & Triton backends (was FlashAttention-only) 🛠 Carried ahead of upstream (still-open PRs, merged + validated): ▸ Triton NVFP4 KV cache → up to 3× KV capacity ▸ DFlash sliding-window drafters + prefix-cache stability for DFlash ▸ High-concurrency DFlash — clean through c=64 🧰 Fixes stock v0.24.0 doesn't have: ▸ Gemma-4 tied embeddings (ModelOpt) — stock crashes at load, ours doesn't ▸ Tool calling out of the box (stock --no-deps builds 500) ▸ Unified-memory hardened — no silent KV overcommit on the 121 GB pool 📊 One image, the whole fleet: Gemma-4-26B-A4B · Qwen3.6-27B · Qwen3.6-35B-A3B (hybrid GDN) 536 tok/s @ c16 · 472 tok/s @ c12 · 262K context · image+video · tools + reasoning parsers · full CUDA-graph capture 🐳 docker pull

Hot off the press 🔥 AEON vLLM Ultimate → v0.24.0. Link below ⤵️ The ultimate container for AI on NVIDIA DGX Spark — vLLM compiled from source for GB10/sm_121a. 🔓 UNLOCKED (impossible until this release): ▸ DFlash + FP8 KV cache — 2× KV capacity, draft acceptance intact, measured on-device ▸ FP8 KV on vision models — the KV-bandwidth lever is back ▸ DFlash on FlashInfer & Triton backends (was FlashAttention-only) 🛠 Carried ahead of upstream (still-open PRs, merged + validated): ▸ Triton NVFP4 KV cache → up to 3× KV capacity ▸ DFlash sliding-window drafters + prefix-cache stability for DFlash ▸ High-concurrency DFlash — clean through c=64 🧰 Fixes stock v0.24.0 doesn't have: ▸ Gemma-4 tied embeddings (ModelOpt) — stock crashes at load, ours doesn't ▸ Tool calling out of the box (stock --no-deps builds 500) ▸ Unified-memory hardened — no silent KV overcommit on the 121 GB pool 📊 One image, the whole fleet: Gemma-4-26B-A4B · Qwen3.6-27B · Qwen3.6-35B-A3B (hybrid GDN) 536 tok/s @ c16 · 472 tok/s @ c12 · 262K context · image+video · tools + reasoning parsers · full CUDA-graph capture 🐳 docker pull

ÆON FORGE ✨

24,800 просмотров • 4 месяцев назад

Someone built a free and better alternative to Claude that runs 100% locally. → works with any LLM (Claude, GPT, Gemini, vLLM) → beats it on deep research → has Cowork-like capabilities → 50+ connectors out of the box → deploy in literally one command 100% open source. MIT license. 28k stars.

Someone built a free and better alternative to Claude that runs 100% locally. → works with any LLM (Claude, GPT, Gemini, vLLM) → beats it on deep research → has Cowork-like capabilities → 50+ connectors out of the box → deploy in literally one command 100% open source. MIT license. 28k stars.

How To Prompt

39,502 просмотров • 6 месяцев назад

A tricky LLM interview question: You're serving a reasoning model on vLLM, and long traces keep pushing it out of GPU memory. You add KV cache compression and evict 90% of the cached tokens. VRAM usage doesn't move, and the GPU still runs out of memory. Why? (answer below) Evicting 90% of the KV cache can free close to zero bytes on a paged server. During decoding, every new token appends its key and value vectors at every layer. vLLM holds all of them until the request finishes. For reasoning models with long chains of thought, this cache is the main memory cost. A Qwen3-32B with 4-bit weights on a 24GB GPU needs KV for every token of a 32K-token trace, and it hits OOM around 24K tokens. Attention over long contexts is sparse, so you can keep the high-importance tokens and drop the rest with little accuracy loss. That reduces the number of cached tokens but it doesn't necessarily reduce allocated memory. vLLM manages the cache with PagedAttention. It splits GPU memory into fixed physical blocks of 16 token slots each. The allocator reclaims a block only when all 16 slots are empty. Importance-based eviction keeps tokens scattered across the full sequence, so the retained tokens are spread over nearly every block. Evict 14K of 16K tokens across 1,000 blocks, and almost every block still holds at least one token. So the allocator reclaims close to nothing. You could write new tokens into the empty slots. If token 16,001 goes into the slot token 40 held, the block now reads positions 38, 16,001, 41. The cache is no longer in position order. Attention still gives the correct result, but each slot now needs a stored position index, and every kernel has to read it. An in-order layout doesn't need that metadata. So the cache shrinks 90% logically and 0% physically. Many compression papers miss this because they benchmark on pre-allocated contiguous tensors instead of a paged server. The second problem is the eviction signal. Most eviction methods rank tokens by their attention scores. FlashAttention never materializes those scores. It computes attention tile by tile in on-chip SRAM and discards each score tile after use, which is where its speed comes from. To get the scores, you have to fall back to eager attention and build the full score matrix in HBM, which removes the speedup FlashAttention provides. NVIDIA's TriAttention handles both problems. It ranks tokens without attention scores. It uses the geometry of the query and key vectors before RoPE is applied, where they form stable clusters. Every 128 decoded tokens, it runs a compaction pass. Retained tokens move forward to fill the evicted slots, so trailing blocks empty out completely and return to the allocator, and the cache stays in position order. On long reasoning traces, it matches full-attention accuracy with 2.5x faster decoding and 10.7x less KV memory. For KV cache compression on a real server, the metric to track is blocks freed, not tokens evicted. NVIDIA's write-up: I also wrote a first-principles breakdown of the KV cache. It covers why the model caches keys and values, why the cache grows with every token, and how generation speed compares with and without caching. Read it below.

A tricky LLM interview question: You're serving a reasoning model on vLLM, and long traces keep pushing it out of GPU memory. You add KV cache compression and evict 90% of the cached tokens. VRAM usage doesn't move, and the GPU still runs out of memory. Why? (answer below) Evicting 90% of the KV cache can free close to zero bytes on a paged server. During decoding, every new token appends its key and value vectors at every layer. vLLM holds all of them until the request finishes. For reasoning models with long chains of thought, this cache is the main memory cost. A Qwen3-32B with 4-bit weights on a 24GB GPU needs KV for every token of a 32K-token trace, and it hits OOM around 24K tokens. Attention over long contexts is sparse, so you can keep the high-importance tokens and drop the rest with little accuracy loss. That reduces the number of cached tokens but it doesn't necessarily reduce allocated memory. vLLM manages the cache with PagedAttention. It splits GPU memory into fixed physical blocks of 16 token slots each. The allocator reclaims a block only when all 16 slots are empty. Importance-based eviction keeps tokens scattered across the full sequence, so the retained tokens are spread over nearly every block. Evict 14K of 16K tokens across 1,000 blocks, and almost every block still holds at least one token. So the allocator reclaims close to nothing. You could write new tokens into the empty slots. If token 16,001 goes into the slot token 40 held, the block now reads positions 38, 16,001, 41. The cache is no longer in position order. Attention still gives the correct result, but each slot now needs a stored position index, and every kernel has to read it. An in-order layout doesn't need that metadata. So the cache shrinks 90% logically and 0% physically. Many compression papers miss this because they benchmark on pre-allocated contiguous tensors instead of a paged server. The second problem is the eviction signal. Most eviction methods rank tokens by their attention scores. FlashAttention never materializes those scores. It computes attention tile by tile in on-chip SRAM and discards each score tile after use, which is where its speed comes from. To get the scores, you have to fall back to eager attention and build the full score matrix in HBM, which removes the speedup FlashAttention provides. NVIDIA's TriAttention handles both problems. It ranks tokens without attention scores. It uses the geometry of the query and key vectors before RoPE is applied, where they form stable clusters. Every 128 decoded tokens, it runs a compaction pass. Retained tokens move forward to fill the evicted slots, so trailing blocks empty out completely and return to the allocator, and the cache stays in position order. On long reasoning traces, it matches full-attention accuracy with 2.5x faster decoding and 10.7x less KV memory. For KV cache compression on a real server, the metric to track is blocks freed, not tokens evicted. NVIDIA's write-up: I also wrote a first-principles breakdown of the KV cache. It covers why the model caches keys and values, why the cache grows with every token, and how generation speed compares with and without caching. Read it below.

Avi Chawla

40,630 просмотров • 6 дней назад

Turn any GitHub repository into LLM-ready text! Simply replace "hub" with "ingest" in a GitHub URL and receive a prompt-friendly text ingest for LLMs. Gitingest is 100% open-source and provides: - Directory structure - A brief summary of the project - The entire content as LLM-ready text Plus, it comes with a nice python package and you can run the UI locally! Stay tuned, I'm working on something really cool with this!✨ Link to the GitHub repo in the next tweet! _____ Find me → Akshay 🚀 ✔️ For more insights and tutorials on ML and AI Engineering!

Turn any GitHub repository into LLM-ready text! Simply replace "hub" with "ingest" in a GitHub URL and receive a prompt-friendly text ingest for LLMs. Gitingest is 100% open-source and provides: - Directory structure - A brief summary of the project - The entire content as LLM-ready text Plus, it comes with a nice python package and you can run the UI locally! Stay tuned, I'm working on something really cool with this!✨ Link to the GitHub repo in the next tweet! _____ Find me → Akshay 🚀 ✔️ For more insights and tutorials on ML and AI Engineering!

Akshay 🚀

191,584 просмотров • 1 год назад

google's notebookLM charges $19.99/mo for pro. this open-source alternative gives you unlimited access for $0. zero subscription. runs locally. MIT licensed. features: → podcast generation → multi-modal ingestion → up to 6 languages → universal LLM selection github repo link in the comments.

google's notebookLM charges $19.99/mo for pro. this open-source alternative gives you unlimited access for $0. zero subscription. runs locally. MIT licensed. features: → podcast generation → multi-modal ingestion → up to 6 languages → universal LLM selection github repo link in the comments.

m0h

42,594 просмотров • 4 месяцев назад

A tricky LLM interview question: You're serving a reasoning model on vLLM, and it keeps running out of GPU memory on long traces. So you add KV cache compression and evict 90% of the cached tokens. VRAM usage stays as is and GPU still runs out of memory. Why? (answer below) Evicting 90% of the KV cache can free almost none of the memory it was using. This sounds counterintuitive, but it follows directly from how production servers store the cache today. The KV cache grows with every token a model generates. Each token appends its key and value vectors across every layer, and nothing is freed while generation continues. This is the dominant memory cost for reasoning models. If a 32K-token CoT caches ~32K tokens of KV vectors, a Qwen3-32B with 4-bit weights will run out-of-memory around 24K tokens on a 24GB GPU. One obvious solution is to keep the important tokens and drop the rest, since attention is sparse enough to allow it. But this does not solve the memory problem yet. The reason is paged attention, which is the memory manager behind vLLM and most production servers. Under the hood, it splits GPU memory into fixed physical blocks, each one holds the KV for about 16 tokens. This block returns to the allocator only when every slot inside it is empty. Since the eviction logic selects tokens by importance, and such tokens are scattered across blocks... ...so despite eviction, almost every block is left with at least some survivor tokens. For instance, if the logic evicts 14k of 16k tokens across 1,000 blocks, most likely every block will still have a token. This means the allocator frees almost nothing. Placing the new tokens into those freed slots is not ideal because it breaks the cache's layout. Say token 16,001 arrives, and it's placed in the slot the 40th token used to hold. The cache now reads position 38, then 16,001, then 41, so the cache is no longer in token order. Attention can still compute the right answer from that, but only if every slot now carries a separate note recording which position it actually holds. This introduces another bookkeeping cost that an in-order layout inherently avoids. So the cache is logically 90% smaller and still physically the same size. Many compression results miss this because they measure on pre-allocated contiguous tensors rather than a paged server. There's another problem. Eviction methods pick which tokens to keep by looking at the attention scores themselves (as expected). But fast attention kernels used in production, like FlashAttention, never save those scores. They compute attention in small pieces and throw the full score grid away as they go, which is also why they're fast. So the exact signal eviction methods need isn't available in memory. The workaround is to fall back to eager attention and build the full matrix, which gives up the speed FlashAttention was there to provide. NVIDIA published a method called TriAttention to solve both these problems. It never needs attention scores. Instead, it scores tokens from the geometry of the model's key and query vectors before RoPE is applied, where those vectors sit in stable clusters. For the memory problem, it runs a compaction pass every 128 decoded tokens. The surviving tokens slide forward to close the holes eviction creates, so whole blocks empty out and return to the allocator while the cache stays in token order. On long reasoning traces, the approach matches full-attention accuracy while decoding 2.5x faster and using 10.7x less KV memory. KV cache compression is a big infrastructure problem. The number that decides whether it works is the count of freed blocks, not the count of evicted tokens. You can find the NVIDIA write-up here: I wrote a first-principles breakdown of how the KV cache works. It walks through why the model stores keys and values at all, why the cache grows with every token, and a comparison of LLM generation speed with and without KV caching. Read it below.

A tricky LLM interview question: You're serving a reasoning model on vLLM, and it keeps running out of GPU memory on long traces. So you add KV cache compression and evict 90% of the cached tokens. VRAM usage stays as is and GPU still runs out of memory. Why? (answer below) Evicting 90% of the KV cache can free almost none of the memory it was using. This sounds counterintuitive, but it follows directly from how production servers store the cache today. The KV cache grows with every token a model generates. Each token appends its key and value vectors across every layer, and nothing is freed while generation continues. This is the dominant memory cost for reasoning models. If a 32K-token CoT caches ~32K tokens of KV vectors, a Qwen3-32B with 4-bit weights will run out-of-memory around 24K tokens on a 24GB GPU. One obvious solution is to keep the important tokens and drop the rest, since attention is sparse enough to allow it. But this does not solve the memory problem yet. The reason is paged attention, which is the memory manager behind vLLM and most production servers. Under the hood, it splits GPU memory into fixed physical blocks, each one holds the KV for about 16 tokens. This block returns to the allocator only when every slot inside it is empty. Since the eviction logic selects tokens by importance, and such tokens are scattered across blocks... ...so despite eviction, almost every block is left with at least some survivor tokens. For instance, if the logic evicts 14k of 16k tokens across 1,000 blocks, most likely every block will still have a token. This means the allocator frees almost nothing. Placing the new tokens into those freed slots is not ideal because it breaks the cache's layout. Say token 16,001 arrives, and it's placed in the slot the 40th token used to hold. The cache now reads position 38, then 16,001, then 41, so the cache is no longer in token order. Attention can still compute the right answer from that, but only if every slot now carries a separate note recording which position it actually holds. This introduces another bookkeeping cost that an in-order layout inherently avoids. So the cache is logically 90% smaller and still physically the same size. Many compression results miss this because they measure on pre-allocated contiguous tensors rather than a paged server. There's another problem. Eviction methods pick which tokens to keep by looking at the attention scores themselves (as expected). But fast attention kernels used in production, like FlashAttention, never save those scores. They compute attention in small pieces and throw the full score grid away as they go, which is also why they're fast. So the exact signal eviction methods need isn't available in memory. The workaround is to fall back to eager attention and build the full matrix, which gives up the speed FlashAttention was there to provide. NVIDIA published a method called TriAttention to solve both these problems. It never needs attention scores. Instead, it scores tokens from the geometry of the model's key and query vectors before RoPE is applied, where those vectors sit in stable clusters. For the memory problem, it runs a compaction pass every 128 decoded tokens. The surviving tokens slide forward to close the holes eviction creates, so whole blocks empty out and return to the allocator while the cache stays in token order. On long reasoning traces, the approach matches full-attention accuracy while decoding 2.5x faster and using 10.7x less KV memory. KV cache compression is a big infrastructure problem. The number that decides whether it works is the count of freed blocks, not the count of evicted tokens. You can find the NVIDIA write-up here: I wrote a first-principles breakdown of how the KV cache works. It walks through why the model stores keys and values at all, why the cache grows with every token, and a comparison of LLM generation speed with and without KV caching. Read it below.

Avi Chawla

273,899 просмотров • 4 месяцев назад

We’re thrilled to open-source LabClaw — the Skill Operating Layer for LabOS by Stanford-Princeton Team One command turns any OpenClaw agent into a full AI Co-Scientist. Demo: Dragon Shrimp Army reporting for duty 🦞🔬 #AIforScience #OpenClaw

We’re thrilled to open-source LabClaw — the Skill Operating Layer for LabOS by Stanford-Princeton Team One command turns any OpenClaw agent into a full AI Co-Scientist. Demo: Dragon Shrimp Army reporting for duty 🦞🔬 #AIforScience #OpenClaw

AI4Science Catalyst

447,161 просмотров • 8 месяцев назад

🚨 YOUR GPU IS PROBABLY WASTING MORE THAN YOU THINK. vLLM is built to squeeze far more useful work out of your GPU when serving LLMs. Running an LLM at scale isn’t just about having a powerful GPU. The real problem is how efficiently you use its memory and compute. That’s where vLLM comes in. → High-throughput LLM inference and serving → PagedAttention for smarter KV-cache memory management → Continuous batching to keep GPUs busy → Prefix caching + chunked prefill → OpenAI-compatible API out of the box → Supports a huge range of modern LLM architectures → Quantization support for running models more efficiently And the crazy part? You can start an OpenAI-compatible inference server with: `vllm serve ` So your application can talk to your own model almost like it’s talking to OpenAI. The bigger idea: Don’t just buy more GPUs. Make the GPUs you already have work harder. That’s why vLLM has become such a major project in LLM inference. 🔥 #vLLM #AI #LLM #Inference #GPU #MachineLearning #AIInfrastructure #OpenSource #AIAgents #Developers

🚨 YOUR GPU IS PROBABLY WASTING MORE THAN YOU THINK. vLLM is built to squeeze far more useful work out of your GPU when serving LLMs. Running an LLM at scale isn’t just about having a powerful GPU. The real problem is how efficiently you use its memory and compute. That’s where vLLM comes in. → High-throughput LLM inference and serving → PagedAttention for smarter KV-cache memory management → Continuous batching to keep GPUs busy → Prefix caching + chunked prefill → OpenAI-compatible API out of the box → Supports a huge range of modern LLM architectures → Quantization support for running models more efficiently And the crazy part? You can start an OpenAI-compatible inference server with: `vllm serve ` So your application can talk to your own model almost like it’s talking to OpenAI. The bigger idea: Don’t just buy more GPUs. Make the GPUs you already have work harder. That’s why vLLM has become such a major project in LLM inference. 🔥 #vLLM #AI #LLM #Inference #GPU #MachineLearning #AIInfrastructure #OpenSource #AIAgents #Developers

Vikas gupta

14,138 просмотров • 1 месяц назад

Sentra just killed Google Research's TurboQuant. SpectralQuant — 5.95× KV cache compression on Mistral 7B at +7.5% perplexity overhead. TurboQuant at the same compression: +22%. 3× less degradation. 15-second calibration. One per-model, then drop-in for any HuggingFace LLM, ViT, ESM, AlphaFold Evoformer, or VideoMAE. Check out the findings and how the mechanism works below. ↓

Sentra just killed Google Research's TurboQuant. SpectralQuant — 5.95× KV cache compression on Mistral 7B at +7.5% perplexity overhead. TurboQuant at the same compression: +22%. 3× less degradation. 15-second calibration. One per-model, then drop-in for any HuggingFace LLM, ViT, ESM, AlphaFold Evoformer, or VideoMAE. Check out the findings and how the mechanism works below. ↓

Ashwin Gopinath

63,001 просмотров • 5 месяцев назад

What compression looks like on vLLM. Same Gemma 4 31B. Red Hat AI's quantized version runs at nearly 2x tokens/sec, half the memory, 99%+ accuracy retained. Open source. Quantized with LLM Compressor. Links in comments. 🙏 Sawyer Bowerman for the 2-minute demo.

What compression looks like on vLLM. Same Gemma 4 31B. Red Hat AI's quantized version runs at nearly 2x tokens/sec, half the memory, 99%+ accuracy retained. Open source. Quantized with LLM Compressor. Links in comments. 🙏 Sawyer Bowerman for the 2-minute demo.

Red Hat AI

34,260 просмотров • 6 месяцев назад

🚀 Excited to release LongLive 2.0! 🎬 An end-to-end infrastructure for long video generation, with FP4 and parallelism at the core of both training and inference. ⚡45.7 FPS generation speed on 5B model⚡ ✨ LongLive 2.0 supports real-video training, few-step distillation, multi-shot training/inference, sequence-parallel acceleration, NVFP4 KV cache, and async VAE decoding deployment. 🧩 To our knowledge, this is the first open-source 4-bit long video generation infra that covers both training and inference. 🙌 Welcome to check it out, try it, and share feedback! 🔗 Code: 📰 Paper: 🎥 Demo: #LongVideoGeneration #VideoGeneration #Realtime #AIInfra #EfficientAI #FP4 #Parallel #NVIDIA

🚀 Excited to release LongLive 2.0! 🎬 An end-to-end infrastructure for long video generation, with FP4 and parallelism at the core of both training and inference. ⚡45.7 FPS generation speed on 5B model⚡ ✨ LongLive 2.0 supports real-video training, few-step distillation, multi-shot training/inference, sequence-parallel acceleration, NVFP4 KV cache, and async VAE decoding deployment. 🧩 To our knowledge, this is the first open-source 4-bit long video generation infra that covers both training and inference. 🙌 Welcome to check it out, try it, and share feedback! 🔗 Code: 📰 Paper: 🎥 Demo: #LongVideoGeneration #VideoGeneration #Realtime #AIInfra #EfficientAI #FP4 #Parallel #NVIDIA

Yukang Chen

60,351 просмотров • 5 месяцев назад

Introducing MiniCPM 4.1-8B: First Open-Source Reasoning LLM with Trainable Sparse Attention ✅ Strong Reasoning Capability: Surpasses similar-sized models on 15 tasks! ✅ Fast Generation: 3x decoding speedup for reasoning ✅ Efficient Architecture: Trainable sparse attention, frequency-ranked speculative decoding Download Models: Huggingface: Github: Technical Report: #AI #MiniCPM #LLM #OpenBMB #ArtificialIntelligence #MachineLearning

Introducing MiniCPM 4.1-8B: First Open-Source Reasoning LLM with Trainable Sparse Attention ✅ Strong Reasoning Capability: Surpasses similar-sized models on 15 tasks! ✅ Fast Generation: 3x decoding speedup for reasoning ✅ Efficient Architecture: Trainable sparse attention, frequency-ranked speculative decoding Download Models: Huggingface: Github: Technical Report: #AI #MiniCPM #LLM #OpenBMB #ArtificialIntelligence #MachineLearning

OpenBMB

19,317 просмотров • 1 год назад

Researchers made LLM inference 14x faster and 90% cheaper. The video below depicts the speed up in action. Providers discount cached input tokens by as much as 90% because a cache hit skips prefill compute entirely. For stable system prompts and tool definitions, hit rates of 60 to 85% are achievable, which makes it the highest-leverage inference optimization. But the cost saving only works when the cached text is an exact, byte-for-byte prefix of the new request. If you change one character anywhere before it, the entire cached region is missed. Three common request patterns produce full cache misses: - A query that needs documents A and B together can't reuse B's standalone cache, because those KV entries were computed without A in front of them. - The same three documents retrieved in a different order produce a full cache miss, even though nothing about the documents changed. - In multi-turn conversations, every new turn invalidates whatever was cached beyond the stable prefix. Alibaba's production data did a study on this and found that just 10% of cached KV blocks serve 77% of all cache hits. So most of what gets cached sits in storage and is never used a single time. And the root cause is that KV entries are position-dependent. Each token's KV encodes attention to everything before it, so a cached block is only valid in the exact context it was computed in. There's a second, less discussed problem as well. Cache management runs inside the inference engine's process. Moving KV tensors between GPU, CPU, and disk competes with inference for the same resources. This is why Google's TurboQuant compresses KV caches to 3 bits with no accuracy loss and still causes a 20%+ slowdown when it runs in-process. Fixing both problems means restructuring where caching lives. Cache management moves into its own process, the engine only exchanges block IDs over shared GPU memory, and heavy data movement runs across GPU, CPU, disk, and remote storage in parallel. Non-prefix reuse gets handled by selectively recomputing only the small set of tokens that attend across document boundaries. LMCache is the open-source project (10k+ stars) that implements this exact architecture, and it plugs into vLLM, SGLang, and TensorRT-LLM. The selective recomputation part is implemented in its CacheBlend technique, which makes cached docs in any order and combination, with 2-4x faster multi-document processing. On H200s running Qwen3-235B with 50 concurrent users, LMCache's multiprocess mode delivers 14x faster time-to-first-token and 4x faster decoding compared to in-process caching. GitHub repo: (don't forget to star 🌟) My co-founder wrote a full breakdown of KV cache management. It covers the disaggregated architecture behind the 14x speed up, how CacheBlend preserves generation quality while skipping recomputation, and how to turn every document in a knowledge base into a reusable cached asset. Read it below.

Researchers made LLM inference 14x faster and 90% cheaper. The video below depicts the speed up in action. Providers discount cached input tokens by as much as 90% because a cache hit skips prefill compute entirely. For stable system prompts and tool definitions, hit rates of 60 to 85% are achievable, which makes it the highest-leverage inference optimization. But the cost saving only works when the cached text is an exact, byte-for-byte prefix of the new request. If you change one character anywhere before it, the entire cached region is missed. Three common request patterns produce full cache misses: - A query that needs documents A and B together can't reuse B's standalone cache, because those KV entries were computed without A in front of them. - The same three documents retrieved in a different order produce a full cache miss, even though nothing about the documents changed. - In multi-turn conversations, every new turn invalidates whatever was cached beyond the stable prefix. Alibaba's production data did a study on this and found that just 10% of cached KV blocks serve 77% of all cache hits. So most of what gets cached sits in storage and is never used a single time. And the root cause is that KV entries are position-dependent. Each token's KV encodes attention to everything before it, so a cached block is only valid in the exact context it was computed in. There's a second, less discussed problem as well. Cache management runs inside the inference engine's process. Moving KV tensors between GPU, CPU, and disk competes with inference for the same resources. This is why Google's TurboQuant compresses KV caches to 3 bits with no accuracy loss and still causes a 20%+ slowdown when it runs in-process. Fixing both problems means restructuring where caching lives. Cache management moves into its own process, the engine only exchanges block IDs over shared GPU memory, and heavy data movement runs across GPU, CPU, disk, and remote storage in parallel. Non-prefix reuse gets handled by selectively recomputing only the small set of tokens that attend across document boundaries. LMCache is the open-source project (10k+ stars) that implements this exact architecture, and it plugs into vLLM, SGLang, and TensorRT-LLM. The selective recomputation part is implemented in its CacheBlend technique, which makes cached docs in any order and combination, with 2-4x faster multi-document processing. On H200s running Qwen3-235B with 50 concurrent users, LMCache's multiprocess mode delivers 14x faster time-to-first-token and 4x faster decoding compared to in-process caching. GitHub repo: (don't forget to star 🌟) My co-founder wrote a full breakdown of KV cache management. It covers the disaggregated architecture behind the 14x speed up, how CacheBlend preserves generation quality while skipping recomputation, and how to turn every document in a knowledge base into a reusable cached asset. Read it below.

Avi Chawla

30,717 просмотров • 3 месяцев назад

🤗 MOSS-Transcribe-Diarize-0.9B is now open source on Hugging Face. Built with an end-to-end audio-to-structured-transcript paradigm: >0.9B open-source ASR model >Apache license 2.0 >128k long-context transcription >Up to ~90-min audio input >Speaker labels + timestamps in one generation >Multi-speaker diarization for meetings, interruptions, and overlapping voices >Hotword biasing for names, terms, and domain-specific vocabulary >~100 token/s on NVIDIA RTX 4090, RTF ~0.017 Thank you SGLang vLLM Prince Canuma Lucas Newman for day-0 support! 🚀 Github: Huggingface: API: Live demo: Technical Report： HF Space: AtomGit： SGLang-Omni: vLLM: MLX-audio: Discord:

🤗 MOSS-Transcribe-Diarize-0.9B is now open source on Hugging Face. Built with an end-to-end audio-to-structured-transcript paradigm: >0.9B open-source ASR model >Apache license 2.0 >128k long-context transcription >Up to ~90-min audio input >Speaker labels + timestamps in one generation >Multi-speaker diarization for meetings, interruptions, and overlapping voices >Hotword biasing for names, terms, and domain-specific vocabulary >~100 token/s on NVIDIA RTX 4090, RTF ~0.017 Thank you SGLang vLLM Prince Canuma Lucas Newman for day-0 support! 🚀 Github: Huggingface: API: Live demo: Technical Report： HF Space: AtomGit： SGLang-Omni: vLLM: MLX-audio: Discord:

MOSS

173,581 просмотров • 4 месяцев назад

Just built an MCP for Ghidra. Now basically any LLM (Claude, Gemini, local...) can Reverse Engineer malware for you. With the right prompting, it automates a *ton* of tedious tasks. One-shot markups of entire binaries with just a click. Open source, on Github now.

Just built an MCP for Ghidra. Now basically any LLM (Claude, Gemini, local...) can Reverse Engineer malware for you. With the right prompting, it automates a ton of tedious tasks. One-shot markups of entire binaries with just a click. Open source, on Github now.

LaurieWired

286,798 просмотров • 1 год назад