Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

Introducing DuoAttention: Our new framework slashes both memory and latency for long-context LLMs without sacrificing performance! By applying full KV cache only to critical heads, we achieve: ⚡ 2.55x memory reduction ⚡ 2.18x decoding speedup ⚡ 3.3M tokens on a single A100 GPU

Guangxuan Xiao

5,470 subscribers

31,108 Aufrufe • vor 2 Jahren •via X (Twitter)

Wissenschaft & Technologie Bildung

Anya Rossi• Live Now

Private livecam show

10 Kommentare

Profilbild von Guangxuan Xiao

Guangxuan Xiaovor 2 Jahren

Paper: Code:

Profilbild von Guangxuan Xiao

Guangxuan Xiaovor 2 Jahren

DuoAttention leverages the insight that only a few attention heads, called Retrieval Heads, need full attention for long contexts, while the rest, Streaming Heads, focus on recent tokens and don't require full attention.

Profilbild von Guangxuan Xiao

Guangxuan Xiaovor 2 Jahren

DuoAttention uses a lightweight, optimization-based algorithm with synthetic data to identify retrieval heads accurately.

Profilbild von Guangxuan Xiao

Guangxuan Xiaovor 2 Jahren

We apply a constant KV cache and efficient "Streaming Attention" to streaming heads, which can accelerate both pre-filling and decoding.

Profilbild von Guangxuan Xiao

Guangxuan Xiaovor 2 Jahren

DuoAttention provides comparable accuracy as full attention on the Needle-in-a-Haystack benchmark using a 25% full attention ratio on the MHA model and a 50% full attention ratio on the GQA model.

Profilbild von Guangxuan Xiao

Guangxuan Xiaovor 2 Jahren

DuoAttention provides a better KV budget and accuracy trade-off on LongBench benchmarks.

Profilbild von Guangxuan Xiao

Guangxuan Xiaovor 2 Jahren

DuoAttention significantly reduces long-context inference memory by up to 2.55× for MHA and 1.67× for GQA models while speeding up decoding by up to 2.18× and 1.50×.

Profilbild von Guangxuan Xiao

Guangxuan Xiaovor 2 Jahren

DuoAttention can also accelerate pre-filling by up to 1.73× and 1.63× for MHA and GQA models, respectively.

Profilbild von Guangxuan Xiao

Guangxuan Xiaovor 2 Jahren

Notably, combined with quantization, DuoAttention enables Llama-3-8B decoding with 3.3 million context length on a single A100 GPU.

Profilbild von Miguel Guerrero

Miguel Guerrerovor 2 Jahren

guys, you always build and share awesome stuff with the community, just bravo!! thanks

Ähnliche Videos

🦍 Introducing Memorilla: memory as a modality for LLMs Memorilla turns everything a frozen LLM needs to remember into just 16 memory embeddings ⚡ It uses up to 4,212× fewer tokens and is 27× faster than full context inference while more accurate 🎯 Pushes the Pareto frontier over RAG and agentic memory in personalization, long-document and knowledge QA A thread 🧵:

🦍 Introducing Memorilla: memory as a modality for LLMs Memorilla turns everything a frozen LLM needs to remember into just 16 memory embeddings ⚡ It uses up to 4,212× fewer tokens and is 27× faster than full context inference while more accurate 🎯 Pushes the Pareto frontier over RAG and agentic memory in personalization, long-document and knowledge QA A thread 🧵:

Adib

97,236 Aufrufe • vor 7 Tagen

I added KV caching and INT8 KV quantization to our transformer inference, improving throughput by 35x. All of this was done from scratch in Rust + CUDA, on top of a homemade ML framework. On a 4-token prompt with 252 generated tokens: - Original: 0.76 tok/s - KV cache fp32: 27.21 tok/s - KV cache int8 (quantized): 27.29 tok/s Try it out yourself here: In practice: - KV caching gave us about a 35x end-to-end speedup - INT8 KV cache kept roughly the same speed as fp32 but cut KV cache memory by 3.78x FP32 cache used 4.5 MB in this run while the INT8 cache used only 1.19 MB This simple change to inference created a huge impact on performance. To learn more about the KV cache and other optimizations like this, check out the blog at

I added KV caching and INT8 KV quantization to our transformer inference, improving throughput by 35x. All of this was done from scratch in Rust + CUDA, on top of a homemade ML framework. On a 4-token prompt with 252 generated tokens: - Original: 0.76 tok/s - KV cache fp32: 27.21 tok/s - KV cache int8 (quantized): 27.29 tok/s Try it out yourself here: In practice: - KV caching gave us about a 35x end-to-end speedup - INT8 KV cache kept roughly the same speed as fp32 but cut KV cache memory by 3.78x FP32 cache used 4.5 MB in this run while the INT8 cache used only 1.19 MB This simple change to inference created a huge impact on performance. To learn more about the KV cache and other optimizations like this, check out the blog at

Reese Chong

53,026 Aufrufe • vor 6 Monaten

GPU tradeoff series: A100 is not much more powerful than 4090 🫠 GPU Perf and Price: - 4090: 330 fp16 TFLOPs, $1,749 - A100 (80GB): 312 fp16 TFLOPs, $20,000 > A100 is 11.4X more pricy Training speed for GPT-2(124M) with llm.c: - 4090: 153K tokens/s - A100 (80GB): 195K tokens/s > A100 is only 1.3X faster (both trained using a single card, A100 llm.c training is shown in the video, 4090 video is in the quoted tweet) Conclusion: 4090 has a much better cost vs performance ratio Why: As in the H100 vs. 4090 comparison, the biggest difference between A100 and 4090 is their GPU memory size/bandwidth and cross-GPU communication bandwidth, which does not matter too much if your model can fit into a single 4090. Specs: 4090: - GPU memory size: 24GB - memory bandwidth: 1 TB/s - communication bandwidth: 64 GB/s A100: - GPU memory size: 80GB - memory bandwidth: 2 TB/s - communication bandwidth: 900 GB/s Nvidia killed off NVLink (a high-speed communication link that connects GPUs) on 4090. (Jensen Huang smiling face) If multiple 4090s could be interconnected via NVLink, their performance would be closer to datacenter-grade A100 GPUs, even for training larger models. Additionally, 4090 isn't allowed in datacenters, that's how Nvidia makes 💰💰💰

GPU tradeoff series: A100 is not much more powerful than 4090 🫠 GPU Perf and Price: - 4090: 330 fp16 TFLOPs, $1,749 - A100 (80GB): 312 fp16 TFLOPs, $20,000 > A100 is 11.4X more pricy Training speed for GPT-2(124M) with llm.c: - 4090: 153K tokens/s - A100 (80GB): 195K tokens/s > A100 is only 1.3X faster (both trained using a single card, A100 llm.c training is shown in the video, 4090 video is in the quoted tweet) Conclusion: 4090 has a much better cost vs performance ratio Why: As in the H100 vs. 4090 comparison, the biggest difference between A100 and 4090 is their GPU memory size/bandwidth and cross-GPU communication bandwidth, which does not matter too much if your model can fit into a single 4090. Specs: 4090: - GPU memory size: 24GB - memory bandwidth: 1 TB/s - communication bandwidth: 64 GB/s A100: - GPU memory size: 80GB - memory bandwidth: 2 TB/s - communication bandwidth: 900 GB/s Nvidia killed off NVLink (a high-speed communication link that connects GPUs) on 4090. (Jensen Huang smiling face) If multiple 4090s could be interconnected via NVLink, their performance would be closer to datacenter-grade A100 GPUs, even for training larger models. Additionally, 4090 isn't allowed in datacenters, that's how Nvidia makes 💰💰💰

Yuchen Jin

234,951 Aufrufe • vor 2 Jahren

A tricky LLM interview question: You're serving a reasoning model on vLLM, and long traces keep pushing it out of GPU memory. You add KV cache compression and evict 90% of the cached tokens. VRAM usage doesn't move, and the GPU still runs out of memory. Why? (answer below) Evicting 90% of the KV cache can free close to zero bytes on a paged server. During decoding, every new token appends its key and value vectors at every layer. vLLM holds all of them until the request finishes. For reasoning models with long chains of thought, this cache is the main memory cost. A Qwen3-32B with 4-bit weights on a 24GB GPU needs KV for every token of a 32K-token trace, and it hits OOM around 24K tokens. Attention over long contexts is sparse, so you can keep the high-importance tokens and drop the rest with little accuracy loss. That reduces the number of cached tokens but it doesn't necessarily reduce allocated memory. vLLM manages the cache with PagedAttention. It splits GPU memory into fixed physical blocks of 16 token slots each. The allocator reclaims a block only when all 16 slots are empty. Importance-based eviction keeps tokens scattered across the full sequence, so the retained tokens are spread over nearly every block. Evict 14K of 16K tokens across 1,000 blocks, and almost every block still holds at least one token. So the allocator reclaims close to nothing. You could write new tokens into the empty slots. If token 16,001 goes into the slot token 40 held, the block now reads positions 38, 16,001, 41. The cache is no longer in position order. Attention still gives the correct result, but each slot now needs a stored position index, and every kernel has to read it. An in-order layout doesn't need that metadata. So the cache shrinks 90% logically and 0% physically. Many compression papers miss this because they benchmark on pre-allocated contiguous tensors instead of a paged server. The second problem is the eviction signal. Most eviction methods rank tokens by their attention scores. FlashAttention never materializes those scores. It computes attention tile by tile in on-chip SRAM and discards each score tile after use, which is where its speed comes from. To get the scores, you have to fall back to eager attention and build the full score matrix in HBM, which removes the speedup FlashAttention provides. NVIDIA's TriAttention handles both problems. It ranks tokens without attention scores. It uses the geometry of the query and key vectors before RoPE is applied, where they form stable clusters. Every 128 decoded tokens, it runs a compaction pass. Retained tokens move forward to fill the evicted slots, so trailing blocks empty out completely and return to the allocator, and the cache stays in position order. On long reasoning traces, it matches full-attention accuracy with 2.5x faster decoding and 10.7x less KV memory. For KV cache compression on a real server, the metric to track is blocks freed, not tokens evicted. NVIDIA's write-up: I also wrote a first-principles breakdown of the KV cache. It covers why the model caches keys and values, why the cache grows with every token, and how generation speed compares with and without caching. Read it below.

A tricky LLM interview question: You're serving a reasoning model on vLLM, and long traces keep pushing it out of GPU memory. You add KV cache compression and evict 90% of the cached tokens. VRAM usage doesn't move, and the GPU still runs out of memory. Why? (answer below) Evicting 90% of the KV cache can free close to zero bytes on a paged server. During decoding, every new token appends its key and value vectors at every layer. vLLM holds all of them until the request finishes. For reasoning models with long chains of thought, this cache is the main memory cost. A Qwen3-32B with 4-bit weights on a 24GB GPU needs KV for every token of a 32K-token trace, and it hits OOM around 24K tokens. Attention over long contexts is sparse, so you can keep the high-importance tokens and drop the rest with little accuracy loss. That reduces the number of cached tokens but it doesn't necessarily reduce allocated memory. vLLM manages the cache with PagedAttention. It splits GPU memory into fixed physical blocks of 16 token slots each. The allocator reclaims a block only when all 16 slots are empty. Importance-based eviction keeps tokens scattered across the full sequence, so the retained tokens are spread over nearly every block. Evict 14K of 16K tokens across 1,000 blocks, and almost every block still holds at least one token. So the allocator reclaims close to nothing. You could write new tokens into the empty slots. If token 16,001 goes into the slot token 40 held, the block now reads positions 38, 16,001, 41. The cache is no longer in position order. Attention still gives the correct result, but each slot now needs a stored position index, and every kernel has to read it. An in-order layout doesn't need that metadata. So the cache shrinks 90% logically and 0% physically. Many compression papers miss this because they benchmark on pre-allocated contiguous tensors instead of a paged server. The second problem is the eviction signal. Most eviction methods rank tokens by their attention scores. FlashAttention never materializes those scores. It computes attention tile by tile in on-chip SRAM and discards each score tile after use, which is where its speed comes from. To get the scores, you have to fall back to eager attention and build the full score matrix in HBM, which removes the speedup FlashAttention provides. NVIDIA's TriAttention handles both problems. It ranks tokens without attention scores. It uses the geometry of the query and key vectors before RoPE is applied, where they form stable clusters. Every 128 decoded tokens, it runs a compaction pass. Retained tokens move forward to fill the evicted slots, so trailing blocks empty out completely and return to the allocator, and the cache stays in position order. On long reasoning traces, it matches full-attention accuracy with 2.5x faster decoding and 10.7x less KV memory. For KV cache compression on a real server, the metric to track is blocks freed, not tokens evicted. NVIDIA's write-up: I also wrote a first-principles breakdown of the KV cache. It covers why the model caches keys and values, why the cache grows with every token, and how generation speed compares with and without caching. Read it below.

Avi Chawla

41,433 Aufrufe • vor 11 Tagen

Big ideas need serious power. Four ways to take your creativity further, from instant GPU performance on the go to full workstation power: ⚡ Real-time AI video generation ⚡ On-premises inference ⚡ Next-level performance A glimpse of what we showcased at #IBC2026. 🎬🔥 #ASUS #ProArt

Big ideas need serious power. Four ways to take your creativity further, from instant GPU performance on the go to full workstation power: ⚡ Real-time AI video generation ⚡ On-premises inference ⚡ Next-level performance A glimpse of what we showcased at #IBC2026. 🎬🔥 #ASUS #ProArt

ROG Global

18,552 Aufrufe • vor 1 Monat

Introducing AMD EXPO™ TECHNOLOGY: FEATURING ULTRA LOW LATENCY. Pushing memory performance even further for gamers on AM5 available through our partners.

Introducing AMD EXPO™ TECHNOLOGY: FEATURING ULTRA LOW LATENCY. Pushing memory performance even further for gamers on AM5 available through our partners.

AMD Gaming

15,861 Aufrufe • vor 5 Monaten

We heavily optimized our GLTF files to fit any GPU memory. We built a custom Electron app to reduce polycount & convert textures to GPU-friendly Basis/ETC1S. Compressed textures used ~10-100x less memory & uploaded instantly, preventing decoding lags while scrolling!

We heavily optimized our GLTF files to fit any GPU memory. We built a custom Electron app to reduce polycount & convert textures to GPU-friendly Basis/ETC1S. Compressed textures used ~10-100x less memory & uploaded instantly, preventing decoding lags while scrolling!

Arnaud Tanielian

17,998 Aufrufe • vor 11 Monaten

New superpowers activated 🦸 ⚡ Kiro powers load tools dynamically based on what you ask for. Only what’s needed activates. Everything else stays out of memory. • No MCP overload • Fewer wasted tokens • Cleaner context Works with tools like Figma, Stripe, Supabase, and Datadog, Inc.. Start today 👉 #Kiro #BuildwithKiro #AICoding

New superpowers activated 🦸 ⚡ Kiro powers load tools dynamically based on what you ask for. Only what’s needed activates. Everything else stays out of memory. • No MCP overload • Fewer wasted tokens • Cleaner context Works with tools like Figma, Stripe, Supabase, and Datadog, Inc.. Start today 👉 #Kiro #BuildwithKiro #AICoding

Kiro

21,247 Aufrufe • vor 10 Monaten

Big O has lied to you. Not intentionally. It's just not telling the whole story. Reading from cache can be ~100x faster than from RAM. Both are O(1). So I built apecs: a cache-friendly ECS framework for performance critical TypeScript apps. → Predictable memory → No GC pressure → Low-level types: u8, i16, f32… All without WASM!

Big O has lied to you. Not intentionally. It's just not telling the whole story. Reading from cache can be ~100x faster than from RAM. Both are O(1). So I built apecs: a cache-friendly ECS framework for performance critical TypeScript apps. → Predictable memory → No GC pressure → Low-level types: u8, i16, f32… All without WASM!

konstantinpaulus

31,839 Aufrufe • vor 2 Monaten

Livewire 4 is FINALLY HERE ⚡⚡⚡ It's insane to me what can be achieved without writing a single line of JavaScript. We did it ❤️

Livewire 4 is FINALLY HERE ⚡⚡⚡ It's insane to me what can be achieved without writing a single line of JavaScript. We did it ❤️

Caleb Porzio ⚡️

75,616 Aufrufe • vor 10 Monaten

A tricky LLM interview question: You're serving a reasoning model on vLLM, and it keeps running out of GPU memory on long traces. So you add KV cache compression and evict 90% of the cached tokens. VRAM usage stays as is and GPU still runs out of memory. Why? (answer below) Evicting 90% of the KV cache can free almost none of the memory it was using. This sounds counterintuitive, but it follows directly from how production servers store the cache today. The KV cache grows with every token a model generates. Each token appends its key and value vectors across every layer, and nothing is freed while generation continues. This is the dominant memory cost for reasoning models. If a 32K-token CoT caches ~32K tokens of KV vectors, a Qwen3-32B with 4-bit weights will run out-of-memory around 24K tokens on a 24GB GPU. One obvious solution is to keep the important tokens and drop the rest, since attention is sparse enough to allow it. But this does not solve the memory problem yet. The reason is paged attention, which is the memory manager behind vLLM and most production servers. Under the hood, it splits GPU memory into fixed physical blocks, each one holds the KV for about 16 tokens. This block returns to the allocator only when every slot inside it is empty. Since the eviction logic selects tokens by importance, and such tokens are scattered across blocks... ...so despite eviction, almost every block is left with at least some survivor tokens. For instance, if the logic evicts 14k of 16k tokens across 1,000 blocks, most likely every block will still have a token. This means the allocator frees almost nothing. Placing the new tokens into those freed slots is not ideal because it breaks the cache's layout. Say token 16,001 arrives, and it's placed in the slot the 40th token used to hold. The cache now reads position 38, then 16,001, then 41, so the cache is no longer in token order. Attention can still compute the right answer from that, but only if every slot now carries a separate note recording which position it actually holds. This introduces another bookkeeping cost that an in-order layout inherently avoids. So the cache is logically 90% smaller and still physically the same size. Many compression results miss this because they measure on pre-allocated contiguous tensors rather than a paged server. There's another problem. Eviction methods pick which tokens to keep by looking at the attention scores themselves (as expected). But fast attention kernels used in production, like FlashAttention, never save those scores. They compute attention in small pieces and throw the full score grid away as they go, which is also why they're fast. So the exact signal eviction methods need isn't available in memory. The workaround is to fall back to eager attention and build the full matrix, which gives up the speed FlashAttention was there to provide. NVIDIA published a method called TriAttention to solve both these problems. It never needs attention scores. Instead, it scores tokens from the geometry of the model's key and query vectors before RoPE is applied, where those vectors sit in stable clusters. For the memory problem, it runs a compaction pass every 128 decoded tokens. The surviving tokens slide forward to close the holes eviction creates, so whole blocks empty out and return to the allocator while the cache stays in token order. On long reasoning traces, the approach matches full-attention accuracy while decoding 2.5x faster and using 10.7x less KV memory. KV cache compression is a big infrastructure problem. The number that decides whether it works is the count of freed blocks, not the count of evicted tokens. You can find the NVIDIA write-up here: I wrote a first-principles breakdown of how the KV cache works. It walks through why the model stores keys and values at all, why the cache grows with every token, and a comparison of LLM generation speed with and without KV caching. Read it below.

A tricky LLM interview question: You're serving a reasoning model on vLLM, and it keeps running out of GPU memory on long traces. So you add KV cache compression and evict 90% of the cached tokens. VRAM usage stays as is and GPU still runs out of memory. Why? (answer below) Evicting 90% of the KV cache can free almost none of the memory it was using. This sounds counterintuitive, but it follows directly from how production servers store the cache today. The KV cache grows with every token a model generates. Each token appends its key and value vectors across every layer, and nothing is freed while generation continues. This is the dominant memory cost for reasoning models. If a 32K-token CoT caches ~32K tokens of KV vectors, a Qwen3-32B with 4-bit weights will run out-of-memory around 24K tokens on a 24GB GPU. One obvious solution is to keep the important tokens and drop the rest, since attention is sparse enough to allow it. But this does not solve the memory problem yet. The reason is paged attention, which is the memory manager behind vLLM and most production servers. Under the hood, it splits GPU memory into fixed physical blocks, each one holds the KV for about 16 tokens. This block returns to the allocator only when every slot inside it is empty. Since the eviction logic selects tokens by importance, and such tokens are scattered across blocks... ...so despite eviction, almost every block is left with at least some survivor tokens. For instance, if the logic evicts 14k of 16k tokens across 1,000 blocks, most likely every block will still have a token. This means the allocator frees almost nothing. Placing the new tokens into those freed slots is not ideal because it breaks the cache's layout. Say token 16,001 arrives, and it's placed in the slot the 40th token used to hold. The cache now reads position 38, then 16,001, then 41, so the cache is no longer in token order. Attention can still compute the right answer from that, but only if every slot now carries a separate note recording which position it actually holds. This introduces another bookkeeping cost that an in-order layout inherently avoids. So the cache is logically 90% smaller and still physically the same size. Many compression results miss this because they measure on pre-allocated contiguous tensors rather than a paged server. There's another problem. Eviction methods pick which tokens to keep by looking at the attention scores themselves (as expected). But fast attention kernels used in production, like FlashAttention, never save those scores. They compute attention in small pieces and throw the full score grid away as they go, which is also why they're fast. So the exact signal eviction methods need isn't available in memory. The workaround is to fall back to eager attention and build the full matrix, which gives up the speed FlashAttention was there to provide. NVIDIA published a method called TriAttention to solve both these problems. It never needs attention scores. Instead, it scores tokens from the geometry of the model's key and query vectors before RoPE is applied, where those vectors sit in stable clusters. For the memory problem, it runs a compaction pass every 128 decoded tokens. The surviving tokens slide forward to close the holes eviction creates, so whole blocks empty out and return to the allocator while the cache stays in token order. On long reasoning traces, the approach matches full-attention accuracy while decoding 2.5x faster and using 10.7x less KV memory. KV cache compression is a big infrastructure problem. The number that decides whether it works is the count of freed blocks, not the count of evicted tokens. You can find the NVIDIA write-up here: I wrote a first-principles breakdown of how the KV cache works. It walks through why the model stores keys and values at all, why the cache grows with every token, and a comparison of LLM generation speed with and without KV caching. Read it below.

Avi Chawla

273,899 Aufrufe • vor 4 Monaten

New course on serving LLMs efficiently -- how do you serve models to many concurrent users at low latency and reasonable cost? This short course is built with Red Hat and taught by Cedric Clyburn. Efficient LLM serving requires efficient memory management. A 70B-parameter model takes ~140 GB just to load the weights. On top of that, every active request needs its own chunk of GPU memory, the KV cache, to store the token context it has built up so far. In this course, you'll learn to reduce a model's memory footprint with quantization and serve it using vLLM, which handles many concurrent requests efficiently through smart memory management. Skills you'll gain: - Quantize a model and measure the accuracy tradeoff - Serve a model with vLLM and watch it handle concurrent requests efficiently - Benchmark your deployment and make informed tradeoffs between speed, cost, and accuracy Join and learn to serve LLMs efficiently:

New course on serving LLMs efficiently -- how do you serve models to many concurrent users at low latency and reasonable cost? This short course is built with Red Hat and taught by Cedric Clyburn. Efficient LLM serving requires efficient memory management. A 70B-parameter model takes ~140 GB just to load the weights. On top of that, every active request needs its own chunk of GPU memory, the KV cache, to store the token context it has built up so far. In this course, you'll learn to reduce a model's memory footprint with quantization and serve it using vLLM, which handles many concurrent requests efficiently through smart memory management. Skills you'll gain: - Quantize a model and measure the accuracy tradeoff - Serve a model with vLLM and watch it handle concurrent requests efficiently - Benchmark your deployment and make informed tradeoffs between speed, cost, and accuracy Join and learn to serve LLMs efficiently:

Andrew Ng

139,195 Aufrufe • vor 5 Monaten

Today we shipped Memory and Dreaming in Devin: persistent self-cleaning memory for agent sessions. The standard is OSS and also works across any harness in any environment. ⚡

Today we shipped Memory and Dreaming in Devin: persistent self-cleaning memory for agent sessions. The standard is OSS and also works across any harness in any environment. ⚡

nader dabit

47,271 Aufrufe • vor 1 Monat

New short course: LLMs as Operating Systems: Agent Memory, created with Letta, and taught by its founders Charles Packer and Sarah Wooders. An LLM's input context window has limited space. Using a longer input context also costs more and results in slower processing. So, managing what's stored in this context window is important. In the innovative paper MemGPT: Towards LLMs as Operating Systems, its authors (which include the instructors) proposed using an LLM agent to manage this context window. Their system uses a large persistent memory that stores everything that could be included in the input context, and an agent decides what is actually included. Take the example of building a chatbot that needs to remember what's been said earlier in a conversation (perhaps over many days of interaction with a user). As the conversation's length grows, the memory management agent will move information from the input context to a persistent searchable database; summarize information to keep relevant facts in the input context; and restore relevant conversation elements from further back in time. This allows a chatbot to keep what's currently most relevant in its input context memory to generate the next response. When I read the original MemGPT paper, I thought it was an innovative technique for handling memory for LLMs. The open-source Letta framework, which we'll use in this course, makes MemGPT easy to implement. It adds memory to your LLM agents and gives them transparent long-term memory. In detail, you’ll learn: - How to build an agent that can edit its own limited input context memory, using tools and multi-step reasoning - What is a memory hierarchy (an idea from computer operating systems, which use a cache to speed up memory access), and how these ideas apply to managing the LLM input context (where the input context window is a "cache" storing the most relevant information; and an agent decides what to move in and out of this to/from a larger persistent storage system) - How to implement multi-agent collaboration by letting different agents share blocks of memory This course will give you a sophisticated understanding of memory management for LLMs, which is important for chatbots having long conversations, and for complex agentic workflows. Please sign up here!

New short course: LLMs as Operating Systems: Agent Memory, created with Letta, and taught by its founders Charles Packer and Sarah Wooders. An LLM's input context window has limited space. Using a longer input context also costs more and results in slower processing. So, managing what's stored in this context window is important. In the innovative paper MemGPT: Towards LLMs as Operating Systems, its authors (which include the instructors) proposed using an LLM agent to manage this context window. Their system uses a large persistent memory that stores everything that could be included in the input context, and an agent decides what is actually included. Take the example of building a chatbot that needs to remember what's been said earlier in a conversation (perhaps over many days of interaction with a user). As the conversation's length grows, the memory management agent will move information from the input context to a persistent searchable database; summarize information to keep relevant facts in the input context; and restore relevant conversation elements from further back in time. This allows a chatbot to keep what's currently most relevant in its input context memory to generate the next response. When I read the original MemGPT paper, I thought it was an innovative technique for handling memory for LLMs. The open-source Letta framework, which we'll use in this course, makes MemGPT easy to implement. It adds memory to your LLM agents and gives them transparent long-term memory. In detail, you’ll learn: - How to build an agent that can edit its own limited input context memory, using tools and multi-step reasoning - What is a memory hierarchy (an idea from computer operating systems, which use a cache to speed up memory access), and how these ideas apply to managing the LLM input context (where the input context window is a "cache" storing the most relevant information; and an agent decides what to move in and out of this to/from a larger persistent storage system) - How to implement multi-agent collaboration by letting different agents share blocks of memory This course will give you a sophisticated understanding of memory management for LLMs, which is important for chatbots having long conversations, and for complex agentic workflows. Please sign up here!

Andrew Ng

201,127 Aufrufe • vor 2 Jahren

Amazing stuff! MatAnyone-E runs 4K video matting at 28 FPS on an H200 and fits on a 3090 without crashing. You get hair strands and soft transparency without a green screen - 7.4x speedup - drops peak GPU memory to 7.6%

Amazing stuff! MatAnyone-E runs 4K video matting at 28 FPS on an H200 and fits on a 3090 without crashing. You get hair strands and soft transparency without a green screen - 7.4x speedup - drops peak GPU memory to 7.6%

Wildminder

17,279 Aufrufe • vor 16 Tagen

🚀 Excited to release LongLive 2.0! 🎬 An end-to-end infrastructure for long video generation, with FP4 and parallelism at the core of both training and inference. ⚡45.7 FPS generation speed on 5B model⚡ ✨ LongLive 2.0 supports real-video training, few-step distillation, multi-shot training/inference, sequence-parallel acceleration, NVFP4 KV cache, and async VAE decoding deployment. 🧩 To our knowledge, this is the first open-source 4-bit long video generation infra that covers both training and inference. 🙌 Welcome to check it out, try it, and share feedback! 🔗 Code: 📰 Paper: 🎥 Demo: #LongVideoGeneration #VideoGeneration #Realtime #AIInfra #EfficientAI #FP4 #Parallel #NVIDIA

🚀 Excited to release LongLive 2.0! 🎬 An end-to-end infrastructure for long video generation, with FP4 and parallelism at the core of both training and inference. ⚡45.7 FPS generation speed on 5B model⚡ ✨ LongLive 2.0 supports real-video training, few-step distillation, multi-shot training/inference, sequence-parallel acceleration, NVFP4 KV cache, and async VAE decoding deployment. 🧩 To our knowledge, this is the first open-source 4-bit long video generation infra that covers both training and inference. 🙌 Welcome to check it out, try it, and share feedback! 🔗 Code: 📰 Paper: 🎥 Demo: #LongVideoGeneration #VideoGeneration #Realtime #AIInfra #EfficientAI #FP4 #Parallel #NVIDIA

Yukang Chen

60,351 Aufrufe • vor 5 Monaten

𝗞𝗩 𝗖𝗮𝗰𝗵𝗶𝗻𝗴 𝗘𝘅𝗽𝗹𝗮𝗶𝗻𝗲𝗱 Ever wondered how LLMs remember thousands of tokens without processing everything from scratch every time? That's where KV Caching comes in. When an LLM generates a response, it doesn't need to recompute the Key and Value representations of all previous tokens at every step. Instead, it stores them in a cache and reuses them while generating the next token. Sounds simple, but this makes a huge difference in inference speed and computational efficiency. Here's what happens: - During prefill, the model processes the prompt and stores the Keys and Values. - During decoding, it computes Q, K and V only for the new token. - The new K and V are added to the existing cache. - Previous KV values are reused instead of being recomputed. But there's one important thing to understand: KV Caching doesn't eliminate attention over previous tokens. It eliminates the need to recompute their Keys and Values. As context grows, KV cache memory can become a major bottleneck. That's why techniques like: - MQA (Multi-Query Attention) - GQA (Grouped-Query Attention) - PagedAttention - Prefix Caching play an important role in making LLM inference more efficient. Understanding KV Caching is essential if you're learning how LLM inference actually works or building systems that serve LLMs at scale.

𝗞𝗩 𝗖𝗮𝗰𝗵𝗶𝗻𝗴 𝗘𝘅𝗽𝗹𝗮𝗶𝗻𝗲𝗱 Ever wondered how LLMs remember thousands of tokens without processing everything from scratch every time? That's where KV Caching comes in. When an LLM generates a response, it doesn't need to recompute the Key and Value representations of all previous tokens at every step. Instead, it stores them in a cache and reuses them while generating the next token. Sounds simple, but this makes a huge difference in inference speed and computational efficiency. Here's what happens: - During prefill, the model processes the prompt and stores the Keys and Values. - During decoding, it computes Q, K and V only for the new token. - The new K and V are added to the existing cache. - Previous KV values are reused instead of being recomputed. But there's one important thing to understand: KV Caching doesn't eliminate attention over previous tokens. It eliminates the need to recompute their Keys and Values. As context grows, KV cache memory can become a major bottleneck. That's why techniques like: - MQA (Multi-Query Attention) - GQA (Grouped-Query Attention) - PagedAttention - Prefix Caching play an important role in making LLM inference more efficient. Understanding KV Caching is essential if you're learning how LLM inference actually works or building systems that serve LLMs at scale.

𝗿𝗮𝗺𝗮𝗸𝗿𝘂𝘀𝗵𝗻𝗮— 𝗲/𝗮𝗰𝗰

22,663 Aufrufe • vor 3 Tagen