Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

got k2 thinking working on vllm completely maxxing out vram on 8xH100 i span up. had to quantize the kv cache to fp8 and decrease seq len to 1024 or else it would OOM. this is not sped up (~3.0 toks/sec). this took about 2 hrs of tweaking serving... show more

Elliot Arledge

32,569 subscribers

39,295 Aufrufe • vor 11 Monaten •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

I added KV caching and INT8 KV quantization to our transformer inference, improving throughput by 35x. All of this was done from scratch in Rust + CUDA, on top of a homemade ML framework. On a 4-token prompt with 252 generated tokens: - Original: 0.76 tok/s - KV cache fp32: 27.21 tok/s - KV cache int8 (quantized): 27.29 tok/s Try it out yourself here: In practice: - KV caching gave us about a 35x end-to-end speedup - INT8 KV cache kept roughly the same speed as fp32 but cut KV cache memory by 3.78x FP32 cache used 4.5 MB in this run while the INT8 cache used only 1.19 MB This simple change to inference created a huge impact on performance. To learn more about the KV cache and other optimizations like this, check out the blog at

I added KV caching and INT8 KV quantization to our transformer inference, improving throughput by 35x. All of this was done from scratch in Rust + CUDA, on top of a homemade ML framework. On a 4-token prompt with 252 generated tokens: - Original: 0.76 tok/s - KV cache fp32: 27.21 tok/s - KV cache int8 (quantized): 27.29 tok/s Try it out yourself here: In practice: - KV caching gave us about a 35x end-to-end speedup - INT8 KV cache kept roughly the same speed as fp32 but cut KV cache memory by 3.78x FP32 cache used 4.5 MB in this run while the INT8 cache used only 1.19 MB This simple change to inference created a huge impact on performance. To learn more about the KV cache and other optimizations like this, check out the blog at

Reese Chong

53,026 Aufrufe • vor 5 Monaten

They really thought no one would catch this on tape... 😳 The mask completely slipped, and the media is working overtime to cover it up.

They really thought no one would catch this on tape... 😳 The mask completely slipped, and the media is working overtime to cover it up.

Watcher Reports

23,594 Aufrufe • vor 1 Monat

Hot off the press 🔥 AEON vLLM Ultimate → v0.24.0. Link below ⤵️ The ultimate container for AI on NVIDIA DGX Spark — vLLM compiled from source for GB10/sm_121a. 🔓 UNLOCKED (impossible until this release): ▸ DFlash + FP8 KV cache — 2× KV capacity, draft acceptance intact, measured on-device ▸ FP8 KV on vision models — the KV-bandwidth lever is back ▸ DFlash on FlashInfer & Triton backends (was FlashAttention-only) 🛠 Carried ahead of upstream (still-open PRs, merged + validated): ▸ Triton NVFP4 KV cache → up to 3× KV capacity ▸ DFlash sliding-window drafters + prefix-cache stability for DFlash ▸ High-concurrency DFlash — clean through c=64 🧰 Fixes stock v0.24.0 doesn't have: ▸ Gemma-4 tied embeddings (ModelOpt) — stock crashes at load, ours doesn't ▸ Tool calling out of the box (stock --no-deps builds 500) ▸ Unified-memory hardened — no silent KV overcommit on the 121 GB pool 📊 One image, the whole fleet: Gemma-4-26B-A4B · Qwen3.6-27B · Qwen3.6-35B-A3B (hybrid GDN) 536 tok/s @ c16 · 472 tok/s @ c12 · 262K context · image+video · tools + reasoning parsers · full CUDA-graph capture 🐳 docker pull

Hot off the press 🔥 AEON vLLM Ultimate → v0.24.0. Link below ⤵️ The ultimate container for AI on NVIDIA DGX Spark — vLLM compiled from source for GB10/sm_121a. 🔓 UNLOCKED (impossible until this release): ▸ DFlash + FP8 KV cache — 2× KV capacity, draft acceptance intact, measured on-device ▸ FP8 KV on vision models — the KV-bandwidth lever is back ▸ DFlash on FlashInfer & Triton backends (was FlashAttention-only) 🛠 Carried ahead of upstream (still-open PRs, merged + validated): ▸ Triton NVFP4 KV cache → up to 3× KV capacity ▸ DFlash sliding-window drafters + prefix-cache stability for DFlash ▸ High-concurrency DFlash — clean through c=64 🧰 Fixes stock v0.24.0 doesn't have: ▸ Gemma-4 tied embeddings (ModelOpt) — stock crashes at load, ours doesn't ▸ Tool calling out of the box (stock --no-deps builds 500) ▸ Unified-memory hardened — no silent KV overcommit on the 121 GB pool 📊 One image, the whole fleet: Gemma-4-26B-A4B · Qwen3.6-27B · Qwen3.6-35B-A3B (hybrid GDN) 536 tok/s @ c16 · 472 tok/s @ c12 · 262K context · image+video · tools + reasoning parsers · full CUDA-graph capture 🐳 docker pull

ÆON FORGE ✨

24,800 Aufrufe • vor 3 Monaten

Interesting approach to long context/continual learning here from Baseten at Cursor Compact long trajectories by compressing a prefix of the KV cache using an MLP/autoencoder. You can train this "compactor" MLP by learning to reconstruct activations that the original KV cache would produce on subsequent tokens. This maximally reconstructs information from long context that's useful for subsequent outputs. If you run an agent over extremely long context and run this compaction recursively, the activations of this compressed KV cache become like trained weights. Similar to task-specific LoRA or "cartridges". Would not be surprised if OpenAI is running a similar algorithm for their blackbox compaction. Clear benefits here if you can avoid busting the cache, as e.g. compacting via writing to a text file would. Seeing many emerging approaches to fixed size latent states for LLMs that seem promising. If building a task-specific KV cache compression ends up being more sample efficient than running backprop, getting this to "work" feels like one of the 2-3 remaining breakthroughs on the path to true AGI

Interesting approach to long context/continual learning here from Baseten at Cursor Compact long trajectories by compressing a prefix of the KV cache using an MLP/autoencoder. You can train this "compactor" MLP by learning to reconstruct activations that the original KV cache would produce on subsequent tokens. This maximally reconstructs information from long context that's useful for subsequent outputs. If you run an agent over extremely long context and run this compaction recursively, the activations of this compressed KV cache become like trained weights. Similar to task-specific LoRA or "cartridges". Would not be surprised if OpenAI is running a similar algorithm for their blackbox compaction. Clear benefits here if you can avoid busting the cache, as e.g. compacting via writing to a text file would. Seeing many emerging approaches to fixed size latent states for LLMs that seem promising. If building a task-specific KV cache compression ends up being more sample efficient than running backprop, getting this to "work" feels like one of the 2-3 remaining breakthroughs on the path to true AGI

Jay Hack

30,031 Aufrufe • vor 3 Monaten

That Is not the song used on that animated music video. This is. ....thinking about it. This and one piece. I really used to listen to a lot of 코요테 songs...

That Is not the song used on that animated music video. This is. ....thinking about it. This and one piece. I really used to listen to a lot of 코요테 songs...

INKD

101,044 Aufrufe • vor 1 Monat

I'm not sure if anyone else did this, but I would rub the end of a mercury thermometer on a blanket to act like I had a fever. One day I overdid it and the thermometer said I had 110° temperature, and the gig was up.

I'm not sure if anyone else did this, but I would rub the end of a mercury thermometer on a blanket to act like I had a fever. One day I overdid it and the thermometer said I had 110° temperature, and the gig was up.

Growing Up Retro

23,766 Aufrufe • vor 1 Monat

I had to pause and rewatch this part. This debate got HEATED fast: blanket immunity vs. letting the market decide, and neither side was backing down. You will NOT believe the argument that came next — I'm still thinking about it. Whose side are you on? Watch to the end and tell me in the comments, because this one is about to blow up. Watch the full episode on my YT channel! Link is in the comments 📷

I had to pause and rewatch this part. This debate got HEATED fast: blanket immunity vs. letting the market decide, and neither side was backing down. You will NOT believe the argument that came next — I'm still thinking about it. Whose side are you on? Watch to the end and tell me in the comments, because this one is about to blow up. Watch the full episode on my YT channel! Link is in the comments 📷

Jillian Michaels

29,631 Aufrufe • vor 1 Monat

can’t stop thinking about this clip of tom holland completely losing it and being unable to stop laughing over amy’s joke about not being in the odyssey since everyone else is

can’t stop thinking about this clip of tom holland completely losing it and being unable to stop laughing over amy’s joke about not being in the odyssey since everyone else is

chris ༯

666,449 Aufrufe • vor 4 Monaten

Russell Wilson is asked if he would request a trade or release from the Giants: "I'm focused on helping this team win...I want to be here. I love this organization, I love the process of it all. I'm not giving up on us and this season."

Russell Wilson is asked if he would request a trade or release from the Giants: "I'm focused on helping this team win...I want to be here. I love this organization, I love the process of it all. I'm not giving up on us and this season."

SNY Giants

190,507 Aufrufe • vor 1 Jahr

BADDIES USA CHAPTER 2 REUNION PART 1 🍿 Finalllly , this is out so I can talk . I been holding it in for so long waiting on it to drop . But this is my opinion from THE AUDIENCE !! Let’s get into the tea & some of the stuff yall missed out on to give more context to what yall seen cause that chopped this up reunion up baby 😭 sped through parts and all that. Let’s talk in the comments 🎤 #baddiesusa #isaiahjaay #zeusnetwork

BADDIES USA CHAPTER 2 REUNION PART 1 🍿 Finalllly , this is out so I can talk . I been holding it in for so long waiting on it to drop . But this is my opinion from THE AUDIENCE !! Let’s get into the tea & some of the stuff yall missed out on to give more context to what yall seen cause that chopped this up reunion up baby 😭 sped through parts and all that. Let’s talk in the comments 🎤 #baddiesusa #isaiahjaay #zeusnetwork

3/12 🎈

48,710 Aufrufe • vor 4 Tagen

xiaojun has a PSA to make 🤣 'from this point on i will NOT be working up to bulk up any more. i think my current built is perfect and i would like to preserve it this way. so from now i won't need to increase my muscle mass, but i will only work out to define my figure'

xiaojun has a PSA to make 🤣 'from this point on i will NOT be working up to bulk up any more. i think my current built is perfect and i would like to preserve it this way. so from now i won't need to increase my muscle mass, but i will only work out to define my figure'

🍒 XIAOJUN WAVES

17,935 Aufrufe • vor 4 Monaten

New course on serving LLMs efficiently -- how do you serve models to many concurrent users at low latency and reasonable cost? This short course is built with Red Hat and taught by Cedric Clyburn. Efficient LLM serving requires efficient memory management. A 70B-parameter model takes ~140 GB just to load the weights. On top of that, every active request needs its own chunk of GPU memory, the KV cache, to store the token context it has built up so far. In this course, you'll learn to reduce a model's memory footprint with quantization and serve it using vLLM, which handles many concurrent requests efficiently through smart memory management. Skills you'll gain: - Quantize a model and measure the accuracy tradeoff - Serve a model with vLLM and watch it handle concurrent requests efficiently - Benchmark your deployment and make informed tradeoffs between speed, cost, and accuracy Join and learn to serve LLMs efficiently:

New course on serving LLMs efficiently -- how do you serve models to many concurrent users at low latency and reasonable cost? This short course is built with Red Hat and taught by Cedric Clyburn. Efficient LLM serving requires efficient memory management. A 70B-parameter model takes ~140 GB just to load the weights. On top of that, every active request needs its own chunk of GPU memory, the KV cache, to store the token context it has built up so far. In this course, you'll learn to reduce a model's memory footprint with quantization and serve it using vLLM, which handles many concurrent requests efficiently through smart memory management. Skills you'll gain: - Quantize a model and measure the accuracy tradeoff - Serve a model with vLLM and watch it handle concurrent requests efficiently - Benchmark your deployment and make informed tradeoffs between speed, cost, and accuracy Join and learn to serve LLMs efficiently:

Andrew Ng

136,970 Aufrufe • vor 4 Monaten

$544K PnL in August. A few years ago, my parents kicked me out. I had no idea where I was going to end up, and honestly, there were times I questioned whether any of this would work out. Trading became something I took seriously when I had nothing else to fall back on. A lot of late nights, losses, mistakes, and starting over led to this. $544K this month means a lot, but not just because of the number. I know exactly where I started.

$544K PnL in August. A few years ago, my parents kicked me out. I had no idea where I was going to end up, and honestly, there were times I questioned whether any of this would work out. Trading became something I took seriously when I had nothing else to fall back on. A lot of late nights, losses, mistakes, and starting over led to this. $544K this month means a lot, but not just because of the number. I know exactly where I started.

rage

72,504 Aufrufe • vor 1 Monat

/// UPDATE /// So, I have now upgraded to the Panasonic UB9000. The second disc that I inserted to test was BLACK PHONE 2 on 4K. It booted up immediately, whereas the UB820 player about nine times out of ten would fail to even read the disc. Here’s a video of it booting up on the UB9000. This is the exact same disc. It boots up within about 15 to 20 seconds. No joke!

/// UPDATE /// So, I have now upgraded to the Panasonic UB9000. The second disc that I inserted to test was BLACK PHONE 2 on 4K. It booted up immediately, whereas the UB820 player about nine times out of ten would fail to even read the disc. Here’s a video of it booting up on the UB9000. This is the exact same disc. It boots up within about 15 to 20 seconds. No joke!

HighDefDiscNews.com

100,534 Aufrufe • vor 9 Monaten

Heading up to the Eurostar One idea I have is to replace the travelator with escalators to free up space BUT giving it more thought I’m thinking luggage on escalators would be the issue to kibosh this.

Heading up to the Eurostar One idea I have is to replace the travelator with escalators to free up space BUT giving it more thought I’m thinking luggage on escalators would be the issue to kibosh this.

AndyBTravels

18,823 Aufrufe • vor 3 Monaten

timelapse #85 (27.5 hrs): - currently cant rely on any other coding models except grok code fast 1 + grok 4 fast (for complex reasoning grok 4 fast is 20 cents for 1M tokens) - wrote qwen3-next trainer entirely from scratch to make it more managable - each piece completely done by grok-code-fast-1 in cursor as it seems to handle this task pretty well without the grok 4 fast reasoning - take on smaller problems and complete them quickly (makes it easier with 400 toks/sec over the api) - got distributed fp8 qwen3-next trainer running at 0.8 seconds per step on 8xH100s (still need to finish checkpoint loading logic) - perfect timing as the fp8 version of qwen3-next drops as im writing this - ill be in LA in 2 days (will visit SF mid way through as well) - 12.5% margarita - steak dinner with family - gained intuition on FlashAttention in very long context settings - caught up w/ Kearm h/eng and Arnie Ramesh

timelapse #85 (27.5 hrs): - currently cant rely on any other coding models except grok code fast 1 + grok 4 fast (for complex reasoning grok 4 fast is 20 cents for 1M tokens) - wrote qwen3-next trainer entirely from scratch to make it more managable - each piece completely done by grok-code-fast-1 in cursor as it seems to handle this task pretty well without the grok 4 fast reasoning - take on smaller problems and complete them quickly (makes it easier with 400 toks/sec over the api) - got distributed fp8 qwen3-next trainer running at 0.8 seconds per step on 8xH100s (still need to finish checkpoint loading logic) - perfect timing as the fp8 version of qwen3-next drops as im writing this - ill be in LA in 2 days (will visit SF mid way through as well) - 12.5% margarita - steak dinner with family - gained intuition on FlashAttention in very long context settings - caught up w/ Kearm h/eng and Arnie Ramesh

Elliot Arledge

283,820 Aufrufe • vor 1 Jahr

See there's 2 sides to hand shuffling. Using it as a thinking tic or to pass time during opponents turn. Then there's this 💀 (sped up x2 btw) This the type of shuffling that annoys people, gets nothing done, and made bandai make a vid about shaka shuffling 💀

See there's 2 sides to hand shuffling. Using it as a thinking tic or to pass time during opponents turn. Then there's this 💀 (sped up x2 btw) This the type of shuffling that annoys people, gets nothing done, and made bandai make a vid about shaka shuffling 💀

ジャスティンJansuko

232,532 Aufrufe • vor 5 Monaten

Gearing up for the start of Lent and thinking of something I may want to change or adjust or give up. This is speaking to me. The best part is the gaining of deeper connections and being more present. It’s a win-win. Try it! Let’s do this!

Gearing up for the start of Lent and thinking of something I may want to change or adjust or give up. This is speaking to me. The best part is the gaining of deeper connections and being more present. It’s a win-win. Try it! Let’s do this!

Bridget Moynahan

17,424 Aufrufe • vor 1 Jahr