正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

got k2 thinking working on vllm completely maxxing out vram on 8xH100 i span up. had to quantize the kv cache to fp8 and decrease seq len to 1024 or else it would OOM. this is not sped up (~3.0 toks/sec). this took about 2 hrs of tweaking serving... show more

Elliot Arledge

32,569 subscribers

39,295 次观看 • 9 个月前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

I added KV caching and INT8 KV quantization to our transformer inference, improving throughput by 35x. All of this was done from scratch in Rust + CUDA, on top of a homemade ML framework. On a 4-token prompt with 252 generated tokens: - Original: 0.76 tok/s - KV cache fp32: 27.21 tok/s - KV cache int8 (quantized): 27.29 tok/s Try it out yourself here: In practice: - KV caching gave us about a 35x end-to-end speedup - INT8 KV cache kept roughly the same speed as fp32 but cut KV cache memory by 3.78x FP32 cache used 4.5 MB in this run while the INT8 cache used only 1.19 MB This simple change to inference created a huge impact on performance. To learn more about the KV cache and other optimizations like this, check out the blog at

I added KV caching and INT8 KV quantization to our transformer inference, improving throughput by 35x. All of this was done from scratch in Rust + CUDA, on top of a homemade ML framework. On a 4-token prompt with 252 generated tokens: - Original: 0.76 tok/s - KV cache fp32: 27.21 tok/s - KV cache int8 (quantized): 27.29 tok/s Try it out yourself here: In practice: - KV caching gave us about a 35x end-to-end speedup - INT8 KV cache kept roughly the same speed as fp32 but cut KV cache memory by 3.78x FP32 cache used 4.5 MB in this run while the INT8 cache used only 1.19 MB This simple change to inference created a huge impact on performance. To learn more about the KV cache and other optimizations like this, check out the blog at

Reese Chong

53,026 次观看 • 3 个月前

When it became clear Roe would be overturned, we had a choice – give up, or fight back. Alongside so many of you, I chose to stand up, to speak out, and to fight. This is what I said on the Senate floor:

When it became clear Roe would be overturned, we had a choice – give up, or fight back. Alongside so many of you, I chose to stand up, to speak out, and to fight. This is what I said on the Senate floor:

Mallory McMorrow

11,760 次观看 • 1 年前

Hot off the press 🔥 AEON vLLM Ultimate → v0.24.0. Link below ⤵️ The ultimate container for AI on NVIDIA DGX Spark — vLLM compiled from source for GB10/sm_121a. 🔓 UNLOCKED (impossible until this release): ▸ DFlash + FP8 KV cache — 2× KV capacity, draft acceptance intact, measured on-device ▸ FP8 KV on vision models — the KV-bandwidth lever is back ▸ DFlash on FlashInfer & Triton backends (was FlashAttention-only) 🛠 Carried ahead of upstream (still-open PRs, merged + validated): ▸ Triton NVFP4 KV cache → up to 3× KV capacity ▸ DFlash sliding-window drafters + prefix-cache stability for DFlash ▸ High-concurrency DFlash — clean through c=64 🧰 Fixes stock v0.24.0 doesn't have: ▸ Gemma-4 tied embeddings (ModelOpt) — stock crashes at load, ours doesn't ▸ Tool calling out of the box (stock --no-deps builds 500) ▸ Unified-memory hardened — no silent KV overcommit on the 121 GB pool 📊 One image, the whole fleet: Gemma-4-26B-A4B · Qwen3.6-27B · Qwen3.6-35B-A3B (hybrid GDN) 536 tok/s @ c16 · 472 tok/s @ c12 · 262K context · image+video · tools + reasoning parsers · full CUDA-graph capture 🐳 docker pull

Hot off the press 🔥 AEON vLLM Ultimate → v0.24.0. Link below ⤵️ The ultimate container for AI on NVIDIA DGX Spark — vLLM compiled from source for GB10/sm_121a. 🔓 UNLOCKED (impossible until this release): ▸ DFlash + FP8 KV cache — 2× KV capacity, draft acceptance intact, measured on-device ▸ FP8 KV on vision models — the KV-bandwidth lever is back ▸ DFlash on FlashInfer & Triton backends (was FlashAttention-only) 🛠 Carried ahead of upstream (still-open PRs, merged + validated): ▸ Triton NVFP4 KV cache → up to 3× KV capacity ▸ DFlash sliding-window drafters + prefix-cache stability for DFlash ▸ High-concurrency DFlash — clean through c=64 🧰 Fixes stock v0.24.0 doesn't have: ▸ Gemma-4 tied embeddings (ModelOpt) — stock crashes at load, ours doesn't ▸ Tool calling out of the box (stock --no-deps builds 500) ▸ Unified-memory hardened — no silent KV overcommit on the 121 GB pool 📊 One image, the whole fleet: Gemma-4-26B-A4B · Qwen3.6-27B · Qwen3.6-35B-A3B (hybrid GDN) 536 tok/s @ c16 · 472 tok/s @ c12 · 262K context · image+video · tools + reasoning parsers · full CUDA-graph capture 🐳 docker pull

ÆON FORGE ✨

24,800 次观看 • 1 个月前

Interesting approach to long context/continual learning here from Baseten at Cursor Compact long trajectories by compressing a prefix of the KV cache using an MLP/autoencoder. You can train this "compactor" MLP by learning to reconstruct activations that the original KV cache would produce on subsequent tokens. This maximally reconstructs information from long context that's useful for subsequent outputs. If you run an agent over extremely long context and run this compaction recursively, the activations of this compressed KV cache become like trained weights. Similar to task-specific LoRA or "cartridges". Would not be surprised if OpenAI is running a similar algorithm for their blackbox compaction. Clear benefits here if you can avoid busting the cache, as e.g. compacting via writing to a text file would. Seeing many emerging approaches to fixed size latent states for LLMs that seem promising. If building a task-specific KV cache compression ends up being more sample efficient than running backprop, getting this to "work" feels like one of the 2-3 remaining breakthroughs on the path to true AGI

Interesting approach to long context/continual learning here from Baseten at Cursor Compact long trajectories by compressing a prefix of the KV cache using an MLP/autoencoder. You can train this "compactor" MLP by learning to reconstruct activations that the original KV cache would produce on subsequent tokens. This maximally reconstructs information from long context that's useful for subsequent outputs. If you run an agent over extremely long context and run this compaction recursively, the activations of this compressed KV cache become like trained weights. Similar to task-specific LoRA or "cartridges". Would not be surprised if OpenAI is running a similar algorithm for their blackbox compaction. Clear benefits here if you can avoid busting the cache, as e.g. compacting via writing to a text file would. Seeing many emerging approaches to fixed size latent states for LLMs that seem promising. If building a task-specific KV cache compression ends up being more sample efficient than running backprop, getting this to "work" feels like one of the 2-3 remaining breakthroughs on the path to true AGI

Jay Hack

29,792 次观看 • 1 个月前

can’t stop thinking about this clip of tom holland completely losing it and being unable to stop laughing over amy’s joke about not being in the odyssey since everyone else is

can’t stop thinking about this clip of tom holland completely losing it and being unable to stop laughing over amy’s joke about not being in the odyssey since everyone else is

chris ༯

666,449 次观看 • 2 个月前

New course on serving LLMs efficiently -- how do you serve models to many concurrent users at low latency and reasonable cost? This short course is built with Red Hat and taught by Cedric Clyburn. Efficient LLM serving requires efficient memory management. A 70B-parameter model takes ~140 GB just to load the weights. On top of that, every active request needs its own chunk of GPU memory, the KV cache, to store the token context it has built up so far. In this course, you'll learn to reduce a model's memory footprint with quantization and serve it using vLLM, which handles many concurrent requests efficiently through smart memory management. Skills you'll gain: - Quantize a model and measure the accuracy tradeoff - Serve a model with vLLM and watch it handle concurrent requests efficiently - Benchmark your deployment and make informed tradeoffs between speed, cost, and accuracy Join and learn to serve LLMs efficiently:

New course on serving LLMs efficiently -- how do you serve models to many concurrent users at low latency and reasonable cost? This short course is built with Red Hat and taught by Cedric Clyburn. Efficient LLM serving requires efficient memory management. A 70B-parameter model takes ~140 GB just to load the weights. On top of that, every active request needs its own chunk of GPU memory, the KV cache, to store the token context it has built up so far. In this course, you'll learn to reduce a model's memory footprint with quantization and serve it using vLLM, which handles many concurrent requests efficiently through smart memory management. Skills you'll gain: - Quantize a model and measure the accuracy tradeoff - Serve a model with vLLM and watch it handle concurrent requests efficiently - Benchmark your deployment and make informed tradeoffs between speed, cost, and accuracy Join and learn to serve LLMs efficiently:

Andrew Ng

130,549 次观看 • 2 个月前

xiaojun has a PSA to make 🤣 'from this point on i will NOT be working up to bulk up any more. i think my current built is perfect and i would like to preserve it this way. so from now i won't need to increase my muscle mass, but i will only work out to define my figure'

xiaojun has a PSA to make 🤣 'from this point on i will NOT be working up to bulk up any more. i think my current built is perfect and i would like to preserve it this way. so from now i won't need to increase my muscle mass, but i will only work out to define my figure'

🍒 XIAOJUN WAVES

17,935 次观看 • 3 个月前

Russell Wilson is asked if he would request a trade or release from the Giants: "I'm focused on helping this team win...I want to be here. I love this organization, I love the process of it all. I'm not giving up on us and this season."

Russell Wilson is asked if he would request a trade or release from the Giants: "I'm focused on helping this team win...I want to be here. I love this organization, I love the process of it all. I'm not giving up on us and this season."

SNY Giants

190,507 次观看 • 10 个月前

timelapse #85 (27.5 hrs): - currently cant rely on any other coding models except grok code fast 1 + grok 4 fast (for complex reasoning grok 4 fast is 20 cents for 1M tokens) - wrote qwen3-next trainer entirely from scratch to make it more managable - each piece completely done by grok-code-fast-1 in cursor as it seems to handle this task pretty well without the grok 4 fast reasoning - take on smaller problems and complete them quickly (makes it easier with 400 toks/sec over the api) - got distributed fp8 qwen3-next trainer running at 0.8 seconds per step on 8xH100s (still need to finish checkpoint loading logic) - perfect timing as the fp8 version of qwen3-next drops as im writing this - ill be in LA in 2 days (will visit SF mid way through as well) - 12.5% margarita - steak dinner with family - gained intuition on FlashAttention in very long context settings - caught up w/ Kearm h/eng and Arnie Ramesh

timelapse #85 (27.5 hrs): - currently cant rely on any other coding models except grok code fast 1 + grok 4 fast (for complex reasoning grok 4 fast is 20 cents for 1M tokens) - wrote qwen3-next trainer entirely from scratch to make it more managable - each piece completely done by grok-code-fast-1 in cursor as it seems to handle this task pretty well without the grok 4 fast reasoning - take on smaller problems and complete them quickly (makes it easier with 400 toks/sec over the api) - got distributed fp8 qwen3-next trainer running at 0.8 seconds per step on 8xH100s (still need to finish checkpoint loading logic) - perfect timing as the fp8 version of qwen3-next drops as im writing this - ill be in LA in 2 days (will visit SF mid way through as well) - 12.5% margarita - steak dinner with family - gained intuition on FlashAttention in very long context settings - caught up w/ Kearm h/eng and Arnie Ramesh

Elliot Arledge

283,820 次观看 • 10 个月前

/// UPDATE /// So, I have now upgraded to the Panasonic UB9000. The second disc that I inserted to test was BLACK PHONE 2 on 4K. It booted up immediately, whereas the UB820 player about nine times out of ten would fail to even read the disc. Here’s a video of it booting up on the UB9000. This is the exact same disc. It boots up within about 15 to 20 seconds. No joke!

/// UPDATE /// So, I have now upgraded to the Panasonic UB9000. The second disc that I inserted to test was BLACK PHONE 2 on 4K. It booted up immediately, whereas the UB820 player about nine times out of ten would fail to even read the disc. Here’s a video of it booting up on the UB9000. This is the exact same disc. It boots up within about 15 to 20 seconds. No joke!

HighDefDiscNews.com

100,503 次观看 • 7 个月前

Heading up to the Eurostar One idea I have is to replace the travelator with escalators to free up space BUT giving it more thought I’m thinking luggage on escalators would be the issue to kibosh this.

Heading up to the Eurostar One idea I have is to replace the travelator with escalators to free up space BUT giving it more thought I’m thinking luggage on escalators would be the issue to kibosh this.

AndyBTravels

18,823 次观看 • 1 个月前

See there's 2 sides to hand shuffling. Using it as a thinking tic or to pass time during opponents turn. Then there's this 💀 (sped up x2 btw) This the type of shuffling that annoys people, gets nothing done, and made bandai make a vid about shaka shuffling 💀

See there's 2 sides to hand shuffling. Using it as a thinking tic or to pass time during opponents turn. Then there's this 💀 (sped up x2 btw) This the type of shuffling that annoys people, gets nothing done, and made bandai make a vid about shaka shuffling 💀

ジャスティンJansuko

232,532 次观看 • 3 个月前

Gearing up for the start of Lent and thinking of something I may want to change or adjust or give up. This is speaking to me. The best part is the gaining of deeper connections and being more present. It’s a win-win. Try it! Let’s do this!

Gearing up for the start of Lent and thinking of something I may want to change or adjust or give up. This is speaking to me. The best part is the gaining of deeper connections and being more present. It’s a win-win. Try it! Let’s do this!

Bridget Moynahan

17,424 次观看 • 1 年前

When isolating our thinking of the ‘pec stretch’ to a position like flip up, we’re not respecting what’s truly going on in order to create a stretch and load on this tissue. They’re actually lengthened as the arms go past the midline, not up or out.

When isolating our thinking of the ‘pec stretch’ to a position like flip up, we’re not respecting what’s truly going on in order to create a stretch and load on this tissue. They’re actually lengthened as the arms go past the midline, not up or out.

Ben Baggett

15,901 次观看 • 1 年前

figured out how Cascadeur – The Future of Animation works. this is mocap i recorded on an #iPhone and converted to #UE5 skeleton with QuickMagic. been working on this pipeline for months, but this particular animation only took like 2 hours. quality like this would have cost $10,000 previously

figured out how Cascadeur – The Future of Animation works. this is mocap i recorded on an #iPhone and converted to #UE5 skeleton with QuickMagic. been working on this pipeline for months, but this particular animation only took like 2 hours. quality like this would have cost $10,000 previously

Andres Hernandez

14,243 次观看 • 1 个月前

.@JamesTalarico: We've got to get to the bottom of this Epstein pedophile ring. We have to know who enabled it, who supported it, and who covered it up. This is not about right versus left. This is right versus wrong`

.@JamesTalarico: We've got to get to the bottom of this Epstein pedophile ring. We have to know who enabled it, who supported it, and who covered it up. This is not about right versus left. This is right versus wrong`

Team Talarico

100,016 次观看 • 9 个月前

Just had to share this video I just downloaded from my camera of the INSANE pulsating aurora last night in Fairbanks, Alaska. Mesmerizing to see the sky light up like this. The video is not sped up at all and the pulsations were easily visible to the naked eye.

Just had to share this video I just downloaded from my camera of the INSANE pulsating aurora last night in Fairbanks, Alaska. Mesmerizing to see the sky light up like this. The video is not sped up at all and the pulsations were easily visible to the naked eye.

Vincent Ledvina

57,461 次观看 • 6 个月前

GLM 5.2 @ 19.8 tok/sec on 2 DGX Sparks ⚡️ I switched to a dspark draft model with K=2 (model from Red Hat AI) Acceptance is ~68% because the drafter was trained with the full FP8 model... Next step is to fine tune the drafter so acceptance on this super quant goes up

GLM 5.2 @ 19.8 tok/sec on 2 DGX Sparks ⚡️ I switched to a dspark draft model with K=2 (model from Red Hat AI) Acceptance is ~68% because the drafter was trained with the full FP8 model... Next step is to fine tune the drafter so acceptance on this super quant goes up

Joe Muller

73,145 次观看 • 27 天前