正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Jointly announcing EAGLE-3 with SGLang: Setting a new record in LLM inference acceleration! - 5x🚀than vanilla (on HF) - 1.4x🚀than EAGLE-2 (on HF) - A record of ~400 TPS on LLama 3.1 8B with a single H100 (on SGLang) - 1.65x🚀in latency even for large bs=64 (on SGLang) -... A new scaling law: more training data, better speedup - Apache 2.0 Paper: Code: SGLang version: ⚒️Takeaway: Introducing training-time test, a novel draft model training technique: we replace feature prediction with direct token prediction and shift from top-layer-only features to multi-layer feature fusion. This approach unlocks a new scaling law previously undiscovered in EAGLE and EAGLE-2. 🙏Acknowledge: We would like to thank the SGLang team (zhyncs Lianmin Zheng Ying Sheng @ SGLang Summit James Liu, Ke Bao, and others LMSYS Org) for their merge and careful evaluation of EAGLE-3 on SGLang. 🤝Want to collaborate? We're a small academic group with limited GPU resources. If you're interested in supporting our next version of EAGLE or would like us to train a preliminary version tailored to a specific model, please get in touch! Joint work with Yuhui Li, Fangyun Wei, and Chao Zhangshow more

Hongyang Zhang

3,794 subscribers

42,856 次观看 • 1 年前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

Introduce EAGLE, a new method for fast LLM decoding based on compression: - 3x🚀than vanilla - 2x🚀 than Lookahead (on its benchmark) - 1.6x🚀 than Medusa (on its benchmark) - provably maintains text distribution - trainable (in 1~2 days) and testable on RTX 3090s Playground: Blog: Code: ⚒️First Principle: Compression! Yi Ma We find that the sequence of second-top-layer features is compressible, making the prediction of subsequent feature vectors from previous ones easy by a small model. 🙏Acknowledge: This project is greatly inspired by the Medusa team (Tianle Cai @yli3521 Zhengyang Geng Hongwu Peng Tri Dao), the Lookahead team (Hao Zhang LMSYS Org), and others. Joint work with Yuhui Li and Chao Zhang

Introduce EAGLE, a new method for fast LLM decoding based on compression: - 3x🚀than vanilla - 2x🚀 than Lookahead (on its benchmark) - 1.6x🚀 than Medusa (on its benchmark) - provably maintains text distribution - trainable (in 1~2 days) and testable on RTX 3090s Playground: Blog: Code: ⚒️First Principle: Compression! Yi Ma We find that the sequence of second-top-layer features is compressible, making the prediction of subsequent feature vectors from previous ones easy by a small model. 🙏Acknowledge: This project is greatly inspired by the Medusa team (Tianle Cai @yli3521 Zhengyang Geng Hongwu Peng Tri Dao), the Lookahead team (Hao Zhang LMSYS Org), and others. Joint work with Yuhui Li and Chao Zhang

Hongyang Zhang

118,934 次观看 • 2 年前

🚀 Self-speculation brings 6.75x real speedup for LLM generation with SGLang inference! Same model drafts future tokens in Diffusion mode → then verifies them in AR (causal) mode. One model and one KV cache. Just different attention masks. Thanks to perfect alignment, we get 2× longer acceptance lengths than MTP techniques (Eagle-3, MTP, dFlash). We run 2 forward passes… but the 2× higher acceptance means we break even - and with zero overhead from extra drafter, KV cache, or LM head that comes with MTP - those are not free. Last week we released Nemotron-Labs-Diffusion + Tri-mode LLMs! We did continued pre-training on Ministral-3 models by switching attention patterns (block causal bidirectional). Result: one model that runs AR mode, Diffusion mode, and Self-Speculation. Diffusion mode already shows high benchmark accuracy - excited to see what happens when someone beats left-to-right acceptance! 🔥 Github: Paper: SGLang inference: Try the models on HF:

🚀 Self-speculation brings 6.75x real speedup for LLM generation with SGLang inference! Same model drafts future tokens in Diffusion mode → then verifies them in AR (causal) mode. One model and one KV cache. Just different attention masks. Thanks to perfect alignment, we get 2× longer acceptance lengths than MTP techniques (Eagle-3, MTP, dFlash). We run 2 forward passes… but the 2× higher acceptance means we break even - and with zero overhead from extra drafter, KV cache, or LM head that comes with MTP - those are not free. Last week we released Nemotron-Labs-Diffusion + Tri-mode LLMs! We did continued pre-training on Ministral-3 models by switching attention patterns (block causal bidirectional). Result: one model that runs AR mode, Diffusion mode, and Self-Speculation. Diffusion mode already shows high benchmark accuracy - excited to see what happens when someone beats left-to-right acceptance! 🔥 Github: Paper: SGLang inference: Try the models on HF:

Pavlo Molchanov

66,604 次观看 • 5 个月前

Can $300 RTX3060 be faster than $10,000 A100 for LLM decoding?🧐Yes Introduce EAGLE-2: - 4.26x🚀vanilla (13B) - 1.4x🚀EAGLE (13B) - No extra training than EAGLE - Maintain distribution Paper: Code: Demo:

Can $300 RTX3060 be faster than $10,000 A100 for LLM decoding?🧐Yes Introduce EAGLE-2: - 4.26x🚀vanilla (13B) - 1.4x🚀EAGLE (13B) - No extra training than EAGLE - Maintain distribution Paper: Code: Demo:

Hongyang Zhang

35,765 次观看 • 2 年前

Added a fun lil widget to the LLM Engineer's Almanac -- a "Token Timing Simulator" so you can get a visceral feel for what a benchmark perf number means. Here's David Wang's latest work with Zhijian Liu's DFlash technique in SGLang -- ~1k TPS!

Added a fun lil widget to the LLM Engineer's Almanac -- a "Token Timing Simulator" so you can get a visceral feel for what a benchmark perf number means. Here's David Wang's latest work with Zhijian Liu's DFlash technique in SGLang -- ~1k TPS!

Charles 🎉 Frye

18,764 次观看 • 6 个月前

MiniMax H3 is more than a video generation model. Its open-weight release enables developers to build on a powerful multimodal system with generation, editing, and consistency features. Supported by ComfyUI, Diffusers, SGLang Diffusion, and Hugging Face, it simplifies integration and could inspire new AI video applications.

MiniMax H3 is more than a video generation model. Its open-weight release enables developers to build on a powerful multimodal system with generation, editing, and consistency features. Supported by ComfyUI, Diffusers, SGLang Diffusion, and Hugging Face, it simplifies integration and could inspire new AI video applications.

Sean Wallace

51,790 次观看 • 3 个月前

First look at Qwen 3.8-Flash-Next on 2 DGX Sparks with SGLang RadixArk’s NVFP4 in OMP

First look at Qwen 3.8-Flash-Next on 2 DGX Sparks with SGLang RadixArk’s NVFP4 in OMP

Joey

16,060 次观看 • 2 个月前

Inkling-small is out today! With SGLang, you can get 648 tok/s decode with DSpark (simulated acc len=4) and 288 tok/s w/o DSpark, under the same setup (8x NVIDIA AI B200, TP 8, NVFP4, bs=1). What makes this model different is the size. 276B total with 12B active is a sweet spot for RL, and both LoRA and full-parameter training become well within reach. Miles is ready and verified for multimodal RL on Inkling-small, so you can turn your multimodal data into real capability gains. At ~1/4 the size, Inkling-small matches the bigger version in capability and even wins on some benchmarks. Run Inkling-small with SGLang, and customize it with Miles.

Inkling-small is out today! With SGLang, you can get 648 tok/s decode with DSpark (simulated acc len=4) and 288 tok/s w/o DSpark, under the same setup (8x NVIDIA AI B200, TP 8, NVFP4, bs=1). What makes this model different is the size. 276B total with 12B active is a sweet spot for RL, and both LoRA and full-parameter training become well within reach. Miles is ready and verified for multimodal RL on Inkling-small, so you can turn your multimodal data into real capability gains. At ~1/4 the size, Inkling-small matches the bigger version in capability and even wins on some benchmarks. Run Inkling-small with SGLang, and customize it with Miles.

LMSYS Org

120,257 次观看 • 3 个月前

Grok 2 mini is now 2x faster than it was yesterday. In the last three days Lianmin Zheng and Saeed Maleki rewrote our inference stack from scratch using SGLang ( This has also allowed us to serve the big Grok 2 model, which requires multi-host inference, at a reasonable speed. Both models didn’t just get faster, but also slightly more accurate. Stay tuned for further speed improvements!

Grok 2 mini is now 2x faster than it was yesterday. In the last three days Lianmin Zheng and Saeed Maleki rewrote our inference stack from scratch using SGLang ( This has also allowed us to serve the big Grok 2 model, which requires multi-host inference, at a reasonable speed. Both models didn’t just get faster, but also slightly more accurate. Stay tuned for further speed improvements!

Igor Babuschkin

1,803,113 次观看 • 2 年前

You want to serve large OCR models at low latency? I've been playing around with speculative decoding for Vik Paruchuri/Datalab's Chandra 2 and I'm releasing an initial EAGLE-3.1 model that gets a ~2x speedup at small batch sizes!

You want to serve large OCR models at low latency? I've been playing around with speculative decoding for Vik Paruchuri/Datalab's Chandra 2 and I'm releasing an initial EAGLE-3.1 model that gets a ~2x speedup at small batch sizes!

Joe Barrow

10,759 次观看 • 4 个月前

SGLang now has Day 0 support for MiniMax-Music 3 from MiniMax (official) 🎵 A music generation model powered by a Qwen3 + RVQ autoregressive backbone and a flow-matching DIT/DAV decoder. POV: you’re parkouring through old Shanghai alleys while Music 3 scores your main-character moment 🎵🧡 Visuals by MiniMax Design (H3), music by MiniMax (official) MiniMax-Music 3. Make your own with SGLang Omni Cookbook👇

SGLang now has Day 0 support for MiniMax-Music 3 from MiniMax (official) 🎵 A music generation model powered by a Qwen3 + RVQ autoregressive backbone and a flow-matching DIT/DAV decoder. POV: you’re parkouring through old Shanghai alleys while Music 3 scores your main-character moment 🎵🧡 Visuals by MiniMax Design (H3), music by MiniMax (official) MiniMax-Music 3. Make your own with SGLang Omni Cookbook👇

SGLang

14,131 次观看 • 3 个月前

You saw the AI & science. Let's talk about the RL infra it took to build Periodic Labs Neon. To minimize training-inference mismatch in RL, SGLang captures inference's MoE routing decisions for each rollout and we "replay" them while training. In agentic (multiturn tool-use) settings, SGLang exports these router decisions in response to each decoding request, i.e. after each conversation turn. So when *any* data-parallel rank finishes a conversation turn, *all* other ranks must wait until routing data finishes exporting. This slowdown is exacerbated because we export routing decisions from the *entire* conversation rather than just the most recent turn! When Byron Hsu Costa Huang discovered this in our Kimi K2 RL setup, they introduced Delta Router Replay: cache previous turns' router decisions on the training client, so you can export only the delta (most recent turn's router decisions) upon each decoding request. Delta Router Replay significantly speeds up our long-context agentic RL runs, and Byron Hsu upstreamed it to SGLang (#24851) a few months ago.

You saw the AI & science. Let's talk about the RL infra it took to build Periodic Labs Neon. To minimize training-inference mismatch in RL, SGLang captures inference's MoE routing decisions for each rollout and we "replay" them while training. In agentic (multiturn tool-use) settings, SGLang exports these router decisions in response to each decoding request, i.e. after each conversation turn. So when any data-parallel rank finishes a conversation turn, all other ranks must wait until routing data finishes exporting. This slowdown is exacerbated because we export routing decisions from the entire conversation rather than just the most recent turn! When Byron Hsu Costa Huang discovered this in our Kimi K2 RL setup, they introduced Delta Router Replay: cache previous turns' router decisions on the training client, so you can export only the delta (most recent turn's router decisions) upon each decoding request. Delta Router Replay significantly speeds up our long-context agentic RL runs, and Byron Hsu upstreamed it to SGLang (#24851) a few months ago.

Rohan Pandey

12,864 次观看 • 1 个月前

Martial eagle (Slide 1) is stronger than bald eagle (Slide 2). It is also a better hunter. It can take down larger, more challenging prey like antelopes or monitor lizards, unlike the bald eagle which relies more on fish and scavenging. The martial eagle’s predatory style is also more aggressive and versatile. Why is the bald eagle now used more by people? That is the power of marketing. The bald eagle is the national bird of the United States. It has been packaged in movies and clips to look more regal and powerful. They even replace its sound with a louder and more majestic call of a red tailed hawk. Watch the second clip and you'll hear their natural sounds. Also, when you type "eagle," the emoji that comes out is 🦅 which is that of bald eagle. That's the power of PR.

Martial eagle (Slide 1) is stronger than bald eagle (Slide 2). It is also a better hunter. It can take down larger, more challenging prey like antelopes or monitor lizards, unlike the bald eagle which relies more on fish and scavenging. The martial eagle’s predatory style is also more aggressive and versatile. Why is the bald eagle now used more by people? That is the power of marketing. The bald eagle is the national bird of the United States. It has been packaged in movies and clips to look more regal and powerful. They even replace its sound with a louder and more majestic call of a red tailed hawk. Watch the second clip and you'll hear their natural sounds. Also, when you type "eagle," the emoji that comes out is 🦅 which is that of bald eagle. That's the power of PR.

Arojinle

99,727 次观看 • 1 年前

Videographed bald eagle In Campbell River. The Haast’s eagle is the largest and heaviest eagle to ever exist, weighing up to 37 pounds and measuring four feet long, with a nine to ten-foot wingspan. This enormous eagle had a bill larger than any living vulture and claws bigger than a tiger’s. Their main prey was moa, which could easily topple 12 feet high and weigh over 500 pounds! While the Haast’s eagle couldn’t necessarily fly off with a moa, they could use their sharp, powerful talons to rip through flesh and crush bone. There is evidence that the Haast’s eagle survived and evolved through multiple ice ages but went extinct around 500 to 600 years ago. In comparison, the biggest of the bald eagles can only reach up to 14 pounds and measure three feet long, with a seven-foot wingspan. #TwitterNatureCommunity #TwitterNaturePhotography #birds #BirdsOfTwitter #birdphotography #birdsofprey #wildlifephotography #NaturePhotography

Videographed bald eagle In Campbell River. The Haast’s eagle is the largest and heaviest eagle to ever exist, weighing up to 37 pounds and measuring four feet long, with a nine to ten-foot wingspan. This enormous eagle had a bill larger than any living vulture and claws bigger than a tiger’s. Their main prey was moa, which could easily topple 12 feet high and weigh over 500 pounds! While the Haast’s eagle couldn’t necessarily fly off with a moa, they could use their sharp, powerful talons to rip through flesh and crush bone. There is evidence that the Haast’s eagle survived and evolved through multiple ice ages but went extinct around 500 to 600 years ago. In comparison, the biggest of the bald eagles can only reach up to 14 pounds and measure three feet long, with a seven-foot wingspan. #TwitterNatureCommunity #TwitterNaturePhotography #birds #BirdsOfTwitter #birdphotography #birdsofprey #wildlifephotography #NaturePhotography

Brad Wildlife Photographer

57,986 次观看 • 3 年前

(1/n) 🚀 With FastVideo, you can now generate a 5-second video in 5 seconds on a single H200 GPU! Introducing FastWan series, a family of fast video generation models trained via a new recipe we term as “sparse distillation”, to speed up video denoising time by 70X! 🖥️ Live demo: (Thanks to @gmicloud for the support!) 🔗 Blog: 🔓 We fully open-source our models, code, and data with Apache-2.0 licenses

(1/n) 🚀 With FastVideo, you can now generate a 5-second video in 5 seconds on a single H200 GPU! Introducing FastWan series, a family of fast video generation models trained via a new recipe we term as “sparse distillation”, to speed up video denoising time by 70X! 🖥️ Live demo: (Thanks to @gmicloud for the support!) 🔗 Blog: 🔓 We fully open-source our models, code, and data with Apache-2.0 licenses

Hao AI Lab

79,140 次观看 • 1 年前

Dasting. Dan Scavino just post an eagle video on IG with the song “Fly like and Eagle”. Bill Clinton’s Secret Service codename is “Eagle”.

Dasting. Dan Scavino just post an eagle video on IG with the song “Fly like and Eagle”. Bill Clinton’s Secret Service codename is “Eagle”.

Hodlr

54,865 次观看 • 2 个月前

A bald eagle rescued in California is believed to be Jackie – one of Big Bear’s famous eagles. The group Friends of Big Bear Valley say the eagle was injured in a fight with two other eagles, and is now getting medical care. David Muir reports.

A bald eagle rescued in California is believed to be Jackie – one of Big Bear’s famous eagles. The group Friends of Big Bear Valley say the eagle was injured in a fight with two other eagles, and is now getting medical care. David Muir reports.

World News Tonight

20,442 次观看 • 3 个月前

🦅 Harpy eagle One of the most powerful eagles on Earth 😮 The harpy eagle lives in the rainforests of Central and South America and rules the jungle skies Its wingspan can exceed 2 meters, and its talons are comparable in size to a bear’s claws ⚡️ Strong enough to hunt monkeys and sloths in the trees

🦅 Harpy eagle One of the most powerful eagles on Earth 😮 The harpy eagle lives in the rainforests of Central and South America and rules the jungle skies Its wingspan can exceed 2 meters, and its talons are comparable in size to a bear’s claws ⚡️ Strong enough to hunt monkeys and sloths in the trees

Curio 𝑿

16,714 次观看 • 2 个月前

It’s a great time to be an eagle! Extremely excited about our guys returning(0 portal entries) and the class we have coming in. Still looking to add 3 impact transfer arms. DM if are looking for a quality institution in the south with top notch facilities/ technology!

It’s a great time to be an eagle! Extremely excited about our guys returning(0 portal entries) and the class we have coming in. Still looking to add 3 impact transfer arms. DM if are looking for a quality institution in the south with top notch facilities/ technology!

Coach Bacon

13,898 次观看 • 1 年前