正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Can $300 RTX3060 be faster than $10,000 A100 for LLM decoding?🧐Yes Introduce EAGLE-2: - 4.26x🚀vanilla (13B) - 1.4x🚀EAGLE (13B) - No extra training than EAGLE - Maintain distribution Paper: Code: Demo:

Hongyang Zhang

2,795 subscribers

35,696 次观看 • 2 年前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

EAGLE-3 introduces two key innovations: Training-Time Testing (TTT) and multi-level feature fusion. By removing the feature prediction constraint used in previous EAGLE versions and leveraging semantic features across multiple layers, EAGLE-3 achieves higher acceptance rates, faster generation, and lossless performance. The result? - 5.6× faster than vanilla decoding (13B) Compared to EAGLE-1, EAGLE-3 delivers a 1.8× speedup on the 13B model, with the future EAGLE-4 release expected to further improve decoding efficiency. *Inference on the video conducted on 2x RTX 3090 GPUs at fp16 precision using the Vicuna 13B model.

EAGLE-3 introduces two key innovations: Training-Time Testing (TTT) and multi-level feature fusion. By removing the feature prediction constraint used in previous EAGLE versions and leveraging semantic features across multiple layers, EAGLE-3 achieves higher acceptance rates, faster generation, and lossless performance. The result? - 5.6× faster than vanilla decoding (13B) Compared to EAGLE-1, EAGLE-3 delivers a 1.8× speedup on the 13B model, with the future EAGLE-4 release expected to further improve decoding efficiency. *Inference on the video conducted on 2x RTX 3090 GPUs at fp16 precision using the Vicuna 13B model.

Eagle

37,069 次观看 • 2 个月前

Jointly announcing EAGLE-3 with SGLang: Setting a new record in LLM inference acceleration! - 5x🚀than vanilla (on HF) - 1.4x🚀than EAGLE-2 (on HF) - A record of ~400 TPS on LLama 3.1 8B with a single H100 (on SGLang) - 1.65x🚀in latency even for large bs=64 (on SGLang) - A new scaling law: more training data, better speedup - Apache 2.0 Paper: Code: SGLang version: ⚒️Takeaway: Introducing training-time test, a novel draft model training technique: we replace feature prediction with direct token prediction and shift from top-layer-only features to multi-layer feature fusion. This approach unlocks a new scaling law previously undiscovered in EAGLE and EAGLE-2. 🙏Acknowledge: We would like to thank the SGLang team (zhyncs Lianmin Zheng Ying Sheng James Liu, Ke Bao, and others LMSYS Org) for their merge and careful evaluation of EAGLE-3 on SGLang. 🤝Want to collaborate? We're a small academic group with limited GPU resources. If you're interested in supporting our next version of EAGLE or would like us to train a preliminary version tailored to a specific model, please get in touch! Joint work with Yuhui Li, Fangyun Wei, and Chao Zhang

Jointly announcing EAGLE-3 with SGLang: Setting a new record in LLM inference acceleration! - 5x🚀than vanilla (on HF) - 1.4x🚀than EAGLE-2 (on HF) - A record of ~400 TPS on LLama 3.1 8B with a single H100 (on SGLang) - 1.65x🚀in latency even for large bs=64 (on SGLang) - A new scaling law: more training data, better speedup - Apache 2.0 Paper: Code: SGLang version: ⚒️Takeaway: Introducing training-time test, a novel draft model training technique: we replace feature prediction with direct token prediction and shift from top-layer-only features to multi-layer feature fusion. This approach unlocks a new scaling law previously undiscovered in EAGLE and EAGLE-2. 🙏Acknowledge: We would like to thank the SGLang team (zhyncs Lianmin Zheng Ying Sheng James Liu, Ke Bao, and others LMSYS Org) for their merge and careful evaluation of EAGLE-3 on SGLang. 🤝Want to collaborate? We're a small academic group with limited GPU resources. If you're interested in supporting our next version of EAGLE or would like us to train a preliminary version tailored to a specific model, please get in touch! Joint work with Yuhui Li, Fangyun Wei, and Chao Zhang

Hongyang Zhang

42,200 次观看 • 1 年前

Introduce EAGLE, a new method for fast LLM decoding based on compression: - 3x🚀than vanilla - 2x🚀 than Lookahead (on its benchmark) - 1.6x🚀 than Medusa (on its benchmark) - provably maintains text distribution - trainable (in 1~2 days) and testable on RTX 3090s Playground: Blog: Code: ⚒️First Principle: Compression! Yi Ma We find that the sequence of second-top-layer features is compressible, making the prediction of subsequent feature vectors from previous ones easy by a small model. 🙏Acknowledge: This project is greatly inspired by the Medusa team (Tianle Cai @yli3521 Zhengyang Geng Hongwu Peng Tri Dao), the Lookahead team (Hao Zhang LMSYS Org), and others. Joint work with Yuhui Li and Chao Zhang

Introduce EAGLE, a new method for fast LLM decoding based on compression: - 3x🚀than vanilla - 2x🚀 than Lookahead (on its benchmark) - 1.6x🚀 than Medusa (on its benchmark) - provably maintains text distribution - trainable (in 1~2 days) and testable on RTX 3090s Playground: Blog: Code: ⚒️First Principle: Compression! Yi Ma We find that the sequence of second-top-layer features is compressible, making the prediction of subsequent feature vectors from previous ones easy by a small model. 🙏Acknowledge: This project is greatly inspired by the Medusa team (Tianle Cai @yli3521 Zhengyang Geng Hongwu Peng Tri Dao), the Lookahead team (Hao Zhang LMSYS Org), and others. Joint work with Yuhui Li and Chao Zhang

Hongyang Zhang

118,857 次观看 • 2 年前

The eagle is stronger than the tiger.😩🥹😁

The eagle is stronger than the tiger.😩🥹😁

Joselyine Chap❤️

45,210 次观看 • 8 个月前

Chip-in eagle for Harry Hall 🦅 This is the first eagle of the week on No. 17 Valero Texas Open.

Chip-in eagle for Harry Hall 🦅 This is the first eagle of the week on No. 17 Valero Texas Open.

PGA TOUR

167,972 次观看 • 3 年前

"EXTRA, EXTRA, READ ALL ABOUT IT" ANOTHER ELITE CALL FROM IAN EAGLE

"EXTRA, EXTRA, READ ALL ABOUT IT" ANOTHER ELITE CALL FROM IAN EAGLE

Knicks Nation

73,477 次观看 • 9 个月前

The Harpy Eagle is the most powerful eagle in the world. Its wingspan can reach to 7.5 feet of width.

The Harpy Eagle is the most powerful eagle in the world. Its wingspan can reach to 7.5 feet of width.

Massimo

18,700 次观看 • 10 个月前

YES!! That's in for an eagle from the fairway! Albane Valenzuela | Honda LPGA Thailand

YES!! That's in for an eagle from the fairway! Albane Valenzuela | Honda LPGA Thailand

LPGA

19,801 次观看 • 2 年前

Callsign Competition…Go‼️ Eddie the Eagle🤓 HR Pufnstuffed😆 Punch’n out’ Judy🤣 #FlyingHigh🚀 📹 brrrrt_corp

Callsign Competition…Go‼️ Eddie the Eagle🤓 HR Pufnstuffed😆 Punch’n out’ Judy🤣 #FlyingHigh🚀 📹 brrrrt_corp

F111Driver

30,703 次观看 • 1 年前

Sydney Sweeney for american eagle.

Sydney Sweeney for american eagle.

Sweeney Daily

42,029 次观看 • 3 个月前

the Philippine eagle possesses telescopic vision that is up to eight times sharper than human sight.

the Philippine eagle possesses telescopic vision that is up to eight times sharper than human sight.

Science girl

172,182 次观看 • 5 个月前

Once an Eagle. Always an Eagle. Hırsına, aidiyetine teşekkürler. Seni unutmayacağız.

Once an Eagle. Always an Eagle. Hırsına, aidiyetine teşekkürler. Seni unutmayacağız.

Medya Siyah Beyaz

16,746 次观看 • 7 个月前

BREAKING: Shooting reported outside Kickapoo Lucky Eagle Casino in Eagle Pass, Texas

BREAKING: Shooting reported outside Kickapoo Lucky Eagle Casino in Eagle Pass, Texas

Intel Point Alert

28,228 次观看 • 10 个月前

Sydney Sweeney for American Eagle.

Sydney Sweeney for American Eagle.

Sweeney NEWS

374,809 次观看 • 28 天前

Sydney Sweeney for American Eagle.

Sydney Sweeney for American Eagle.

Sydney Sweeney Fans

27,547 次观看 • 4 个月前

Sydney Sweeney for American Eagle.

Sydney Sweeney for American Eagle.

Sweeney NEWS

16,438 次观看 • 8 天前

Sydney Sweeney for American Eagle.

Sydney Sweeney for American Eagle.

Sydney Sweeney Files

89,582 次观看 • 1 年前

American Eagle x Sydney Sweeney Ring The Opening Bell $AEO American Eagle | Sydney Sweeney

American Eagle x Sydney Sweeney Ring The Opening Bell $AEO American Eagle | Sydney Sweeney

NYSE 🏛

13,544 次观看 • 6 个月前