Loading video...

Video Failed to Load

Go Home

Can $300 RTX3060 be faster than $10,000 A100 for LLM decoding?🧐Yes Introduce EAGLE-2: - 4.26x🚀vanilla (13B) - 1.4x🚀EAGLE (13B) - No extra training than EAGLE - Maintain distribution Paper: Code: Demo:

hongyangzh's profile picture

Hongyang Zhang

2,795 subscribers

35,696 views • 2 years ago •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

EAGLE-3 introduces two key innovations: Training-Time Testing (TTT) and multi-level feature fusion. By removing the feature prediction constraint used in previous EAGLE versions and leveraging semantic features across multiple layers, EAGLE-3 achieves higher acceptance rates, faster generation, and lossless performance. The result? - 5.6× faster than vanilla decoding (13B) Compared to EAGLE-1, EAGLE-3 delivers a 1.8× speedup on the 13B model, with the future EAGLE-4 release expected to further improve decoding efficiency. *Inference on the video conducted on 2x RTX 3090 GPUs at fp16 precision using the Vicuna 13B model.
0:12

EAGLE-3 introduces two key innovations: Training-Time Testing (TTT) and multi-level feature fusion. By removing the feature prediction constraint used in previous EAGLE versions and leveraging semantic features across multiple layers, EAGLE-3 achieves higher acceptance rates, faster generation, and lossless performance. The result? - 5.6× faster than vanilla decoding (13B) Compared to EAGLE-1, EAGLE-3 delivers a 1.8× speedup on the 13B model, with the future EAGLE-4 release expected to further improve decoding efficiency. *Inference on the video conducted on 2x RTX 3090 GPUs at fp16 precision using the Vicuna 13B model.

Eagle

37,069 views • 2 months ago

Jointly announcing EAGLE-3 with SGLang: Setting a new record in LLM inference acceleration! - 5x🚀than vanilla (on HF) - 1.4x🚀than EAGLE-2 (on HF) - A record of ~400 TPS on LLama 3.1 8B with a single H100 (on SGLang) - 1.65x🚀in latency even for large bs=64 (on SGLang) - A new scaling law: more training data, better speedup - Apache 2.0 Paper: Code: SGLang version: ⚒️Takeaway: Introducing training-time test, a novel draft model training technique: we replace feature prediction with direct token prediction and shift from top-layer-only features to multi-layer feature fusion. This approach unlocks a new scaling law previously undiscovered in EAGLE and EAGLE-2. 🙏Acknowledge: We would like to thank the SGLang team (zhyncs Lianmin Zheng Ying Sheng James Liu, Ke Bao, and others LMSYS Org) for their merge and careful evaluation of EAGLE-3 on SGLang. 🤝Want to collaborate? We're a small academic group with limited GPU resources. If you're interested in supporting our next version of EAGLE or would like us to train a preliminary version tailored to a specific model, please get in touch! Joint work with Yuhui Li, Fangyun Wei, and Chao Zhang
0:12

Jointly announcing EAGLE-3 with SGLang: Setting a new record in LLM inference acceleration! - 5x🚀than vanilla (on HF) - 1.4x🚀than EAGLE-2 (on HF) - A record of ~400 TPS on LLama 3.1 8B with a single H100 (on SGLang) - 1.65x🚀in latency even for large bs=64 (on SGLang) - A new scaling law: more training data, better speedup - Apache 2.0 Paper: Code: SGLang version: ⚒️Takeaway: Introducing training-time test, a novel draft model training technique: we replace feature prediction with direct token prediction and shift from top-layer-only features to multi-layer feature fusion. This approach unlocks a new scaling law previously undiscovered in EAGLE and EAGLE-2. 🙏Acknowledge: We would like to thank the SGLang team (zhyncs Lianmin Zheng Ying Sheng James Liu, Ke Bao, and others LMSYS Org) for their merge and careful evaluation of EAGLE-3 on SGLang. 🤝Want to collaborate? We're a small academic group with limited GPU resources. If you're interested in supporting our next version of EAGLE or would like us to train a preliminary version tailored to a specific model, please get in touch! Joint work with Yuhui Li, Fangyun Wei, and Chao Zhang

Hongyang Zhang

42,200 views • 1 year ago

Introduce EAGLE, a new method for fast LLM decoding based on compression: - 3x🚀than vanilla - 2x🚀 than Lookahead (on its benchmark) - 1.6x🚀 than Medusa (on its benchmark) - provably maintains text distribution - trainable (in 1~2 days) and testable on RTX 3090s Playground: Blog: Code: ⚒️First Principle: Compression! Yi Ma We find that the sequence of second-top-layer features is compressible, making the prediction of subsequent feature vectors from previous ones easy by a small model. 🙏Acknowledge: This project is greatly inspired by the Medusa team (Tianle Cai @yli3521 Zhengyang Geng Hongwu Peng Tri Dao), the Lookahead team (Hao Zhang LMSYS Org), and others. Joint work with Yuhui Li and Chao Zhang
0:15

Introduce EAGLE, a new method for fast LLM decoding based on compression: - 3x🚀than vanilla - 2x🚀 than Lookahead (on its benchmark) - 1.6x🚀 than Medusa (on its benchmark) - provably maintains text distribution - trainable (in 1~2 days) and testable on RTX 3090s Playground: Blog: Code: ⚒️First Principle: Compression! Yi Ma We find that the sequence of second-top-layer features is compressible, making the prediction of subsequent feature vectors from previous ones easy by a small model. 🙏Acknowledge: This project is greatly inspired by the Medusa team (Tianle Cai @yli3521 Zhengyang Geng Hongwu Peng Tri Dao), the Lookahead team (Hao Zhang LMSYS Org), and others. Joint work with Yuhui Li and Chao Zhang

Hongyang Zhang

118,857 views • 2 years ago

The eagle is stronger than the tiger.😩🥹😁
0:12

The eagle is stronger than the tiger.😩🥹😁

Joselyine Chap❤️

45,210 views • 8 months ago

"EXTRA, EXTRA, READ ALL ABOUT IT" ANOTHER ELITE CALL FROM IAN EAGLE
0:10

"EXTRA, EXTRA, READ ALL ABOUT IT" ANOTHER ELITE CALL FROM IAN EAGLE

Knicks Nation

73,477 views • 9 months ago

Chip-in eagle for Harry Hall 🦅 This is the first eagle of the week on No. 17 Valero Texas Open.
0:16

Chip-in eagle for Harry Hall 🦅 This is the first eagle of the week on No. 17 Valero Texas Open.

PGA TOUR

167,972 views • 3 years ago

The Harpy Eagle is the most powerful eagle in the world. Its wingspan can reach to 7.5 feet of width.
0:10

The Harpy Eagle is the most powerful eagle in the world. Its wingspan can reach to 7.5 feet of width.

Massimo

18,700 views • 10 months ago

YES!! That's in for an eagle from the fairway! Albane Valenzuela | Honda LPGA Thailand
0:17

YES!! That's in for an eagle from the fairway! Albane Valenzuela | Honda LPGA Thailand

LPGA

19,801 views • 2 years ago

Callsign Competition…Go‼️ Eddie the Eagle🤓 HR Pufnstuffed😆 Punch’n out’ Judy🤣 #FlyingHigh🚀 📹 brrrrt_corp
0:18

Callsign Competition…Go‼️ Eddie the Eagle🤓 HR Pufnstuffed😆 Punch’n out’ Judy🤣 #FlyingHigh🚀 📹 brrrrt_corp

F111Driver

30,703 views • 1 year ago

Sydney Sweeney for american eagle.
0:15

Sydney Sweeney for american eagle.

Sweeney Daily

42,029 views • 3 months ago

the Philippine eagle possesses telescopic vision that is up to eight times sharper than human sight.
0:10

the Philippine eagle possesses telescopic vision that is up to eight times sharper than human sight.

Science girl

172,182 views • 5 months ago

BREAKING: Shooting reported outside Kickapoo Lucky Eagle Casino in Eagle Pass, Texas
0:12

BREAKING: Shooting reported outside Kickapoo Lucky Eagle Casino in Eagle Pass, Texas

Intel Point Alert

28,228 views • 10 months ago

Once an Eagle. Always an Eagle. Hırsına, aidiyetine teşekkürler. Seni unutmayacağız.
0:12

Once an Eagle. Always an Eagle. Hırsına, aidiyetine teşekkürler. Seni unutmayacağız.

Medya Siyah Beyaz

16,746 views • 6 months ago

Sydney Sweeney for American Eagle.
0:19

Sydney Sweeney for American Eagle.

Sydney Sweeney Fans

27,547 views • 4 months ago

Sydney Sweeney for American Eagle.
0:14

Sydney Sweeney for American Eagle.

Sydney Sweeney Files

89,582 views • 1 year ago

Sydney Sweeney for American Eagle.
0:14

Sydney Sweeney for American Eagle.

Sweeney NEWS

374,809 views • 23 days ago

Justin Thomas starts his round with an eagle on No. 2. #themasters
0:18

Justin Thomas starts his round with an eagle on No. 2. #themasters

The Masters

317,021 views • 1 year ago

American Eagle x Sydney Sweeney Ring The Opening Bell $AEO American Eagle | Sydney Sweeney
0:19

American Eagle x Sydney Sweeney Ring The Opening Bell $AEO American Eagle | Sydney Sweeney

NYSE 🏛

13,544 views • 6 months ago