Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Can $300 RTX3060 be faster than $10,000 A100 for LLM decoding?🧐Yes Introduce EAGLE-2: - 4.26x🚀vanilla (13B) - 1.4x🚀EAGLE (13B) - No extra training than EAGLE - Maintain distribution Paper: Code: Demo:

hongyangzh's profile picture

Hongyang Zhang

2,739 subscribers

35,677 Aufrufe • vor 2 Jahren •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

EAGLE-3 introduces two key innovations: Training-Time Testing (TTT) and multi-level feature fusion. By removing the feature prediction constraint used in previous EAGLE versions and leveraging semantic features across multiple layers, EAGLE-3 achieves higher acceptance rates, faster generation, and lossless performance. The result? - 5.6× faster than vanilla decoding (13B) Compared to EAGLE-1, EAGLE-3 delivers a 1.8× speedup on the 13B model, with the future EAGLE-4 release expected to further improve decoding efficiency. *Inference on the video conducted on 2x RTX 3090 GPUs at fp16 precision using the Vicuna 13B model.
0:12

EAGLE-3 introduces two key innovations: Training-Time Testing (TTT) and multi-level feature fusion. By removing the feature prediction constraint used in previous EAGLE versions and leveraging semantic features across multiple layers, EAGLE-3 achieves higher acceptance rates, faster generation, and lossless performance. The result? - 5.6× faster than vanilla decoding (13B) Compared to EAGLE-1, EAGLE-3 delivers a 1.8× speedup on the 13B model, with the future EAGLE-4 release expected to further improve decoding efficiency. *Inference on the video conducted on 2x RTX 3090 GPUs at fp16 precision using the Vicuna 13B model.

Eagle

36,689 Aufrufe • vor 9 Tagen

Jointly announcing EAGLE-3 with SGLang: Setting a new record in LLM inference acceleration! - 5x🚀than vanilla (on HF) - 1.4x🚀than EAGLE-2 (on HF) - A record of ~400 TPS on LLama 3.1 8B with a single H100 (on SGLang) - 1.65x🚀in latency even for large bs=64 (on SGLang) - A new scaling law: more training data, better speedup - Apache 2.0 Paper: Code: SGLang version: ⚒️Takeaway: Introducing training-time test, a novel draft model training technique: we replace feature prediction with direct token prediction and shift from top-layer-only features to multi-layer feature fusion. This approach unlocks a new scaling law previously undiscovered in EAGLE and EAGLE-2. 🙏Acknowledge: We would like to thank the SGLang team (zhyncs Lianmin Zheng Ying Sheng James Liu, Ke Bao, and others LMSYS Org) for their merge and careful evaluation of EAGLE-3 on SGLang. 🤝Want to collaborate? We're a small academic group with limited GPU resources. If you're interested in supporting our next version of EAGLE or would like us to train a preliminary version tailored to a specific model, please get in touch! Joint work with Yuhui Li, Fangyun Wei, and Chao Zhang
0:12

Jointly announcing EAGLE-3 with SGLang: Setting a new record in LLM inference acceleration! - 5x🚀than vanilla (on HF) - 1.4x🚀than EAGLE-2 (on HF) - A record of ~400 TPS on LLama 3.1 8B with a single H100 (on SGLang) - 1.65x🚀in latency even for large bs=64 (on SGLang) - A new scaling law: more training data, better speedup - Apache 2.0 Paper: Code: SGLang version: ⚒️Takeaway: Introducing training-time test, a novel draft model training technique: we replace feature prediction with direct token prediction and shift from top-layer-only features to multi-layer feature fusion. This approach unlocks a new scaling law previously undiscovered in EAGLE and EAGLE-2. 🙏Acknowledge: We would like to thank the SGLang team (zhyncs Lianmin Zheng Ying Sheng James Liu, Ke Bao, and others LMSYS Org) for their merge and careful evaluation of EAGLE-3 on SGLang. 🤝Want to collaborate? We're a small academic group with limited GPU resources. If you're interested in supporting our next version of EAGLE or would like us to train a preliminary version tailored to a specific model, please get in touch! Joint work with Yuhui Li, Fangyun Wei, and Chao Zhang

Hongyang Zhang

41,821 Aufrufe • vor 1 Jahr

Introduce EAGLE, a new method for fast LLM decoding based on compression: - 3x🚀than vanilla - 2x🚀 than Lookahead (on its benchmark) - 1.6x🚀 than Medusa (on its benchmark) - provably maintains text distribution - trainable (in 1~2 days) and testable on RTX 3090s Playground: Blog: Code: ⚒️First Principle: Compression! Yi Ma We find that the sequence of second-top-layer features is compressible, making the prediction of subsequent feature vectors from previous ones easy by a small model. 🙏Acknowledge: This project is greatly inspired by the Medusa team (Tianle Cai @yli3521 Zhengyang Geng Hongwu Peng Tri Dao), the Lookahead team (Hao Zhang LMSYS Org), and others. Joint work with Yuhui Li and Chao Zhang
0:15

Introduce EAGLE, a new method for fast LLM decoding based on compression: - 3x🚀than vanilla - 2x🚀 than Lookahead (on its benchmark) - 1.6x🚀 than Medusa (on its benchmark) - provably maintains text distribution - trainable (in 1~2 days) and testable on RTX 3090s Playground: Blog: Code: ⚒️First Principle: Compression! Yi Ma We find that the sequence of second-top-layer features is compressible, making the prediction of subsequent feature vectors from previous ones easy by a small model. 🙏Acknowledge: This project is greatly inspired by the Medusa team (Tianle Cai @yli3521 Zhengyang Geng Hongwu Peng Tri Dao), the Lookahead team (Hao Zhang LMSYS Org), and others. Joint work with Yuhui Li and Chao Zhang

Hongyang Zhang

118,810 Aufrufe • vor 2 Jahren

The eagle is stronger than the tiger.😩🥹😁
0:12

The eagle is stronger than the tiger.😩🥹😁

Joselyine Chap❤️

45,210 Aufrufe • vor 6 Monaten

"EXTRA, EXTRA, READ ALL ABOUT IT" ANOTHER ELITE CALL FROM IAN EAGLE
0:10

"EXTRA, EXTRA, READ ALL ABOUT IT" ANOTHER ELITE CALL FROM IAN EAGLE

KnicksNation

73,319 Aufrufe • vor 7 Monaten

Callsign Competition…Go‼️ Eddie the Eagle🤓 HR Pufnstuffed😆 Punch’n out’ Judy🤣 #FlyingHigh🚀 📹 brrrrt_corp
0:18

Callsign Competition…Go‼️ Eddie the Eagle🤓 HR Pufnstuffed😆 Punch’n out’ Judy🤣 #FlyingHigh🚀 📹 brrrrt_corp

F111Driver

30,703 Aufrufe • vor 10 Monaten

The Harpy Eagle is the most powerful eagle in the world. Its wingspan can reach to 7.5 feet of width.
0:10

The Harpy Eagle is the most powerful eagle in the world. Its wingspan can reach to 7.5 feet of width.

Massimo

18,700 Aufrufe • vor 8 Monaten

YES!! That's in for an eagle from the fairway! Albane Valenzuela | Honda LPGA Thailand
0:17

YES!! That's in for an eagle from the fairway! Albane Valenzuela | Honda LPGA Thailand

LPGA

19,787 Aufrufe • vor 2 Jahren

Sydney Sweeney for american eagle.
0:15

Sydney Sweeney for american eagle.

Sweeney Daily

42,029 Aufrufe • vor 1 Monat

the Philippine eagle possesses telescopic vision that is up to eight times sharper than human sight.
0:10

the Philippine eagle possesses telescopic vision that is up to eight times sharper than human sight.

Science girl

171,059 Aufrufe • vor 3 Monaten

BREAKING: Shooting reported outside Kickapoo Lucky Eagle Casino in Eagle Pass, Texas
0:12

BREAKING: Shooting reported outside Kickapoo Lucky Eagle Casino in Eagle Pass, Texas

Intel Point Alert

28,219 Aufrufe • vor 8 Monaten

Sydney Sweeney for American Eagle.
0:19

Sydney Sweeney for American Eagle.

Sydney Sweeney Fans

27,547 Aufrufe • vor 2 Monaten

Once an Eagle. Always an Eagle. Hırsına, aidiyetine teşekkürler. Seni unutmayacağız.
0:12

Once an Eagle. Always an Eagle. Hırsına, aidiyetine teşekkürler. Seni unutmayacağız.

Medya Siyah Beyaz

16,746 Aufrufe • vor 4 Monaten

Sydney Sweeney for American Eagle.
0:14

Sydney Sweeney for American Eagle.

Sydney Sweeney Files

89,582 Aufrufe • vor 11 Monaten

Justin Thomas starts his round with an eagle on No. 2. #themasters
0:18

Justin Thomas starts his round with an eagle on No. 2. #themasters

The Masters

317,021 Aufrufe • vor 1 Jahr

American Eagle x Sydney Sweeney Ring The Opening Bell $AEO American Eagle | Sydney Sweeney
0:19

American Eagle x Sydney Sweeney Ring The Opening Bell $AEO American Eagle | Sydney Sweeney

NYSE 🏛

13,544 Aufrufe • vor 4 Monaten

can confirm 🕺😂 Richard Jefferson | Noah Eagle
0:12

can confirm 🕺😂 Richard Jefferson | Noah Eagle

Brooklyn Nets

16,974 Aufrufe • vor 1 Jahr

Brooksy. Eagle.
0:11

Brooksy. Eagle.

Fore Play

155,725 Aufrufe • vor 1 Jahr