Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

⚡ Speed of flash. Just 2 days after launch, DFlash is already running in SGLang (SGLang). With serving-engine support, we can now unlock speedup with higher concurrency, and we’ve quickly worked on a new demo based on it. We'll be cooking up more and better draft models over the...

zhijianliu_'s profile picture

Zhijian Liu

5,545 subscribers

21,233 Aufrufe • vor 8 Monaten •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

📢 Qwen's Qwen3.8-2.4T-A95B weights just dropped, and SGLang already has day-0 support with DSpark. HF: Cookbook: 🖖 What kind of model can help optimize an inference engine? Qwen3.8-2.4T-A95B can take on complex engineering tasks, plan and iterate on its own, and keep going until the job is done. We even asked it to optimize its own serving on SGLang, and it ran unattended for 4.5 hours with verified performance improvements. Thanks to the Qwen team Qwen, NVIDIA AI, and AI at AMD for the close collaboration.
0:36

📢 Qwen's Qwen3.8-2.4T-A95B weights just dropped, and SGLang already has day-0 support with DSpark. HF: Cookbook: 🖖 What kind of model can help optimize an inference engine? Qwen3.8-2.4T-A95B can take on complex engineering tasks, plan and iterate on its own, and keep going until the job is done. We even asked it to optimize its own serving on SGLang, and it ran unattended for 4.5 hours with verified performance improvements. Thanks to the Qwen team Qwen, NVIDIA AI, and AI at AMD for the close collaboration.

SGLang

27,295 Aufrufe • vor 1 Monat

Inkling, Thinking Machines' first open model, dropped today: 975B total / 41B active MoE, up to 1M context, reasoning natively over text, images, and audio. Serving and RL support are already live: you can run and shape it on an open stack, starting now. Day 0 support on SGLang SGLang and Miles RadixArk👇 - Inkling's new architecture (ShortConv, attention with relative positional embedding, shared expert sink MoE) is natively implemented and deeply optimized, with prefill full CUDA graph and MXFP8 KV cache - Full parameter and LoRA RL in a customized Megatron backend, train inference consistency via customized kernels, routing replay, and cross-runtime parameter synchronization - DFlash speculative decoding from Modal for low-latency serving Launching now, blog and cookbook in the comments ⬇️
1:01

Inkling, Thinking Machines' first open model, dropped today: 975B total / 41B active MoE, up to 1M context, reasoning natively over text, images, and audio. Serving and RL support are already live: you can run and shape it on an open stack, starting now. Day 0 support on SGLang SGLang and Miles RadixArk👇 - Inkling's new architecture (ShortConv, attention with relative positional embedding, shared expert sink MoE) is natively implemented and deeply optimized, with prefill full CUDA graph and MXFP8 KV cache - Full parameter and LoRA RL in a customized Megatron backend, train inference consistency via customized kernels, routing replay, and cross-runtime parameter synchronization - DFlash speculative decoding from Modal for low-latency serving Launching now, blog and cookbook in the comments ⬇️

LMSYS Org

146,339 Aufrufe • vor 2 Monaten

SGLang now supports DSpark, enabling confidence-driven, variable-length verification for speculative decoding 🎉 DSpark addresses a key bottleneck under load: instead of verifying every draft token, it verifies only where the draft model is confident, so the gains hold even as batch size scales. We heavily optimized variable-length verification in SGLang. Across batch sizes 1 to 256, DSpark gives the best throughput/latency tradeoff on DeepSeek-V4-Flash, ahead of both MTP and non-spec. At high concurrency, dynamic scheduling provides up to ~20% higher throughput compared to a fixed budget, while maintaining high verification quality across workloads. With fused kernels and zero-overhead scheduling, DeepSeek-V4-Pro reaches 383.7 tok/s at B=1 on B300. DSpark is now available in SGLang with support for Qwen3 and DeepSeek-V4. Thanks DeepSeek for open-sourcing! Blog with full technical details and commands to run below 👇
0:33

SGLang now supports DSpark, enabling confidence-driven, variable-length verification for speculative decoding 🎉 DSpark addresses a key bottleneck under load: instead of verifying every draft token, it verifies only where the draft model is confident, so the gains hold even as batch size scales. We heavily optimized variable-length verification in SGLang. Across batch sizes 1 to 256, DSpark gives the best throughput/latency tradeoff on DeepSeek-V4-Flash, ahead of both MTP and non-spec. At high concurrency, dynamic scheduling provides up to ~20% higher throughput compared to a fixed budget, while maintaining high verification quality across workloads. With fused kernels and zero-overhead scheduling, DeepSeek-V4-Pro reaches 383.7 tok/s at B=1 on B300. DSpark is now available in SGLang with support for Qwen3 and DeepSeek-V4. Thanks DeepSeek for open-sourcing! Blog with full technical details and commands to run below 👇

LMSYS Org

170,431 Aufrufe • vor 2 Monaten

Serving GLM5.2 NVFP4 Agentic Workload with SGLang: How We Reached 500 TPS on 8xB300 at bs=1 In this deep dive, we explain how SGLang reaches 500+ tok/s/user at bs=1 on 8xB300, with 18 to 34% higher single-user interactivity within two weeks since day-0, and 6 to 11% better peak throughput at high concurrency, benchmarked on a real multi-turn agentic coding workload. Our new TopK-V2 kernel is 2.33x faster at 80K ISL, scaling to 10.17x at 1M ISL, keeping interactivity essentially flat out to 1M tokens. Part of the story is the architecture itself. GLM-5.2 applies IndexShare to its DSA layers and ships a stronger MTP head reusing IndexShare and KVShare. The rest comes from our serving optimizations. Special thanks to NVIDIA AI for the help in day-0 support of GLM-5.2 NVFP4, and to Z.ai for IndexShare in SGLang!
0:47

Serving GLM5.2 NVFP4 Agentic Workload with SGLang: How We Reached 500 TPS on 8xB300 at bs=1 In this deep dive, we explain how SGLang reaches 500+ tok/s/user at bs=1 on 8xB300, with 18 to 34% higher single-user interactivity within two weeks since day-0, and 6 to 11% better peak throughput at high concurrency, benchmarked on a real multi-turn agentic coding workload. Our new TopK-V2 kernel is 2.33x faster at 80K ISL, scaling to 10.17x at 1M ISL, keeping interactivity essentially flat out to 1M tokens. Part of the story is the architecture itself. GLM-5.2 applies IndexShare to its DSA layers and ships a stronger MTP head reusing IndexShare and KVShare. The rest comes from our serving optimizations. Special thanks to NVIDIA AI for the help in day-0 support of GLM-5.2 NVFP4, and to Z.ai for IndexShare in SGLang!

LMSYS Org

107,511 Aufrufe • vor 2 Monaten

It’s been an incredibly exciting few weeks for all of us involved with #Alvara. We’re delighted to finally be up and running but the real work has just started. All eyes are firmly on the testnet launch in a few weeks. An exciting few months lay ahead. Stay tuned. $ALVA
0:54

It’s been an incredibly exciting few weeks for all of us involved with #Alvara. We’re delighted to finally be up and running but the real work has just started. All eyes are firmly on the testnet launch in a few weeks. An exciting few months lay ahead. Stay tuned. $ALVA

Alvara

31,927 Aufrufe • vor 2 Jahren

AI at Meta's Muse Glimmer (30B dense, open-weights) launches with SGLang day-0 support. We got ~230 tok/s on a single RTX 5090, with NVFP4 + DFlash on. It also works out of the box on NVIDIA AI Developer RTX Pro 6000, DGX Spark, and MLX for Mac. Speed and reliability have always been the blockers for a local agent. SGLang and Muse Glimmer make a local agent that's blazing fast, always on, and gets things done. Go build and run a local agent on your own machine!🚀
0:32

AI at Meta's Muse Glimmer (30B dense, open-weights) launches with SGLang day-0 support. We got ~230 tok/s on a single RTX 5090, with NVFP4 + DFlash on. It also works out of the box on NVIDIA AI Developer RTX Pro 6000, DGX Spark, and MLX for Mac. Speed and reliability have always been the blockers for a local agent. SGLang and Muse Glimmer make a local agent that's blazing fast, always on, and gets things done. Go build and run a local agent on your own machine!🚀

LMSYS Org

43,450 Aufrufe • vor 1 Monat

Executing a Helio payment on Solana and Polygon | POL in parallel🔥 Solana rivals a Visa payment in speed and poly is still pretty quick. we'll do one over the next few weeks with a 3rd screen executing on ETH :)
0:35

Executing a Helio payment on Solana and Polygon | POL in parallel🔥 Solana rivals a Visa payment in speed and poly is still pretty quick. we'll do one over the next few weeks with a 3rd screen executing on ETH :)

MoonPay Commerce

74,791 Aufrufe • vor 3 Jahren

Laguna S 2.1 is built to run locally. We heard the ask for a serious open model in the 80–120B range. Laguna S 2.1 is our answer, and we want it to redefine what this size class can do. Weights ship in BF16, FP8, INT4 and NVFP4, with official GGUF and MLX conversions, plus official DFlash draft models for faster inference. Day-one support across llama.cpp, ollama, Valerie Lee Make Up, SGLang, NVIDIA TRT-LLM, Hugging Face transformers and ZML. Watch it run on an NVIDIA AI DGX Spark.
0:39

Laguna S 2.1 is built to run locally. We heard the ask for a serious open model in the 80–120B range. Laguna S 2.1 is our answer, and we want it to redefine what this size class can do. Weights ship in BF16, FP8, INT4 and NVFP4, with official GGUF and MLX conversions, plus official DFlash draft models for faster inference. Day-one support across llama.cpp, ollama, Valerie Lee Make Up, SGLang, NVIDIA TRT-LLM, Hugging Face transformers and ZML. Watch it run on an NVIDIA AI DGX Spark.

Poolside

31,242 Aufrufe • vor 1 Monat

We ran a live demo NVIDIA GTC last week, but the real story is how quickly we got it running. The system was up and running in days, not weeks. This is a step toward robots that can be deployed quickly without task-by-task programming. How we made it happen👇 🧵 (1/6)
0:54

We ran a live demo NVIDIA GTC last week, but the real story is how quickly we got it running. The system was up and running in days, not weeks. This is a step toward robots that can be deployed quickly without task-by-task programming. How we made it happen👇 🧵 (1/6)

Generalist

41,800 Aufrufe • vor 5 Monaten

YES! We’re releasing a whole NEW 3D multiplayer game! Toca Boca Days will be live and available in Australia and New Zealand from May 2nd, with Sweden and Canada next in line! We'll add more countries over time leading up to a global release. Stay tuned for more info and updates!
0:34

YES! We’re releasing a whole NEW 3D multiplayer game! Toca Boca Days will be live and available in Australia and New Zealand from May 2nd, with Sweden and Canada next in line! We'll add more countries over time leading up to a global release. Stay tuned for more info and updates!

Toca Boca

21,835 Aufrufe • vor 2 Jahren

IN 2 DAYS WE STEP THROUGH THE PORTAL 🔥 BURN 1 PASS = MINT 1 PFP/PASS With this we're also launching: - Member dashboard - Stardust 2.0 - Questing This is only the beginning of a new JRNY. New utility and products will be rolling out over the next few months after launch
0:35

IN 2 DAYS WE STEP THROUGH THE PORTAL 🔥 BURN 1 PASS = MINT 1 PFP/PASS With this we're also launching: - Member dashboard - Stardust 2.0 - Questing This is only the beginning of a new JRNY. New utility and products will be rolling out over the next few months after launch

JRNY CLUB

77,358 Aufrufe • vor 2 Jahren

Please believe me when I say the next few months are going to be historic. What’s happening in macro is something that’s never happened before and GCoin will black hole all liquidity. here’s the lowdown: It already has 3,000+ holders and is sitting at a sub-$1.7M market cap just two weeks in. Insane early-stage potential! There’s a ton of buzz around $GCOIN right now, and I’m definitely here for it. I’ll be dropping some serious alpha on this project in the next few days, so stay tuned and stay bullish 📈 Might just stack a bit more soon:
0:27

Please believe me when I say the next few months are going to be historic. What’s happening in macro is something that’s never happened before and GCoin will black hole all liquidity. here’s the lowdown: It already has 3,000+ holders and is sitting at a sub-$1.7M market cap just two weeks in. Insane early-stage potential! There’s a ton of buzz around $GCOIN right now, and I’m definitely here for it. I’ll be dropping some serious alpha on this project in the next few days, so stay tuned and stay bullish 📈 Might just stack a bit more soon:

Henry

12,364 Aufrufe • vor 1 Jahr

Saving more is better than trying to chase higher returns. You'll always feel like giving up SIPs when you find a "loss" after a few years of investor. It may be better to simply save a little more, and end up with higher. The power of a "step up" SIP, in a new format at Capitalmind - focus more on earning and saving more, at least in the initial parts of your investing journey.
2:57

Saving more is better than trying to chase higher returns. You'll always feel like giving up SIPs when you find a "loss" after a few years of investor. It may be better to simply save a little more, and end up with higher. The power of a "step up" SIP, in a new format at Capitalmind - focus more on earning and saving more, at least in the initial parts of your investing journey.

Deepak Shenoy

41,808 Aufrufe • vor 1 Jahr

DFlash 2 is here! Qwen3.8-27B at 70 tok/s on an M5 Max MacBook Pro. ⚡ Up to 4.6× the speed of autoregressive decoding, with the same output. This is the next generation of DFlash, seeded at Z Lab and upgraded at Inco AI. Get one more accepted token on every pass, for free!
0:26

DFlash 2 is here! Qwen3.8-27B at 70 tok/s on an M5 Max MacBook Pro. ⚡ Up to 4.6× the speed of autoregressive decoding, with the same output. This is the next generation of DFlash, seeded at Z Lab and upgraded at Inco AI. Get one more accepted token on every pass, for free!

Zhijian Liu

1,129,832 Aufrufe • vor 28 Tagen

🚨 Speaker Alert! 🚨 We're incredibly proud to bring together an exceptional group of speakers who are pushing the boundaries of web development with Vite ⚡. Over the next few days, we'll be highlighting each incredible speaker who will be on our ViteConf 2025 stage in Amsterdam on Oct 9-10! Stay tuned! 🚀
0:36

🚨 Speaker Alert! 🚨 We're incredibly proud to bring together an exceptional group of speakers who are pushing the boundaries of web development with Vite ⚡. Over the next few days, we'll be highlighting each incredible speaker who will be on our ViteConf 2025 stage in Amsterdam on Oct 9-10! Stay tuned! 🚀

ViteConf

336,774 Aufrufe • vor 1 Jahr

MiniMax (official) H3 is live in SGLang Diffusion, with day-0 serving support 🎬 This open model matches Seedance 2.0 at 1/3 the cost, or $0 if you run it locally on 2x 5090 or 1 RTX 6000. With SGLang Diffusion, you can build visual concepts, motion design, e-commerce creatives, precise video edits, animation and stylized visuals, all locally on your own machine. No API bill, no waitlist. H3 takes text, image, video and audio in a single context, and generates 5-15s clips at native 2K, 24fps with native stereo audio. SGLang Diffusion also runs on NVIDIA AI Blackwell and Hopper, and AMD MI355X / MI300X. Let's create something with MiniMax H3 👇
0:31

MiniMax (official) H3 is live in SGLang Diffusion, with day-0 serving support 🎬 This open model matches Seedance 2.0 at 1/3 the cost, or $0 if you run it locally on 2x 5090 or 1 RTX 6000. With SGLang Diffusion, you can build visual concepts, motion design, e-commerce creatives, precise video edits, animation and stylized visuals, all locally on your own machine. No API bill, no waitlist. H3 takes text, image, video and audio in a single context, and generates 5-15s clips at native 2K, 24fps with native stereo audio. SGLang Diffusion also runs on NVIDIA AI Blackwell and Hopper, and AMD MI355X / MI300X. Let's create something with MiniMax H3 👇

LMSYS Org

99,992 Aufrufe • vor 1 Monat

SGLang day-0 speed on Kimi K3: 423 tok/s (measured on gsm8k), plus RL support ready in Miles RadixArk! How the largest open-source model runs this fast: we natively implemented and deeply optimized K3’s new architecture with fused KDA decode kernels, DP attention, DSpark, PD disagg, and KDA-aware prefix caching. We've passed Kimi Vendor Verifier and are ready for production! Thanks to Kimi.ai, NVIDIA, AMD, KVCache.AI, Modal, and Baseten for building this with us, and to Google Cloud, Nebius Token Factory, fal, DigitalOcean, Runpod, DeepInfra and GMI Cloud for serving K3 on SGLang. Blog, cookbook, benchmarks in the comments. P.S. This demo video? Kimi K3 made it itself. Play the game 👇
0:52

SGLang day-0 speed on Kimi K3: 423 tok/s (measured on gsm8k), plus RL support ready in Miles RadixArk! How the largest open-source model runs this fast: we natively implemented and deeply optimized K3’s new architecture with fused KDA decode kernels, DP attention, DSpark, PD disagg, and KDA-aware prefix caching. We've passed Kimi Vendor Verifier and are ready for production! Thanks to Kimi.ai, NVIDIA, AMD, KVCache.AI, Modal, and Baseten for building this with us, and to Google Cloud, Nebius Token Factory, fal, DigitalOcean, Runpod, DeepInfra and GMI Cloud for serving K3 on SGLang. Blog, cookbook, benchmarks in the comments. P.S. This demo video? Kimi K3 made it itself. Play the game 👇

LMSYS Org

204,807 Aufrufe • vor 1 Monat

I just turned 37 today and started this journey when I was 30. After years of building the engine with 💕 and optimizing it, this is the final gameplay running on #webgl with #threejs After converting Jamir from WebGL to WebGPU, I hope to release the X3JS, built on top of Jamir's game engine, in 2025, initially for creating FPP and TPP games, and make it available for everyone. New version will be upcoming in a few days, btw! 🥳 #gamedev #gamedevelopment
9:27

I just turned 37 today and started this journey when I was 30. After years of building the engine with 💕 and optimizing it, this is the final gameplay running on #webgl with #threejs After converting Jamir from WebGL to WebGPU, I hope to release the X3JS, built on top of Jamir's game engine, in 2025, initially for creating FPP and TPP games, and make it available for everyone. New version will be upcoming in a few days, btw! 🥳 #gamedev #gamedevelopment

mert

31,651 Aufrufe • vor 1 Jahr