vLLM

@vllm_project • 46,592 subscribers

A high-throughput and memory-efficient inference and serving engine for LLMs. Join https://t.co/lxJ0SfX5pJ to discuss together with the community!

Shorts

🎉 Congrats to MiniMax (official) on releasing the open weights for MiniMax H3! Day-0 support in vLLM-Omni! One model reads text, images, video, and audio as a single context and returns video with native stereo audio. Text-to-video, first/last-frame, and multi-reference generation, 4 to 15 seconds at up to 2K and 24 FPS. The MP4 comes back with H.264 video and a synchronized stereo track already muxed in. It serves over the OpenAI-compatible /v1/videos endpoint, synchronous or async with job polling. 🔊 The video ⬇️ was made with H3, served on vLLM-Omni.

🎉 Congrats to MiniMax (official) on releasing the open weights for MiniMax H3! Day-0 support in vLLM-Omni! One model reads text, images, video, and audio as a single context and returns video with native stereo audio. Text-to-video, first/last-frame, and multi-reference generation, 4 to 15 seconds at up to 2K and 24 FPS. The MP4 comes back with H.264 video and a synchronized stereo track already muxed in. It serves over the OpenAI-compatible /v1/videos endpoint, synchronous or async with job polling. 🔊 The video ⬇️ was made with H3, served on vLLM-Omni.

127,714 Aufrufe

🎉 Congrats to Thinking Machines on TML Inkling—a 1T-parameter open-weight model supported in vLLM from Day 0. Highlights: • Natively multimodal across text, image, and audio • Up to 1M-token context • New architecture with relative attention, short convolutions, and MoE expert sinks • 8 MTP heads for speculative decoding vLLM supports both NVFP4 and BF16 checkpoints, optimized for NVIDIA Blackwell and Hopper, reaching up to 380 tok/s/user on 4× GB200 with MTP. Huge thanks to the Thinking Machines Lab team for the close collaboration 🙏 Read about implementation below 👇

🎉 Congrats to Thinking Machines on TML Inkling—a 1T-parameter open-weight model supported in vLLM from Day 0. Highlights: • Natively multimodal across text, image, and audio • Up to 1M-token context • New architecture with relative attention, short convolutions, and MoE expert sinks • 8 MTP heads for speculative decoding vLLM supports both NVFP4 and BF16 checkpoints, optimized for NVIDIA Blackwell and Hopper, reaching up to 380 tok/s/user on 4× GB200 with MTP. Huge thanks to the Thinking Machines Lab team for the close collaboration 🙏 Read about implementation below 👇

49,212 Aufrufe

🎉Announcing Gemma4 on vLLM model launch blog at Explore our detailed blogpost covering Gemma 4's capabilities, first-ever day-0 support across diverse hardware platforms, and ready-to-go deployment recipes! #Gemma4 #vLLM

🎉Announcing Gemma4 on vLLM model launch blog at Explore our detailed blogpost covering Gemma 4's capabilities, first-ever day-0 support across diverse hardware platforms, and ready-to-go deployment recipes! #Gemma4 #vLLM

12,090 Aufrufe

Videos

Anya Rossi

sweetdream.ai

SweetDream.ai•Sponsored•Livecam

Watch Anya Live

Anya is streaming live right now! Join her private show and enjoy exclusive content.

Exclusive private shows

1.2k viewers online

Private Show

Join now for exclusive access

Free preview available • Premium content

With Kimi K3 Day-0 on vLLM: Open Frontier Intelligence for Everyone 🚀 At 2.8 trillion parameters, Moonshot AI's Kimi K3 is one of the most powerful open-weight models ever released. Starting today, you can serve it on vLLM the moment the weights are public. What K3 brings: 🧠 2.8T-parameter Mixture-of-Experts (16 of 896 experts active per token) 📚 1M-token context window 🛠️ Native multimodal understanding, including vision ⚡ Kimi Delta Attention: a hybrid of linear and full attention that makes million-token context affordable Huge thank you to Kimi.ai AI for the model release and partnership, Inferact for leading the vLLM optimizations, and to our partners at NVIDIA, AMD, and the broader vLLM community. 1/6

With Kimi K3 Day-0 on vLLM: Open Frontier Intelligence for Everyone 🚀 At 2.8 trillion parameters, Moonshot AI's Kimi K3 is one of the most powerful open-weight models ever released. Starting today, you can serve it on vLLM the moment the weights are public. What K3 brings: 🧠 2.8T-parameter Mixture-of-Experts (16 of 896 experts active per token) 📚 1M-token context window 🛠️ Native multimodal understanding, including vision ⚡ Kimi Delta Attention: a hybrid of linear and full attention that makes million-token context affordable Huge thank you to Kimi.ai AI for the model release and partnership, Inferact for leading the vLLM optimizations, and to our partners at NVIDIA, AMD, and the broader vLLM community. 1/6

102,101 Aufrufe • vor 20 Tagen

vLLM v0.24.0 is out! 571 commits from 256 contributors (77 new). 🎉 Highlights: MiniMax-M3 support (FP8/MXFP4 + broad AMD tuning), DeepSeek-V4 keeps maturing (FlashInfer sparse index cache, prefill chunk-planning, now on SM120), Model Runner V2 now handles quantized models by default, a new unified Streaming Parser Engine for tool-calls + reasoning, DiffusionGemma, DeepEP v2 for wide expert parallelism, and a maturing Rust frontend. Thread 👇

vLLM v0.24.0 is out! 571 commits from 256 contributors (77 new). 🎉 Highlights: MiniMax-M3 support (FP8/MXFP4 + broad AMD tuning), DeepSeek-V4 keeps maturing (FlashInfer sparse index cache, prefill chunk-planning, now on SM120), Model Runner V2 now handles quantized models by default, a new unified Streaming Parser Engine for tool-calls + reasoning, DiffusionGemma, DeepEP v2 for wide expert parallelism, and a maturing Rust frontend. Thread 👇

52,291 Aufrufe • vor 1 Monat

🎉 Meet vLLM-Omni v0.22.0, a major upgrade for omnimodal world models and production-grade multimodal serving. 🌍 Day-0 NVIDIA AI Cosmos 3 world models: text, image, audio, video, and action, in and out. 🤖 Robot serving: DreamZero + OpenPI realtime API. 🎙️ Production TTS: Qwen3-TTS, Qwen3-Omni, VoxCPM2 and more. 🎨 Faster image/video/diffusion: Wan 2.2, HunyuanVideo 1.5, LTX-2.3. ⚡ Broader quantization (FP8/INT8, MXFP4/MXFP8, W4A16, ModelOpt) and hardware coverage. 339 commits, 124 contributors, 52 of them new. Thank you all. 🙌 🔗

🎉 Meet vLLM-Omni v0.22.0, a major upgrade for omnimodal world models and production-grade multimodal serving. 🌍 Day-0 NVIDIA AI Cosmos 3 world models: text, image, audio, video, and action, in and out. 🤖 Robot serving: DreamZero + OpenPI realtime API. 🎙️ Production TTS: Qwen3-TTS, Qwen3-Omni, VoxCPM2 and more. 🎨 Faster image/video/diffusion: Wan 2.2, HunyuanVideo 1.5, LTX-2.3. ⚡ Broader quantization (FP8/INT8, MXFP4/MXFP8, W4A16, ModelOpt) and hardware coverage. 339 commits, 124 contributors, 52 of them new. Thank you all. 🙌 🔗

42,708 Aufrufe • vor 2 Monaten

🎉 Congrats to MiniMax (official) on releasing MiniMax M3! Frontier coding and agentic capabilities, native image and video input, computer use, and a 1M-token context window, all in a single open model. At the heart of M3 is MSA, a new sparse attention architecture: instead of attending densely over the full KV cache, each query scores 128-token KV blocks and runs attention only over the top blocks. That is what makes 1M-token context practical to serve. M3 runs in vLLM with day-0 support, verified on NVIDIA and AMD hardware: ✨ MSA sparse attention with dedicated prefill and decode kernels ✨ 1M-token context serving with prefix caching and chunked prefill ✨ BF16 and MXFP8 checkpoints, with MoE backends for both Hopper and Blackwell ✨ Native multimodal input (image + video) ✨ Tool calling, reasoning parsing, and thinking-mode control for agent workloads Day-0 support like this is a true team effort. Grateful to the teams at MiniMax (official), NVIDIA AI, AI at AMD, and Inferact, and to the vLLM community for making it happen. 🙏 Deep dive into the implementation, kernel work, and deployment recipes: 🔗

🎉 Congrats to MiniMax (official) on releasing MiniMax M3! Frontier coding and agentic capabilities, native image and video input, computer use, and a 1M-token context window, all in a single open model. At the heart of M3 is MSA, a new sparse attention architecture: instead of attending densely over the full KV cache, each query scores 128-token KV blocks and runs attention only over the top blocks. That is what makes 1M-token context practical to serve. M3 runs in vLLM with day-0 support, verified on NVIDIA and AMD hardware: ✨ MSA sparse attention with dedicated prefill and decode kernels ✨ 1M-token context serving with prefix caching and chunked prefill ✨ BF16 and MXFP8 checkpoints, with MoE backends for both Hopper and Blackwell ✨ Native multimodal input (image + video) ✨ Tool calling, reasoning parsing, and thinking-mode control for agent workloads Day-0 support like this is a true team effort. Grateful to the teams at MiniMax (official), NVIDIA AI, AI at AMD, and Inferact, and to the vLLM community for making it happen. 🙏 Deep dive into the implementation, kernel work, and deployment recipes: 🔗

40,306 Aufrufe • vor 2 Monaten

🎉 Day-0 support for in vLLM, available today in v0.23.0! Congrats to Z.ai on GLM-5.2, a flagship model built for long-horizon coding agents. ✨ 1M-token context, built to hold project-scale engineering work in a single run ✨ Tuned for long-horizon coding: large-scale implementation, automated research, and performance optimization ✨ One task can carry a full dev workflow, from requirements to a deployable product across platforms ✨ Client-side and mobile engineering, including an on-device debugging loop Try it out running it on vLLM today: 🔗

🎉 Day-0 support for in vLLM, available today in v0.23.0! Congrats to Z.ai on GLM-5.2, a flagship model built for long-horizon coding agents. ✨ 1M-token context, built to hold project-scale engineering work in a single run ✨ Tuned for long-horizon coding: large-scale implementation, automated research, and performance optimization ✨ One task can carry a full dev workflow, from requirements to a deployable product across platforms ✨ Client-side and mobile engineering, including an on-device debugging loop Try it out running it on vLLM today: 🔗

36,240 Aufrufe • vor 2 Monaten

Keine weiteren Inhalte verfügbar