正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

InferenceMAX, vLLM TPU, compressed-tensors, MoE support via transformers, DeepSeek-OCR, and more. Here’s what’s new in the vLLM community over the past two weeks:

Red Hat AI

10,748 subscribers

24,429 次观看 • 8 个月前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

Jensen spent 2 min on InferenceMAX at the GTC keynote 😎 InferenceMAX is completely open source and completely impartial, with the support of AMD, OpenAI, Nvidia, Microsoft, Oracle, Pytorch, SGLang, vLLM, Cursoe, Nebius, Tensorwave, etc

Jensen spent 2 min on InferenceMAX at the GTC keynote 😎 InferenceMAX is completely open source and completely impartial, with the support of AMD, OpenAI, Nvidia, Microsoft, Oracle, Pytorch, SGLang, vLLM, Cursoe, Nebius, Tensorwave, etc

Dylan Patel

86,497 次观看 • 7 个月前

DeepSeek's (DeepSeek) latest—MLA, Multi-Token Prediction, 256 Experts, FP8 block quantization—shines with vLLM. Catch the office hours session were we discuss all the DeepSeek goodies and explore their integration and benchmarks with #vLLM.

DeepSeek's (DeepSeek) latest—MLA, Multi-Token Prediction, 256 Experts, FP8 block quantization—shines with vLLM. Catch the office hours session were we discuss all the DeepSeek goodies and explore their integration and benchmarks with #vLLM.

Red Hat AI

14,093 次观看 • 1 年前

Today we announced a $150M seed round in Inferact, a new startup led by the maintainers of the vLLM project. Inferact will support the vLLM open source project through dedicated financial and developer resources and build what they see as the next generation commercial inference engine. Cofounders Simon Mo and Woosuk Kwon joined a16z GP Matt Bornstein for a conversation on how vLLM came to be, what they’ve learned from building it, and what they’re now doing at Inferact. They cover how vLLM began as a side project, why open source is essential to the world’s AI infrastructure, why inference is getting harder, and more. 00:00 Introduction 11:41 Community and collaboration in vLLM 19:19 Understanding inference engines 24:27 Cluster scale and GPU deployment 31:19 Belief in open source AI 35:45 Founding Inferact 40:00 The future of Inference at scale Simon Mo Woosuk Kwon Matt Bornstein

Today we announced a $150M seed round in Inferact, a new startup led by the maintainers of the vLLM project. Inferact will support the vLLM open source project through dedicated financial and developer resources and build what they see as the next generation commercial inference engine. Cofounders Simon Mo and Woosuk Kwon joined a16z GP Matt Bornstein for a conversation on how vLLM came to be, what they’ve learned from building it, and what they’re now doing at Inferact. They cover how vLLM began as a side project, why open source is essential to the world’s AI infrastructure, why inference is getting harder, and more. 00:00 Introduction 11:41 Community and collaboration in vLLM 19:19 Understanding inference engines 24:27 Cluster scale and GPU deployment 31:19 Belief in open source AI 35:45 Founding Inferact 40:00 The future of Inference at scale Simon Mo Woosuk Kwon Matt Bornstein

a16z

87,993 次观看 • 5 个月前

A full year of vLLM in 30 minutes by vLLM Lead from UC Berkeley, Simon Mo. Model and hardware usage trends, model architectures, API evolution, V1 engine rebuild, multimodal progress, expanding hardware support, and more. Plus how we are thinking about 2026. Enjoy!

A full year of vLLM in 30 minutes by vLLM Lead from UC Berkeley, Simon Mo. Model and hardware usage trends, model architectures, API evolution, V1 engine rebuild, multimodal progress, expanding hardware support, and more. Plus how we are thinking about 2026. Enjoy!

Red Hat AI

15,702 次观看 • 6 个月前

One adapter. Every OpenAI-compatible provider. 🔌 openaiCompatible({ baseURL, apiKey, models }) Connects DeepSeek, Kimi, Together, Fireworks, Qwen, local LM Studio/vLLM, and more. Type-safe model unions, no package per vendor. 🚀

One adapter. Every OpenAI-compatible provider. 🔌 openaiCompatible({ baseURL, apiKey, models }) Connects DeepSeek, Kimi, Together, Fireworks, Qwen, local LM Studio/vLLM, and more. Type-safe model unions, no package per vendor. 🚀

TANSTACK

19,494 次观看 • 19 天前

introducing simple-llm: a ~950 line, powerful & extensible inference engine that performs on par with vllm. enjoy :) performance (gpt-oss-120b, on an h100): - batch=1: 135 tok/s (vllm: 138) - batch=64: 4,041 tok/s (vllm: 3,846)

introducing simple-llm: a ~950 line, powerful & extensible inference engine that performs on par with vllm. enjoy :) performance (gpt-oss-120b, on an h100): - batch=1: 135 tok/s (vllm: 138) - batch=64: 4,041 tok/s (vllm: 3,846)

naklecha

59,730 次观看 • 5 个月前

DeepSeek-V4-Flash-Spark and Spark-Mini were born today. 1 command setup to Pi / vllm-studio / docker deployment. All this is running local on a single DGX Spark.

DeepSeek-V4-Flash-Spark and Spark-Mini were born today. 1 command setup to Pi / vllm-studio / docker deployment. All this is running local on a single DGX Spark.

0xSero

20,019 次观看 • 25 天前

vllm-studio > claude-desktop officially confirmed

vllm-studio > claude-desktop officially confirmed

0xSero

23,270 次观看 • 1 个月前

Laguna XS.2 from Poolside is a 33B MoE built for agentic coding. Red Hat AI trained a DFlash speculator for it: 0.6B drafter, 8 tokens per pass, no quality loss. FP8, NVFP4, and INT4 checkpoints via LLM Compressor. Models in comments. Speedup with vLLM:

Laguna XS.2 from Poolside is a 33B MoE built for agentic coding. Red Hat AI trained a DFlash speculator for it: 0.6B drafter, 8 tokens per pass, no quality loss. FP8, NVFP4, and INT4 checkpoints via LLM Compressor. Models in comments. Speedup with vLLM:

Red Hat AI

20,828 次观看 • 23 天前

We asked Gemma to analyze the sentiment in the Uniswap community over the past 7 days. Prompt was: “What is the sentiment in the Uniswap community in the past 7 days?” Here’s what she found:

We asked Gemma to analyze the sentiment in the Uniswap community over the past 7 days. Prompt was: “What is the sentiment in the Uniswap community in the past 7 days?” Here’s what she found:

Hey Anon

36,580 次观看 • 1 年前

These are the faces of some of the 5,700 Palestinians killed in Gaza over the past two weeks.

These are the faces of some of the 5,700 Palestinians killed in Gaza over the past two weeks.

AJ+

6,542,737 次观看 • 2 年前

.Nicki Minaj’s “Beez In The Trap and What’s Up” audio have been going viral on TikTok for two weeks straight. -Gaining over 5k new videos every day.

.Nicki Minaj’s “Beez In The Trap and What’s Up” audio have been going viral on TikTok for two weeks straight. -Gaining over 5k new videos every day.

Zina

12,207 次观看 • 7 个月前

Michael Goin (Michael Goin) walks through what's new in vLLM v0.17, v0.18, and v0.19 in ~8 minutes. Flash Attention 4, new performance modes, zero-bubble async scheduling, online MXFP4 quantization, Gemma 4, and a lot more. 1,592 commits. 682 contributors (163 new). 🎉 🚀

Michael Goin (Michael Goin) walks through what's new in vLLM v0.17, v0.18, and v0.19 in ~8 minutes. Flash Attention 4, new performance modes, zero-bubble async scheduling, online MXFP4 quantization, Gemma 4, and a lot more. 1,592 commits. 682 contributors (163 new). 🎉 🚀

Red Hat AI

23,103 次观看 • 2 个月前

New Year, New InferenceMAX updates! Over the last month, we've made significant improvements to the UI, particularly based on feedback from the community. Here's an example of viewing a changelog and comparing across dates. (1/3)

New Year, New InferenceMAX updates! Over the last month, we've made significant improvements to the UI, particularly based on feedback from the community. Here's an example of viewing a changelog and comparing across dates. (1/3)

SemiAnalysis

24,489 次观看 • 5 个月前

The past can be a monster. The new season of #DarkWinds premieres in just two weeks, exclusively on AMC and AMC+.

The past can be a monster. The new season of #DarkWinds premieres in just two weeks, exclusively on AMC and AMC+.

AMC TV

13,948 次观看 • 1 年前

We co-led Inferact's $150M seed round to support them in their mission to build the inference engine for all current and future AI. In this episode of The Investment Memo, Lightspeed's Bucky Moore and James Alcorn sit down with Simon Mo (Co-Founder & CEO Inferact) to cover: - How vLLM grew to 60K+ GitHub stars - Why inference is shifting to the majority of compute - How vLLM evolved from a research project into the industry standard - Why building a company was the next step to push open-source inference forward 00:00 Introduction 02:03 The investment memo 04:47 Latency vs throughput vs cost 06:19 Paged attention explained 08:04 The evolution of attention 09:42 Growing the vLLM open source community 11:41 Working with hardware vendors 14:45 Deploying vLLM at large scale 16:03 Inferact's culture of openness 18:45 Building an open ecosystem and horizontal stack 19:45 Inferact's approach to fundraising 22:14 What is the future of inference? Simon Mo Bucky Moore James Alcorn

We co-led Inferact's $150M seed round to support them in their mission to build the inference engine for all current and future AI. In this episode of The Investment Memo, Lightspeed's Bucky Moore and James Alcorn sit down with Simon Mo (Co-Founder & CEO Inferact) to cover: - How vLLM grew to 60K+ GitHub stars - Why inference is shifting to the majority of compute - How vLLM evolved from a research project into the industry standard - Why building a company was the next step to push open-source inference forward 00:00 Introduction 02:03 The investment memo 04:47 Latency vs throughput vs cost 06:19 Paged attention explained 08:04 The evolution of attention 09:42 Growing the vLLM open source community 11:41 Working with hardware vendors 14:45 Deploying vLLM at large scale 16:03 Inferact's culture of openness 18:45 Building an open ecosystem and horizontal stack 19:45 Inferact's approach to fundraising 22:14 What is the future of inference? Simon Mo Bucky Moore James Alcorn

Lightspeed

27,040 次观看 • 5 个月前

Finnish President Alexander Stubb: "I think in the past two weeks we've probably had more progress in ending this war than we have in the past three and a half years."

Finnish President Alexander Stubb: "I think in the past two weeks we've probably had more progress in ending this war than we have in the past three and a half years."

Fox News

116,649 次观看 • 10 个月前

🎉 Congrats to MiniMax (official) on releasing MiniMax M3! Frontier coding and agentic capabilities, native image and video input, computer use, and a 1M-token context window, all in a single open model. At the heart of M3 is MSA, a new sparse attention architecture: instead of attending densely over the full KV cache, each query scores 128-token KV blocks and runs attention only over the top blocks. That is what makes 1M-token context practical to serve. M3 runs in vLLM with day-0 support, verified on NVIDIA and AMD hardware: ✨ MSA sparse attention with dedicated prefill and decode kernels ✨ 1M-token context serving with prefix caching and chunked prefill ✨ BF16 and MXFP8 checkpoints, with MoE backends for both Hopper and Blackwell ✨ Native multimodal input (image + video) ✨ Tool calling, reasoning parsing, and thinking-mode control for agent workloads Day-0 support like this is a true team effort. Grateful to the teams at MiniMax (official), NVIDIA AI, AI at AMD, and Inferact, and to the vLLM community for making it happen. 🙏 Deep dive into the implementation, kernel work, and deployment recipes: 🔗

🎉 Congrats to MiniMax (official) on releasing MiniMax M3! Frontier coding and agentic capabilities, native image and video input, computer use, and a 1M-token context window, all in a single open model. At the heart of M3 is MSA, a new sparse attention architecture: instead of attending densely over the full KV cache, each query scores 128-token KV blocks and runs attention only over the top blocks. That is what makes 1M-token context practical to serve. M3 runs in vLLM with day-0 support, verified on NVIDIA and AMD hardware: ✨ MSA sparse attention with dedicated prefill and decode kernels ✨ 1M-token context serving with prefix caching and chunked prefill ✨ BF16 and MXFP8 checkpoints, with MoE backends for both Hopper and Blackwell ✨ Native multimodal input (image + video) ✨ Tool calling, reasoning parsing, and thinking-mode control for agent workloads Day-0 support like this is a true team effort. Grateful to the teams at MiniMax (official), NVIDIA AI, AI at AMD, and Inferact, and to the vLLM community for making it happen. 🙏 Deep dive into the implementation, kernel work, and deployment recipes: 🔗

vLLM

39,777 次观看 • 10 天前

We asked Gemma AI about the community sentiment in Algorand Technologies social channels. The prompt we used was: “What’s the community sentiment in Algorand socials and chats over the last 3 days? Give examples.” Here’s what she found:

We asked Gemma AI about the community sentiment in Algorand Technologies social channels. The prompt we used was: “What’s the community sentiment in Algorand socials and chats over the last 3 days? Give examples.” Here’s what she found:

Hey Anon

17,465 次观看 • 1 年前

Among the fastest DeepSeek V3.2, MiniMax-M2.5, and Qwen 3.5 397B inference in the market, per Artificial Analysis benchmarks (April 2026). ⚡️🤖 Sub-1-second TTFT. 230 tokens per second. Co-designed every layer of the stack with Inferact, performance optimized vLLM, all on NVIDIA HGX B300. Live on DigitalOcean Serverless Inference now. Full breakdown in the comments. ⬇️

Among the fastest DeepSeek V3.2, MiniMax-M2.5, and Qwen 3.5 397B inference in the market, per Artificial Analysis benchmarks (April 2026). ⚡️🤖 Sub-1-second TTFT. 230 tokens per second. Co-designed every layer of the stack with Inferact, performance optimized vLLM, all on NVIDIA HGX B300. Live on DigitalOcean Serverless Inference now. Full breakdown in the comments. ⬇️

DigitalOcean

39,063 次观看 • 1 个月前