Loading video...

Video Failed to Load

Go Home

InferenceMAX, vLLM TPU, compressed-tensors, MoE support via transformers, DeepSeek-OCR, and more. Here’s what’s new in the vLLM community over the past two weeks:

RedHat_AI's profile picture

Red Hat AI

10,748 subscribers

24,429 views • 9 months ago •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

Jensen spent 2 min on InferenceMAX at the GTC keynote 😎 InferenceMAX is completely open source and completely impartial, with the support of AMD, OpenAI, Nvidia, Microsoft, Oracle, Pytorch, SGLang, vLLM, Cursoe, Nebius, Tensorwave, etc
1:44

Jensen spent 2 min on InferenceMAX at the GTC keynote 😎 InferenceMAX is completely open source and completely impartial, with the support of AMD, OpenAI, Nvidia, Microsoft, Oracle, Pytorch, SGLang, vLLM, Cursoe, Nebius, Tensorwave, etc

Dylan Patel

86,497 views • 9 months ago

Today we announced a $150M seed round in Inferact, a new startup led by the maintainers of the vLLM project. Inferact will support the vLLM open source project through dedicated financial and developer resources and build what they see as the next generation commercial inference engine. Cofounders Simon Mo and Woosuk Kwon joined a16z GP Matt Bornstein for a conversation on how vLLM came to be, what they’ve learned from building it, and what they’re now doing at Inferact. They cover how vLLM began as a side project, why open source is essential to the world’s AI infrastructure, why inference is getting harder, and more. 00:00 Introduction 11:41 Community and collaboration in vLLM 19:19 Understanding inference engines 24:27 Cluster scale and GPU deployment 31:19 Belief in open source AI 35:45 Founding Inferact 40:00 The future of Inference at scale Simon Mo Woosuk Kwon Matt Bornstein
44:14

Today we announced a $150M seed round in Inferact, a new startup led by the maintainers of the vLLM project. Inferact will support the vLLM open source project through dedicated financial and developer resources and build what they see as the next generation commercial inference engine. Cofounders Simon Mo and Woosuk Kwon joined a16z GP Matt Bornstein for a conversation on how vLLM came to be, what they’ve learned from building it, and what they’re now doing at Inferact. They cover how vLLM began as a side project, why open source is essential to the world’s AI infrastructure, why inference is getting harder, and more. 00:00 Introduction 11:41 Community and collaboration in vLLM 19:19 Understanding inference engines 24:27 Cluster scale and GPU deployment 31:19 Belief in open source AI 35:45 Founding Inferact 40:00 The future of Inference at scale Simon Mo Woosuk Kwon Matt Bornstein

a16z

87,993 views • 6 months ago

We asked Gemma to analyze the sentiment in the Uniswap community over the past 7 days. Prompt was: “What is the sentiment in the Uniswap community in the past 7 days?” Here’s what she found:
0:20

We asked Gemma to analyze the sentiment in the Uniswap community over the past 7 days. Prompt was: “What is the sentiment in the Uniswap community in the past 7 days?” Here’s what she found:

Hey Anon

36,580 views • 1 year ago

.Nicki Minaj’s “Beez In The Trap and What’s Up” audio have been going viral on TikTok for two weeks straight. -Gaining over 5k new videos every day.
0:20

.Nicki Minaj’s “Beez In The Trap and What’s Up” audio have been going viral on TikTok for two weeks straight. -Gaining over 5k new videos every day.

Zina

12,207 views • 9 months ago

These are the faces of some of the 5,700 Palestinians killed in Gaza over the past two weeks.
2:19

These are the faces of some of the 5,700 Palestinians killed in Gaza over the past two weeks.

AJ+

6,544,915 views • 2 years ago

Unlimited-OCR 🔥New OCR from PaddlePaddle It can parse hundreds of pages in a single pass while maintaining stable speed. The key idea is R-SWA (Reference Sliding Window Attention), which keeps KV cache constant during decoding. 🏆 93% on OmniDocBench 📈 +6% over DeepSeek-OCR
0:50

Unlimited-OCR 🔥New OCR from PaddlePaddle It can parse hundreds of pages in a single pass while maintaining stable speed. The key idea is R-SWA (Reference Sliding Window Attention), which keeps KV cache constant during decoding. 🏆 93% on OmniDocBench 📈 +6% over DeepSeek-OCR

Adina Yakup

289,305 views • 1 month ago

New Year, New InferenceMAX updates! Over the last month, we've made significant improvements to the UI, particularly based on feedback from the community. Here's an example of viewing a changelog and comparing across dates. (1/3)
0:42

New Year, New InferenceMAX updates! Over the last month, we've made significant improvements to the UI, particularly based on feedback from the community. Here's an example of viewing a changelog and comparing across dates. (1/3)

SemiAnalysis

24,489 views • 7 months ago

DUAL INTEL ARC PRO B70 GPUS AT $1,900 RUN QWEN 3 CODER 30B AT 268 TOKENS PER SECOND VIA VLLM ON 8 CONCURRENT REQUESTS, A NEW INTEL TIER UNDER YOUR MAP THAT BEATS RTX 3090 STACKS ON THROUGHPUT PER DOLLAR 02:22 the operator reads off his benchmark sheet, "vllm was at 268 tokens per second which is just wild to me, getting 250+ tokens per second out of these overall kind of slow Intel cards" dual Intel Arc Pro B70 ships at $950 per card, 32GB GDDR6 each, 64GB combined VRAM for $1,900 total, the same budget gets you two used RTX 3090s at 48GB combined and worse multi-user throughput Qwen 3 coder 30B on dual B70 with vLLM hits 268 t/s at 8 concurrent users, llama.cpp tops out at 84 t/s on the same hardware, the gap is not 2x or 3x, vLLM with Intel's tweaks pulls 3.2x on parallel inference prefill on a 16,000 word context took 26 seconds on llama.cpp and 3 seconds on vLLM, agents spawning subagents or households sharing one box turn that 8x speedup into 250 t/s of usable throughput your map skips Intel because the Arc cards looked weak in 2025, the B70 plus the vLLM stack flipped that, $1,900 in dual Intel silicon now serves a 4 person team at the speed openai serves enterprise tiers for $400 a month per seat bookmark this and read the article below
4:27

DUAL INTEL ARC PRO B70 GPUS AT $1,900 RUN QWEN 3 CODER 30B AT 268 TOKENS PER SECOND VIA VLLM ON 8 CONCURRENT REQUESTS, A NEW INTEL TIER UNDER YOUR MAP THAT BEATS RTX 3090 STACKS ON THROUGHPUT PER DOLLAR 02:22 the operator reads off his benchmark sheet, "vllm was at 268 tokens per second which is just wild to me, getting 250+ tokens per second out of these overall kind of slow Intel cards" dual Intel Arc Pro B70 ships at $950 per card, 32GB GDDR6 each, 64GB combined VRAM for $1,900 total, the same budget gets you two used RTX 3090s at 48GB combined and worse multi-user throughput Qwen 3 coder 30B on dual B70 with vLLM hits 268 t/s at 8 concurrent users, llama.cpp tops out at 84 t/s on the same hardware, the gap is not 2x or 3x, vLLM with Intel's tweaks pulls 3.2x on parallel inference prefill on a 16,000 word context took 26 seconds on llama.cpp and 3 seconds on vLLM, agents spawning subagents or households sharing one box turn that 8x speedup into 250 t/s of usable throughput your map skips Intel because the Arc cards looked weak in 2025, the B70 plus the vLLM stack flipped that, $1,900 in dual Intel silicon now serves a 4 person team at the speed openai serves enterprise tiers for $400 a month per seat bookmark this and read the article below

starmex

28,401 views • 1 month ago

We co-led Inferact's $150M seed round to support them in their mission to build the inference engine for all current and future AI. In this episode of The Investment Memo, Lightspeed's Bucky Moore and James Alcorn sit down with Simon Mo (Co-Founder & CEO Inferact) to cover: - How vLLM grew to 60K+ GitHub stars - Why inference is shifting to the majority of compute - How vLLM evolved from a research project into the industry standard - Why building a company was the next step to push open-source inference forward 00:00 Introduction 02:03 The investment memo 04:47 Latency vs throughput vs cost 06:19 Paged attention explained 08:04 The evolution of attention 09:42 Growing the vLLM open source community 11:41 Working with hardware vendors 14:45 Deploying vLLM at large scale 16:03 Inferact's culture of openness 18:45 Building an open ecosystem and horizontal stack 19:45 Inferact's approach to fundraising 22:14 What is the future of inference? Simon Mo Bucky Moore James Alcorn
26:09

We co-led Inferact's $150M seed round to support them in their mission to build the inference engine for all current and future AI. In this episode of The Investment Memo, Lightspeed's Bucky Moore and James Alcorn sit down with Simon Mo (Co-Founder & CEO Inferact) to cover: - How vLLM grew to 60K+ GitHub stars - Why inference is shifting to the majority of compute - How vLLM evolved from a research project into the industry standard - Why building a company was the next step to push open-source inference forward 00:00 Introduction 02:03 The investment memo 04:47 Latency vs throughput vs cost 06:19 Paged attention explained 08:04 The evolution of attention 09:42 Growing the vLLM open source community 11:41 Working with hardware vendors 14:45 Deploying vLLM at large scale 16:03 Inferact's culture of openness 18:45 Building an open ecosystem and horizontal stack 19:45 Inferact's approach to fundraising 22:14 What is the future of inference? Simon Mo Bucky Moore James Alcorn

Lightspeed

27,040 views • 6 months ago

Finnish President Alexander Stubb: "I think in the past two weeks we've probably had more progress in ending this war than we have in the past three and a half years."
0:37

Finnish President Alexander Stubb: "I think in the past two weeks we've probably had more progress in ending this war than we have in the past three and a half years."

Fox News

116,678 views • 1 year ago

CMU PhD who built the kernels NVIDIA now ships in TensorRT-LLM explained fast attention in 68 minutes - better than $1200 GPU programming courses. pick the attention pattern -> generate a fused CUDA kernel -> drop it into vLLM/SGLang -> same GPU, way more tokens per second. That loop is why FlashInfer now powers inference at NVIDIA, vLLM, SGLang, and half the serving stacks you use. FlashInfer + Triton + JIT-compiled kernels + paged-KV attention - that's the stack.
1:08:48

CMU PhD who built the kernels NVIDIA now ships in TensorRT-LLM explained fast attention in 68 minutes - better than $1200 GPU programming courses. pick the attention pattern -> generate a fused CUDA kernel -> drop it into vLLM/SGLang -> same GPU, way more tokens per second. That loop is why FlashInfer now powers inference at NVIDIA, vLLM, SGLang, and half the serving stacks you use. FlashInfer + Triton + JIT-compiled kernels + paged-KV attention - that's the stack.

h100envy

32,605 views • 1 month ago

We asked Gemma AI about the community sentiment in Algorand Technologies social channels. The prompt we used was: “What’s the community sentiment in Algorand socials and chats over the last 3 days? Give examples.” Here’s what she found:
0:45

We asked Gemma AI about the community sentiment in Algorand Technologies social channels. The prompt we used was: “What’s the community sentiment in Algorand socials and chats over the last 3 days? Give examples.” Here’s what she found:

Hey Anon

17,465 views • 1 year ago

🎉 Congrats to MiniMax (official) on releasing MiniMax M3! Frontier coding and agentic capabilities, native image and video input, computer use, and a 1M-token context window, all in a single open model. At the heart of M3 is MSA, a new sparse attention architecture: instead of attending densely over the full KV cache, each query scores 128-token KV blocks and runs attention only over the top blocks. That is what makes 1M-token context practical to serve. M3 runs in vLLM with day-0 support, verified on NVIDIA and AMD hardware: ✨ MSA sparse attention with dedicated prefill and decode kernels ✨ 1M-token context serving with prefix caching and chunked prefill ✨ BF16 and MXFP8 checkpoints, with MoE backends for both Hopper and Blackwell ✨ Native multimodal input (image + video) ✨ Tool calling, reasoning parsing, and thinking-mode control for agent workloads Day-0 support like this is a true team effort. Grateful to the teams at MiniMax (official), NVIDIA AI, AI at AMD, and Inferact, and to the vLLM community for making it happen. 🙏 Deep dive into the implementation, kernel work, and deployment recipes: 🔗
0:21

🎉 Congrats to MiniMax (official) on releasing MiniMax M3! Frontier coding and agentic capabilities, native image and video input, computer use, and a 1M-token context window, all in a single open model. At the heart of M3 is MSA, a new sparse attention architecture: instead of attending densely over the full KV cache, each query scores 128-token KV blocks and runs attention only over the top blocks. That is what makes 1M-token context practical to serve. M3 runs in vLLM with day-0 support, verified on NVIDIA and AMD hardware: ✨ MSA sparse attention with dedicated prefill and decode kernels ✨ 1M-token context serving with prefix caching and chunked prefill ✨ BF16 and MXFP8 checkpoints, with MoE backends for both Hopper and Blackwell ✨ Native multimodal input (image + video) ✨ Tool calling, reasoning parsing, and thinking-mode control for agent workloads Day-0 support like this is a true team effort. Grateful to the teams at MiniMax (official), NVIDIA AI, AI at AMD, and Inferact, and to the vLLM community for making it happen. 🙏 Deep dive into the implementation, kernel work, and deployment recipes: 🔗

vLLM

40,306 views • 2 months ago

NEW: As support pours in from several parties for what’s been dubbed the “largest mobilization” of the 2SLGBTQIA+ community in decades, one party is conspicuously absent. What is going on inside the Conservative Party? Here’s what we know:
2:14

NEW: As support pours in from several parties for what’s been dubbed the “largest mobilization” of the 2SLGBTQIA+ community in decades, one party is conspicuously absent. What is going on inside the Conservative Party? Here’s what we know:

Rachel Gilmore

37,728 views • 2 years ago

Among the fastest DeepSeek V3.2, MiniMax-M2.5, and Qwen 3.5 397B inference in the market, per Artificial Analysis benchmarks (April 2026). ⚡️🤖 Sub-1-second TTFT. 230 tokens per second. Co-designed every layer of the stack with Inferact, performance optimized vLLM, all on NVIDIA HGX B300. Live on DigitalOcean Serverless Inference now. Full breakdown in the comments. ⬇️
0:25

Among the fastest DeepSeek V3.2, MiniMax-M2.5, and Qwen 3.5 397B inference in the market, per Artificial Analysis benchmarks (April 2026). ⚡️🤖 Sub-1-second TTFT. 230 tokens per second. Co-designed every layer of the stack with Inferact, performance optimized vLLM, all on NVIDIA HGX B300. Live on DigitalOcean Serverless Inference now. Full breakdown in the comments. ⬇️

DigitalOcean

39,063 views • 3 months ago

AI can now hack everything. In 3 weeks Anthropic’s new AI built more hacks than the entire global security community discovered in the past decade. Treasury is already holding emergency meetings with banks.
3:15

AI can now hack everything. In 3 weeks Anthropic’s new AI built more hacks than the entire global security community discovered in the past decade. Treasury is already holding emergency meetings with banks.

Peter St Onge, Ph.D.

166,857 views • 3 months ago

🎉 Happy New Year from #IOST! ✨ To our partners, builders, and community around the world: THANK YOU for your trust, collaboration, and support. Here’s to building even more together in the year ahead 🚀
0:20

🎉 Happy New Year from #IOST! ✨ To our partners, builders, and community around the world: THANK YOU for your trust, collaboration, and support. Here’s to building even more together in the year ahead 🚀

IOST

13,410 views • 7 months ago

Here’s Jordan Bell today on why he stayed at Memphis and how he’s grown over the past two years:
1:18

Here’s Jordan Bell today on why he stayed at Memphis and how he’s grown over the past two years:

Jonah Dylan

23,669 views • 4 months ago