Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

InferenceMAX, vLLM TPU, compressed-tensors, MoE support via transformers, DeepSeek-OCR, and more. Here’s what’s new in the vLLM community over the past two weeks:

RedHat_AI's profile picture

Red Hat AI

10,748 subscribers

24,429 Aufrufe • vor 8 Monaten •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

Jensen spent 2 min on InferenceMAX at the GTC keynote 😎 InferenceMAX is completely open source and completely impartial, with the support of AMD, OpenAI, Nvidia, Microsoft, Oracle, Pytorch, SGLang, vLLM, Cursoe, Nebius, Tensorwave, etc
1:44

Jensen spent 2 min on InferenceMAX at the GTC keynote 😎 InferenceMAX is completely open source and completely impartial, with the support of AMD, OpenAI, Nvidia, Microsoft, Oracle, Pytorch, SGLang, vLLM, Cursoe, Nebius, Tensorwave, etc

Dylan Patel

86,497 Aufrufe • vor 8 Monaten

Today we announced a $150M seed round in Inferact, a new startup led by the maintainers of the vLLM project. Inferact will support the vLLM open source project through dedicated financial and developer resources and build what they see as the next generation commercial inference engine. Cofounders Simon Mo and Woosuk Kwon joined a16z GP Matt Bornstein for a conversation on how vLLM came to be, what they’ve learned from building it, and what they’re now doing at Inferact. They cover how vLLM began as a side project, why open source is essential to the world’s AI infrastructure, why inference is getting harder, and more. 00:00 Introduction 11:41 Community and collaboration in vLLM 19:19 Understanding inference engines 24:27 Cluster scale and GPU deployment 31:19 Belief in open source AI 35:45 Founding Inferact 40:00 The future of Inference at scale Simon Mo Woosuk Kwon Matt Bornstein
44:14

Today we announced a $150M seed round in Inferact, a new startup led by the maintainers of the vLLM project. Inferact will support the vLLM open source project through dedicated financial and developer resources and build what they see as the next generation commercial inference engine. Cofounders Simon Mo and Woosuk Kwon joined a16z GP Matt Bornstein for a conversation on how vLLM came to be, what they’ve learned from building it, and what they’re now doing at Inferact. They cover how vLLM began as a side project, why open source is essential to the world’s AI infrastructure, why inference is getting harder, and more. 00:00 Introduction 11:41 Community and collaboration in vLLM 19:19 Understanding inference engines 24:27 Cluster scale and GPU deployment 31:19 Belief in open source AI 35:45 Founding Inferact 40:00 The future of Inference at scale Simon Mo Woosuk Kwon Matt Bornstein

a16z

87,993 Aufrufe • vor 5 Monaten

We asked Gemma to analyze the sentiment in the Uniswap community over the past 7 days. Prompt was: “What is the sentiment in the Uniswap community in the past 7 days?” Here’s what she found:
0:20

We asked Gemma to analyze the sentiment in the Uniswap community over the past 7 days. Prompt was: “What is the sentiment in the Uniswap community in the past 7 days?” Here’s what she found:

Hey Anon

36,580 Aufrufe • vor 1 Jahr

These are the faces of some of the 5,700 Palestinians killed in Gaza over the past two weeks.
2:19

These are the faces of some of the 5,700 Palestinians killed in Gaza over the past two weeks.

AJ+

6,543,774 Aufrufe • vor 2 Jahren

.Nicki Minaj’s “Beez In The Trap and What’s Up” audio have been going viral on TikTok for two weeks straight. -Gaining over 5k new videos every day.
0:20

.Nicki Minaj’s “Beez In The Trap and What’s Up” audio have been going viral on TikTok for two weeks straight. -Gaining over 5k new videos every day.

Zina

12,207 Aufrufe • vor 8 Monaten

Unlimited-OCR 🔥New OCR from PaddlePaddle It can parse hundreds of pages in a single pass while maintaining stable speed. The key idea is R-SWA (Reference Sliding Window Attention), which keeps KV cache constant during decoding. 🏆 93% on OmniDocBench 📈 +6% over DeepSeek-OCR
0:50

Unlimited-OCR 🔥New OCR from PaddlePaddle It can parse hundreds of pages in a single pass while maintaining stable speed. The key idea is R-SWA (Reference Sliding Window Attention), which keeps KV cache constant during decoding. 🏆 93% on OmniDocBench 📈 +6% over DeepSeek-OCR

Adina Yakup

287,221 Aufrufe • vor 23 Tagen

The past can be a monster. The new season of #DarkWinds premieres in just two weeks, exclusively on AMC and AMC+.
0:30

The past can be a monster. The new season of #DarkWinds premieres in just two weeks, exclusively on AMC and AMC+.

AMC TV

13,948 Aufrufe • vor 1 Jahr

New Year, New InferenceMAX updates! Over the last month, we've made significant improvements to the UI, particularly based on feedback from the community. Here's an example of viewing a changelog and comparing across dates. (1/3)
0:42

New Year, New InferenceMAX updates! Over the last month, we've made significant improvements to the UI, particularly based on feedback from the community. Here's an example of viewing a changelog and comparing across dates. (1/3)

SemiAnalysis

24,489 Aufrufe • vor 6 Monaten

We co-led Inferact's $150M seed round to support them in their mission to build the inference engine for all current and future AI. In this episode of The Investment Memo, Lightspeed's Bucky Moore and James Alcorn sit down with Simon Mo (Co-Founder & CEO Inferact) to cover: - How vLLM grew to 60K+ GitHub stars - Why inference is shifting to the majority of compute - How vLLM evolved from a research project into the industry standard - Why building a company was the next step to push open-source inference forward 00:00 Introduction 02:03 The investment memo 04:47 Latency vs throughput vs cost 06:19 Paged attention explained 08:04 The evolution of attention 09:42 Growing the vLLM open source community 11:41 Working with hardware vendors 14:45 Deploying vLLM at large scale 16:03 Inferact's culture of openness 18:45 Building an open ecosystem and horizontal stack 19:45 Inferact's approach to fundraising 22:14 What is the future of inference? Simon Mo Bucky Moore James Alcorn
26:09

We co-led Inferact's $150M seed round to support them in their mission to build the inference engine for all current and future AI. In this episode of The Investment Memo, Lightspeed's Bucky Moore and James Alcorn sit down with Simon Mo (Co-Founder & CEO Inferact) to cover: - How vLLM grew to 60K+ GitHub stars - Why inference is shifting to the majority of compute - How vLLM evolved from a research project into the industry standard - Why building a company was the next step to push open-source inference forward 00:00 Introduction 02:03 The investment memo 04:47 Latency vs throughput vs cost 06:19 Paged attention explained 08:04 The evolution of attention 09:42 Growing the vLLM open source community 11:41 Working with hardware vendors 14:45 Deploying vLLM at large scale 16:03 Inferact's culture of openness 18:45 Building an open ecosystem and horizontal stack 19:45 Inferact's approach to fundraising 22:14 What is the future of inference? Simon Mo Bucky Moore James Alcorn

Lightspeed

27,040 Aufrufe • vor 5 Monaten

Finnish President Alexander Stubb: "I think in the past two weeks we've probably had more progress in ending this war than we have in the past three and a half years."
0:37

Finnish President Alexander Stubb: "I think in the past two weeks we've probably had more progress in ending this war than we have in the past three and a half years."

Fox News

116,674 Aufrufe • vor 11 Monaten

CMU PhD who built the kernels NVIDIA now ships in TensorRT-LLM explained fast attention in 68 minutes - better than $1200 GPU programming courses. pick the attention pattern -> generate a fused CUDA kernel -> drop it into vLLM/SGLang -> same GPU, way more tokens per second. That loop is why FlashInfer now powers inference at NVIDIA, vLLM, SGLang, and half the serving stacks you use. FlashInfer + Triton + JIT-compiled kernels + paged-KV attention - that's the stack.
1:08:48

CMU PhD who built the kernels NVIDIA now ships in TensorRT-LLM explained fast attention in 68 minutes - better than $1200 GPU programming courses. pick the attention pattern -> generate a fused CUDA kernel -> drop it into vLLM/SGLang -> same GPU, way more tokens per second. That loop is why FlashInfer now powers inference at NVIDIA, vLLM, SGLang, and half the serving stacks you use. FlashInfer + Triton + JIT-compiled kernels + paged-KV attention - that's the stack.

h100envy

32,605 Aufrufe • vor 13 Tagen

We asked Gemma AI about the community sentiment in Algorand Technologies social channels. The prompt we used was: “What’s the community sentiment in Algorand socials and chats over the last 3 days? Give examples.” Here’s what she found:
0:45

We asked Gemma AI about the community sentiment in Algorand Technologies social channels. The prompt we used was: “What’s the community sentiment in Algorand socials and chats over the last 3 days? Give examples.” Here’s what she found:

Hey Anon

17,465 Aufrufe • vor 1 Jahr

🎉 Congrats to MiniMax (official) on releasing MiniMax M3! Frontier coding and agentic capabilities, native image and video input, computer use, and a 1M-token context window, all in a single open model. At the heart of M3 is MSA, a new sparse attention architecture: instead of attending densely over the full KV cache, each query scores 128-token KV blocks and runs attention only over the top blocks. That is what makes 1M-token context practical to serve. M3 runs in vLLM with day-0 support, verified on NVIDIA and AMD hardware: ✨ MSA sparse attention with dedicated prefill and decode kernels ✨ 1M-token context serving with prefix caching and chunked prefill ✨ BF16 and MXFP8 checkpoints, with MoE backends for both Hopper and Blackwell ✨ Native multimodal input (image + video) ✨ Tool calling, reasoning parsing, and thinking-mode control for agent workloads Day-0 support like this is a true team effort. Grateful to the teams at MiniMax (official), NVIDIA AI, AI at AMD, and Inferact, and to the vLLM community for making it happen. 🙏 Deep dive into the implementation, kernel work, and deployment recipes: 🔗
0:21

🎉 Congrats to MiniMax (official) on releasing MiniMax M3! Frontier coding and agentic capabilities, native image and video input, computer use, and a 1M-token context window, all in a single open model. At the heart of M3 is MSA, a new sparse attention architecture: instead of attending densely over the full KV cache, each query scores 128-token KV blocks and runs attention only over the top blocks. That is what makes 1M-token context practical to serve. M3 runs in vLLM with day-0 support, verified on NVIDIA and AMD hardware: ✨ MSA sparse attention with dedicated prefill and decode kernels ✨ 1M-token context serving with prefix caching and chunked prefill ✨ BF16 and MXFP8 checkpoints, with MoE backends for both Hopper and Blackwell ✨ Native multimodal input (image + video) ✨ Tool calling, reasoning parsing, and thinking-mode control for agent workloads Day-0 support like this is a true team effort. Grateful to the teams at MiniMax (official), NVIDIA AI, AI at AMD, and Inferact, and to the vLLM community for making it happen. 🙏 Deep dive into the implementation, kernel work, and deployment recipes: 🔗

vLLM

40,306 Aufrufe • vor 1 Monat

🎉 Happy New Year from #IOST! ✨ To our partners, builders, and community around the world: THANK YOU for your trust, collaboration, and support. Here’s to building even more together in the year ahead 🚀
0:20

🎉 Happy New Year from #IOST! ✨ To our partners, builders, and community around the world: THANK YOU for your trust, collaboration, and support. Here’s to building even more together in the year ahead 🚀

IOST

13,410 Aufrufe • vor 6 Monaten

Among the fastest DeepSeek V3.2, MiniMax-M2.5, and Qwen 3.5 397B inference in the market, per Artificial Analysis benchmarks (April 2026). ⚡️🤖 Sub-1-second TTFT. 230 tokens per second. Co-designed every layer of the stack with Inferact, performance optimized vLLM, all on NVIDIA HGX B300. Live on DigitalOcean Serverless Inference now. Full breakdown in the comments. ⬇️
0:25

Among the fastest DeepSeek V3.2, MiniMax-M2.5, and Qwen 3.5 397B inference in the market, per Artificial Analysis benchmarks (April 2026). ⚡️🤖 Sub-1-second TTFT. 230 tokens per second. Co-designed every layer of the stack with Inferact, performance optimized vLLM, all on NVIDIA HGX B300. Live on DigitalOcean Serverless Inference now. Full breakdown in the comments. ⬇️

DigitalOcean

39,063 Aufrufe • vor 2 Monaten

AI can now hack everything. In 3 weeks Anthropic’s new AI built more hacks than the entire global security community discovered in the past decade. Treasury is already holding emergency meetings with banks.
3:15

AI can now hack everything. In 3 weeks Anthropic’s new AI built more hacks than the entire global security community discovered in the past decade. Treasury is already holding emergency meetings with banks.

Peter St Onge, Ph.D.

166,651 Aufrufe • vor 2 Monaten

Here’s Jordan Bell today on why he stayed at Memphis and how he’s grown over the past two years:
1:18

Here’s Jordan Bell today on why he stayed at Memphis and how he’s grown over the past two years:

Jonah Dylan

23,587 Aufrufe • vor 3 Monaten

#HRC55 Over the past two weeks, about 430 people, predominantly #women and schoolchildren, have been abducted in #Nigeria, by Boko Haram and other terrorists. The international community must stand together and demand the immediate and unconditional release of these captives. For Al Qaeda, ISIS, Al Shabab, Boko Haram, and Hamas, the use of hostage taking is not new. 134 Israelis are still being held in #Gaza for the past 5 months. The world must do more to address the scourge of terrorism. We must bring our people home.
1:26

#HRC55 Over the past two weeks, about 430 people, predominantly #women and schoolchildren, have been abducted in #Nigeria, by Boko Haram and other terrorists. The international community must stand together and demand the immediate and unconditional release of these captives. For Al Qaeda, ISIS, Al Shabab, Boko Haram, and Hamas, the use of hostage taking is not new. 134 Israelis are still being held in #Gaza for the past 5 months. The world must do more to address the scourge of terrorism. We must bring our people home.

Meirav Eilon Shahar 🇮🇱

31,949 Aufrufe • vor 2 Jahren

You can’t look at the atrocities Hamas terrorists committed in Israel over past two weeks and not scream out for justice. Justice must be done.
0:42

You can’t look at the atrocities Hamas terrorists committed in Israel over past two weeks and not scream out for justice. Justice must be done.

President Biden Archived

8,051,653 Aufrufe • vor 2 Jahren