Загрузка видео...

Не удалось загрузить видео

Возникла проблема при загрузке этого видео. Это может быть связано с временными проблемами сети или видео может быть недоступно.

На главную

Luminal fuses entire models into a single GPU kernel, automatically. Let's talk about why this matters for inference at the speed of light:

Joe Fioti

3,040 subscribers

24,436 просмотров • 8 месяцев назад •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

Reasoning LLMs generate very long chains-of-thought, so even small quantization errors add up. With AWQ, Qwen3-4B drops 71.0 → 68.2 on MMLU-Pro (~4% relative loss). 😬 ParoQuant fixes this! It keeps only the critical rotation pairs and fuses everything into a single kernel. Recovers most of the lost reasoning accuracy with minimal overhead — so 4-bit models stay strong at reasoning. 💪💪

Reasoning LLMs generate very long chains-of-thought, so even small quantization errors add up. With AWQ, Qwen3-4B drops 71.0 → 68.2 on MMLU-Pro (~4% relative loss). 😬 ParoQuant fixes this! It keeps only the critical rotation pairs and fuses everything into a single kernel. Recovers most of the lost reasoning accuracy with minimal overhead — so 4-bit models stay strong at reasoning. 💪💪

Zhijian Liu

171,517 просмотров • 6 месяцев назад

Manual QA is dead Engineers used to wait hours, sometimes days, for QA to green-light a single change. That loop is collapsing with agents. Verify in minutes, fix in minutes, and ship with confidence at the speed of inference.

Manual QA is dead Engineers used to wait hours, sometimes days, for QA to green-light a single change. That loop is collapsing with agents. Verify in minutes, fix in minutes, and ship with confidence at the speed of inference.

hayden

19,717 просмотров • 4 месяцев назад

I didn't hear a single Democrat talk about this while it was happening in front of the entire world.

I didn't hear a single Democrat talk about this while it was happening in front of the entire world.

aka

113,705 просмотров • 8 месяцев назад

$500b wiped out because of a single tweet? Private market valuations of Anthropic, OpenAI, SpaceXai crashed yesterday...let's talk about why. (Please don't make me explain the rug pull joke at the beginning 🤦‍♂️)

$500b wiped out because of a single tweet? Private market valuations of Anthropic, OpenAI, SpaceXai crashed yesterday...let's talk about why. (Please don't make me explain the rug pull joke at the beginning 🤦‍♂️)

Matthew Berman

143,034 просмотров • 4 месяцев назад

Why does NVIDIA spend millions of GPU hours training models just to give it all away? Here's a recording to the talk at NeurIPS for anyone who couldn't make it, wish we recorded the questions afterwards!

Why does NVIDIA spend millions of GPU hours training models just to give it all away? Here's a recording to the talk at NeurIPS for anyone who couldn't make it, wish we recorded the questions afterwards!

Nader Khalil🍊

126,208 просмотров • 9 месяцев назад

PyTorch core engineer at Meta turned CUDA kernel writing into a sport in 13 minutes - better than $1500 GPU programming bootcamps. profile the kernel -> find the bottleneck -> rewrite -> benchmark -> merge the winning code into PyTorch. That loop is how the open community now beats hand-tuned vendor kernels. GPU MODE community + KernelBot competition + winning kernel merged into the framework - that's the stack. Watch it, then steal the loop below.

h100envy

35,390 просмотров • 2 месяцев назад

PowerInfer - a high-speed inference engine for deploying LLMs locally. Just came across this super interesting project on speeding up inference. It's not MoE but it's a simple approach that exploits the high locality in LLM inference to design a GPU-CPU hybrid inference engine. Hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons (the majority) are computed on the CPU. This approach significantly reduces GPU memory demands and CPU-GPU data transfer. It achieves an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs on a single NVIDIA RTX 4090 GPU. It's on only 18% lower than that achieved by a top-tier server-grade A100 GPU. It also significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy. There is a lot more innovation around inference that's coming fast. Really encouraged by the study on sparse computation to enhance the computational efficiency of LLMs. It's now possible to use PowerInfer with Llama 2 and Faclon 40B. Mistral-7B support is coming soon!

PowerInfer - a high-speed inference engine for deploying LLMs locally. Just came across this super interesting project on speeding up inference. It's not MoE but it's a simple approach that exploits the high locality in LLM inference to design a GPU-CPU hybrid inference engine. Hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons (the majority) are computed on the CPU. This approach significantly reduces GPU memory demands and CPU-GPU data transfer. It achieves an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs on a single NVIDIA RTX 4090 GPU. It's on only 18% lower than that achieved by a top-tier server-grade A100 GPU. It also significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy. There is a lot more innovation around inference that's coming fast. Really encouraged by the study on sparse computation to enhance the computational efficiency of LLMs. It's now possible to use PowerInfer with Llama 2 and Faclon 40B. Mistral-7B support is coming soon!

elvis

261,622 просмотров • 2 лет назад

Every single time there is a spirit-filled prophetic pastor, the entire world begins to align against that man. This is not new. There are plenty of examples of this in the Bible. Let's talk about the Phillip Anthony Mitchell "scandal."

Every single time there is a spirit-filled prophetic pastor, the entire world begins to align against that man. This is not new. There are plenty of examples of this in the Bible. Let's talk about the Phillip Anthony Mitchell "scandal."

Josh Howerton

107,766 просмотров • 17 дней назад

at the speed of light (no pun intended) My talk for Deep Tech Week: optics, cancer, diagnostics, and Precigenetics

at the speed of light (no pun intended) My talk for Deep Tech Week: optics, cancer, diagnostics, and Precigenetics

Parmita Mishra

33,942 просмотров • 1 год назад

The Metaphysics of Slop Let's talk about why slop is bad.

The Metaphysics of Slop Let's talk about why slop is bad.

Carl Benjamin 🏴󠁧󠁢󠁥󠁮󠁧󠁿

168,898 просмотров • 1 год назад

This Shabbat, let's talk about a different kind of candles - how much do you know about the reason behind candle-lighting on Hannukah? Have a listen to find out why even when choosing light, you have to do it for the right reasons. Shabbat shalom! 🕯️🕯️

This Shabbat, let's talk about a different kind of candles - how much do you know about the reason behind candle-lighting on Hannukah? Have a listen to find out why even when choosing light, you have to do it for the right reasons. Shabbat shalom! 🕯️🕯️

Ambassador Yechiel (Michael) Leiter

13,612 просмотров • 9 месяцев назад

Let's talk about the thriving business of poverty. In this personal story, I want to show you how easy it is to fall below the poverty line in a struggling economy, and why it is very dangerous for the entire nation.

Let's talk about the thriving business of poverty. In this personal story, I want to show you how easy it is to fall below the poverty line in a struggling economy, and why it is very dangerous for the entire nation.

Jude Bela

65,949 просмотров • 2 лет назад

What is AI inference engineering, why is it such an in-demand skill, and how do you break into the field? With author of Inference Engineering Philip Kiely and head of training at Baseten Charlie O'Neill 0:00: What is inference? 2:47: History of inference 4:59: Downstream effects of AI research on inference 13:54: What you'll learn from Inference Engineering 16:14: Advice for engineers transitioning into AI 19:00: Open source models driving inference growth 20:55: Specialization vs. frontier closed models 23:51: "Big Token" and the importance of open source AI 27:18: Where to get Inference Engineering

What is AI inference engineering, why is it such an in-demand skill, and how do you break into the field? With author of Inference Engineering Philip Kiely and head of training at Baseten Charlie O'Neill 0:00: What is inference? 2:47: History of inference 4:59: Downstream effects of AI research on inference 13:54: What you'll learn from Inference Engineering 16:14: Advice for engineers transitioning into AI 19:00: Open source models driving inference growth 20:55: Specialization vs. frontier closed models 23:51: "Big Token" and the importance of open source AI 27:18: Where to get Inference Engineering

Madison Kanna

119,268 просмотров • 5 месяцев назад

2026: the year of explosive AI innovation. Fireworks AI Co-founder & CEO Lin Qiao speaks on how the massive expansion of customized inference and models is creating speed of light production to scale timelines.

2026: the year of explosive AI innovation. Fireworks AI Co-founder & CEO Lin Qiao speaks on how the massive expansion of customized inference and models is creating speed of light production to scale timelines.

NVIDIA

35,963 просмотров • 4 месяцев назад

Astrophysicist Neil deGrasse Tyson explains, the faster you move through space, the slower you move through time. As you approach the speed of light, time stretches and slows down until it reaches the ultimate cosmic speed limit, where time completely stops. This means that for a photon of light created at the Big Bang 13.8 billion years ago, the exact moment it was born is the exact same instant it strikes your eye; a mind-bending reality proving that for light, the entire history of the universe happens in a single, frozen moment.

Astrophysicist Neil deGrasse Tyson explains, the faster you move through space, the slower you move through time. As you approach the speed of light, time stretches and slows down until it reaches the ultimate cosmic speed limit, where time completely stops. This means that for a photon of light created at the Big Bang 13.8 billion years ago, the exact moment it was born is the exact same instant it strikes your eye; a mind-bending reality proving that for light, the entire history of the universe happens in a single, frozen moment.

Cosmos Archive

88,054 просмотров • 4 месяцев назад

🚨 YOUR GPU IS PROBABLY WASTING MORE THAN YOU THINK. vLLM is built to squeeze far more useful work out of your GPU when serving LLMs. Running an LLM at scale isn’t just about having a powerful GPU. The real problem is how efficiently you use its memory and compute. That’s where vLLM comes in. → High-throughput LLM inference and serving → PagedAttention for smarter KV-cache memory management → Continuous batching to keep GPUs busy → Prefix caching + chunked prefill → OpenAI-compatible API out of the box → Supports a huge range of modern LLM architectures → Quantization support for running models more efficiently And the crazy part? You can start an OpenAI-compatible inference server with: `vllm serve ` So your application can talk to your own model almost like it’s talking to OpenAI. The bigger idea: Don’t just buy more GPUs. Make the GPUs you already have work harder. That’s why vLLM has become such a major project in LLM inference. 🔥 #vLLM #AI #LLM #Inference #GPU #MachineLearning #AIInfrastructure #OpenSource #AIAgents #Developers

🚨 YOUR GPU IS PROBABLY WASTING MORE THAN YOU THINK. vLLM is built to squeeze far more useful work out of your GPU when serving LLMs. Running an LLM at scale isn’t just about having a powerful GPU. The real problem is how efficiently you use its memory and compute. That’s where vLLM comes in. → High-throughput LLM inference and serving → PagedAttention for smarter KV-cache memory management → Continuous batching to keep GPUs busy → Prefix caching + chunked prefill → OpenAI-compatible API out of the box → Supports a huge range of modern LLM architectures → Quantization support for running models more efficiently And the crazy part? You can start an OpenAI-compatible inference server with: `vllm serve ` So your application can talk to your own model almost like it’s talking to OpenAI. The bigger idea: Don’t just buy more GPUs. Make the GPUs you already have work harder. That’s why vLLM has become such a major project in LLM inference. 🔥 #vLLM #AI #LLM #Inference #GPU #MachineLearning #AIInfrastructure #OpenSource #AIAgents #Developers

Vikas gupta

14,138 просмотров • 2 дней назад

CMU PhD who built the kernels NVIDIA now ships in TensorRT-LLM explained fast attention in 68 minutes - better than $1200 GPU programming courses. pick the attention pattern -> generate a fused CUDA kernel -> drop it into vLLM/SGLang -> same GPU, way more tokens per second. That loop is why FlashInfer now powers inference at NVIDIA, vLLM, SGLang, and half the serving stacks you use. FlashInfer + Triton + JIT-compiled kernels + paged-KV attention - that's the stack.

CMU PhD who built the kernels NVIDIA now ships in TensorRT-LLM explained fast attention in 68 minutes - better than $1200 GPU programming courses. pick the attention pattern -> generate a fused CUDA kernel -> drop it into vLLM/SGLang -> same GPU, way more tokens per second. That loop is why FlashInfer now powers inference at NVIDIA, vLLM, SGLang, and half the serving stacks you use. FlashInfer + Triton + JIT-compiled kernels + paged-KV attention - that's the stack.

h100envy

32,605 просмотров • 2 месяцев назад