Загрузка видео...

Не удалось загрузить видео

На главную

Luminal fuses entire models into a single GPU kernel, automatically. Let's talk about why this matters for inference at the speed of light:

joefioti's profile picture

Joe Fioti

2,611 subscribers

24,326 просмотров • 5 месяцев назад •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

Reasoning LLMs generate very long chains-of-thought, so even small quantization errors add up. With AWQ, Qwen3-4B drops 71.0 → 68.2 on MMLU-Pro (~4% relative loss). 😬 ParoQuant fixes this! It keeps only the critical rotation pairs and fuses everything into a single kernel. Recovers most of the lost reasoning accuracy with minimal overhead — so 4-bit models stay strong at reasoning. 💪💪
0:27

Reasoning LLMs generate very long chains-of-thought, so even small quantization errors add up. With AWQ, Qwen3-4B drops 71.0 → 68.2 on MMLU-Pro (~4% relative loss). 😬 ParoQuant fixes this! It keeps only the critical rotation pairs and fuses everything into a single kernel. Recovers most of the lost reasoning accuracy with minimal overhead — so 4-bit models stay strong at reasoning. 💪💪

Zhijian Liu

170,883 просмотров • 3 месяцев назад

Manual QA is dead Engineers used to wait hours, sometimes days, for QA to green-light a single change. That loop is collapsing with agents. Verify in minutes, fix in minutes, and ship with confidence at the speed of inference.
0:28

Manual QA is dead Engineers used to wait hours, sometimes days, for QA to green-light a single change. That loop is collapsing with agents. Verify in minutes, fix in minutes, and ship with confidence at the speed of inference.

hayden

19,627 просмотров • 1 месяц назад

Why does NVIDIA spend millions of GPU hours training models just to give it all away? Here's a recording to the talk at NeurIPS for anyone who couldn't make it, wish we recorded the questions afterwards!
6:47

Why does NVIDIA spend millions of GPU hours training models just to give it all away? Here's a recording to the talk at NeurIPS for anyone who couldn't make it, wish we recorded the questions afterwards!

Nader Khalil🍊

126,208 просмотров • 6 месяцев назад

I didn't hear a single Democrat talk about this while it was happening in front of the entire world.
0:45

I didn't hear a single Democrat talk about this while it was happening in front of the entire world.

aka

113,705 просмотров • 5 месяцев назад

Is 1500 transactions per second on the XRP Ledger suitable for global finance? Let's talk about how it works - speed matters, but efficiency and reliability define its future.
0:51

Is 1500 transactions per second on the XRP Ledger suitable for global finance? Let's talk about how it works - speed matters, but efficiency and reliability define its future.

Jake Claver, QFOP

29,066 просмотров • 8 месяцев назад

$500b wiped out because of a single tweet? Private market valuations of Anthropic, OpenAI, SpaceXai crashed yesterday...let's talk about why. (Please don't make me explain the rug pull joke at the beginning 🤦‍♂️)
15:13

$500b wiped out because of a single tweet? Private market valuations of Anthropic, OpenAI, SpaceXai crashed yesterday...let's talk about why. (Please don't make me explain the rug pull joke at the beginning 🤦‍♂️)

Matthew Berman

142,921 просмотров • 1 месяц назад

Let's talk about the "feminist conservative" women. This is one of the most controversial topics you can talk about. That's why we need to talk about it.
0:56

Let's talk about the "feminist conservative" women. This is one of the most controversial topics you can talk about. That's why we need to talk about it.

Jedediah Bila

79,090 просмотров • 1 год назад

PowerInfer - a high-speed inference engine for deploying LLMs locally. Just came across this super interesting project on speeding up inference. It's not MoE but it's a simple approach that exploits the high locality in LLM inference to design a GPU-CPU hybrid inference engine. Hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons (the majority) are computed on the CPU. This approach significantly reduces GPU memory demands and CPU-GPU data transfer. It achieves an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs on a single NVIDIA RTX 4090 GPU. It's on only 18% lower than that achieved by a top-tier server-grade A100 GPU. It also significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy. There is a lot more innovation around inference that's coming fast. Really encouraged by the study on sparse computation to enhance the computational efficiency of LLMs. It's now possible to use PowerInfer with Llama 2 and Faclon 40B. Mistral-7B support is coming soon!
0:56

PowerInfer - a high-speed inference engine for deploying LLMs locally. Just came across this super interesting project on speeding up inference. It's not MoE but it's a simple approach that exploits the high locality in LLM inference to design a GPU-CPU hybrid inference engine. Hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons (the majority) are computed on the CPU. This approach significantly reduces GPU memory demands and CPU-GPU data transfer. It achieves an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs on a single NVIDIA RTX 4090 GPU. It's on only 18% lower than that achieved by a top-tier server-grade A100 GPU. It also significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy. There is a lot more innovation around inference that's coming fast. Really encouraged by the study on sparse computation to enhance the computational efficiency of LLMs. It's now possible to use PowerInfer with Llama 2 and Faclon 40B. Mistral-7B support is coming soon!

elvis

261,583 просмотров • 2 лет назад

The Entire Financial System Will Move at the Speed of Light With Blockchain #XRP #Crypto
0:38

The Entire Financial System Will Move at the Speed of Light With Blockchain #XRP #Crypto

mickle

12,677 просмотров • 1 год назад

at the speed of light (no pun intended) My talk for Deep Tech Week: optics, cancer, diagnostics, and Precigenetics
6:20

at the speed of light (no pun intended) My talk for Deep Tech Week: optics, cancer, diagnostics, and Precigenetics

Parmita Mishra

33,942 просмотров • 1 год назад

What is AI inference engineering, why is it such an in-demand skill, and how do you break into the field? With author of Inference Engineering Philip Kiely and head of training at Baseten Charlie O'Neill 0:00: What is inference? 2:47: History of inference 4:59: Downstream effects of AI research on inference 13:54: What you'll learn from Inference Engineering 16:14: Advice for engineers transitioning into AI 19:00: Open source models driving inference growth 20:55: Specialization vs. frontier closed models 23:51: "Big Token" and the importance of open source AI 27:18: Where to get Inference Engineering
27:58

What is AI inference engineering, why is it such an in-demand skill, and how do you break into the field? With author of Inference Engineering Philip Kiely and head of training at Baseten Charlie O'Neill 0:00: What is inference? 2:47: History of inference 4:59: Downstream effects of AI research on inference 13:54: What you'll learn from Inference Engineering 16:14: Advice for engineers transitioning into AI 19:00: Open source models driving inference growth 20:55: Specialization vs. frontier closed models 23:51: "Big Token" and the importance of open source AI 27:18: Where to get Inference Engineering

Madison Kanna

119,268 просмотров • 2 месяцев назад

The Metaphysics of Slop Let's talk about why slop is bad.
15:57

The Metaphysics of Slop Let's talk about why slop is bad.

Carl Benjamin 🏴󠁧󠁢󠁥󠁮󠁧󠁿

168,898 просмотров • 1 год назад

This Shabbat, let's talk about a different kind of candles - how much do you know about the reason behind candle-lighting on Hannukah? Have a listen to find out why even when choosing light, you have to do it for the right reasons. Shabbat shalom! 🕯️🕯️
1:03

This Shabbat, let's talk about a different kind of candles - how much do you know about the reason behind candle-lighting on Hannukah? Have a listen to find out why even when choosing light, you have to do it for the right reasons. Shabbat shalom! 🕯️🕯️

Ambassador Yechiel (Michael) Leiter

13,598 просмотров • 6 месяцев назад

2026: the year of explosive AI innovation. Fireworks AI Co-founder & CEO Lin Qiao speaks on how the massive expansion of customized inference and models is creating speed of light production to scale timelines.
0:20

2026: the year of explosive AI innovation. Fireworks AI Co-founder & CEO Lin Qiao speaks on how the massive expansion of customized inference and models is creating speed of light production to scale timelines.

NVIDIA

35,716 просмотров • 1 месяц назад

Let's talk about the thriving business of poverty. In this personal story, I want to show you how easy it is to fall below the poverty line in a struggling economy, and why it is very dangerous for the entire nation.
8:02

Let's talk about the thriving business of poverty. In this personal story, I want to show you how easy it is to fall below the poverty line in a struggling economy, and why it is very dangerous for the entire nation.

Jude Bela

65,827 просмотров • 1 год назад

Astrophysicist Neil deGrasse Tyson explains, the faster you move through space, the slower you move through time. As you approach the speed of light, time stretches and slows down until it reaches the ultimate cosmic speed limit, where time completely stops. This means that for a photon of light created at the Big Bang 13.8 billion years ago, the exact moment it was born is the exact same instant it strikes your eye; a mind-bending reality proving that for light, the entire history of the universe happens in a single, frozen moment.
0:58

Astrophysicist Neil deGrasse Tyson explains, the faster you move through space, the slower you move through time. As you approach the speed of light, time stretches and slows down until it reaches the ultimate cosmic speed limit, where time completely stops. This means that for a photon of light created at the Big Bang 13.8 billion years ago, the exact moment it was born is the exact same instant it strikes your eye; a mind-bending reality proving that for light, the entire history of the universe happens in a single, frozen moment.

Cosmos Archive

87,906 просмотров • 2 месяцев назад

Young people have so much on the line this election. That’s why I was so grateful to talk with the young women of Theta at MSU (at my former sorority house!) about what’s at stake and why their vote matters.
1:18

Young people have so much on the line this election. That’s why I was so grateful to talk with the young women of Theta at MSU (at my former sorority house!) about what’s at stake and why their vote matters.

Gretchen Whitmer

557,920 просмотров • 1 год назад

BBC: "You changed the Covid misinformation labels... Why do that?" Musk: "Does the BBC hold itself at all responsible for misinformation regarding masking, and side effects of vaccinations?" BBC: "Let's talk about something else." Musk: "You weren't expecting that." BBC: "Let's talk about something else."
1:16

BBC: "You changed the Covid misinformation labels... Why do that?" Musk: "Does the BBC hold itself at all responsible for misinformation regarding masking, and side effects of vaccinations?" BBC: "Let's talk about something else." Musk: "You weren't expecting that." BBC: "Let's talk about something else."

Wide Awake Media

4,135,885 просмотров • 1 год назад

Slash AI costs by 99%, run 1000+ models on a single GPU! OpenLoRA lets you run thousands of models on one GPU instead of one per model, drastically cutting costs and boosting flexibility. Excited to work with Aethir, Hyperbolic for infrastructure and for verifiability. Watch the demo:
3:18

Slash AI costs by 99%, run 1000+ models on a single GPU! OpenLoRA lets you run thousands of models on one GPU instead of one per model, drastically cutting costs and boosting flexibility. Excited to work with Aethir, Hyperbolic for infrastructure and for verifiability. Watch the demo:

OpenLedger

155,266 просмотров • 1 год назад