Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Luminal fuses entire models into a single GPU kernel, automatically. Let's talk about why this matters for inference at the speed of light:

joefioti's profile picture

Joe Fioti

2,611 subscribers

24,326 Aufrufe • vor 5 Monaten •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

Reasoning LLMs generate very long chains-of-thought, so even small quantization errors add up. With AWQ, Qwen3-4B drops 71.0 → 68.2 on MMLU-Pro (~4% relative loss). 😬 ParoQuant fixes this! It keeps only the critical rotation pairs and fuses everything into a single kernel. Recovers most of the lost reasoning accuracy with minimal overhead — so 4-bit models stay strong at reasoning. 💪💪
0:27

Reasoning LLMs generate very long chains-of-thought, so even small quantization errors add up. With AWQ, Qwen3-4B drops 71.0 → 68.2 on MMLU-Pro (~4% relative loss). 😬 ParoQuant fixes this! It keeps only the critical rotation pairs and fuses everything into a single kernel. Recovers most of the lost reasoning accuracy with minimal overhead — so 4-bit models stay strong at reasoning. 💪💪

Zhijian Liu

170,758 Aufrufe • vor 3 Monaten

Manual QA is dead Engineers used to wait hours, sometimes days, for QA to green-light a single change. That loop is collapsing with agents. Verify in minutes, fix in minutes, and ship with confidence at the speed of inference.
0:28

Manual QA is dead Engineers used to wait hours, sometimes days, for QA to green-light a single change. That loop is collapsing with agents. Verify in minutes, fix in minutes, and ship with confidence at the speed of inference.

hayden

19,627 Aufrufe • vor 1 Monat

Why does NVIDIA spend millions of GPU hours training models just to give it all away? Here's a recording to the talk at NeurIPS for anyone who couldn't make it, wish we recorded the questions afterwards!
6:47

Why does NVIDIA spend millions of GPU hours training models just to give it all away? Here's a recording to the talk at NeurIPS for anyone who couldn't make it, wish we recorded the questions afterwards!

Nader Khalil🍊

126,208 Aufrufe • vor 6 Monaten

I didn't hear a single Democrat talk about this while it was happening in front of the entire world.
0:45

I didn't hear a single Democrat talk about this while it was happening in front of the entire world.

aka

113,705 Aufrufe • vor 5 Monaten

Is 1500 transactions per second on the XRP Ledger suitable for global finance? Let's talk about how it works - speed matters, but efficiency and reliability define its future.
0:51

Is 1500 transactions per second on the XRP Ledger suitable for global finance? Let's talk about how it works - speed matters, but efficiency and reliability define its future.

Jake Claver, QFOP

29,066 Aufrufe • vor 8 Monaten

Let's talk about the "feminist conservative" women. This is one of the most controversial topics you can talk about. That's why we need to talk about it.
0:56

Let's talk about the "feminist conservative" women. This is one of the most controversial topics you can talk about. That's why we need to talk about it.

Jedediah Bila

79,090 Aufrufe • vor 1 Jahr

$500b wiped out because of a single tweet? Private market valuations of Anthropic, OpenAI, SpaceXai crashed yesterday...let's talk about why. (Please don't make me explain the rug pull joke at the beginning 🤦‍♂️)
15:13

$500b wiped out because of a single tweet? Private market valuations of Anthropic, OpenAI, SpaceXai crashed yesterday...let's talk about why. (Please don't make me explain the rug pull joke at the beginning 🤦‍♂️)

Matthew Berman

142,921 Aufrufe • vor 1 Monat

PowerInfer - a high-speed inference engine for deploying LLMs locally. Just came across this super interesting project on speeding up inference. It's not MoE but it's a simple approach that exploits the high locality in LLM inference to design a GPU-CPU hybrid inference engine. Hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons (the majority) are computed on the CPU. This approach significantly reduces GPU memory demands and CPU-GPU data transfer. It achieves an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs on a single NVIDIA RTX 4090 GPU. It's on only 18% lower than that achieved by a top-tier server-grade A100 GPU. It also significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy. There is a lot more innovation around inference that's coming fast. Really encouraged by the study on sparse computation to enhance the computational efficiency of LLMs. It's now possible to use PowerInfer with Llama 2 and Faclon 40B. Mistral-7B support is coming soon!
0:56

PowerInfer - a high-speed inference engine for deploying LLMs locally. Just came across this super interesting project on speeding up inference. It's not MoE but it's a simple approach that exploits the high locality in LLM inference to design a GPU-CPU hybrid inference engine. Hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons (the majority) are computed on the CPU. This approach significantly reduces GPU memory demands and CPU-GPU data transfer. It achieves an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs on a single NVIDIA RTX 4090 GPU. It's on only 18% lower than that achieved by a top-tier server-grade A100 GPU. It also significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy. There is a lot more innovation around inference that's coming fast. Really encouraged by the study on sparse computation to enhance the computational efficiency of LLMs. It's now possible to use PowerInfer with Llama 2 and Faclon 40B. Mistral-7B support is coming soon!

elvis

261,583 Aufrufe • vor 2 Jahren

The Entire Financial System Will Move at the Speed of Light With Blockchain #XRP #Crypto
0:38

The Entire Financial System Will Move at the Speed of Light With Blockchain #XRP #Crypto

mickle

12,677 Aufrufe • vor 1 Jahr

at the speed of light (no pun intended) My talk for Deep Tech Week: optics, cancer, diagnostics, and Precigenetics
6:20

at the speed of light (no pun intended) My talk for Deep Tech Week: optics, cancer, diagnostics, and Precigenetics

Parmita Mishra

33,942 Aufrufe • vor 1 Jahr

What is AI inference engineering, why is it such an in-demand skill, and how do you break into the field? With author of Inference Engineering Philip Kiely and head of training at Baseten Charlie O'Neill 0:00: What is inference? 2:47: History of inference 4:59: Downstream effects of AI research on inference 13:54: What you'll learn from Inference Engineering 16:14: Advice for engineers transitioning into AI 19:00: Open source models driving inference growth 20:55: Specialization vs. frontier closed models 23:51: "Big Token" and the importance of open source AI 27:18: Where to get Inference Engineering
27:58

What is AI inference engineering, why is it such an in-demand skill, and how do you break into the field? With author of Inference Engineering Philip Kiely and head of training at Baseten Charlie O'Neill 0:00: What is inference? 2:47: History of inference 4:59: Downstream effects of AI research on inference 13:54: What you'll learn from Inference Engineering 16:14: Advice for engineers transitioning into AI 19:00: Open source models driving inference growth 20:55: Specialization vs. frontier closed models 23:51: "Big Token" and the importance of open source AI 27:18: Where to get Inference Engineering

Madison Kanna

119,268 Aufrufe • vor 2 Monaten

The Metaphysics of Slop Let's talk about why slop is bad.
15:57

The Metaphysics of Slop Let's talk about why slop is bad.

Carl Benjamin 🏴󠁧󠁢󠁥󠁮󠁧󠁿

168,898 Aufrufe • vor 1 Jahr

This Shabbat, let's talk about a different kind of candles - how much do you know about the reason behind candle-lighting on Hannukah? Have a listen to find out why even when choosing light, you have to do it for the right reasons. Shabbat shalom! 🕯️🕯️
1:03

This Shabbat, let's talk about a different kind of candles - how much do you know about the reason behind candle-lighting on Hannukah? Have a listen to find out why even when choosing light, you have to do it for the right reasons. Shabbat shalom! 🕯️🕯️

Ambassador Yechiel (Michael) Leiter

13,598 Aufrufe • vor 6 Monaten

2026: the year of explosive AI innovation. Fireworks AI Co-founder & CEO Lin Qiao speaks on how the massive expansion of customized inference and models is creating speed of light production to scale timelines.
0:20

2026: the year of explosive AI innovation. Fireworks AI Co-founder & CEO Lin Qiao speaks on how the massive expansion of customized inference and models is creating speed of light production to scale timelines.

NVIDIA

35,716 Aufrufe • vor 1 Monat

Let's talk about the thriving business of poverty. In this personal story, I want to show you how easy it is to fall below the poverty line in a struggling economy, and why it is very dangerous for the entire nation.
8:02

Let's talk about the thriving business of poverty. In this personal story, I want to show you how easy it is to fall below the poverty line in a struggling economy, and why it is very dangerous for the entire nation.

Jude Bela

65,827 Aufrufe • vor 1 Jahr

Astrophysicist Neil deGrasse Tyson explains, the faster you move through space, the slower you move through time. As you approach the speed of light, time stretches and slows down until it reaches the ultimate cosmic speed limit, where time completely stops. This means that for a photon of light created at the Big Bang 13.8 billion years ago, the exact moment it was born is the exact same instant it strikes your eye; a mind-bending reality proving that for light, the entire history of the universe happens in a single, frozen moment.
0:58

Astrophysicist Neil deGrasse Tyson explains, the faster you move through space, the slower you move through time. As you approach the speed of light, time stretches and slows down until it reaches the ultimate cosmic speed limit, where time completely stops. This means that for a photon of light created at the Big Bang 13.8 billion years ago, the exact moment it was born is the exact same instant it strikes your eye; a mind-bending reality proving that for light, the entire history of the universe happens in a single, frozen moment.

Cosmos Archive

87,895 Aufrufe • vor 1 Monat

Young people have so much on the line this election. That’s why I was so grateful to talk with the young women of Theta at MSU (at my former sorority house!) about what’s at stake and why their vote matters.
1:18

Young people have so much on the line this election. That’s why I was so grateful to talk with the young women of Theta at MSU (at my former sorority house!) about what’s at stake and why their vote matters.

Gretchen Whitmer

557,920 Aufrufe • vor 1 Jahr

BBC: "You changed the Covid misinformation labels... Why do that?" Musk: "Does the BBC hold itself at all responsible for misinformation regarding masking, and side effects of vaccinations?" BBC: "Let's talk about something else." Musk: "You weren't expecting that." BBC: "Let's talk about something else."
1:16

BBC: "You changed the Covid misinformation labels... Why do that?" Musk: "Does the BBC hold itself at all responsible for misinformation regarding masking, and side effects of vaccinations?" BBC: "Let's talk about something else." Musk: "You weren't expecting that." BBC: "Let's talk about something else."

Wide Awake Media

4,135,885 Aufrufe • vor 1 Jahr

Slash AI costs by 99%, run 1000+ models on a single GPU! OpenLoRA lets you run thousands of models on one GPU instead of one per model, drastically cutting costs and boosting flexibility. Excited to work with Aethir, Hyperbolic for infrastructure and for verifiability. Watch the demo:
3:18

Slash AI costs by 99%, run 1000+ models on a single GPU! OpenLoRA lets you run thousands of models on one GPU instead of one per model, drastically cutting costs and boosting flexibility. Excited to work with Aethir, Hyperbolic for infrastructure and for verifiability. Watch the demo:

OpenLedger

155,266 Aufrufe • vor 1 Jahr