Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

The bottleneck in LLM inference isn't compute. It's how fast you can move the weights. Our CTO Mathias Lechner, Mathias Lechner, joins Piotr Mazurek, Piotr Mazurek (in SF 🌉), from our inference team, to discuss what actually limits token throughput and how we're optimizing for it.

Liquid AI

28,532 subscribers

21,393 views • 3 months ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

Building a model is just the start. Post-training makes it useful. Our CTO Mathias Lechner (Mathias Lechner) sits down for a conversation with Maxime Labonne (Maxime Labonne), our head of post-training, on the pipeline that takes a base model from autocomplete to something that can reason and follow instructions.

Building a model is just the start. Post-training makes it useful. Our CTO Mathias Lechner (Mathias Lechner) sits down for a conversation with Maxime Labonne (Maxime Labonne), our head of post-training, on the pipeline that takes a base model from autocomplete to something that can reason and follow instructions.

Liquid AI

26,121 views • 4 months ago

Training LFMs at scale means solving parallelism across every layer of the architecture. And not all layers are the same. Our CTO Mathias Lechner (Mathias Lechner) sits down with Liquid's founding engineer Paul Pak (Paul Pak) to talk training infrastructure: Data, tensor, pipeline, expert, and context parallelism, and how they make context parallelism work across hybrid architectures with both attention and convolution operators.

Training LFMs at scale means solving parallelism across every layer of the architecture. And not all layers are the same. Our CTO Mathias Lechner (Mathias Lechner) sits down with Liquid's founding engineer Paul Pak (Paul Pak) to talk training infrastructure: Data, tensor, pipeline, expert, and context parallelism, and how they make context parallelism work across hybrid architectures with both attention and convolution operators.

Liquid AI

17,908 views • 3 months ago

The Inference Engineering Masterclass: 10x faster models, quantization, speculative decoding, Rubin, & self-optimizing AI Baseten Philip Kiely and ali explain what actually happens after a model is trained, why turning weights into a fast and reliable product creates an entirely new optimization problem, how quantization errors can cancel out to unlock more throughput, why inference teams are still finding 20–200% performance gains, how video generation runs into a quadratic attention wall, and how GLM-5.2 helped rewrite and optimize the GPU kernels serving GLM-5.2 itself.

The Inference Engineering Masterclass: 10x faster models, quantization, speculative decoding, Rubin, & self-optimizing AI Baseten Philip Kiely and ali explain what actually happens after a model is trained, why turning weights into a fast and reliable product creates an entirely new optimization problem, how quantization errors can cancel out to unlock more throughput, why inference teams are still finding 20–200% performance gains, how video generation runs into a quadratic attention wall, and how GLM-5.2 helped rewrite and optimize the GPU kernels serving GLM-5.2 itself.

Latent.Space

148,203 views • 1 month ago

“Nillion aims to provide privacy to the people who have none.” John Woods CTO of Nillion joins The Defiant to unpack the “blind computer”—a federated network for private storage and compute—with NIL 2.0 weaving tokenomics directly into the tech stack. From private LLM inference to genomics, this is privacy-by-design in action. Listen now👇

“Nillion aims to provide privacy to the people who have none.” John Woods CTO of Nillion joins The Defiant to unpack the “blind computer”—a federated network for private storage and compute—with NIL 2.0 weaving tokenomics directly into the tech stack. From private LLM inference to genomics, this is privacy-by-design in action. Listen now👇

The Defiant

15,601 views • 10 months ago

You can't buffer an LLM. Unlike video streaming, where you can render lower-resolution frames while data loads, an LLM requires 100% of its KV cache to generate the next token. This means your slowest chunk dictates your entire inference speed. If you are managing LLM infrastructure, optimizing for average latency isn't enough; your architecture is entirely bound by P99 performance. Samuel Shen breaks down how dropping variable-sized metadata and enforcing fixed-size chunking doubled LMCache's throughput from 2GB/s to 5GB/s. Watch the full technical breakdown here: #LMCache #KVCache #LLM #AIInfrastructure #OpenSource

You can't buffer an LLM. Unlike video streaming, where you can render lower-resolution frames while data loads, an LLM requires 100% of its KV cache to generate the next token. This means your slowest chunk dictates your entire inference speed. If you are managing LLM infrastructure, optimizing for average latency isn't enough; your architecture is entirely bound by P99 performance. Samuel Shen breaks down how dropping variable-sized metadata and enforcing fixed-size chunking doubled LMCache's throughput from 2GB/s to 5GB/s. Watch the full technical breakdown here: #LMCache #KVCache #LLM #AIInfrastructure #OpenSource

LMCache Lab

10,360 views • 2 months ago

What is AI inference engineering, why is it such an in-demand skill, and how do you break into the field? With author of Inference Engineering Philip Kiely and head of training at Baseten Charlie O'Neill 0:00: What is inference? 2:47: History of inference 4:59: Downstream effects of AI research on inference 13:54: What you'll learn from Inference Engineering 16:14: Advice for engineers transitioning into AI 19:00: Open source models driving inference growth 20:55: Specialization vs. frontier closed models 23:51: "Big Token" and the importance of open source AI 27:18: Where to get Inference Engineering

What is AI inference engineering, why is it such an in-demand skill, and how do you break into the field? With author of Inference Engineering Philip Kiely and head of training at Baseten Charlie O'Neill 0:00: What is inference? 2:47: History of inference 4:59: Downstream effects of AI research on inference 13:54: What you'll learn from Inference Engineering 16:14: Advice for engineers transitioning into AI 19:00: Open source models driving inference growth 20:55: Specialization vs. frontier closed models 23:51: "Big Token" and the importance of open source AI 27:18: Where to get Inference Engineering

Madison Kanna

119,268 views • 6 months ago

An AI analyzed the scan and produced an answer. But can you prove what it actually saw? In high-stakes systems, trust requires verification. Verifiable inference changes how AI is trusted.

An AI analyzed the scan and produced an answer. But can you prove what it actually saw? In high-stakes systems, trust requires verification. Verifiable inference changes how AI is trusted.

OpenGradient (∇, ∇)

191,263 views • 7 months ago

Introducing the Cerebras Ambassador Program 🌏 We're looking for AI community builders to build your local ecosystem around the world's fastest inference. In the last few months, our community has hosted Cafe Compute Meetups in Berlin, Mexico City, Mumbai, SF, Seattle, and NYC. We want your city to be next! Application link in comments.

Introducing the Cerebras Ambassador Program 🌏 We're looking for AI community builders to build your local ecosystem around the world's fastest inference. In the last few months, our community has hosted Cafe Compute Meetups in Berlin, Mexico City, Mumbai, SF, Seattle, and NYC. We want your city to be next! Application link in comments.

Cerebras

102,053 views • 2 months ago

Andrew Feldman, co-founder and CEO of Cerebras gives the best explanation of why Cerebras' wafer-scale architecture is 2,500X faster than a GPU during LLM inference. During inference, there are 2 stages: - pre-fill, where the model first processes the user's prompt, and - decode, where it generates the answer 1 token at a time in sequence. During that sequencial Decode phase, before each token is calculated the model weights have to be moved from memory into compute. On a GPU those weights are moved from HBM, while Cerebras keeps them in much faster SRAM spread across its very large wafer-scale processor, so the memory-to-compute movement that must happen for every token is about 2,500× faster ---- From The MAD Podcast with Matt Turck and Cerebras YouTube channel, (full video link in comment)

Andrew Feldman, co-founder and CEO of Cerebras gives the best explanation of why Cerebras' wafer-scale architecture is 2,500X faster than a GPU during LLM inference. During inference, there are 2 stages: - pre-fill, where the model first processes the user's prompt, and - decode, where it generates the answer 1 token at a time in sequence. During that sequencial Decode phase, before each token is calculated the model weights have to be moved from memory into compute. On a GPU those weights are moved from HBM, while Cerebras keeps them in much faster SRAM spread across its very large wafer-scale processor, so the memory-to-compute movement that must happen for every token is about 2,500× faster ---- From The MAD Podcast with Matt Turck and Cerebras YouTube channel, (full video link in comment)

Rohan Paul

39,999 views • 26 days ago

🚨 YOUR GPU IS PROBABLY WASTING MORE THAN YOU THINK. vLLM is built to squeeze far more useful work out of your GPU when serving LLMs. Running an LLM at scale isn’t just about having a powerful GPU. The real problem is how efficiently you use its memory and compute. That’s where vLLM comes in. → High-throughput LLM inference and serving → PagedAttention for smarter KV-cache memory management → Continuous batching to keep GPUs busy → Prefix caching + chunked prefill → OpenAI-compatible API out of the box → Supports a huge range of modern LLM architectures → Quantization support for running models more efficiently And the crazy part? You can start an OpenAI-compatible inference server with: `vllm serve ` So your application can talk to your own model almost like it’s talking to OpenAI. The bigger idea: Don’t just buy more GPUs. Make the GPUs you already have work harder. That’s why vLLM has become such a major project in LLM inference. 🔥 #vLLM #AI #LLM #Inference #GPU #MachineLearning #AIInfrastructure #OpenSource #AIAgents #Developers

🚨 YOUR GPU IS PROBABLY WASTING MORE THAN YOU THINK. vLLM is built to squeeze far more useful work out of your GPU when serving LLMs. Running an LLM at scale isn’t just about having a powerful GPU. The real problem is how efficiently you use its memory and compute. That’s where vLLM comes in. → High-throughput LLM inference and serving → PagedAttention for smarter KV-cache memory management → Continuous batching to keep GPUs busy → Prefix caching + chunked prefill → OpenAI-compatible API out of the box → Supports a huge range of modern LLM architectures → Quantization support for running models more efficiently And the crazy part? You can start an OpenAI-compatible inference server with: `vllm serve ` So your application can talk to your own model almost like it’s talking to OpenAI. The bigger idea: Don’t just buy more GPUs. Make the GPUs you already have work harder. That’s why vLLM has become such a major project in LLM inference. 🔥 #vLLM #AI #LLM #Inference #GPU #MachineLearning #AIInfrastructure #OpenSource #AIAgents #Developers

Vikas gupta

14,138 views • 12 days ago

The real bottleneck in AI’s next leap isn’t compute, it’s the human data it can actually learn from. Our co-founder Manish | HumynLabs; KGeN addresses our community on why KGeN has the moat to solve it. Sneak peek below 👀

The real bottleneck in AI’s next leap isn’t compute, it’s the human data it can actually learn from. Our co-founder Manish | HumynLabs; KGeN addresses our community on why KGeN has the moat to solve it. Sneak peek below 👀

KGeN Foundation 🟩

68,990 views • 1 month ago

$NBIS cofounder Roman Chernin describes how their recent acquisitions of Eigen AI and Clarifai were all about speed, incredible talent, and acceleration: "The philosophy is very simple. We need to build so many things, and we need to move so fast, that we're always looking for people who can accelerate us. It should be exceptional talent, and/or something that has a great adoption." "Our two recent acquisitions [were] two teams that work on inference optimization. A big part of our business is how efficiently we convert GPUs into tokens. And these two teams — Eigen AI and Clarifai — one is focused on model optimization, the engine of inference. How you run specific models and all the techniques around spec decoding, quantization, and so on." "And the other is system optimization. All the routing, KV caching, and orchestration across the big cluster of compute and so on." "We have a very strong internal team working on inference. But we felt that we needed to move faster, bring more capabilities. Because the market is so fast."

$NBIS cofounder Roman Chernin describes how their recent acquisitions of Eigen AI and Clarifai were all about speed, incredible talent, and acceleration: "The philosophy is very simple. We need to build so many things, and we need to move so fast, that we're always looking for people who can accelerate us. It should be exceptional talent, and/or something that has a great adoption." "Our two recent acquisitions [were] two teams that work on inference optimization. A big part of our business is how efficiently we convert GPUs into tokens. And these two teams — Eigen AI and Clarifai — one is focused on model optimization, the engine of inference. How you run specific models and all the techniques around spec decoding, quantization, and so on." "And the other is system optimization. All the routing, KV caching, and orchestration across the big cluster of compute and so on." "We have a very strong internal team working on inference. But we felt that we needed to move faster, bring more capabilities. Because the market is so fast."

TBPN

30,608 views • 4 months ago

I sat down with Silas Alberti, Head of Research Cognition, to chat about the intersection of RL and inference -- from training frontier coding models to running inference at scale. Watch till the end for bonus doggo! :o 0:00 — Intros 1:04 — What's hardest to get right in an RL run 3:39 — How Cognition started using Modal 4:03 — What to weigh when buying inference 4:58 — The latency/throughput Pareto frontier 7:14 — DFlash reveal (!!) 8:44 — Why faster runtime matters 9:56 — Memory-bound vs. compute-bound 10:57 — Tree-based speculation 14:20 — Hot take: RL and inference optimization converging 15:28 — Online training of the speculator 16:22 — "Auto inference" 17:27 — Agents competing on optimization problems 18:27 — What inference providers still get wrong

I sat down with Silas Alberti, Head of Research Cognition, to chat about the intersection of RL and inference -- from training frontier coding models to running inference at scale. Watch till the end for bonus doggo! :o 0:00 — Intros 1:04 — What's hardest to get right in an RL run 3:39 — How Cognition started using Modal 4:03 — What to weigh when buying inference 4:58 — The latency/throughput Pareto frontier 7:14 — DFlash reveal (!!) 8:44 — Why faster runtime matters 9:56 — Memory-bound vs. compute-bound 10:57 — Tree-based speculation 14:20 — Hot take: RL and inference optimization converging 15:28 — Online training of the speculator 16:22 — "Auto inference" 17:27 — Agents competing on optimization problems 18:27 — What inference providers still get wrong

gongy

81,494 views • 1 month ago

"You could imagine yourself actually being 10x more productive." Nolan Dey from the Cerebras Core ML team speaks with Alycia Cary on why speed contributes to intelligence. In agentic workloads like Claude Code, queries can run for hours. Faster inference means more test time compute in the same budget.

"You could imagine yourself actually being 10x more productive." Nolan Dey from the Cerebras Core ML team speaks with Alycia Cary on why speed contributes to intelligence. In agentic workloads like Claude Code, queries can run for hours. Faster inference means more test time compute in the same budget.

Cerebras

132,844 views • 1 month ago

I'd like to welcome you to Flagship! In this video, I'll walk you through what we're exactly building, our philosophy, how the $FYI token and Agents work and more. Anything that's missing for you?

I'd like to welcome you to Flagship! In this video, I'll walk you through what we're exactly building, our philosophy, how the $FYI token and Agents work and more. Anything that's missing for you?

Jorn

286,638 views • 1 year ago

"100 million words context window is already possible, which is roughly what a human hears in a lifetime. Inference support is the only bottleneck to achieve it. And AI Models actually do learn during the context window, without changing the weights." ~ Anthropic CEO Dario Amodei (On the 2nd point, there was this brilliant Google Paper published last week that says, LLMs can learn in context from examples in the prompt, can pick up new patterns while answering, yet their stored weights never change.) --- From 'Alex Kantrowitz' YT Channel (Full Video link in comment)

"100 million words context window is already possible, which is roughly what a human hears in a lifetime. Inference support is the only bottleneck to achieve it. And AI Models actually do learn during the context window, without changing the weights." ~ Anthropic CEO Dario Amodei (On the 2nd point, there was this brilliant Google Paper published last week that says, LLMs can learn in context from examples in the prompt, can pick up new patterns while answering, yet their stored weights never change.) --- From 'Alex Kantrowitz' YT Channel (Full Video link in comment)

Rohan Paul

631,729 views • 11 months ago

Sachin Katti says OpenAI had Astra optimize its own inference on NVIDIA's Rubin chip, adding another 2x in 72 hours "It's a testament to NVIDIA's hardware that out of the box we got a 3x improvement in throughput. We literally took the same code that we had optimized for Blackwell, ran it on Rubin, got 3x improvement." "Then, can we actually make Astra run agents using Astra as the base model, go optimize this inference of Astra itself on Rubin? It's a recursive loop. The model is figuring out how to optimize its own inference serving on Rubin chips, and over 72 hours the model produced another 2x improvement on the throughput." "Things that would take weeks, if not months, to optimize in the past, we now are getting models that are very capable of actually producing over a weekend." _______ Link and more from Sachin Katti's conversation:

Sachin Katti says OpenAI had Astra optimize its own inference on NVIDIA's Rubin chip, adding another 2x in 72 hours "It's a testament to NVIDIA's hardware that out of the box we got a 3x improvement in throughput. We literally took the same code that we had optimized for Blackwell, ran it on Rubin, got 3x improvement." "Then, can we actually make Astra run agents using Astra as the base model, go optimize this inference of Astra itself on Rubin? It's a recursive loop. The model is figuring out how to optimize its own inference serving on Rubin chips, and over 72 hours the model produced another 2x improvement on the throughput." "Things that would take weeks, if not months, to optimize in the past, we now are getting models that are very capable of actually producing over a weekend." _______ Link and more from Sachin Katti's conversation:

Fireside Alpha

40,013 views • 6 days ago

✨ Inferium 2024 Wrapped! ✨ This year, we shattered boundaries, innovated at lightning speed, and revolutionized the way you interact with AI inference 🤖🌌 📊 Milestones, magic, and momentum – all wrapped up just for you. 🎥 Watch our 2024 Wrapped to see how we’ve transformed #inference forever👇 Here’s to shaping the future in 2025!

✨ Inferium 2024 Wrapped! ✨ This year, we shattered boundaries, innovated at lightning speed, and revolutionized the way you interact with AI inference 🤖🌌 📊 Milestones, magic, and momentum – all wrapped up just for you. 🎥 Watch our 2024 Wrapped to see how we’ve transformed #inference forever👇 Here’s to shaping the future in 2025!

Inferium

26,258 views • 1 year ago

AI performance isn't just about compute. It's about how fast, and how far, data can move. Marvell Technology CTO Chris Koopmans joined Patrick Moorhead and Daniel Newman to dig into why scaling AI means rethinking connectivity at every layer. From millimeters to thousands of kilometers, no single technology wins. Copper, optics, and new interconnect innovations all matter because AI systems live or die by how efficiently bits move. Watch the full discussion here:

AI performance isn't just about compute. It's about how fast, and how far, data can move. Marvell Technology CTO Chris Koopmans joined Patrick Moorhead and Daniel Newman to dig into why scaling AI means rethinking connectivity at every layer. From millimeters to thousands of kilometers, no single technology wins. Copper, optics, and new interconnect innovations all matter because AI systems live or die by how efficiently bits move. Watch the full discussion here:

Six Five Media

67,333 views • 7 months ago

🚨New Feature Alert🚨: Our Geometry Tool now includes vectors you can compute with in the expression list! How will you use our new vector tool? (Bonus: What do you notice about the vector tokens in the expression list as we move the vector around?)

🚨New Feature Alert🚨: Our Geometry Tool now includes vectors you can compute with in the expression list! How will you use our new vector tool? (Bonus: What do you notice about the vector tokens in the expression list as we move the vector around?)

Desmos

30,529 views • 3 years ago