Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

The Inference Engineering Masterclass: 10x faster models, quantization, speculative decoding, Rubin, & self-optimizing AI Baseten Philip Kiely and ali explain what actually happens after a model is trained, why turning weights into a fast and reliable product creates an entirely new optimization problem, how quantization errors can cancel out... show more

Latent.Space

33,462 subscribers

149,249 views • 3 months ago •via X (Twitter)

Science & Technology Education News & Politics

Anya Rossi• Live Now

Private livecam show

3 Comments

John3 months ago

speculative decoding was the one that surprised me most in practice. the draft model rejection rate changes a lot based on prompt style, so batching similar prompt types together before running speculative decode gave a much bigger throughput lift than just tuning the draft model itself

Aayaan Naqvi3 months ago

@baseten @philipkiely @waterloo_intern woahhhhh @waterloo_intern sick

Jong Hyun Park3 months ago

@baseten @philipkiely @waterloo_intern Is a durable moat possible in inference tech on a multi year timescale?

Related Videos

What is AI inference engineering, why is it such an in-demand skill, and how do you break into the field? With author of Inference Engineering Philip Kiely and head of training at Baseten Charlie O'Neill 0:00: What is inference? 2:47: History of inference 4:59: Downstream effects of AI research on inference 13:54: What you'll learn from Inference Engineering 16:14: Advice for engineers transitioning into AI 19:00: Open source models driving inference growth 20:55: Specialization vs. frontier closed models 23:51: "Big Token" and the importance of open source AI 27:18: Where to get Inference Engineering

What is AI inference engineering, why is it such an in-demand skill, and how do you break into the field? With author of Inference Engineering Philip Kiely and head of training at Baseten Charlie O'Neill 0:00: What is inference? 2:47: History of inference 4:59: Downstream effects of AI research on inference 13:54: What you'll learn from Inference Engineering 16:14: Advice for engineers transitioning into AI 19:00: Open source models driving inference growth 20:55: Specialization vs. frontier closed models 23:51: "Big Token" and the importance of open source AI 27:18: Where to get Inference Engineering

Madison Kanna

119,268 views • 7 months ago

Loading DeepSeek v4 Flash just reminded me how good and fast this model is and how well it runs on DGX Sparks. GLM-5.2 is really awesome, but if you want fast and concurrent inference, DS4F is still a top choice IMO. Created by DS4F in single shot The coffee guy

Loading DeepSeek v4 Flash just reminded me how good and fast this model is and how well it runs on DGX Sparks. GLM-5.2 is really awesome, but if you want fast and concurrent inference, DS4F is still a top choice IMO. Created by DS4F in single shot The coffee guy

Mia

40,899 views • 3 months ago

Today, we're introducing GLM-5.2 Fast: our GLM-5.2 Model API designed for the most demanding real-time use cases. The Fast tier delivers 2-3x higher TPS than the standard GLM-5.2 MAPI.

Today, we're introducing GLM-5.2 Fast: our GLM-5.2 Model API designed for the most demanding real-time use cases. The Fast tier delivers 2-3x higher TPS than the standard GLM-5.2 MAPI.

Baseten

115,573 views • 3 months ago

Long video generation is a systems problem. Introducing LongLive-2.0 from NVIDIA Research: an end-to-end NVFP4 training and inference system for long video generation. Low-precision deployment often relies on post-training quantization, creating a gap between how models are trained and how they run. LongLive-2.0 aligns NVFP4-aware training, distillation, and W4A4 inference, maintaining strong benchmark quality while improving speed and memory efficiency.

Long video generation is a systems problem. Introducing LongLive-2.0 from NVIDIA Research: an end-to-end NVFP4 training and inference system for long video generation. Low-precision deployment often relies on post-training quantization, creating a gap between how models are trained and how they run. LongLive-2.0 aligns NVFP4-aware training, distillation, and W4A4 inference, maintaining strong benchmark quality while improving speed and memory efficiency.

NVIDIA AI

60,715 views • 5 months ago

$NBIS cofounder Roman Chernin describes how their recent acquisitions of Eigen AI and Clarifai were all about speed, incredible talent, and acceleration: "The philosophy is very simple. We need to build so many things, and we need to move so fast, that we're always looking for people who can accelerate us. It should be exceptional talent, and/or something that has a great adoption." "Our two recent acquisitions [were] two teams that work on inference optimization. A big part of our business is how efficiently we convert GPUs into tokens. And these two teams — Eigen AI and Clarifai — one is focused on model optimization, the engine of inference. How you run specific models and all the techniques around spec decoding, quantization, and so on." "And the other is system optimization. All the routing, KV caching, and orchestration across the big cluster of compute and so on." "We have a very strong internal team working on inference. But we felt that we needed to move faster, bring more capabilities. Because the market is so fast."

$NBIS cofounder Roman Chernin describes how their recent acquisitions of Eigen AI and Clarifai were all about speed, incredible talent, and acceleration: "The philosophy is very simple. We need to build so many things, and we need to move so fast, that we're always looking for people who can accelerate us. It should be exceptional talent, and/or something that has a great adoption." "Our two recent acquisitions [were] two teams that work on inference optimization. A big part of our business is how efficiently we convert GPUs into tokens. And these two teams — Eigen AI and Clarifai — one is focused on model optimization, the engine of inference. How you run specific models and all the techniques around spec decoding, quantization, and so on." "And the other is system optimization. All the routing, KV caching, and orchestration across the big cluster of compute and so on." "We have a very strong internal team working on inference. But we felt that we needed to move faster, bring more capabilities. Because the market is so fast."

TBPN

30,676 views • 5 months ago

The bottleneck in LLM inference isn't compute. It's how fast you can move the weights. Our CTO Mathias Lechner, Mathias Lechner, joins Piotr Mazurek, Piotr Mazurek (in SF 🌉), from our inference team, to discuss what actually limits token throughput and how we're optimizing for it.

The bottleneck in LLM inference isn't compute. It's how fast you can move the weights. Our CTO Mathias Lechner, Mathias Lechner, joins Piotr Mazurek, Piotr Mazurek (in SF 🌉), from our inference team, to discuss what actually limits token throughput and how we're optimizing for it.

Liquid AI

21,393 views • 5 months ago

Sachin Katti says OpenAI had Astra optimize its own inference on NVIDIA's Rubin chip, adding another 2x in 72 hours "It's a testament to NVIDIA's hardware that out of the box we got a 3x improvement in throughput. We literally took the same code that we had optimized for Blackwell, ran it on Rubin, got 3x improvement." "Then, can we actually make Astra run agents using Astra as the base model, go optimize this inference of Astra itself on Rubin? It's a recursive loop. The model is figuring out how to optimize its own inference serving on Rubin chips, and over 72 hours the model produced another 2x improvement on the throughput." "Things that would take weeks, if not months, to optimize in the past, we now are getting models that are very capable of actually producing over a weekend." _______ Link and more from Sachin Katti's conversation:

Sachin Katti says OpenAI had Astra optimize its own inference on NVIDIA's Rubin chip, adding another 2x in 72 hours "It's a testament to NVIDIA's hardware that out of the box we got a 3x improvement in throughput. We literally took the same code that we had optimized for Blackwell, ran it on Rubin, got 3x improvement." "Then, can we actually make Astra run agents using Astra as the base model, go optimize this inference of Astra itself on Rubin? It's a recursive loop. The model is figuring out how to optimize its own inference serving on Rubin chips, and over 72 hours the model produced another 2x improvement on the throughput." "Things that would take weeks, if not months, to optimize in the past, we now are getting models that are very capable of actually producing over a weekend." _______ Link and more from Sachin Katti's conversation:

Fireside Alpha

40,013 views • 1 month ago

GLM-5.2 running on Mac Studio Clusters or RTX GPU rigs isn't how Local AI will get mass adopted. For mass adoption we need to focus on two axes: 1. Intelligence/Speed Frontier. This is being tracked on local dot ai. Aggressive co-design is needed across the entire stack to push up the frontier specifically for local hardware. Better kernels, inference tricks like spec decoding, quantizing models, model routing. 2. Usability / UX. To use Local AI today, you need to install CLIs, select a model, a quantization, a harness that works best with that model, and you need to configure your inference engine. It's not very accessible. More work needs to be done on making this seamless, so the user doesn't even know the AI is running locally.

GLM-5.2 running on Mac Studio Clusters or RTX GPU rigs isn't how Local AI will get mass adopted. For mass adoption we need to focus on two axes: 1. Intelligence/Speed Frontier. This is being tracked on local dot ai. Aggressive co-design is needed across the entire stack to push up the frontier specifically for local hardware. Better kernels, inference tricks like spec decoding, quantizing models, model routing. 2. Usability / UX. To use Local AI today, you need to install CLIs, select a model, a quantization, a harness that works best with that model, and you need to configure your inference engine. It's not very accessible. More work needs to be done on making this seamless, so the user doesn't even know the AI is running locally.

Alex Cheema

30,854 views • 4 months ago

GLM 5.2 Easy Setup with Baseten + Droid harness. 00:00 Fastest GLM 5.2 Setup Goal 00:32 Getting a Baseten API Key 01:09 Automated Droid AI Configuration 02:23 Testing the GLM 5.2 Integration 03:07 Alternative Setups and Limitations 04:14 Bonus: Setup for Other IDEs

GLM 5.2 Easy Setup with Baseten + Droid harness. 00:00 Fastest GLM 5.2 Setup Goal 00:32 Getting a Baseten API Key 01:09 Automated Droid AI Configuration 02:23 Testing the GLM 5.2 Integration 03:07 Alternative Setups and Limitations 04:14 Bonus: Setup for Other IDEs

Ray Fernando

10,580 views • 4 months ago

Understanding GPUs will change your AI Engineering journey beyond agents and LLM. Rise of LLMs and deep learning introduced dedicated GPU hardware which serves your models for training and inference. You don't need to start with C++, CUDA, and Maths, but: > Components of a GPU and roles > How a token flows through hardware > Compute vs memory, the two limits > What FLOPS means in practice > Training vs inference hardware needs > What is CUDA Kernels > Calculating GPU memory from model size > Quantization to fit bigger models > Multi-GPU and splitting large models Hugging Face has the feature of calculating FLOps and recommending GPUs for your use case. You must explore. In terms of inference engineering, GPU understanding are much helpful your platform performance and handling multiple requests. Read this article for complete breakdown of GPUs for AI Engineers doesn't require any pre-requisite.

Understanding GPUs will change your AI Engineering journey beyond agents and LLM. Rise of LLMs and deep learning introduced dedicated GPU hardware which serves your models for training and inference. You don't need to start with C++, CUDA, and Maths, but: > Components of a GPU and roles > How a token flows through hardware > Compute vs memory, the two limits > What FLOPS means in practice > Training vs inference hardware needs > What is CUDA Kernels > Calculating GPU memory from model size > Quantization to fit bigger models > Multi-GPU and splitting large models Hugging Face has the feature of calculating FLOps and recommending GPUs for your use case. You must explore. In terms of inference engineering, GPU understanding are much helpful your platform performance and handling multiple requests. Read this article for complete breakdown of GPUs for AI Engineers doesn't require any pre-requisite.

Karan🧋

33,453 views • 1 month ago

While exploring diffusion language models (DLMs), we kept asking ourselves: given speculative decoding, a lossless acceleration technique, why do we still need DLMs? 🤔 Our latest work, ORL, tries to answer this question: AR optimizes intelligence 👆, speculative decoding optimizes speed 👉, and DLMs optimize the Pareto frontier of speed and intelligence 🚀. Through co-design across the algorithms, the training and inference engine, and the kernels, ORL pushes out a new Pareto frontier of intelligence and speed. Similar performance, much faster than speculative decoding. You say lossy spec? ORL-DLM is better too! 😎 🌐 Project + live demos:

While exploring diffusion language models (DLMs), we kept asking ourselves: given speculative decoding, a lossless acceleration technique, why do we still need DLMs? 🤔 Our latest work, ORL, tries to answer this question: AR optimizes intelligence 👆, speculative decoding optimizes speed 👉, and DLMs optimize the Pareto frontier of speed and intelligence 🚀. Through co-design across the algorithms, the training and inference engine, and the kernels, ORL pushes out a new Pareto frontier of intelligence and speed. Similar performance, much faster than speculative decoding. You say lossy spec? ORL-DLM is better too! 😎 🌐 Project + live demos:

Chenfeng_X

21,292 views • 22 days ago

Chinese labs aren't just dropping model weights anymore, they're building full ecosystems. Z.ai's ZCode + GLM 5.2 = native IDE, 1M context, agentic workflows, and now dSpark speculative decoding. The open model platform war is here. 👇

Chinese labs aren't just dropping model weights anymore, they're building full ecosystems. Z.ai's ZCode + GLM 5.2 = native IDE, 1M context, agentic workflows, and now dSpark speculative decoding. The open model platform war is here. 👇

Boxmining

14,831 views • 4 months ago

🚨 YOUR GPU IS PROBABLY WASTING MORE THAN YOU THINK. vLLM is built to squeeze far more useful work out of your GPU when serving LLMs. Running an LLM at scale isn’t just about having a powerful GPU. The real problem is how efficiently you use its memory and compute. That’s where vLLM comes in. → High-throughput LLM inference and serving → PagedAttention for smarter KV-cache memory management → Continuous batching to keep GPUs busy → Prefix caching + chunked prefill → OpenAI-compatible API out of the box → Supports a huge range of modern LLM architectures → Quantization support for running models more efficiently And the crazy part? You can start an OpenAI-compatible inference server with: `vllm serve ` So your application can talk to your own model almost like it’s talking to OpenAI. The bigger idea: Don’t just buy more GPUs. Make the GPUs you already have work harder. That’s why vLLM has become such a major project in LLM inference. 🔥 #vLLM #AI #LLM #Inference #GPU #MachineLearning #AIInfrastructure #OpenSource #AIAgents #Developers

🚨 YOUR GPU IS PROBABLY WASTING MORE THAN YOU THINK. vLLM is built to squeeze far more useful work out of your GPU when serving LLMs. Running an LLM at scale isn’t just about having a powerful GPU. The real problem is how efficiently you use its memory and compute. That’s where vLLM comes in. → High-throughput LLM inference and serving → PagedAttention for smarter KV-cache memory management → Continuous batching to keep GPUs busy → Prefix caching + chunked prefill → OpenAI-compatible API out of the box → Supports a huge range of modern LLM architectures → Quantization support for running models more efficiently And the crazy part? You can start an OpenAI-compatible inference server with: `vllm serve ` So your application can talk to your own model almost like it’s talking to OpenAI. The bigger idea: Don’t just buy more GPUs. Make the GPUs you already have work harder. That’s why vLLM has become such a major project in LLM inference. 🔥 #vLLM #AI #LLM #Inference #GPU #MachineLearning #AIInfrastructure #OpenSource #AIAgents #Developers

Vikas gupta

14,138 views • 1 month ago

you can now get unlimited free GLM 5.2 for the first time in history 😳 this is bigger than ever. here's how to do it: 1 / cancel your existing subscriptions: cancel your Lovable, Cursor, and Claude Code subscriptions. you don't need them anymore 2 / sign up for an account on freebuff you can now use unlimited GPT 5.6 and deepseek as well. 3 / to get GLM 5.2: go to the earn tab, and complete the bounties. a bounty provides unlimited GLM 5.2 messages for an hour, each. this isn't going to last forever.

you can now get unlimited free GLM 5.2 for the first time in history 😳 this is bigger than ever. here's how to do it: 1 / cancel your existing subscriptions: cancel your Lovable, Cursor, and Claude Code subscriptions. you don't need them anymore 2 / sign up for an account on freebuff you can now use unlimited GPT 5.6 and deepseek as well. 3 / to get GLM 5.2: go to the earn tab, and complete the bounties. a bounty provides unlimited GLM 5.2 messages for an hour, each. this isn't going to last forever.

Victor Cheng

55,165 views • 2 months ago

JUST IN: 🇨🇳 China’s new open-source AI model, GLM-5.2 by Zhipu AI, reportedly matches Anthropic’s Claude Mythos in security bug detection. The model outperformed some Claude versions in independent cybersecurity benchmarks and can achieve Mythos-level performance. Unlike Claude Mythos, GLM-5.2 is open-source and significantly cheaper to run. The gap between US and Chinese AI models is narrowing rapidly.

JUST IN: 🇨🇳 China’s new open-source AI model, GLM-5.2 by Zhipu AI, reportedly matches Anthropic’s Claude Mythos in security bug detection. The model outperformed some Claude versions in independent cybersecurity benchmarks and can achieve Mythos-level performance. Unlike Claude Mythos, GLM-5.2 is open-source and significantly cheaper to run. The gap between US and Chinese AI models is narrowing rapidly.

CryptoTweets

63,496 views • 4 months ago

GLM-5.2 is a bigger deal than you think. > what GLM-5.2 actually scores (0:46) > "it's just a rich person's toy" (4:27) > why GLM-5.2 matters even if you never run it (7:19)

GLM-5.2 is a bigger deal than you think. > what GLM-5.2 actually scores (0:46) > "it's just a rich person's toy" (4:27) > why GLM-5.2 matters even if you never run it (7:19)

Rob Hallam

19,906 views • 4 months ago

CHINA IS CATCHING UP IN AI 🇨🇳 China’s new open-source AI model, GLM-5.2 by Zhipu AI, reportedly matches Anthropic’s Claude Mythos in security bug detection. The model outperformed some Claude versions in independent cybersecurity benchmarks and can achieve Mythos-level performance. Unlike Claude Mythos, GLM-5.2 is open-source and significantly cheaper to run. The gap between US and Chinese AI models is narrowing rapidly.

CHINA IS CATCHING UP IN AI 🇨🇳 China’s new open-source AI model, GLM-5.2 by Zhipu AI, reportedly matches Anthropic’s Claude Mythos in security bug detection. The model outperformed some Claude versions in independent cybersecurity benchmarks and can achieve Mythos-level performance. Unlike Claude Mythos, GLM-5.2 is open-source and significantly cheaper to run. The gap between US and Chinese AI models is narrowing rapidly.

CryptoGoos

71,944 views • 4 months ago

OpenAI's New Agent Stack: Superhuman Computer Use, Dots, Decisions API, UltraFast, & the AI Cloud OpenAI’s Ari Weinstein and Nikunj Handa explain why Computer Use is already becoming faster than humans at real-world tasks, how Dots gives agents their own cloud computers, why better models can now debug and recover from failures, how Decisions API went from idea to product in weeks, and why async tools, UltraFast inference, caching, compaction, and higher-level agent primitives are turning OpenAI’s API into an AI-native cloud.

OpenAI's New Agent Stack: Superhuman Computer Use, Dots, Decisions API, UltraFast, & the AI Cloud OpenAI’s Ari Weinstein and Nikunj Handa explain why Computer Use is already becoming faster than humans at real-world tasks, how Dots gives agents their own cloud computers, why better models can now debug and recover from failures, how Decisions API went from idea to product in weeks, and why async tools, UltraFast inference, caching, compaction, and higher-level agent primitives are turning OpenAI’s API into an AI-native cloud.

Latent.Space: AI News & Views

25,690 views • 1 month ago

he spent $50,000 self-hosting AI models local AI, GLM 5.2, Fable 5 and the exact setup that pays it back watch before you rent another GPU

he spent $50,000 self-hosting AI models local AI, GLM 5.2, Fable 5 and the exact setup that pays it back watch before you rent another GPU

David Ondrej

66,427 views • 4 months ago

Truly unbelievable GLM 5.2 just released and it's an open weights model you can run locally The insane part is, it's just as good as Opus 4.8 Unlimited, free super intelligence running on your desk In this video I cover how it works, and how to set up your first local model:

Truly unbelievable GLM 5.2 just released and it's an open weights model you can run locally The insane part is, it's just as good as Opus 4.8 Unlimited, free super intelligence running on your desk In this video I cover how it works, and how to set up your first local model:

Alex Finn

125,273 views • 4 months ago