Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

The Inference Engineering Masterclass: 10x faster models, quantization, speculative decoding, Rubin, & self-optimizing AI Baseten Philip Kiely and ali explain what actually happens after a model is trained, why turning weights into a fast and reliable product creates an entirely new optimization problem, how quantization errors can cancel out... show more

Latent.Space

31,983 subscribers

148,203 Aufrufe • vor 1 Monat •via X (Twitter)

Wissenschaft & Technologie Bildung Nachrichten & Politik

Anya Rossi• Live Now

Private livecam show

3 Kommentare

Profilbild von John

Johnvor 1 Monat

speculative decoding was the one that surprised me most in practice. the draft model rejection rate changes a lot based on prompt style, so batching similar prompt types together before running speculative decode gave a much bigger throughput lift than just tuning the draft model itself

Profilbild von Aayaan Naqvi

Aayaan Naqvivor 1 Monat

@baseten @philipkiely @waterloo_intern woahhhhh @waterloo_intern sick

Profilbild von Jong Hyun Park

Jong Hyun Parkvor 1 Monat

@baseten @philipkiely @waterloo_intern Is a durable moat possible in inference tech on a multi year timescale?

Ähnliche Videos

What is AI inference engineering, why is it such an in-demand skill, and how do you break into the field? With author of Inference Engineering Philip Kiely and head of training at Baseten Charlie O'Neill 0:00: What is inference? 2:47: History of inference 4:59: Downstream effects of AI research on inference 13:54: What you'll learn from Inference Engineering 16:14: Advice for engineers transitioning into AI 19:00: Open source models driving inference growth 20:55: Specialization vs. frontier closed models 23:51: "Big Token" and the importance of open source AI 27:18: Where to get Inference Engineering

What is AI inference engineering, why is it such an in-demand skill, and how do you break into the field? With author of Inference Engineering Philip Kiely and head of training at Baseten Charlie O'Neill 0:00: What is inference? 2:47: History of inference 4:59: Downstream effects of AI research on inference 13:54: What you'll learn from Inference Engineering 16:14: Advice for engineers transitioning into AI 19:00: Open source models driving inference growth 20:55: Specialization vs. frontier closed models 23:51: "Big Token" and the importance of open source AI 27:18: Where to get Inference Engineering

Madison Kanna

119,268 Aufrufe • vor 5 Monaten

$NBIS cofounder Roman Chernin describes how their recent acquisitions of Eigen AI and Clarifai were all about speed, incredible talent, and acceleration: "The philosophy is very simple. We need to build so many things, and we need to move so fast, that we're always looking for people who can accelerate us. It should be exceptional talent, and/or something that has a great adoption." "Our two recent acquisitions [were] two teams that work on inference optimization. A big part of our business is how efficiently we convert GPUs into tokens. And these two teams — Eigen AI and Clarifai — one is focused on model optimization, the engine of inference. How you run specific models and all the techniques around spec decoding, quantization, and so on." "And the other is system optimization. All the routing, KV caching, and orchestration across the big cluster of compute and so on." "We have a very strong internal team working on inference. But we felt that we needed to move faster, bring more capabilities. Because the market is so fast."

$NBIS cofounder Roman Chernin describes how their recent acquisitions of Eigen AI and Clarifai were all about speed, incredible talent, and acceleration: "The philosophy is very simple. We need to build so many things, and we need to move so fast, that we're always looking for people who can accelerate us. It should be exceptional talent, and/or something that has a great adoption." "Our two recent acquisitions [were] two teams that work on inference optimization. A big part of our business is how efficiently we convert GPUs into tokens. And these two teams — Eigen AI and Clarifai — one is focused on model optimization, the engine of inference. How you run specific models and all the techniques around spec decoding, quantization, and so on." "And the other is system optimization. All the routing, KV caching, and orchestration across the big cluster of compute and so on." "We have a very strong internal team working on inference. But we felt that we needed to move faster, bring more capabilities. Because the market is so fast."

TBPN

30,608 Aufrufe • vor 4 Monaten

Sachin Katti says OpenAI had Astra optimize its own inference on NVIDIA's Rubin chip, adding another 2x in 72 hours "It's a testament to NVIDIA's hardware that out of the box we got a 3x improvement in throughput. We literally took the same code that we had optimized for Blackwell, ran it on Rubin, got 3x improvement." "Then, can we actually make Astra run agents using Astra as the base model, go optimize this inference of Astra itself on Rubin? It's a recursive loop. The model is figuring out how to optimize its own inference serving on Rubin chips, and over 72 hours the model produced another 2x improvement on the throughput." "Things that would take weeks, if not months, to optimize in the past, we now are getting models that are very capable of actually producing over a weekend." _______ Link and more from Sachin Katti's conversation:

Sachin Katti says OpenAI had Astra optimize its own inference on NVIDIA's Rubin chip, adding another 2x in 72 hours "It's a testament to NVIDIA's hardware that out of the box we got a 3x improvement in throughput. We literally took the same code that we had optimized for Blackwell, ran it on Rubin, got 3x improvement." "Then, can we actually make Astra run agents using Astra as the base model, go optimize this inference of Astra itself on Rubin? It's a recursive loop. The model is figuring out how to optimize its own inference serving on Rubin chips, and over 72 hours the model produced another 2x improvement on the throughput." "Things that would take weeks, if not months, to optimize in the past, we now are getting models that are very capable of actually producing over a weekend." _______ Link and more from Sachin Katti's conversation:

Fireside Alpha

40,013 Aufrufe • vor 4 Tagen

GLM-5.2 running on Mac Studio Clusters or RTX GPU rigs isn't how Local AI will get mass adopted. For mass adoption we need to focus on two axes: 1. Intelligence/Speed Frontier. This is being tracked on local dot ai. Aggressive co-design is needed across the entire stack to push up the frontier specifically for local hardware. Better kernels, inference tricks like spec decoding, quantizing models, model routing. 2. Usability / UX. To use Local AI today, you need to install CLIs, select a model, a quantization, a harness that works best with that model, and you need to configure your inference engine. It's not very accessible. More work needs to be done on making this seamless, so the user doesn't even know the AI is running locally.

GLM-5.2 running on Mac Studio Clusters or RTX GPU rigs isn't how Local AI will get mass adopted. For mass adoption we need to focus on two axes: 1. Intelligence/Speed Frontier. This is being tracked on local dot ai. Aggressive co-design is needed across the entire stack to push up the frontier specifically for local hardware. Better kernels, inference tricks like spec decoding, quantizing models, model routing. 2. Usability / UX. To use Local AI today, you need to install CLIs, select a model, a quantization, a harness that works best with that model, and you need to configure your inference engine. It's not very accessible. More work needs to be done on making this seamless, so the user doesn't even know the AI is running locally.

Alex Cheema

30,854 Aufrufe • vor 2 Monaten

🚨 YOUR GPU IS PROBABLY WASTING MORE THAN YOU THINK. vLLM is built to squeeze far more useful work out of your GPU when serving LLMs. Running an LLM at scale isn’t just about having a powerful GPU. The real problem is how efficiently you use its memory and compute. That’s where vLLM comes in. → High-throughput LLM inference and serving → PagedAttention for smarter KV-cache memory management → Continuous batching to keep GPUs busy → Prefix caching + chunked prefill → OpenAI-compatible API out of the box → Supports a huge range of modern LLM architectures → Quantization support for running models more efficiently And the crazy part? You can start an OpenAI-compatible inference server with: `vllm serve ` So your application can talk to your own model almost like it’s talking to OpenAI. The bigger idea: Don’t just buy more GPUs. Make the GPUs you already have work harder. That’s why vLLM has become such a major project in LLM inference. 🔥 #vLLM #AI #LLM #Inference #GPU #MachineLearning #AIInfrastructure #OpenSource #AIAgents #Developers

🚨 YOUR GPU IS PROBABLY WASTING MORE THAN YOU THINK. vLLM is built to squeeze far more useful work out of your GPU when serving LLMs. Running an LLM at scale isn’t just about having a powerful GPU. The real problem is how efficiently you use its memory and compute. That’s where vLLM comes in. → High-throughput LLM inference and serving → PagedAttention for smarter KV-cache memory management → Continuous batching to keep GPUs busy → Prefix caching + chunked prefill → OpenAI-compatible API out of the box → Supports a huge range of modern LLM architectures → Quantization support for running models more efficiently And the crazy part? You can start an OpenAI-compatible inference server with: `vllm serve ` So your application can talk to your own model almost like it’s talking to OpenAI. The bigger idea: Don’t just buy more GPUs. Make the GPUs you already have work harder. That’s why vLLM has become such a major project in LLM inference. 🔥 #vLLM #AI #LLM #Inference #GPU #MachineLearning #AIInfrastructure #OpenSource #AIAgents #Developers

Vikas gupta

14,138 Aufrufe • vor 10 Tagen

World models are moving beyond offline generation towards interactive, real-time experiences. Introducing ⚡FlashDreams⚡: an open-source high-performance inference and serving library built for autoregressive world models: 🔥 Up to 3.10× faster LingBot-World inference 🔥 Up to 2.12× faster Self-Forcing inference 🔥 Up to 1.40× faster Wan2.1 inference 🔥 8 integrated models 🔥 Multi-GPU, streaming, low-latency serving 🔥 Agentic skills that teach you how to use it FlashDreams is designed for a new generation of AI systems that continuously evolve over time while responding to user interactions. It powers applications across robotics, autonomous vehicle simulation, gaming, and virtual worlds. Github: Docs: Research page: Join the #flashdreams Discord channel at FlashDreams is also the runtime backbone behind NVIDIA OmniDreams ( 1/n #AI #WorldModels #FastInference #PhysicalAI #OpenSource #NVIDIA

World models are moving beyond offline generation towards interactive, real-time experiences. Introducing ⚡FlashDreams⚡: an open-source high-performance inference and serving library built for autoregressive world models: 🔥 Up to 3.10× faster LingBot-World inference 🔥 Up to 2.12× faster Self-Forcing inference 🔥 Up to 1.40× faster Wan2.1 inference 🔥 8 integrated models 🔥 Multi-GPU, streaming, low-latency serving 🔥 Agentic skills that teach you how to use it FlashDreams is designed for a new generation of AI systems that continuously evolve over time while responding to user interactions. It powers applications across robotics, autonomous vehicle simulation, gaming, and virtual worlds. Github: Docs: Research page: Join the #flashdreams Discord channel at FlashDreams is also the runtime backbone behind NVIDIA OmniDreams ( 1/n #AI #WorldModels #FastInference #PhysicalAI #OpenSource #NVIDIA

Ruilong Li

99,521 Aufrufe • vor 3 Monaten

Interview with Nebius Co-Founder Roman Chernin Please like & share this video so that all $NBIS investors on X will see it! :) If you prefer watching on YouTube: Timestamps: 00:00 - Why AI Infrastructure Is So Hard to Understand 00:24 - Market Fragmentation and What Actually Differentiates Providers 01:30 - Consolidation, Segmentation, and the Future AI Cloud Landscape 02:56 - What Analysts and VCs Still Get Wrong About AI Infrastructure 05:34 - Nebius Cloud: Product Readiness and Customer Proof Points 07:42 - Why Inference Workloads Are Exploding 09:11 - Training vs. Inference: How AI Models Actually Reach Production 10:10 - Why Inference Market Share May Concentrate Around a Few Winners 12:36 - Customer Use Cases: Coding, Enterprise AI, and Real-World Adoption 14:01 - Why Integrated Training and Inference Matter Strategically 16:01 - Building Scalable AI Infrastructure With High Utilization 18:24 - Token Factory: Inference as a Managed Service 20:24 - Revolut Case Study: AI-Driven Product Enhancements 22:56 - Token Factory Performance Optimization and Competitive Advantage 25:07 - Scale, Capacity, and Efficiency as Growth Drivers 28:36 - Why Inference Capacity Could Become the Next Major Bottleneck 30:10 - How Nebius Benchmarks Performance Across Providers 33:14 - The Future Size and Shape of the Inference Market 36:38 - Value-Based Pricing: Moving Beyond Cost per GPU Hour 40:55 - How Nebius Wins Deals: Quality, Performance, and Customer Experience 44:53 - Autonomous AI Platforms and the Rise of Agent-Based Models 47:28 - Tavily, Agentic Applications, and the Next Layer of the AI Stack 50:45 - Strategic Trade-Offs: Scaling, Product Roadmap, and Customer Relevance 55:40 - Final Thoughts: Adapting to the Next Shift in AI Workloads Nebius Roman Chernin

Interview with Nebius Co-Founder Roman Chernin Please like & share this video so that all $NBIS investors on X will see it! :) If you prefer watching on YouTube: Timestamps: 00:00 - Why AI Infrastructure Is So Hard to Understand 00:24 - Market Fragmentation and What Actually Differentiates Providers 01:30 - Consolidation, Segmentation, and the Future AI Cloud Landscape 02:56 - What Analysts and VCs Still Get Wrong About AI Infrastructure 05:34 - Nebius Cloud: Product Readiness and Customer Proof Points 07:42 - Why Inference Workloads Are Exploding 09:11 - Training vs. Inference: How AI Models Actually Reach Production 10:10 - Why Inference Market Share May Concentrate Around a Few Winners 12:36 - Customer Use Cases: Coding, Enterprise AI, and Real-World Adoption 14:01 - Why Integrated Training and Inference Matter Strategically 16:01 - Building Scalable AI Infrastructure With High Utilization 18:24 - Token Factory: Inference as a Managed Service 20:24 - Revolut Case Study: AI-Driven Product Enhancements 22:56 - Token Factory Performance Optimization and Competitive Advantage 25:07 - Scale, Capacity, and Efficiency as Growth Drivers 28:36 - Why Inference Capacity Could Become the Next Major Bottleneck 30:10 - How Nebius Benchmarks Performance Across Providers 33:14 - The Future Size and Shape of the Inference Market 36:38 - Value-Based Pricing: Moving Beyond Cost per GPU Hour 40:55 - How Nebius Wins Deals: Quality, Performance, and Customer Experience 44:53 - Autonomous AI Platforms and the Rise of Agent-Based Models 47:28 - Tavily, Agentic Applications, and the Next Layer of the AI Stack 50:45 - Strategic Trade-Offs: Scaling, Product Roadmap, and Customer Relevance 55:40 - Final Thoughts: Adapting to the Next Shift in AI Workloads Nebius Roman Chernin

Daniel Koss

204,656 Aufrufe • vor 4 Monaten

New course: Transformers in Practice. You'll get a practical view of how transformer-based LLMs work, so you can reason about their behavior, diagnose problems like slow inference, and make smarter decisions about deployment. This course is built in partnership with AMD and taught by Sharon Zhou. You'll see how transformers generate text one token at a time, how the model decides which earlier words matter most when predicting the next one, and how techniques like quantization speed up inference on GPUs. This is not a video-only course; interactive visualizations throughout let you play with these concepts and build intuition that sticks. Skills you'll gain: - Understand why LLMs hallucinate, and RAG and chain-of-thought shape what they generate - Look inside the model to see how attention and layers combine to predict the next token - Diagnose inference bottlenecks and learn the techniques that speed up transformers on GPUs Join and understand what's really happening inside your LLMs:

New course: Transformers in Practice. You'll get a practical view of how transformer-based LLMs work, so you can reason about their behavior, diagnose problems like slow inference, and make smarter decisions about deployment. This course is built in partnership with AMD and taught by Sharon Zhou. You'll see how transformers generate text one token at a time, how the model decides which earlier words matter most when predicting the next one, and how techniques like quantization speed up inference on GPUs. This is not a video-only course; interactive visualizations throughout let you play with these concepts and build intuition that sticks. Skills you'll gain: - Understand why LLMs hallucinate, and RAG and chain-of-thought shape what they generate - Look inside the model to see how attention and layers combine to predict the next token - Diagnose inference bottlenecks and learn the techniques that speed up transformers on GPUs Join and understand what's really happening inside your LLMs:

Andrew Ng

124,677 Aufrufe • vor 4 Monaten

Serving GLM5.2 NVFP4 Agentic Workload with SGLang: How We Reached 500 TPS on 8xB300 at bs=1 In this deep dive, we explain how SGLang reaches 500+ tok/s/user at bs=1 on 8xB300, with 18 to 34% higher single-user interactivity within two weeks since day-0, and 6 to 11% better peak throughput at high concurrency, benchmarked on a real multi-turn agentic coding workload. Our new TopK-V2 kernel is 2.33x faster at 80K ISL, scaling to 10.17x at 1M ISL, keeping interactivity essentially flat out to 1M tokens. Part of the story is the architecture itself. GLM-5.2 applies IndexShare to its DSA layers and ships a stronger MTP head reusing IndexShare and KVShare. The rest comes from our serving optimizations. Special thanks to NVIDIA AI for the help in day-0 support of GLM-5.2 NVFP4, and to Z.ai for IndexShare in SGLang!

Serving GLM5.2 NVFP4 Agentic Workload with SGLang: How We Reached 500 TPS on 8xB300 at bs=1 In this deep dive, we explain how SGLang reaches 500+ tok/s/user at bs=1 on 8xB300, with 18 to 34% higher single-user interactivity within two weeks since day-0, and 6 to 11% better peak throughput at high concurrency, benchmarked on a real multi-turn agentic coding workload. Our new TopK-V2 kernel is 2.33x faster at 80K ISL, scaling to 10.17x at 1M ISL, keeping interactivity essentially flat out to 1M tokens. Part of the story is the architecture itself. GLM-5.2 applies IndexShare to its DSA layers and ships a stronger MTP head reusing IndexShare and KVShare. The rest comes from our serving optimizations. Special thanks to NVIDIA AI for the help in day-0 support of GLM-5.2 NVFP4, and to Z.ai for IndexShare in SGLang!

LMSYS Org

107,571 Aufrufe • vor 2 Monaten

Learn how to build an optimized LLM inference system from the ground up in our new short course, Efficiently Serving LLMs, built in collaboration with Predibase by Rubrik and taught by Travis Addair. Whether you're serving your own LLM or using a model hosting service, this course will give you a deep understanding of the optimizations required to efficiently serve many users at once. - Learn how LLMs generate text one token at a time, and how techniques like KV caching, continuous batching, and quantization speed things up and optimize memory usage for serving multiple users. - Benchmark the performance of these LLM optimizations to explore the trade-offs between quickly responding to an individual user’s request vs. serving many users at once. - Use techniques like low-rank adaptation (LoRA) to efficiently serve hundreds of unique, custom fine-tuned models on a single device, without sacrificing throughput. - Use Predibase's LoRAX framework to see optimization techniques in action on a real LLM server. Sign up here:

Learn how to build an optimized LLM inference system from the ground up in our new short course, Efficiently Serving LLMs, built in collaboration with Predibase by Rubrik and taught by Travis Addair. Whether you're serving your own LLM or using a model hosting service, this course will give you a deep understanding of the optimizations required to efficiently serve many users at once. - Learn how LLMs generate text one token at a time, and how techniques like KV caching, continuous batching, and quantization speed things up and optimize memory usage for serving multiple users. - Benchmark the performance of these LLM optimizations to explore the trade-offs between quickly responding to an individual user’s request vs. serving many users at once. - Use techniques like low-rank adaptation (LoRA) to efficiently serve hundreds of unique, custom fine-tuned models on a single device, without sacrificing throughput. - Use Predibase's LoRAX framework to see optimization techniques in action on a real LLM server. Sign up here:

Andrew Ng

104,727 Aufrufe • vor 2 Jahren

Structured Outputs: The Building Blocks for Reliable AI! 🏗️ I am SUPER EXCITED to publish our newest Weaviate Podcast featuring Will Kurt (Will Kurt) and Cameron Pfiffer (Cameron) from .txt! 🎙️🎉 Dottxt is the company behind Outlines, reshaping how we control LLM outputs with constrained decoding! 🚀 This podcast dives into all sorts of details from new applications in metadata and information extraction, such as self-expanding knowledge graphs, to structured reasoning, report generation, and more! We also dove into the details of how this leverages Automata Theory and Finite State Machines to achieve this, how it is packaged with inference engines such as vLLM, and a rebuttal to Let Me Speak Freely, discussing how Structured Generation actually improve the quality of LLM outputs, in addition to speeding up inference, and of course, making it more reliable! I hope you find the podcast interesting, this was a super fun one! Links below!

Structured Outputs: The Building Blocks for Reliable AI! 🏗️ I am SUPER EXCITED to publish our newest Weaviate Podcast featuring Will Kurt (Will Kurt) and Cameron Pfiffer (Cameron) from .txt! 🎙️🎉 Dottxt is the company behind Outlines, reshaping how we control LLM outputs with constrained decoding! 🚀 This podcast dives into all sorts of details from new applications in metadata and information extraction, such as self-expanding knowledge graphs, to structured reasoning, report generation, and more! We also dove into the details of how this leverages Automata Theory and Finite State Machines to achieve this, how it is packaged with inference engines such as vLLM, and a rebuttal to Let Me Speak Freely, discussing how Structured Generation actually improve the quality of LLM outputs, in addition to speeding up inference, and of course, making it more reliable! I hope you find the podcast interesting, this was a super fun one! Links below!

Connor Shorten

12,742 Aufrufe • vor 1 Jahr

Fireworks AI is now available first-party in Microsoft Foundry, as part of a multi-year partnership, starting today in private preview. Enterprise AI has a speed problem. Most infrastructure can't keep up with how fast open models are moving. New architectures, fine-tuned weights, models that aren't on anyone's approved list yet. The usual answer was "manage your own serving infrastructure." That's a real cost. Fireworks runs the inference. Azure owns the governance. Your weights stay yours. Beyond custom weights, teams can also access the latest open models on day zero. No waiting for the marketplace to catch up. Serverless inference for prototyping, and a path to production that doesn't require re-architecting anything. Fireworks handles the serving layer. Azure handles the rest. Your boss handles the congratulations. Want in →

Fireworks AI is now available first-party in Microsoft Foundry, as part of a multi-year partnership, starting today in private preview. Enterprise AI has a speed problem. Most infrastructure can't keep up with how fast open models are moving. New architectures, fine-tuned weights, models that aren't on anyone's approved list yet. The usual answer was "manage your own serving infrastructure." That's a real cost. Fireworks runs the inference. Azure owns the governance. Your weights stay yours. Beyond custom weights, teams can also access the latest open models on day zero. No waiting for the marketplace to catch up. Serverless inference for prototyping, and a path to production that doesn't require re-architecting anything. Fireworks handles the serving layer. Azure handles the rest. Your boss handles the congratulations. Want in →

Fireworks AI

51,657 Aufrufe • vor 6 Monaten

“What happens when AI infrastructure stops chasing everything — and focuses on making intelligence actually reliable?” Sachi Takahara / 0699.eth joins Karan Sirdesai (Karan), Founder of Mira, for a deep dive into how Mira is taking a sharper path in decentralized AI: high-trust, reliable intelligence as the core product. They break down why Mira isn’t trying to be another full-stack AI protocol, how it differs from networks like Bittensor, Ritual, and Sahara, and why its narrow focus could make it easier to plug into the broader AI infra ecosystem instead of competing with every layer. The conversation also dives into Mira’s reliability model — from core tech to node-level consensus — and why the future of AI APIs may depend less on raw model access, and more on whether users can actually trust the intelligence they receive.

“What happens when AI infrastructure stops chasing everything — and focuses on making intelligence actually reliable?” Sachi Takahara / 0699.eth joins Karan Sirdesai (Karan), Founder of Mira, for a deep dive into how Mira is taking a sharper path in decentralized AI: high-trust, reliable intelligence as the core product. They break down why Mira isn’t trying to be another full-stack AI protocol, how it differs from networks like Bittensor, Ritual, and Sahara, and why its narrow focus could make it easier to plug into the broader AI infra ecosystem instead of competing with every layer. The conversation also dives into Mira’s reliability model — from core tech to node-level consensus — and why the future of AI APIs may depend less on raw model access, and more on whether users can actually trust the intelligence they receive.

BlockLayer Podcast

48,028 Aufrufe • vor 4 Monaten

Nemotron 3 Full Breakdown With the help of Joey Conway from NVIDIA AI getting into the specifics around why Nemotron 3 is kind of a big deal Biggest headline with Nemotron is: Hybrid Mamba Transformer, Latent MoE, and MTP Hybrid Mamba Transformer essentially attacks right at the Attention mechanism to make the overhead sub-quadratic, but unlike quantizing KV Cache or swapping out attention head, NVIDIA chose Mamba-2 Latent MoE helps further optimize on sparsity by down projecting the dimensions so you're doing less math and less memory movement between HBM and SRAM, you're saving a ton, and NVIDIA made a conscious choice to add more experts given the surplus Finally, MTP or multi token prediction where the model can see future tokens to be more expressive in training and also option to use for speculative decoding during inference Oh, also the model adopts the new OpenMDW 1.1 License

Nemotron 3 Full Breakdown With the help of Joey Conway from NVIDIA AI getting into the specifics around why Nemotron 3 is kind of a big deal Biggest headline with Nemotron is: Hybrid Mamba Transformer, Latent MoE, and MTP Hybrid Mamba Transformer essentially attacks right at the Attention mechanism to make the overhead sub-quadratic, but unlike quantizing KV Cache or swapping out attention head, NVIDIA chose Mamba-2 Latent MoE helps further optimize on sparsity by down projecting the dimensions so you're doing less math and less memory movement between HBM and SRAM, you're saving a ton, and NVIDIA made a conscious choice to add more experts given the surplus Finally, MTP or multi token prediction where the model can see future tokens to be more expressive in training and also option to use for speculative decoding during inference Oh, also the model adopts the new OpenMDW 1.1 License

Caleb Eom

50,425 Aufrufe • vor 3 Monaten

Today we're incredibly excited to announce Dreamscale Labs (YC F26) (YC F26)! We're an applied research company putting robot brains in the cloud. Robot AI models are growing larger and on-board compute is getting increasingly inaccessible due to cost. We're taking a contrarian bet that the future is moving inference off the robot and into the cloud, while still meeting real-time deadlines required by robot motion. We're building the real-time inference infrastructure that enables frontier physical AI models to run smoothly on your robot. We optimize for low latency across the full stack: from the model, down to the GPU kernels, and over the network. Our mission is to unlock a future of safe and intelligent robots doing meaningful work, and we think this is only possible by letting you deploy and run robot models in the cloud. Reach out to us at founders@dreamscalelabs.com or visit our website at to learn more. Kevin L Antoine Nguyen Eric R

Today we're incredibly excited to announce Dreamscale Labs (YC F26) (YC F26)! We're an applied research company putting robot brains in the cloud. Robot AI models are growing larger and on-board compute is getting increasingly inaccessible due to cost. We're taking a contrarian bet that the future is moving inference off the robot and into the cloud, while still meeting real-time deadlines required by robot motion. We're building the real-time inference infrastructure that enables frontier physical AI models to run smoothly on your robot. We optimize for low latency across the full stack: from the model, down to the GPU kernels, and over the network. Our mission is to unlock a future of safe and intelligent robots doing meaningful work, and we think this is only possible by letting you deploy and run robot models in the cloud. Reach out to us at [email protected] or visit our website at to learn more. Kevin L Antoine Nguyen Eric R

Chris Yoo

46,032 Aufrufe • vor 4 Tagen

$BREAKING: SambaNova Announces First Close of $1 Billion Series E at $11B Valuation Led by General Atlantic, with participation from Seligman Ventures, T. Rowe Price, & Capital Group SambaNova CEO Rodrigo Liang (Rodrigo Liang) says AI is entering a new era of "premium inference" "The world of inference is running into an economics problem." Enter: new SambaNova "SM50" "A cloud-scale RDU that allows us to deliver premium inference at significantly higher performance than you can get with GPUs, at a fraction of the cost & a fraction of the power." Why does that matter? "When you're running faster with the high-accuracy models, you can charge more at a lower cost, allowing our service providers to generate more margin." Liang says the shift is just beginning: "The world's moving into inference." And after today's funding announcement: "We're using this capital to accelerate our supply chain and make sure we can deliver the racks to customers worldwide." Full interview from Paris coming soon.$

BREAKING: SambaNova Announces First Close of $1 Billion Series E at $11B Valuation Led by General Atlantic, with participation from Seligman Ventures, T. Rowe Price, & Capital Group SambaNova CEO Rodrigo Liang (Rodrigo Liang) says AI is entering a new era of "premium inference" "The world of inference is running into an economics problem." Enter: new SambaNova "SM50" "A cloud-scale RDU that allows us to deliver premium inference at significantly higher performance than you can get with GPUs, at a fraction of the cost & a fraction of the power." Why does that matter? "When you're running faster with the high-accuracy models, you can charge more at a lower cost, allowing our service providers to generate more margin." Liang says the shift is just beginning: "The world's moving into inference." And after today's funding announcement: "We're using this capital to accelerate our supply chain and make sure we can deliver the racks to customers worldwide." Full interview from Paris coming soon.

Molly O’Shea

14,162 Aufrufe • vor 2 Monaten

Tokenization -- turning text into a sequence of integers -- is a key part of generative AI, and most API providers charge per million tokens. How does tokenization work? Learn the details of tokenization and RAG optimization in Retrieval Optimization: From Tokenization to Vector Quantization, created in collaboration with Qdrant and taught by its Developer Relations Lead, Kacper Łukawski. This course focuses on Retrieval augmented generation (RAG), which has two steps: First, a retriever finds relevant information; then, the generator uses what’s retrieved as context to produce a response. You’ll learn to optimize the first step (the retriever) by understanding how tokenization works and how it impacts the relevance of your search. In addition, you will also learn to measure and improve retrieval quality, speed, and memory. In detail, you’ll: - Learn about the internal workings of the embedding models and how your text turns into vectors. - Understand how several tokenizers, such as Byte-Pair Encoding, WordPiece, Unigram, and SentencePiece work. - Explore common challenges with tokenizers, such as unknown tokens, domain-specific identifiers, and numerical values, that can negatively affect your vector search. - Understand how to measure the quality of your search across relevance, ranking, and score-related metrics. - Understand how the main parameters in "HNSW", a graph-based algorithm, affect the relevance and speed of vector search, and how to tune its parameters. - Experiment with the three major quantization methods – product, scalar, and binary – and learn how they impact memory requirements, search quality, and speed. By the end of this course, you’ll have a solid understanding of how tokenization functions and how to optimize vector search in your RAG systems. Please sign up here!

Tokenization -- turning text into a sequence of integers -- is a key part of generative AI, and most API providers charge per million tokens. How does tokenization work? Learn the details of tokenization and RAG optimization in Retrieval Optimization: From Tokenization to Vector Quantization, created in collaboration with Qdrant and taught by its Developer Relations Lead, Kacper Łukawski. This course focuses on Retrieval augmented generation (RAG), which has two steps: First, a retriever finds relevant information; then, the generator uses what’s retrieved as context to produce a response. You’ll learn to optimize the first step (the retriever) by understanding how tokenization works and how it impacts the relevance of your search. In addition, you will also learn to measure and improve retrieval quality, speed, and memory. In detail, you’ll: - Learn about the internal workings of the embedding models and how your text turns into vectors. - Understand how several tokenizers, such as Byte-Pair Encoding, WordPiece, Unigram, and SentencePiece work. - Explore common challenges with tokenizers, such as unknown tokens, domain-specific identifiers, and numerical values, that can negatively affect your vector search. - Understand how to measure the quality of your search across relevance, ranking, and score-related metrics. - Understand how the main parameters in "HNSW", a graph-based algorithm, affect the relevance and speed of vector search, and how to tune its parameters. - Experiment with the three major quantization methods – product, scalar, and binary – and learn how they impact memory requirements, search quality, and speed. By the end of this course, you’ll have a solid understanding of how tokenization functions and how to optimize vector search in your RAG systems. Please sign up here!

Andrew Ng

146,313 Aufrufe • vor 1 Jahr

The first company I ever joined was Rubrik, Arvind Jain's previous company. It's where I learned how to build systems and engineering teams. Years later, when we started Composio, Glean became one of our first customers, one of the first to believe in what we were building. So sitting down with Arvind felt like closing a loop. In this episode: •How Glean brought transformers to enterprise search before "semantic search" had a name •Glean's partner-first strategy and why they chose Composio for actions •Why Arvind has never worried about competing with OpenAI or Anthropic •GLM 5.2 as the open-source inflection: 90%+ of enterprise AI tasks, majority of inference within 12–18 months •Measuring real ROI: the US telco that cut case-resolution time by 48% •Why agents built on MCP alone act like day-one employees and how context graphs make them tenured CHAPTERS: (00:00) – Rubrik days: how Arvind and Karan met (01:05) – Glean's origin story: transformers before "generative AI" existed (03:04) – What Glean is today: superset of ChatGPT and Claude (05:39) – From embedding search to agents: was it a pivot? (11:39) – The Composio partnership and why actions are hard (14:43) – Why competition doesn't matter yet (20:11) – The open-source inflection point (GLM 5.2) (24:41) – Contracts and model risk: who absorbs the volatility? (26:57) – Enterprise ROI: cost, tokens, and what to measure (39:57) – Measuring value team by team (44:11) – Day-one employees vs. tenured agents (46:09) – Closing thoughts

The first company I ever joined was Rubrik, Arvind Jain's previous company. It's where I learned how to build systems and engineering teams. Years later, when we started Composio, Glean became one of our first customers, one of the first to believe in what we were building. So sitting down with Arvind felt like closing a loop. In this episode: •How Glean brought transformers to enterprise search before "semantic search" had a name •Glean's partner-first strategy and why they chose Composio for actions •Why Arvind has never worried about competing with OpenAI or Anthropic •GLM 5.2 as the open-source inflection: 90%+ of enterprise AI tasks, majority of inference within 12–18 months •Measuring real ROI: the US telco that cut case-resolution time by 48% •Why agents built on MCP alone act like day-one employees and how context graphs make them tenured CHAPTERS: (00:00) – Rubrik days: how Arvind and Karan met (01:05) – Glean's origin story: transformers before "generative AI" existed (03:04) – What Glean is today: superset of ChatGPT and Claude (05:39) – From embedding search to agents: was it a pivot? (11:39) – The Composio partnership and why actions are hard (14:43) – Why competition doesn't matter yet (20:11) – The open-source inflection point (GLM 5.2) (24:41) – Contracts and model risk: who absorbs the volatility? (26:57) – Enterprise ROI: cost, tokens, and what to measure (39:57) – Measuring value team by team (44:11) – Day-one employees vs. tenured agents (46:09) – Closing thoughts

Karan Vaidya

62,488 Aufrufe • vor 1 Monat

$AMD | Inference's estimated to be 80%+ in 2027👑🆕 Dr. Su told everyone the world will need a lot more CPUs and Inference will dominate most of compute long term from 2022. Nobody believed her, but I did along with other high conviction investors. 2026 is the first time Inference surpassed training at 65%+ vs 33-35% for training. Early 2026 infrastructure spend: ~55% inference With Agentic AI, 2027 Inference Infrastructure spend is projected to be 80%+ Training still grows in absolute terms (bigger clusters, more experiments). Inference grows faster because usage, agents, and reasoning traces multiply token volume continuously. That’s why hardware and data center design are shifting toward latency, utilization, and cost per token rather than peak training FLOPS. What "token efficient" actually means when AI labs produces better/smarter models? When models get more token efficient, the expensive part of an agent the GPU “thinking” step gets shorter and cheaper. The rest of the loop does not. The agent still has to parse the answer, pick a tool, run code in a sandbox, query a database, open a browser, apply guardrails, and feed the result back. Those steps live on CPUs where AMD has the best CPU in the world. So a more efficient model does not shrink the agent; it shrinks the model’s share of the agent. Wall clock time and cost tilt toward orchestration and sandboxes, which is why you provision more CPU racks even as tokens per decision fall. Cheaper thinking also unlocks more doing. Teams stop designing one shot answers and start adding retries, parallel branches, sub-agents, and24/7 digital workers, Jevons paradox for agents. Each extra loop is another isolated environment, and unlike GPU batching, sandbox demand scales almost linearly with concurrency: fifty candidate patches means fifty containers, not one fatter GPU job. Token saving tricks often push even more work onto that layer. The result is a fleet that thinks less per step and acts more often, so the volume product becomes CPU/sandbox capacity, not just accelerators. Agentic workloads are a big part of why inference is pulling ahead. I’ll pull the latest numbers on token multipliers and how that shows up in 2026 compute mix.Yes. The inference flip is mostly agents + reasoning, not more people chatting. Chat was one prompt in, a few hundred tokens out. Agents turn a single user request into a loop: plan, tool call, read the result, think, retry, hand off to a sub-agent. That is why Gartner’s 2026 range is 5–30× more tokens per task than a chatbot, with coding and research agents often landing higher. On OpenRouter, agentic workloads 14x’d in six months, passed human usage in February 2026, and by early August were ~5× human tokens (~7.3T/day agentic vs ~1.4T human). A typical agentic request there used 15× the tokens of a human query. One important thing to understand, unit cost per token is still falling, but tokens per useful outcome rose faster. An “agentic seat” can burn 50–100× the tokens of a chat subscription. That is impressive growth for inference. It is also why KV cache, speculative decoding, quantization, and inference specific silicon suddenly matter more than another giant training cluster. Not Financial Advice! DYOR!

$AMD | Inference's estimated to be 80%+ in 2027👑🆕 Dr. Su told everyone the world will need a lot more CPUs and Inference will dominate most of compute long term from 2022. Nobody believed her, but I did along with other high conviction investors. 2026 is the first time Inference surpassed training at 65%+ vs 33-35% for training. Early 2026 infrastructure spend: ~55% inference With Agentic AI, 2027 Inference Infrastructure spend is projected to be 80%+ Training still grows in absolute terms (bigger clusters, more experiments). Inference grows faster because usage, agents, and reasoning traces multiply token volume continuously. That’s why hardware and data center design are shifting toward latency, utilization, and cost per token rather than peak training FLOPS. What "token efficient" actually means when AI labs produces better/smarter models? When models get more token efficient, the expensive part of an agent the GPU “thinking” step gets shorter and cheaper. The rest of the loop does not. The agent still has to parse the answer, pick a tool, run code in a sandbox, query a database, open a browser, apply guardrails, and feed the result back. Those steps live on CPUs where AMD has the best CPU in the world. So a more efficient model does not shrink the agent; it shrinks the model’s share of the agent. Wall clock time and cost tilt toward orchestration and sandboxes, which is why you provision more CPU racks even as tokens per decision fall. Cheaper thinking also unlocks more doing. Teams stop designing one shot answers and start adding retries, parallel branches, sub-agents, and24/7 digital workers, Jevons paradox for agents. Each extra loop is another isolated environment, and unlike GPU batching, sandbox demand scales almost linearly with concurrency: fifty candidate patches means fifty containers, not one fatter GPU job. Token saving tricks often push even more work onto that layer. The result is a fleet that thinks less per step and acts more often, so the volume product becomes CPU/sandbox capacity, not just accelerators. Agentic workloads are a big part of why inference is pulling ahead. I’ll pull the latest numbers on token multipliers and how that shows up in 2026 compute mix.Yes. The inference flip is mostly agents + reasoning, not more people chatting. Chat was one prompt in, a few hundred tokens out. Agents turn a single user request into a loop: plan, tool call, read the result, think, retry, hand off to a sub-agent. That is why Gartner’s 2026 range is 5–30× more tokens per task than a chatbot, with coding and research agents often landing higher. On OpenRouter, agentic workloads 14x’d in six months, passed human usage in February 2026, and by early August were ~5× human tokens (~7.3T/day agentic vs ~1.4T human). A typical agentic request there used 15× the tokens of a human query. One important thing to understand, unit cost per token is still falling, but tokens per useful outcome rose faster. An “agentic seat” can burn 50–100× the tokens of a chat subscription. That is impressive growth for inference. It is also why KV cache, speculative decoding, quantization, and inference specific silicon suddenly matter more than another giant training cluster. Not Financial Advice! DYOR!

Mike

16,350 Aufrufe • vor 15 Tagen

Announcing the Open Model Series: a new series on open source AI models, with merve of Hugging Face. Hugging Face is the open model hub: it hosts model weights, datasets, evals, & demo apps, and maintains the open-source libraries (like transformers) used to run and train them. I think of it like GitHub, but for AI models. You can download open models and run them on your own machines. In this series we'll cover what open source and open weight models are, how they work, how to run them yourself, and why they're important. By the way - this video is edited entirely with Claude! I can make a tutorial on how I did it. If there's a question you want answered about open models, leave it in the comments. We will read them all. Episode 1 is out soon.

Announcing the Open Model Series: a new series on open source AI models, with merve of Hugging Face. Hugging Face is the open model hub: it hosts model weights, datasets, evals, & demo apps, and maintains the open-source libraries (like transformers) used to run and train them. I think of it like GitHub, but for AI models. You can download open models and run them on your own machines. In this series we'll cover what open source and open weight models are, how they work, how to run them yourself, and why they're important. By the way - this video is edited entirely with Claude! I can make a tutorial on how I did it. If there's a question you want answered about open models, leave it in the comments. We will read them all. Episode 1 is out soon.

Harper Carroll

30,392 Aufrufe • vor 1 Monat