Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

I interviewed Neil Movva for Substrate on latency-insensitive inference, alternative chips, and why massive datacenters face diseconomies of scale. "We've always predicted that async inference is going to eat the world. That could easily be 95% or more of inference demand." Neil's been at the frontier of inference for... show more

CAT

1,664 subscribers

38,600 Aufrufe • vor 12 Tagen •via X (Twitter)

Wissenschaft & Technologie

Anya Rossi• Live Now

Private livecam show

37 Kommentare

Profilbild von CAT

CATvor 12 Tagen

Links: Youtube: Spotify: Transcript:

Profilbild von Linda Xue

Linda Xuevor 12 Tagen

@neilmovva @substratemag this is so well produced

Profilbild von CAT

CATvor 12 Tagen

@neilmovva @substratemag thank you! shoutout to @jlinbio

Profilbild von Bryan Huang

Bryan Huangvor 12 Tagen

@neilmovva @substratemag let’s go videooo

Profilbild von CAT

CATvor 12 Tagen

@neilmovva @substratemag always more mediums!

Profilbild von James Lin

James Linvor 12 Tagen

@neilmovva @substratemag insanely good q&a through and through

Profilbild von CAT

CATvor 12 Tagen

@neilmovva @substratemag really enjoyed this one!

Profilbild von Benjamin

Benjaminvor 12 Tagen

@cupertinohoops @neilmovva @substratemag This is so good!!

Profilbild von CAT

CATvor 12 Tagen

@cupertinohoops @neilmovva @substratemag thank you! glad you liked

Profilbild von aadilpickle

aadilpicklevor 12 Tagen

@neilmovva @substratemag hype

Profilbild von CAT

CATvor 12 Tagen

@neilmovva @substratemag 🥳🥳

Profilbild von Kavin Anand

Kavin Anandvor 12 Tagen

@neilmovva @substratemag another banger

Profilbild von CAT

CATvor 12 Tagen

@neilmovva @substratemag @neilmovva banger factory

Profilbild von gitika

gitikavor 12 Tagen

@neilmovva @substratemag So so good!

Profilbild von CAT

CATvor 12 Tagen

@neilmovva @substratemag so appreciate you!

Profilbild von Aurash Vatan

Aurash Vatanvor 12 Tagen

@neilmovva @substratemag Where are the short-form vertical-vid outtakes @cattttang for those with shorter attention spans

Profilbild von CAT

CATvor 12 Tagen

@neilmovva @substratemag my bad...insufficiently new media

Profilbild von Alex Brogan

Alex Broganvor 12 Tagen

@neilmovva @substratemag awesome work, Cat!

Profilbild von CAT

CATvor 12 Tagen

@neilmovva @substratemag thank you!!

Profilbild von amogh

amoghvor 12 Tagen

@neilmovva @substratemag Genuinely a good watch 🫡

Profilbild von CAT

CATvor 12 Tagen

@neilmovva @substratemag appreciate it, hope you learned as much as I did!

Profilbild von Maria Patni

Maria Patnivor 12 Tagen

@cupertinohoops @neilmovva @substratemag excited to watch this after work tdy 🫣

Profilbild von CAT

CATvor 12 Tagen

@cupertinohoops @neilmovva @substratemag pending opinion!

Profilbild von Ryan

Ryanvor 12 Tagen

@neilmovva @substratemag sickkkk

Profilbild von CAT

CATvor 12 Tagen

@neilmovva @substratemag thank you!! was such a fun conversation

Profilbild von Alex Marley

Alex Marleyvor 12 Tagen

@neilmovva @substratemag nicely done!

Profilbild von CAT

CATvor 12 Tagen

@neilmovva @substratemag thanks so much!

Profilbild von will hang

will hangvor 12 Tagen

@neilmovva @substratemag BOOM as fuck

Profilbild von CAT

CATvor 12 Tagen

@neilmovva @substratemag 💥💥💥

Profilbild von andi

andivor 12 Tagen

@neilmovva @substratemag wowowowowow

Profilbild von CAT

CATvor 12 Tagen

@neilmovva @substratemag 🤓

Profilbild von Charlie Ochim

Charlie Ochimvor 12 Tagen

@neilmovva @substratemag Async eating 95% of inference is wild to me. Also the part about huge datacenters getting worse as they scale is not what I expected. Going to read this tonight.

Profilbild von Jan

Janvor 12 Tagen

@neilmovva @substratemag so well produced!

Profilbild von Umesh Khanna 🇨🇦🇺🇸

Umesh Khanna 🇨🇦🇺🇸vor 12 Tagen

@neilmovva @substratemag Soo good!

Profilbild von perper

perpervor 12 Tagen

@neilmovva @substratemag 95% of inference demand being async is a bold number to say out loud

Profilbild von Cooper Veit

Cooper Veitvor 12 Tagen

@ttunguz @neilmovva @substratemag 🐐

Profilbild von Michael Tan

Michael Tanvor 12 Tagen

@neilmovva @substratemag nice

Ähnliche Videos

We're excited to back Sail Research, the inference platform built for long-horizon, asynchronous agents. The next wave of AI isn't a person typing into a box. It's agents running for minutes or hours, burning orders of magnitude more tokens per task. At that scale, cost and capacity beat raw latency, and teams are already hitting budget caps. Neil Movva and Samir Menon are solving this full-stack, from CUDA kernels to the API. Hear Neil and Aditya Naganath on why inference is being rebuilt for the age of agents.

We're excited to back Sail Research, the inference platform built for long-horizon, asynchronous agents. The next wave of AI isn't a person typing into a box. It's agents running for minutes or hours, burning orders of magnitude more tokens per task. At that scale, cost and capacity beat raw latency, and teams are already hitting budget caps. Neil Movva and Samir Menon are solving this full-stack, from CUDA kernels to the API. Hear Neil and Aditya Naganath on why inference is being rebuilt for the age of agents.

Kleiner Perkins

17,495 Aufrufe • vor 4 Monaten

This is why Nvidia bought Groq. If Groq deploys a ton of low-cost inference chips, demand for training will increase massively. The more inference you have, the more training you need, and vice versa. Nvidia can keep its high margins selling chips for training forever.

This is why Nvidia bought Groq. If Groq deploys a ton of low-cost inference chips, demand for training will increase massively. The more inference you have, the more training you need, and vice versa. Nvidia can keep its high margins selling chips for training forever.

Lin

92,571 Aufrufe • vor 10 Monaten

What is AI inference engineering, why is it such an in-demand skill, and how do you break into the field? With author of Inference Engineering Philip Kiely and head of training at Baseten Charlie O'Neill 0:00: What is inference? 2:47: History of inference 4:59: Downstream effects of AI research on inference 13:54: What you'll learn from Inference Engineering 16:14: Advice for engineers transitioning into AI 19:00: Open source models driving inference growth 20:55: Specialization vs. frontier closed models 23:51: "Big Token" and the importance of open source AI 27:18: Where to get Inference Engineering

What is AI inference engineering, why is it such an in-demand skill, and how do you break into the field? With author of Inference Engineering Philip Kiely and head of training at Baseten Charlie O'Neill 0:00: What is inference? 2:47: History of inference 4:59: Downstream effects of AI research on inference 13:54: What you'll learn from Inference Engineering 16:14: Advice for engineers transitioning into AI 19:00: Open source models driving inference growth 20:55: Specialization vs. frontier closed models 23:51: "Big Token" and the importance of open source AI 27:18: Where to get Inference Engineering

Madison Kanna

119,268 Aufrufe • vor 7 Monaten

We are excited to share a preview of our peer-to-peer decentralized inference stack Engineered for consumer GPUs and high-latency networks — plus a research roadmap to scale it to a planetary-scale decentralized inference engine.

We are excited to share a preview of our peer-to-peer decentralized inference stack Engineered for consumer GPUs and high-latency networks — plus a research roadmap to scale it to a planetary-scale decentralized inference engine.

Prime Intellect

103,807 Aufrufe • vor 1 Jahr

Introducing Prime Inference: We've served trillions of tokens for RL and dedicated customer deployments To own your intelligence, you need to own your inference Unpacking our inference stack

Introducing Prime Inference: We've served trillions of tokens for RL and dedicated customer deployments To own your intelligence, you need to own your inference Unpacking our inference stack

Prime Intellect

230,645 Aufrufe • vor 26 Tagen

"The inflection point for inference has arrived." — Jensen Huang, Founder & CEO of NVIDIA We’ve officially crossed a new milestone in the inference era — where widespread adoption of AI shifts from learning to doing. The breakthrough: extreme codesign across hardware and software driving down cost per token. Lower cost → more inference More inference → more users and applications More users and applications → exponential AI revenue growth

"The inflection point for inference has arrived." — Jensen Huang, Founder & CEO of NVIDIA We’ve officially crossed a new milestone in the inference era — where widespread adoption of AI shifts from learning to doing. The breakthrough: extreme codesign across hardware and software driving down cost per token. Lower cost → more inference More inference → more users and applications More users and applications → exponential AI revenue growth

NVIDIA

82,086 Aufrufe • vor 6 Monaten

Jesse Pollak on the ecosystem forming around tokenized inference "There's this really cool tokenized inference market that Venice has innovated, where they've taken inference and turned it into a token called DIEM where you can buy it and you get $1 of inference per day" "That innovation of tokenizing inference has unlocked a ton of innovation around it, where people are using that tokenized inference to do things" "There's other projects that let people sell their tokenized inference at a discount so other people can get cheaper inference and the whole system works more efficiently" "I feel like that's the first time we're really seeing the intersection of AI plus markets plus crypto and it's thanks to Venice and it's happening on Base"

Jesse Pollak on the ecosystem forming around tokenized inference "There's this really cool tokenized inference market that Venice has innovated, where they've taken inference and turned it into a token called DIEM where you can buy it and you get $1 of inference per day" "That innovation of tokenizing inference has unlocked a ton of innovation around it, where people are using that tokenized inference to do things" "There's other projects that let people sell their tokenized inference at a discount so other people can get cheaper inference and the whole system works more efficiently" "I feel like that's the first time we're really seeing the intersection of AI plus markets plus crypto and it's thanks to Venice and it's happening on Base"

Market Bubble

26,240 Aufrufe • vor 3 Monaten

I sat down with Silas Alberti, Head of Research Cognition, to chat about the intersection of RL and inference -- from training frontier coding models to running inference at scale. Watch till the end for bonus doggo! :o 0:00 — Intros 1:04 — What's hardest to get right in an RL run 3:39 — How Cognition started using Modal 4:03 — What to weigh when buying inference 4:58 — The latency/throughput Pareto frontier 7:14 — DFlash reveal (!!) 8:44 — Why faster runtime matters 9:56 — Memory-bound vs. compute-bound 10:57 — Tree-based speculation 14:20 — Hot take: RL and inference optimization converging 15:28 — Online training of the speculator 16:22 — "Auto inference" 17:27 — Agents competing on optimization problems 18:27 — What inference providers still get wrong

I sat down with Silas Alberti, Head of Research Cognition, to chat about the intersection of RL and inference -- from training frontier coding models to running inference at scale. Watch till the end for bonus doggo! :o 0:00 — Intros 1:04 — What's hardest to get right in an RL run 3:39 — How Cognition started using Modal 4:03 — What to weigh when buying inference 4:58 — The latency/throughput Pareto frontier 7:14 — DFlash reveal (!!) 8:44 — Why faster runtime matters 9:56 — Memory-bound vs. compute-bound 10:57 — Tree-based speculation 14:20 — Hot take: RL and inference optimization converging 15:28 — Online training of the speculator 16:22 — "Auto inference" 17:27 — Agents competing on optimization problems 18:27 — What inference providers still get wrong

gongy

82,724 Aufrufe • vor 3 Monaten

$A single API or inference call can cost a fraction of a cent. Gordon Liao explains why agentic payments need rails built for nanopayments, from data and inference calls to more complex commerce workflows. Circle Agent Stack helps developers build for that shift.$

A single API or inference call can cost a fraction of a cent. Gordon Liao explains why agentic payments need rails built for nanopayments, from data and inference calls to more complex commerce workflows. Circle Agent Stack helps developers build for that shift.

Circle

14,692 Aufrufe • vor 3 Monaten

$NVDA CEO: “Demand for inference will go up by a billion times.” $AMD will be one of the biggest winners as it has the upper hand in inference thanks to larger memory bandwith and capacity of its chips. Stock is still at 22x 2027 earnings. How can’t you be bullish on $AMD?

$NVDA CEO: “Demand for inference will go up by a billion times.” $AMD will be one of the biggest winners as it has the upper hand in inference thanks to larger memory bandwith and capacity of its chips. Stock is still at 22x 2027 earnings. How can’t you be bullish on $AMD?

Oguz Erkan

288,805 Aufrufe • vor 6 Monaten

AI inference is already live on UOMI Router. Access frontier open-source models through a decentralized inference network with: • Low-cost inference • OpenAI-compatible APIs • Verifiable compute • On-chain settlement Start building: And if you have idle GPUs, there's now a second way to participate. The UOMI Provider whitelist is officially open. Instead of letting your hardware sit unused, you can contribute GPU power to the UOMI Inference Network and earn from real AI demand. Join the Provider whitelist:

AI inference is already live on UOMI Router. Access frontier open-source models through a decentralized inference network with: • Low-cost inference • OpenAI-compatible APIs • Verifiable compute • On-chain settlement Start building: And if you have idle GPUs, there's now a second way to participate. The UOMI Provider whitelist is officially open. Instead of letting your hardware sit unused, you can contribute GPU power to the UOMI Inference Network and earn from real AI demand. Join the Provider whitelist:

Uomi

19,009 Aufrufe • vor 4 Monaten

Terrible acting? Yes! Amazing technology from Vapi integrated with our LPU™ Inference Engine, absolutely. Imagine a world where NPCs in video games could be this dynamic. Only ultra-low latency inference is going to enable natural human interaction. What are you building?

Terrible acting? Yes! Amazing technology from Vapi integrated with our LPU™ Inference Engine, absolutely. Imagine a world where NPCs in video games could be this dynamic. Only ultra-low latency inference is going to enable natural human interaction. What are you building?

Groq

19,102 Aufrufe • vor 2 Jahren

Free, instant access to 1M inference tok/day inference is 20x faster than on NVIDIA. You probably don't even know what that feels like. Now you can Building our new chat and showcase were so fun to ship. I promise this will be your *holy shit* moment of 2025.

Free, instant access to 1M inference tok/day inference is 20x faster than on NVIDIA. You probably don't even know what that feels like. Now you can Building our new chat and showcase were so fun to ship. I promise this will be your holy shit moment of 2025.

Sarah Chieng

20,416 Aufrufe • vor 1 Jahr

From POC to production inference. 🤖🌁 Join us at #DOdeploy in San Francisco where we'll discuss inference-first architectures, predictable inference economics, operational simplicity at scale, & more.

From POC to production inference. 🤖🌁 Join us at #DOdeploy in San Francisco where we'll discuss inference-first architectures, predictable inference economics, operational simplicity at scale, & more.

DigitalOcean

12,195 Aufrufe • vor 8 Monaten

AI answers your questions in seconds, but behind that speed is something called inference—the compute-intensive process where trained models generate responses. At AWS, we've built custom chips like Trainium, intelligent routing systems, and unified infrastructure to make inference faster and more affordable. As AI agents handle complex multi-step tasks, inference accounts for 80-90% of AI computing power. We're engineering at planetary scale to keep those milliseconds reliable.

AI answers your questions in seconds, but behind that speed is something called inference—the compute-intensive process where trained models generate responses. At AWS, we've built custom chips like Trainium, intelligent routing systems, and unified infrastructure to make inference faster and more affordable. As AI agents handle complex multi-step tasks, inference accounts for 80-90% of AI computing power. We're engineering at planetary scale to keep those milliseconds reliable.

Amazon Web Services

27,021 Aufrufe • vor 8 Monaten

NVIDIA Vera Rubin is in full production and arrives just in time for the next frontier of AI. The #NVIDIARubin platform uses extreme co-design across six new chips to accelerate agentic AI, advanced reasoning and massive-scale MoE model inference. NVIDIA Vera Rubin NVL72: Six new chips, one AI supercomputer — see how we built it.

NVIDIA Vera Rubin is in full production and arrives just in time for the next frontier of AI. The #NVIDIARubin platform uses extreme co-design across six new chips to accelerate agentic AI, advanced reasoning and massive-scale MoE model inference. NVIDIA Vera Rubin NVL72: Six new chips, one AI supercomputer — see how we built it.

NVIDIA Data Center

27,643 Aufrufe • vor 9 Monaten

Introducing RSI-Inference Bench from 🍎 Apple ML Research 🌐 💻 Can AI agents improve the systems that run AI? We spent $100K API tokens, 50K GPU hours to find out. We built environments and benchmark from real-world #vLLM and #SGLang tasks, covering the full lifecycle of model inference engineering: 🚀 Day-0 Model Support ⚡ Inference Optimization 🔧 Bug Fix → Unsaturated: best agent at 26.7% pass@1 → Comprehensive coverage of the inference stack → Verifiable rewards: binary and continuous Watch the video to find out more!

Introducing RSI-Inference Bench from 🍎 Apple ML Research 🌐 💻 Can AI agents improve the systems that run AI? We spent $100K API tokens, 50K GPU hours to find out. We built environments and benchmark from real-world #vLLM and #SGLang tasks, covering the full lifecycle of model inference engineering: 🚀 Day-0 Model Support ⚡ Inference Optimization 🔧 Bug Fix → Unsaturated: best agent at 26.7% pass@1 → Comprehensive coverage of the inference stack → Verifiable rewards: binary and continuous Watch the video to find out more!

Ruixiang Zhang

78,128 Aufrufe • vor 12 Tagen