Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

✨ The #NVIDIARubin platform is engineered to master multi-step problem-solving and massive long-context reasoning workflows at scale. Built using extreme co-design, Rubin delivers more AI tokens per watt and lower cost per token compared to the previous architecture. ✔️ 50 PFLOPS of NVFP4 inference ✔️ 22 TB/s of HBM4... show more

NVIDIA AI Infrastructure

68,835 subscribers

10,475 views • 5 months ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

The NVIDIA Vera Rubin NVL72 scale-up fabric, built on the sixth-generation NVIDIA NVLink. The NVLink 6 Switch trays connect 72 Rubin GPUs in an all-to-all configuration. The NVLink spine—over 5,000 copper cables make up the vertical NVLink traverse. It securely moves more data than the entire bandwidth of the internet. The single-wide NVIDIA MGX rack design unlocks a cable-free, horizontal NVLink scale-up traverse, running directly on PCB for resiliency at scale. This is how NVIDIA Vera Rubin NVL72 works as one, designed to deliver the lowest token cost and 10x more performance per watt than the previous generation. #NVIDIAVeraRubin

The NVIDIA Vera Rubin NVL72 scale-up fabric, built on the sixth-generation NVIDIA NVLink. The NVLink 6 Switch trays connect 72 Rubin GPUs in an all-to-all configuration. The NVLink spine—over 5,000 copper cables make up the vertical NVLink traverse. It securely moves more data than the entire bandwidth of the internet. The single-wide NVIDIA MGX rack design unlocks a cable-free, horizontal NVLink scale-up traverse, running directly on PCB for resiliency at scale. This is how NVIDIA Vera Rubin NVL72 works as one, designed to deliver the lowest token cost and 10x more performance per watt than the previous generation. #NVIDIAVeraRubin

NVIDIA AI Infrastructure

35,194 views • 10 days ago

NVIDIA Vera Rubin is in full production and arrives just in time for the next frontier of AI. The #NVIDIARubin platform uses extreme co-design across six new chips to accelerate agentic AI, advanced reasoning and massive-scale MoE model inference. NVIDIA Vera Rubin NVL72: Six new chips, one AI supercomputer — see how we built it.

NVIDIA Vera Rubin is in full production and arrives just in time for the next frontier of AI. The #NVIDIARubin platform uses extreme co-design across six new chips to accelerate agentic AI, advanced reasoning and massive-scale MoE model inference. NVIDIA Vera Rubin NVL72: Six new chips, one AI supercomputer — see how we built it.

NVIDIA Data Center

27,643 views • 6 months ago

“NVIDIA’s cost per token is the lowest in the world.” — Jensen Huang, Founder & CEO of NVIDIA Token generation cost is a direct result of architecture excellence and extreme co-design, not just compute cost. Lowest cost per token and highest performance per watt are definitive measures of AI economics and the key to unlocking maximum profitability and AI revenue. ➡️

“NVIDIA’s cost per token is the lowest in the world.” — Jensen Huang, Founder & CEO of NVIDIA Token generation cost is a direct result of architecture excellence and extreme co-design, not just compute cost. Lowest cost per token and highest performance per watt are definitive measures of AI economics and the key to unlocking maximum profitability and AI revenue. ➡️

NVIDIA

88,929 views • 3 months ago

The NVIDIA Vera Rubin platform is a multi-rack pod-scale system, built to process agentic AI and is now in full production. Built through extreme co-design, Vera Rubin unifies five connected rack-scale systems — the NVIDIA Vera Rubin NVL72, Vera CPU rack, Groq 3 LPX, Vera BlueField-4 STX, and Spectrum-6 SPX Ethernet — all purpose-built for global-scale agentic AI infrastructure. Vera Rubin delivers intelligence at scale, with the performance, efficiency and security needed to power the next industrial revolution. Congratulations to Microsoft, Dell Technologies, and CoreWeave for each standing up their Vera Rubin NVL72 engineering racks. Learn more:

The NVIDIA Vera Rubin platform is a multi-rack pod-scale system, built to process agentic AI and is now in full production. Built through extreme co-design, Vera Rubin unifies five connected rack-scale systems — the NVIDIA Vera Rubin NVL72, Vera CPU rack, Groq 3 LPX, Vera BlueField-4 STX, and Spectrum-6 SPX Ethernet — all purpose-built for global-scale agentic AI infrastructure. Vera Rubin delivers intelligence at scale, with the performance, efficiency and security needed to power the next industrial revolution. Congratulations to Microsoft, Dell Technologies, and CoreWeave for each standing up their Vera Rubin NVL72 engineering racks. Learn more:

NVIDIA

49,432 views • 1 month ago

"The inflection point for inference has arrived." — Jensen Huang, Founder & CEO of NVIDIA We’ve officially crossed a new milestone in the inference era — where widespread adoption of AI shifts from learning to doing. The breakthrough: extreme codesign across hardware and software driving down cost per token. Lower cost → more inference More inference → more users and applications More users and applications → exponential AI revenue growth

"The inflection point for inference has arrived." — Jensen Huang, Founder & CEO of NVIDIA We’ve officially crossed a new milestone in the inference era — where widespread adoption of AI shifts from learning to doing. The breakthrough: extreme codesign across hardware and software driving down cost per token. Lower cost → more inference More inference → more users and applications More users and applications → exponential AI revenue growth

NVIDIA

81,834 views • 3 months ago

📈 NVIDIA tops AI leaderboards and benchmarks with open models driven by extreme co-design across compute, networking, memory, storage, and software. This includes models for biology, AI physics, agentic AI, physical AI, robotics, and autonomous vehicles. By being vertically integrated and horizontally open, NVIDIA enables its ecosystem to scale agentic workflows at every layer of the stack with the lowest cost per token.

📈 NVIDIA tops AI leaderboards and benchmarks with open models driven by extreme co-design across compute, networking, memory, storage, and software. This includes models for biology, AI physics, agentic AI, physical AI, robotics, and autonomous vehicles. By being vertically integrated and horizontally open, NVIDIA enables its ecosystem to scale agentic workflows at every layer of the stack with the lowest cost per token.

NVIDIA

85,416 views • 2 months ago

As AI labs race to train and deploy new frontier models, existing models become more affordable with better tokenomics. ✨ "Everybody's trying to get to the next frontier. And every time they get to the next frontier, the last generation AI tokens, the cost starts to decline about a factor of 10x every year," said NVIDIA CEO Jensen Huang in a recent keynote. Model optimization techniques such as speculative decoding and multi-token prediction, combined with inference serving platforms like NVIDIA Dynamo on NVIDIA Blackwell NVL72 systems, enable AI factories to boost throughput by 10x with one-tenth of the cost per token. Learn more about AI factory tokenomics ➡️

As AI labs race to train and deploy new frontier models, existing models become more affordable with better tokenomics. ✨ "Everybody's trying to get to the next frontier. And every time they get to the next frontier, the last generation AI tokens, the cost starts to decline about a factor of 10x every year," said NVIDIA CEO Jensen Huang in a recent keynote. Model optimization techniques such as speculative decoding and multi-token prediction, combined with inference serving platforms like NVIDIA Dynamo on NVIDIA Blackwell NVL72 systems, enable AI factories to boost throughput by 10x with one-tenth of the cost per token. Learn more about AI factory tokenomics ➡️

NVIDIA AI

16,053 views • 6 months ago

GPU tradeoff series: A100 is not much more powerful than 4090 🫠 GPU Perf and Price: - 4090: 330 fp16 TFLOPs, $1,749 - A100 (80GB): 312 fp16 TFLOPs, $20,000 > A100 is 11.4X more pricy Training speed for GPT-2(124M) with llm.c: - 4090: 153K tokens/s - A100 (80GB): 195K tokens/s > A100 is only 1.3X faster (both trained using a single card, A100 llm.c training is shown in the video, 4090 video is in the quoted tweet) Conclusion: 4090 has a much better cost vs performance ratio Why: As in the H100 vs. 4090 comparison, the biggest difference between A100 and 4090 is their GPU memory size/bandwidth and cross-GPU communication bandwidth, which does not matter too much if your model can fit into a single 4090. Specs: 4090: - GPU memory size: 24GB - memory bandwidth: 1 TB/s - communication bandwidth: 64 GB/s A100: - GPU memory size: 80GB - memory bandwidth: 2 TB/s - communication bandwidth: 900 GB/s Nvidia killed off NVLink (a high-speed communication link that connects GPUs) on 4090. (Jensen Huang smiling face) If multiple 4090s could be interconnected via NVLink, their performance would be closer to datacenter-grade A100 GPUs, even for training larger models. Additionally, 4090 isn't allowed in datacenters, that's how Nvidia makes 💰💰💰

GPU tradeoff series: A100 is not much more powerful than 4090 🫠 GPU Perf and Price: - 4090: 330 fp16 TFLOPs, $1,749 - A100 (80GB): 312 fp16 TFLOPs, $20,000 > A100 is 11.4X more pricy Training speed for GPT-2(124M) with llm.c: - 4090: 153K tokens/s - A100 (80GB): 195K tokens/s > A100 is only 1.3X faster (both trained using a single card, A100 llm.c training is shown in the video, 4090 video is in the quoted tweet) Conclusion: 4090 has a much better cost vs performance ratio Why: As in the H100 vs. 4090 comparison, the biggest difference between A100 and 4090 is their GPU memory size/bandwidth and cross-GPU communication bandwidth, which does not matter too much if your model can fit into a single 4090. Specs: 4090: - GPU memory size: 24GB - memory bandwidth: 1 TB/s - communication bandwidth: 64 GB/s A100: - GPU memory size: 80GB - memory bandwidth: 2 TB/s - communication bandwidth: 900 GB/s Nvidia killed off NVLink (a high-speed communication link that connects GPUs) on 4090. (Jensen Huang smiling face) If multiple 4090s could be interconnected via NVLink, their performance would be closer to datacenter-grade A100 GPUs, even for training larger models. Additionally, 4090 isn't allowed in datacenters, that's how Nvidia makes 💰💰💰

Yuchen Jin

234,951 views • 1 year ago

Jensen Huang said Grok 5 will be 7 Trillion parameter model. On time to train, with the training window fixed at 1 month, the new Vera-Rubin GPU system of Nvidia needs 1/4 the number of systems compared with Blackwell to train the same frontier model. “factory throughput” improves by about 10x over Blackwell, and Blackwell itself was about 10x over Hopper. i.e. overall Rubin is roughly 100x Hopper in factory throughput per watt, which matters hugely, because a 1 GW, $50B data center is power-limited and revenue scales with throughput per watt. --- From Nvidia YT channel

Jensen Huang said Grok 5 will be 7 Trillion parameter model. On time to train, with the training window fixed at 1 month, the new Vera-Rubin GPU system of Nvidia needs 1/4 the number of systems compared with Blackwell to train the same frontier model. “factory throughput” improves by about 10x over Blackwell, and Blackwell itself was about 10x over Hopper. i.e. overall Rubin is roughly 100x Hopper in factory throughput per watt, which matters hugely, because a 1 GW, $50B data center is power-limited and revenue scales with throughput per watt. --- From Nvidia YT channel

Rohan Paul

165,770 views • 6 months ago

🔥 NVIDIA GTC 2026: The Vera Rubin platform just dropped, introducing 10x inference throughput per watt, at 1/10th the cost per token vs. Blackwell. Jensen Huang called it: every company will now run a "token factory." Here's the thing: Aethir has been running distributed token factories since day one, with nearly 440,000+ GPU containers across 94 countries, delivering 1.5B+ compute hours to enterprises, powered by ATH. When the world's most powerful AI chips land, you need a cloud that can actually deploy them at scale, without the centralized cloud limitations and bottlenecks. That's Aethir’s advantage over centralized compute providers. ⚡

🔥 NVIDIA GTC 2026: The Vera Rubin platform just dropped, introducing 10x inference throughput per watt, at 1/10th the cost per token vs. Blackwell. Jensen Huang called it: every company will now run a "token factory." Here's the thing: Aethir has been running distributed token factories since day one, with nearly 440,000+ GPU containers across 94 countries, delivering 1.5B+ compute hours to enterprises, powered by ATH. When the world's most powerful AI chips land, you need a cloud that can actually deploy them at scale, without the centralized cloud limitations and bottlenecks. That's Aethir’s advantage over centralized compute providers. ⚡

Aethir Ecosystem

14,319 views • 4 months ago

Etched is deploying two new technologies in chip design: low-voltage inference and cluster-scale memory. CEO Gavin Uberti says they'll make their chips much more power-efficient and way, way faster than today's leading GPUs. He breaks it down: "We looked at a lot of early research directions, and we realized the key things that models need are way more compute and way faster memory." "If you think about inference, there are two key parts: prefill and decode. For prefill, it's a compute-bound problem. You need to have more FLOPS, more operations per second on each of your chips." "On our GPU, the bottleneck's actually thermals. You can't really run a GPU at more than around 50% of what it could theoretically do, or it'll melt." "So we're using a new technology today called low-voltage inference to try to solve this problem. You bring the voltage of the chip down dramatically, which allows us to have way, way better efficiency in terms of how much power is drawn per unit of math, and thus fit way way more flops onto the chip..." "For decode, it's all about bandwidth. Not just bandwidth on a chip, but bandwidth across your cluster. That's why we have this technology we call cluster-scale memory. It reduces the amount of time it takes to communicate from one chip to another dramatically." "As a result we can go use all of our HBM, HBM bandwidth, SRAM, SRAM bandwidth, and our scale-up domain as a single coherent pool. And that means if you're a user, you can go get much faster tokens per second, while still keeping your costs low."

Etched is deploying two new technologies in chip design: low-voltage inference and cluster-scale memory. CEO Gavin Uberti says they'll make their chips much more power-efficient and way, way faster than today's leading GPUs. He breaks it down: "We looked at a lot of early research directions, and we realized the key things that models need are way more compute and way faster memory." "If you think about inference, there are two key parts: prefill and decode. For prefill, it's a compute-bound problem. You need to have more FLOPS, more operations per second on each of your chips." "On our GPU, the bottleneck's actually thermals. You can't really run a GPU at more than around 50% of what it could theoretically do, or it'll melt." "So we're using a new technology today called low-voltage inference to try to solve this problem. You bring the voltage of the chip down dramatically, which allows us to have way, way better efficiency in terms of how much power is drawn per unit of math, and thus fit way way more flops onto the chip..." "For decode, it's all about bandwidth. Not just bandwidth on a chip, but bandwidth across your cluster. That's why we have this technology we call cluster-scale memory. It reduces the amount of time it takes to communicate from one chip to another dramatically." "As a result we can go use all of our HBM, HBM bandwidth, SRAM, SRAM bandwidth, and our scale-up domain as a single coherent pool. And that means if you're a user, you can go get much faster tokens per second, while still keeping your costs low."

TBPN

20,404 views • 24 days ago

🤝 Alphabet and NVIDIA are expanding their decade-long partnership to advance agentic AI, robotics, drug discovery, and more. This involves deep co-engineering with integrated platforms, open-source frameworks, and managed services. ✅ Google Cloud is one of the first to bring the NVIDIA Blackwell platform to the cloud. ✅ Google Distributed Cloud using NVIDIA Confidential Computing on #NVIDIABlackwell for enterprises to run Google Gemini on-premises. ✅ Integration of the NVIDIA AI platform across Vertex AI, Cluster Director and Google Kubernetes Engine. ✅ Bringing the NVIDIA Nemotron family of open models to Vertex AI Model Garden. Watch the full video to learn more ➡️

🤝 Alphabet and NVIDIA are expanding their decade-long partnership to advance agentic AI, robotics, drug discovery, and more. This involves deep co-engineering with integrated platforms, open-source frameworks, and managed services. ✅ Google Cloud is one of the first to bring the NVIDIA Blackwell platform to the cloud. ✅ Google Distributed Cloud using NVIDIA Confidential Computing on #NVIDIABlackwell for enterprises to run Google Gemini on-premises. ✅ Integration of the NVIDIA AI platform across Vertex AI, Cluster Director and Google Kubernetes Engine. ✅ Bringing the NVIDIA Nemotron family of open models to Vertex AI Model Garden. Watch the full video to learn more ➡️

NVIDIA AI Infrastructure

268,567 views • 6 months ago

✨ Meet our new open family of models: NVIDIA Nemotron 3 Open in weights, data, tools, and training, Nemotron 3 is built for multi-agent apps and features: • An efficient hybrid Mamba‑Transformer MoE architecture • 1M token context for long-term memory and improved reasoning • Multi‑environment reinforcement learning via NeMo Gym for advanced skill adaptation Plus NVFP4 pre-training, latent MoE, 1T tokens of data, and more. 📗Read the details in our tech blog: 🤗 Try the model on Hugging Face:

✨ Meet our new open family of models: NVIDIA Nemotron 3 Open in weights, data, tools, and training, Nemotron 3 is built for multi-agent apps and features: • An efficient hybrid Mamba‑Transformer MoE architecture • 1M token context for long-term memory and improved reasoning • Multi‑environment reinforcement learning via NeMo Gym for advanced skill adaptation Plus NVFP4 pre-training, latent MoE, 1T tokens of data, and more. 📗Read the details in our tech blog: 🤗 Try the model on Hugging Face:

NVIDIA AI Developer

68,886 views • 7 months ago

We're excited to back Sail Research, the inference platform built for long-horizon, asynchronous agents. The next wave of AI isn't a person typing into a box. It's agents running for minutes or hours, burning orders of magnitude more tokens per task. At that scale, cost and capacity beat raw latency, and teams are already hitting budget caps. Neil Movva and Samir Menon are solving this full-stack, from CUDA kernels to the API. Hear Neil and Aditya Naganath on why inference is being rebuilt for the age of agents.

We're excited to back Sail Research, the inference platform built for long-horizon, asynchronous agents. The next wave of AI isn't a person typing into a box. It's agents running for minutes or hours, burning orders of magnitude more tokens per task. At that scale, cost and capacity beat raw latency, and teams are already hitting budget caps. Neil Movva and Samir Menon are solving this full-stack, from CUDA kernels to the API. Hear Neil and Aditya Naganath on why inference is being rebuilt for the age of agents.

Kleiner Perkins

17,095 views • 29 days ago

Frontier models that use reasoning to "think" during inference are generating 5X more AI tokens per year. ✨ "Inference is now a thinking process. And in order to teach AI how to think, reinforcement learning and very significant computation was introduced into post-training," said NVIDIA CEO Jensen Huang in a recent keynote at #CES2026. Reinforcement learning is increasing computation demands across all AI scaling laws: pre-training, post-training, and test-time scaling. Learn more about reinforcement learning and AI scaling ➡️

Frontier models that use reasoning to "think" during inference are generating 5X more AI tokens per year. ✨ "Inference is now a thinking process. And in order to teach AI how to think, reinforcement learning and very significant computation was introduced into post-training," said NVIDIA CEO Jensen Huang in a recent keynote at #CES2026. Reinforcement learning is increasing computation demands across all AI scaling laws: pre-training, post-training, and test-time scaling. Learn more about reinforcement learning and AI scaling ➡️

NVIDIA AI Infrastructure

13,329 views • 6 months ago

NVIDIA sent us 2 DGX Sparks. For a while we wondered what we would do with them. The memory bandwidth is 273GB/s making it 3x slower than an M3 Ultra (819GB/s) for batch_size=1 inference. But it has 4x more FLOPS (100 TFLOPS compared to 26 TFLOPS). So we thought, what if we could combine the DGX Spark & M3 Ultra, and make use of both the massive compute on the DGX Spark and the massive memory-bandwidth on the M3 Ultra. We came up with a way to split inference across both devices and achieve a speedup of up to 4x for long prompts compared to the M3 Ultra on its own. Full details in the blog post linked below.

NVIDIA sent us 2 DGX Sparks. For a while we wondered what we would do with them. The memory bandwidth is 273GB/s making it 3x slower than an M3 Ultra (819GB/s) for batch_size=1 inference. But it has 4x more FLOPS (100 TFLOPS compared to 26 TFLOPS). So we thought, what if we could combine the DGX Spark & M3 Ultra, and make use of both the massive compute on the DGX Spark and the massive memory-bandwidth on the M3 Ultra. We came up with a way to split inference across both devices and achieve a speedup of up to 4x for long prompts compared to the M3 Ultra on its own. Full details in the blog post linked below.

Alex Cheema

281,225 views • 9 months ago

Jensen Huang just doubled NVIDIA's demand forecast to $1 Trillion through 2027 🤯 Then spent two hours explaining why that number is conservative… Here's everything today from GTC: - NemoClaw: NVIDIA's open-source enterprise AI agent stack built around OpenClaw. Jensen called OpenClaw "the operating system for personal AI" and said every company needs a strategy for it. - Space-1: NVIDIA is putting Vera Rubin data centers in orbit. Not a concept. An actual system being designed for space deployment right now. - DLSS 5: 3D-guided neural rendering that blends raw graphics with generative AI. Jensen called it the future of real-time rendering. - AWS: Deploying 1 million+ NVIDIA GPUs starting this year. Azure was the first hyperscaler to power up Vera Rubin. - Vera Rubin: NVIDIA's next-gen AI supercomputer. 10x more performance per watt than Blackwell, 700 million tokens per second, shipping later this year. - Groq 3 LPU: First chip from NVIDIA's $20B Groq acquisition. A purpose-built inference accelerator that ships Q3. NVIDIA now owns training AND inference. -Feynman: The architecture after Rubin, coming 2028. New GPU, new LPU, new CPU. NVIDIA is on a 12-month chip cadence and the treadmill never stops. - Autonomous driving: BYD, Hyundai, Nissan, and Geely building Level 4 vehicles on NVIDIA. Uber deploying NVIDIA-powered robotaxis across 28 cities by 2028. The man doubled his demand forecast to a trillion dollars, announced data centers in space, and closed the show with a robot singing country music. This is NVIDIA's world. Everyone else is just renting compute in it.

Jensen Huang just doubled NVIDIA's demand forecast to $1 Trillion through 2027 🤯 Then spent two hours explaining why that number is conservative… Here's everything today from GTC: - NemoClaw: NVIDIA's open-source enterprise AI agent stack built around OpenClaw. Jensen called OpenClaw "the operating system for personal AI" and said every company needs a strategy for it. - Space-1: NVIDIA is putting Vera Rubin data centers in orbit. Not a concept. An actual system being designed for space deployment right now. - DLSS 5: 3D-guided neural rendering that blends raw graphics with generative AI. Jensen called it the future of real-time rendering. - AWS: Deploying 1 million+ NVIDIA GPUs starting this year. Azure was the first hyperscaler to power up Vera Rubin. - Vera Rubin: NVIDIA's next-gen AI supercomputer. 10x more performance per watt than Blackwell, 700 million tokens per second, shipping later this year. - Groq 3 LPU: First chip from NVIDIA's $20B Groq acquisition. A purpose-built inference accelerator that ships Q3. NVIDIA now owns training AND inference. -Feynman: The architecture after Rubin, coming 2028. New GPU, new LPU, new CPU. NVIDIA is on a 12-month chip cadence and the treadmill never stops. - Autonomous driving: BYD, Hyundai, Nissan, and Geely building Level 4 vehicles on NVIDIA. Uber deploying NVIDIA-powered robotaxis across 28 cities by 2028. The man doubled his demand forecast to a trillion dollars, announced data centers in space, and closed the show with a robot singing country music. This is NVIDIA's world. Everyone else is just renting compute in it.

Josh Kale

45,875 views • 4 months ago

"Mac Minis for example are a very good fit" - Andrej Karpathy Andrej Karpathy shouted out my work on EXO Labs in his keynote at Y Combinator AI SUS! Here's the breakdown: Right now most AI workloads run in the cloud where requests from different users are continuously batched together. These workloads are FLOPS-bound and favors hardware with the best unit economics of $ per FLOP, i.e. enterprise GPUs. The personal computing revolution will shift these workloads to personal devices with lower batch sizes (mostly batch_size=1). batch_size=1 inference is memory-bound, because all of the model parameters need to be loaded into the GPU every time a token is generated. Apple Silicon with its Unified Memory architecture has a lot of memory and memory bandwidth per $ compared to other hardware: - M4 Pro Mac Mini, 24GB @ 273GB/s, $58.33/GB, $5.13/GB/s - H100, 80GB @ 3350GB/s, $625/GB, $14.93/GB/s The unit economics of Apple Silicon are becoming more compelling with every release of the Mac. The future of AI inference looks more like open weights models (OpenAI open weights model soonTM) run at low batch_size on personal devices.

"Mac Minis for example are a very good fit" - Andrej Karpathy Andrej Karpathy shouted out my work on EXO Labs in his keynote at Y Combinator AI SUS! Here's the breakdown: Right now most AI workloads run in the cloud where requests from different users are continuously batched together. These workloads are FLOPS-bound and favors hardware with the best unit economics of $ per FLOP, i.e. enterprise GPUs. The personal computing revolution will shift these workloads to personal devices with lower batch sizes (mostly batch_size=1). batch_size=1 inference is memory-bound, because all of the model parameters need to be loaded into the GPU every time a token is generated. Apple Silicon with its Unified Memory architecture has a lot of memory and memory bandwidth per $ compared to other hardware: - M4 Pro Mac Mini, 24GB @ 273GB/s, $58.33/GB, $5.13/GB/s - H100, 80GB @ 3350GB/s, $625/GB, $14.93/GB/s The unit economics of Apple Silicon are becoming more compelling with every release of the Mac. The future of AI inference looks more like open weights models (OpenAI open weights model soonTM) run at low batch_size on personal devices.

Alex Cheema

103,566 views • 1 year ago

Micron is going to $4,000 and once you understand what inference actually is, the number stops sounding crazy (Save this). Dylan Patel just said that by 2030, OpenAI and Anthropic alone will need over 100 gigawatts of compute combined and by 2040, we may not even be measuring AI infrastructure in gigawatts anymore. We may be talking about terawatts. Every single one of those gigawatts needs memory to function. Without it, the compute is worthless. Most people heard that and thought about Nvidia but they should be thinking about Micron. Every AI model generating a response has two phases. The first is prefill, processing your prompt which is compute-heavy and the second is decode generating each word one token at a time and that phase is almost entirely memory-bound, not compute-bound. During decode, the GPU's processing units sit idle more than 95% of the time, waiting for data to arrive from memory. Google confirmed it in a research paper that decode-phase bottlenecks are dominated by memory bandwidth and capacity not raw compute. The GPU is not the bottleneck but the memory feeding the GPU is. This matters because inference is now where all the money lives. Training a model happens once, Inference happens billions of times a day every ChatGPT response, every Claude output, every agentic workflow running in the background and every one of those token streams is a billing event tied directly to memory performance. Adding more GPUs does not fix this because GPUs are already underutilized in inference because they are sitting idle waiting on memory. Adding more memory bandwidth and capacity is what directly reduces token cost, reduces latency, and allows the same cluster to serve dramatically more users simultaneously. Longer context windows compound the problem further, a model running a 1 million token context window requires dramatically more memory per session than a 10,000 token window, and every new model generation pushes context longer. The market treats memory as a downstream beneficiary of Nvidia orders. The correct framework is the opposite, Micron is the upstream constraint on how much value every Nvidia GPU can actually generate at inference scale. Micron guided Q4 to $50 billion in revenue, has HBM4 ramping at twice the pace of the prior generation, and CEO Sanjay Mehrotra has said supply will not catch demand before the end of 2027. At 8x forward earnings on $112 projected FY2027 EPS, Micron is the most undervalued infrastructure company in the entire AI stack. Inference is memory. Memory is Micron and the inference ramp has barely started. Milk Road Pro members are already up massively on this position and we're just getting started. If you want the full breakdown of what we're buying and why, come join us for just a dollar using the link below!

Micron is going to $4,000 and once you understand what inference actually is, the number stops sounding crazy (Save this). Dylan Patel just said that by 2030, OpenAI and Anthropic alone will need over 100 gigawatts of compute combined and by 2040, we may not even be measuring AI infrastructure in gigawatts anymore. We may be talking about terawatts. Every single one of those gigawatts needs memory to function. Without it, the compute is worthless. Most people heard that and thought about Nvidia but they should be thinking about Micron. Every AI model generating a response has two phases. The first is prefill, processing your prompt which is compute-heavy and the second is decode generating each word one token at a time and that phase is almost entirely memory-bound, not compute-bound. During decode, the GPU's processing units sit idle more than 95% of the time, waiting for data to arrive from memory. Google confirmed it in a research paper that decode-phase bottlenecks are dominated by memory bandwidth and capacity not raw compute. The GPU is not the bottleneck but the memory feeding the GPU is. This matters because inference is now where all the money lives. Training a model happens once, Inference happens billions of times a day every ChatGPT response, every Claude output, every agentic workflow running in the background and every one of those token streams is a billing event tied directly to memory performance. Adding more GPUs does not fix this because GPUs are already underutilized in inference because they are sitting idle waiting on memory. Adding more memory bandwidth and capacity is what directly reduces token cost, reduces latency, and allows the same cluster to serve dramatically more users simultaneously. Longer context windows compound the problem further, a model running a 1 million token context window requires dramatically more memory per session than a 10,000 token window, and every new model generation pushes context longer. The market treats memory as a downstream beneficiary of Nvidia orders. The correct framework is the opposite, Micron is the upstream constraint on how much value every Nvidia GPU can actually generate at inference scale. Micron guided Q4 to $50 billion in revenue, has HBM4 ramping at twice the pace of the prior generation, and CEO Sanjay Mehrotra has said supply will not catch demand before the end of 2027. At 8x forward earnings on $112 projected FY2027 EPS, Micron is the most undervalued infrastructure company in the entire AI stack. Inference is memory. Memory is Micron and the inference ramp has barely started. Milk Road Pro members are already up massively on this position and we're just getting started. If you want the full breakdown of what we're buying and why, come join us for just a dollar using the link below!

Milk Road AI

128,522 views • 24 days ago

$AMD's heading to $5T MC LT| Lowest $/M tokens 🧵 The real reason why Institutions are FOMOing into AMD while other Semi stocks are underperforming ($NVDA $AVGO) Not Financial Advice! DYOR! Under Dr. Lisa Su’s leadership, AMD has transformed from a distant challenger into a formidable force in AI infrastructure, delivering the industry’s most compelling TCO story for high-volume inference. Her clear vision open ecosystems, aggressive annual roadmaps, rack-scale innovation, and relentless focus on tokens-per-dollar has positioned AMD’s Helios racks as the go-to solution for hyperscalers and AI natives struggling with exploding token costs, collapsing the cost down to $0.0003-$0.0005/M tokens. I will link various threads on this analysis to supply chain and wafer ratio if you are interested in understanding the full picture. In the last 3-4 months, explosive Agentic AI demand significantly increased Inference demand for Agentic AI models with 5-10 agents. If you are a listener of CNBC or Bloomberg, u should know enterprises and companies are complaining abt cost of token, and how it starts to spike up way too much to make sense. The fact that most data center today are run by $NVDA Chips, where the cost is way too high for Training or Inference. 1. Token cost Here are some quick comp, so u understand why $META OpenAI Anthropic $MSFT $AMZN Softbank $GOOGL and many more small to medium AI Natives are buying AMD CPUs and GPUs as much as they want, or pretty much AMD chips are sold out for the next 3-5 years. Inference (Cost per Million Tokens) ~$NVDA B200 / HGX: ~$0.02–$0.08 on optimized workloads (FP4/MXFP4, speculative decoding). Significant improvement over Hopper but still premium-priced. GB200 NVL72 rack-scale: $0.05–$0.25+ ~$AMD Helios Racks: $0.0003-$0.0005 per M tokens, dramatically lower than NVIDIA equivalents in owned infra. MI355X node-level: Up to 40% more tokens per dollar vs. competing solutions ( B200), driven by higher memory capacity (up to 288GB+ HBM), strong bandwidth, and lower acquisition costs. Training ~$NVDA Rubin Rack is estimated $0.7-$1.2/M Tokens ~$AMD Helios Rack is estimated $0.65-$1.0/M Tokens 2. Why Hyperscalers and AI Natives Are Choosing AMD Token consumption (especially Agentic) is outpacing even NVIDIA’s efficiency gains, making diversification mandatory for economic viability. Massive deals reflect this reality like $META, OpenAI, $MSFT, Softbank, $AMZN, Oracle, LumaAI, G42... Dr. Lisa Su’s Vision in Action: Since taking the helm, Su has driven AMD’s turnaround with disciplined execution, annual GPU cadence (MI300 → MI350 → MI400), full-stack software (ROCm 7), open ecosystems (UALink, OCP designs), and customer-centric rack-scale solutions like Helios. Her emphasis on “tokens per dollar” and TCO has turned AMD into the pragmatic choice for sustainable AI scaling. Power/Energy Efficiency: ~Helios Rack-level is estimated at 120kW-140kW with 50% more HBM4 where Inference and Training cost matter ~Rubin Rack-Level is estimated at 160kW-230kw AMD Helios shines in owned TCO, memory density, and energy flexibility at hyperscale. Cost to build 1GW data center 1GW Helios Rack full build is estimated $30-$35B 1GW Rubin Rack full build is estimated $45-$55B 3. Superior CPUs to pair with GPUs on massive scale 5-10-20GW Agentic AI. autonomous, multi-step workflows with orchestration, tool use, parallel agents, data movement, and enterprise integration has dramatically increased the importance of strong host CPUs alongside GPUs. This shifts the CPU-to-GPU ratio higher and makes balanced systems critical toward 1:1 to 5:1 as enterprises testing more than 5-10 agents. AMD EPYC Venice excels ~Leadership core density (up to 256 Zen 6 cores per socket) for running many agents in parallel, orchestration layers, and high-throughput control-plane tasks. ~Superior performance-per-core and power efficiency ( up to 2.1x higher perf/core and 2.26x better SPECpower vs. NVIDIA Grace in benchmarks). ~Tight integration in Helios: One Venice CPU + multiple MI450 GPUs per node, enabling efficient data feeding to GPUs ("zero-copy"), parallel execution, and full rack utilization for complex agentic loops. Hyperscalers (Meta, Microsoft, Amazon, Google, Softbank) and AI natives (OpenAI, Anthropic...) are adopting high-core EPYC at scale specifically for these agentic demands, as CPUs now handle a larger share of non-model work (orchestration, policy enforcement, tool calls). This complements AMD’s lower-cost GPUs for overall TCO wins. Conclusion: NVIDIA’s Vera Rubin cannot compete with a 2 years old EPYC Turin, but AMD under Dr. Lisa Su has engineered the lowest cost-per-million-tokens, highly competitive energy-efficient solutions, and superior CPU orchestration for agentic AI at scale with Helios. Dr. Su has championed this shift since at least 2023, foreseeing the rise of agentic workflows that demand far more orchestration, parallel agents, and balanced compute well before the industry fully embraced it. Her long-term vision of AI moving from simple prompts to always-on, multi-agent systems has driven AMD’s investments in high-core EPYC CPUs and integrated rack-scale solutions, perfectly positioning the company for today’s realities. Hyperscalers and AI natives effectively have no choice but to buy more AMD system for Agentic AI as leadership in economical, power-aware, high-volume internal + agentic use. However, due to supply constraints where Supply is far behind Demand, this makes multi-vendor reality along with in-house chips drive faster industry progress, lower overall costs, and better sustainability. Not Financial Advice! DYOR! Video source: Microsoft Build 2026

$AMD's heading to $5T MC LT| Lowest $/M tokens 🧵 The real reason why Institutions are FOMOing into AMD while other Semi stocks are underperforming ($NVDA $AVGO) Not Financial Advice! DYOR! Under Dr. Lisa Su’s leadership, AMD has transformed from a distant challenger into a formidable force in AI infrastructure, delivering the industry’s most compelling TCO story for high-volume inference. Her clear vision open ecosystems, aggressive annual roadmaps, rack-scale innovation, and relentless focus on tokens-per-dollar has positioned AMD’s Helios racks as the go-to solution for hyperscalers and AI natives struggling with exploding token costs, collapsing the cost down to $0.0003-$0.0005/M tokens. I will link various threads on this analysis to supply chain and wafer ratio if you are interested in understanding the full picture. In the last 3-4 months, explosive Agentic AI demand significantly increased Inference demand for Agentic AI models with 5-10 agents. If you are a listener of CNBC or Bloomberg, u should know enterprises and companies are complaining abt cost of token, and how it starts to spike up way too much to make sense. The fact that most data center today are run by $NVDA Chips, where the cost is way too high for Training or Inference. 1. Token cost Here are some quick comp, so u understand why $META OpenAI Anthropic $MSFT $AMZN Softbank $GOOGL and many more small to medium AI Natives are buying AMD CPUs and GPUs as much as they want, or pretty much AMD chips are sold out for the next 3-5 years. Inference (Cost per Million Tokens) ~$NVDA B200 / HGX: ~$0.02–$0.08 on optimized workloads (FP4/MXFP4, speculative decoding). Significant improvement over Hopper but still premium-priced. GB200 NVL72 rack-scale: $0.05–$0.25+ ~$AMD Helios Racks: $0.0003-$0.0005 per M tokens, dramatically lower than NVIDIA equivalents in owned infra. MI355X node-level: Up to 40% more tokens per dollar vs. competing solutions ( B200), driven by higher memory capacity (up to 288GB+ HBM), strong bandwidth, and lower acquisition costs. Training ~$NVDA Rubin Rack is estimated $0.7-$1.2/M Tokens ~$AMD Helios Rack is estimated $0.65-$1.0/M Tokens 2. Why Hyperscalers and AI Natives Are Choosing AMD Token consumption (especially Agentic) is outpacing even NVIDIA’s efficiency gains, making diversification mandatory for economic viability. Massive deals reflect this reality like $META, OpenAI, $MSFT, Softbank, $AMZN, Oracle, LumaAI, G42... Dr. Lisa Su’s Vision in Action: Since taking the helm, Su has driven AMD’s turnaround with disciplined execution, annual GPU cadence (MI300 → MI350 → MI400), full-stack software (ROCm 7), open ecosystems (UALink, OCP designs), and customer-centric rack-scale solutions like Helios. Her emphasis on “tokens per dollar” and TCO has turned AMD into the pragmatic choice for sustainable AI scaling. Power/Energy Efficiency: ~Helios Rack-level is estimated at 120kW-140kW with 50% more HBM4 where Inference and Training cost matter ~Rubin Rack-Level is estimated at 160kW-230kw AMD Helios shines in owned TCO, memory density, and energy flexibility at hyperscale. Cost to build 1GW data center 1GW Helios Rack full build is estimated $30-$35B 1GW Rubin Rack full build is estimated $45-$55B 3. Superior CPUs to pair with GPUs on massive scale 5-10-20GW Agentic AI. autonomous, multi-step workflows with orchestration, tool use, parallel agents, data movement, and enterprise integration has dramatically increased the importance of strong host CPUs alongside GPUs. This shifts the CPU-to-GPU ratio higher and makes balanced systems critical toward 1:1 to 5:1 as enterprises testing more than 5-10 agents. AMD EPYC Venice excels ~Leadership core density (up to 256 Zen 6 cores per socket) for running many agents in parallel, orchestration layers, and high-throughput control-plane tasks. ~Superior performance-per-core and power efficiency ( up to 2.1x higher perf/core and 2.26x better SPECpower vs. NVIDIA Grace in benchmarks). ~Tight integration in Helios: One Venice CPU + multiple MI450 GPUs per node, enabling efficient data feeding to GPUs ("zero-copy"), parallel execution, and full rack utilization for complex agentic loops. Hyperscalers (Meta, Microsoft, Amazon, Google, Softbank) and AI natives (OpenAI, Anthropic...) are adopting high-core EPYC at scale specifically for these agentic demands, as CPUs now handle a larger share of non-model work (orchestration, policy enforcement, tool calls). This complements AMD’s lower-cost GPUs for overall TCO wins. Conclusion: NVIDIA’s Vera Rubin cannot compete with a 2 years old EPYC Turin, but AMD under Dr. Lisa Su has engineered the lowest cost-per-million-tokens, highly competitive energy-efficient solutions, and superior CPU orchestration for agentic AI at scale with Helios. Dr. Su has championed this shift since at least 2023, foreseeing the rise of agentic workflows that demand far more orchestration, parallel agents, and balanced compute well before the industry fully embraced it. Her long-term vision of AI moving from simple prompts to always-on, multi-agent systems has driven AMD’s investments in high-core EPYC CPUs and integrated rack-scale solutions, perfectly positioning the company for today’s realities. Hyperscalers and AI natives effectively have no choice but to buy more AMD system for Agentic AI as leadership in economical, power-aware, high-volume internal + agentic use. However, due to supply constraints where Supply is far behind Demand, this makes multi-vendor reality along with in-house chips drive faster industry progress, lower overall costs, and better sustainability. Not Financial Advice! DYOR! Video source: Microsoft Build 2026

Mike

145,778 views • 1 month ago