Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

AI inference is memory-bound more often than compute-bound. High Bandwidth Compute (HBC) flips the traditional model by moving compute to the data, helping unlock dramatic gains in memory bandwidth and efficiency. Learn more:

Qualcomm

441,964 subscribers

11,901 views • 1 month ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

Speaking of spot on. Gavin Gavin Baker was asked in early Jan'25 his prediction for best performing asset class of the year (as @jason references in latest The All-In Podcast): "I think the companies that make high bandwidth memory. Going back to "we're going to run out of compute," it's actually a pretty shocking stat: high bandwidth memory is a bigger part of NVIDIA's COGS on GPUs than Taiwan Semi is. And today there's two companies that can make it, SK Hynix and Micron. We'll see if Samsung gets their act together. But HBM, it's in NVIDIA GPUs, AMD GPUs, Amazon Trainiums. And particularly in a world of test-time compute and inference being so important, high bandwidth memory is arguably more important than it ever was. And it's been sold out for the last two years. So high bandwidth memory would be my pick."

Speaking of spot on. Gavin Gavin Baker was asked in early Jan'25 his prediction for best performing asset class of the year (as @jason references in latest The All-In Podcast): "I think the companies that make high bandwidth memory. Going back to "we're going to run out of compute," it's actually a pretty shocking stat: high bandwidth memory is a bigger part of NVIDIA's COGS on GPUs than Taiwan Semi is. And today there's two companies that can make it, SK Hynix and Micron. We'll see if Samsung gets their act together. But HBM, it's in NVIDIA GPUs, AMD GPUs, Amazon Trainiums. And particularly in a world of test-time compute and inference being so important, high bandwidth memory is arguably more important than it ever was. And it's been sold out for the last two years. So high bandwidth memory would be my pick."

Fireside Alpha

66,227 views • 2 months ago

Micron’s new DRAM-based Compute Express Links (CXLs) are high-speed devices that help interconnect processors, accelerators, memory, storage and IO while simultaneously increasing compute efficiency and scalability. Learn More → Mouser Electronics

Micron’s new DRAM-based Compute Express Links (CXLs) are high-speed devices that help interconnect processors, accelerators, memory, storage and IO while simultaneously increasing compute efficiency and scalability. Learn More → Mouser Electronics

Supplyframe

139,526 views • 1 year ago

$MU $SKHY $DRAM Holy Shit $AMD's Lisa Su just spent two full minutes at the 2026 AI keynote raving about HBM4 capacity as the thing that separates Helios from the competition. Not compute. Memory. "When you compare Helios to the competition, we're delivering 15% more compute, 50% more HBM4 memory capacity and memory bandwidth, and 50% more scale-out bandwidth." Notice which number is bigger. "What that means is that every Helios can deliver more performance for the largest models, more capacity for longer context, and the bandwidth to scale across thousands of racks.... More than 18,000 CDNA five GPU compute units, over 4,600 Zen 6 CPU cores, and 31 terabytes of HBM4 memory, all in a single rack." "That's what it takes to run agentic AI at scale. Today, I'm excited to announce that Helios is in full production. We have shipments on track to start at the end of the third quarter and ramping into the fourth quarter in the second half of the year. I can tell you, customer demand for Helios is extremely strong, we're extremely proud of the work that we have done across the leading AI labs to adopt Helios."

$MU $SKHY $DRAM Holy Shit $AMD's Lisa Su just spent two full minutes at the 2026 AI keynote raving about HBM4 capacity as the thing that separates Helios from the competition. Not compute. Memory. "When you compare Helios to the competition, we're delivering 15% more compute, 50% more HBM4 memory capacity and memory bandwidth, and 50% more scale-out bandwidth." Notice which number is bigger. "What that means is that every Helios can deliver more performance for the largest models, more capacity for longer context, and the bandwidth to scale across thousands of racks.... More than 18,000 CDNA five GPU compute units, over 4,600 Zen 6 CPU cores, and 31 terabytes of HBM4 memory, all in a single rack." "That's what it takes to run agentic AI at scale. Today, I'm excited to announce that Helios is in full production. We have shipments on track to start at the end of the third quarter and ramping into the fourth quarter in the second half of the year. I can tell you, customer demand for Helios is extremely strong, we're extremely proud of the work that we have done across the leading AI labs to adopt Helios."

Trade Whisperer

180,513 views • 2 months ago

Modal engineer who teaches Full-Stack Deep Learning explained everything an AI engineer needs to know about GPUs in 20 minutes - better than $800 CUDA workshops. memory bandwidth -> SM occupancy -> kernel fusion -> arithmetic intensity -> know when you're compute-bound vs memory-bound. That loop is how you stop overpaying for GPUs you're barely using. VRAM math + roofline model + Modal serverless GPUs + kernel profiling - that's the stack. Watch and save it, then check your own GPU utilization tonight.

Modal engineer who teaches Full-Stack Deep Learning explained everything an AI engineer needs to know about GPUs in 20 minutes - better than $800 CUDA workshops. memory bandwidth -> SM occupancy -> kernel fusion -> arithmetic intensity -> know when you're compute-bound vs memory-bound. That loop is how you stop overpaying for GPUs you're barely using. VRAM math + roofline model + Modal serverless GPUs + kernel profiling - that's the stack. Watch and save it, then check your own GPU utilization tonight.

h100envy

18,442 views • 2 months ago

Micron is going to $4,000 and once you understand what inference actually is, the number stops sounding crazy (Save this). Dylan Patel just said that by 2030, OpenAI and Anthropic alone will need over 100 gigawatts of compute combined and by 2040, we may not even be measuring AI infrastructure in gigawatts anymore. We may be talking about terawatts. Every single one of those gigawatts needs memory to function. Without it, the compute is worthless. Most people heard that and thought about Nvidia but they should be thinking about Micron. Every AI model generating a response has two phases. The first is prefill, processing your prompt which is compute-heavy and the second is decode generating each word one token at a time and that phase is almost entirely memory-bound, not compute-bound. During decode, the GPU's processing units sit idle more than 95% of the time, waiting for data to arrive from memory. Google confirmed it in a research paper that decode-phase bottlenecks are dominated by memory bandwidth and capacity not raw compute. The GPU is not the bottleneck but the memory feeding the GPU is. This matters because inference is now where all the money lives. Training a model happens once, Inference happens billions of times a day every ChatGPT response, every Claude output, every agentic workflow running in the background and every one of those token streams is a billing event tied directly to memory performance. Adding more GPUs does not fix this because GPUs are already underutilized in inference because they are sitting idle waiting on memory. Adding more memory bandwidth and capacity is what directly reduces token cost, reduces latency, and allows the same cluster to serve dramatically more users simultaneously. Longer context windows compound the problem further, a model running a 1 million token context window requires dramatically more memory per session than a 10,000 token window, and every new model generation pushes context longer. The market treats memory as a downstream beneficiary of Nvidia orders. The correct framework is the opposite, Micron is the upstream constraint on how much value every Nvidia GPU can actually generate at inference scale. Micron guided Q4 to $50 billion in revenue, has HBM4 ramping at twice the pace of the prior generation, and CEO Sanjay Mehrotra has said supply will not catch demand before the end of 2027. At 8x forward earnings on $112 projected FY2027 EPS, Micron is the most undervalued infrastructure company in the entire AI stack. Inference is memory. Memory is Micron and the inference ramp has barely started. Milk Road Pro members are already up massively on this position and we're just getting started. If you want the full breakdown of what we're buying and why, come join us for just a dollar using the link below!

Micron is going to $4,000 and once you understand what inference actually is, the number stops sounding crazy (Save this). Dylan Patel just said that by 2030, OpenAI and Anthropic alone will need over 100 gigawatts of compute combined and by 2040, we may not even be measuring AI infrastructure in gigawatts anymore. We may be talking about terawatts. Every single one of those gigawatts needs memory to function. Without it, the compute is worthless. Most people heard that and thought about Nvidia but they should be thinking about Micron. Every AI model generating a response has two phases. The first is prefill, processing your prompt which is compute-heavy and the second is decode generating each word one token at a time and that phase is almost entirely memory-bound, not compute-bound. During decode, the GPU's processing units sit idle more than 95% of the time, waiting for data to arrive from memory. Google confirmed it in a research paper that decode-phase bottlenecks are dominated by memory bandwidth and capacity not raw compute. The GPU is not the bottleneck but the memory feeding the GPU is. This matters because inference is now where all the money lives. Training a model happens once, Inference happens billions of times a day every ChatGPT response, every Claude output, every agentic workflow running in the background and every one of those token streams is a billing event tied directly to memory performance. Adding more GPUs does not fix this because GPUs are already underutilized in inference because they are sitting idle waiting on memory. Adding more memory bandwidth and capacity is what directly reduces token cost, reduces latency, and allows the same cluster to serve dramatically more users simultaneously. Longer context windows compound the problem further, a model running a 1 million token context window requires dramatically more memory per session than a 10,000 token window, and every new model generation pushes context longer. The market treats memory as a downstream beneficiary of Nvidia orders. The correct framework is the opposite, Micron is the upstream constraint on how much value every Nvidia GPU can actually generate at inference scale. Micron guided Q4 to $50 billion in revenue, has HBM4 ramping at twice the pace of the prior generation, and CEO Sanjay Mehrotra has said supply will not catch demand before the end of 2027. At 8x forward earnings on $112 projected FY2027 EPS, Micron is the most undervalued infrastructure company in the entire AI stack. Inference is memory. Memory is Micron and the inference ramp has barely started. Milk Road Pro members are already up massively on this position and we're just getting started. If you want the full breakdown of what we're buying and why, come join us for just a dollar using the link below!

Milk Road AI

130,756 views • 2 months ago

The compute continuum is complete. Introducing Qualcomm Dragonfly. More soon at #QCOMInvestorDay.

The compute continuum is complete. Introducing Qualcomm Dragonfly. More soon at #QCOMInvestorDay.

Qualcomm

32,806 views • 3 months ago

Everyone’s still talking about GPUs. Meanwhile, memory is now the real constraint for AI at scale. From GTC, Patrick Moorhead & Daniel Newman are joined by Paul Cho & Indong Kim from Samsung Semiconductor US to discuss why bandwidth, not compute, is starting to define AI performance.

Everyone’s still talking about GPUs. Meanwhile, memory is now the real constraint for AI at scale. From GTC, Patrick Moorhead & Daniel Newman are joined by Paul Cho & Indong Kim from Samsung Semiconductor US to discuss why bandwidth, not compute, is starting to define AI performance.

Six Five Media

56,637 views • 6 months ago

$MU CEO Sanjay Mehrotra says AI is turning memory from a commodity into “strategic infrastructure.” AI cannot scale without more HBM DRAM and SSD capacity which is making memory far more valuable across the entire compute stack.

$MU CEO Sanjay Mehrotra says AI is turning memory from a commodity into “strategic infrastructure.” AI cannot scale without more HBM DRAM and SSD capacity which is making memory far more valuable across the entire compute stack.

Shay Boloor

194,460 views • 1 month ago

Etched is deploying two new technologies in chip design: low-voltage inference and cluster-scale memory. CEO Gavin Uberti says they'll make their chips much more power-efficient and way, way faster than today's leading GPUs. He breaks it down: "We looked at a lot of early research directions, and we realized the key things that models need are way more compute and way faster memory." "If you think about inference, there are two key parts: prefill and decode. For prefill, it's a compute-bound problem. You need to have more FLOPS, more operations per second on each of your chips." "On our GPU, the bottleneck's actually thermals. You can't really run a GPU at more than around 50% of what it could theoretically do, or it'll melt." "So we're using a new technology today called low-voltage inference to try to solve this problem. You bring the voltage of the chip down dramatically, which allows us to have way, way better efficiency in terms of how much power is drawn per unit of math, and thus fit way way more flops onto the chip..." "For decode, it's all about bandwidth. Not just bandwidth on a chip, but bandwidth across your cluster. That's why we have this technology we call cluster-scale memory. It reduces the amount of time it takes to communicate from one chip to another dramatically." "As a result we can go use all of our HBM, HBM bandwidth, SRAM, SRAM bandwidth, and our scale-up domain as a single coherent pool. And that means if you're a user, you can go get much faster tokens per second, while still keeping your costs low."

Etched is deploying two new technologies in chip design: low-voltage inference and cluster-scale memory. CEO Gavin Uberti says they'll make their chips much more power-efficient and way, way faster than today's leading GPUs. He breaks it down: "We looked at a lot of early research directions, and we realized the key things that models need are way more compute and way faster memory." "If you think about inference, there are two key parts: prefill and decode. For prefill, it's a compute-bound problem. You need to have more FLOPS, more operations per second on each of your chips." "On our GPU, the bottleneck's actually thermals. You can't really run a GPU at more than around 50% of what it could theoretically do, or it'll melt." "So we're using a new technology today called low-voltage inference to try to solve this problem. You bring the voltage of the chip down dramatically, which allows us to have way, way better efficiency in terms of how much power is drawn per unit of math, and thus fit way way more flops onto the chip..." "For decode, it's all about bandwidth. Not just bandwidth on a chip, but bandwidth across your cluster. That's why we have this technology we call cluster-scale memory. It reduces the amount of time it takes to communicate from one chip to another dramatically." "As a result we can go use all of our HBM, HBM bandwidth, SRAM, SRAM bandwidth, and our scale-up domain as a single coherent pool. And that means if you're a user, you can go get much faster tokens per second, while still keeping your costs low."

TBPN

20,404 views • 2 months ago

What comes after the OptimAI Lite & Edge Node? The answer: OptimAI Core Node, and it's going to be a game-changer. Built for the next evolution of Agentic AI, the Core Node brings AI + Edge Computing, real-time data processing, and reinforcement-grade human validation straight to your desktop. ⚙️ Tap into your CPU, GPU, RAM, and bandwidth to not just contribute — but to compute, validate, and train intelligent agents in real-time. 💡 The more contributors, the smarter the system. The more compute, the more rewards flow back to the community.

What comes after the OptimAI Lite & Edge Node? The answer: OptimAI Core Node, and it's going to be a game-changer. Built for the next evolution of Agentic AI, the Core Node brings AI + Edge Computing, real-time data processing, and reinforcement-grade human validation straight to your desktop. ⚙️ Tap into your CPU, GPU, RAM, and bandwidth to not just contribute — but to compute, validate, and train intelligent agents in real-time. 💡 The more contributors, the smarter the system. The more compute, the more rewards flow back to the community.

OptimAI Network

46,067 views • 1 year ago

OpenAI's Noam Brown says that while AI model performance scales roughly equivalently with more training or inference compute, the cost of inference is on the order of 100 billion times cheaper

OpenAI's Noam Brown says that while AI model performance scales roughly equivalently with more training or inference compute, the cost of inference is on the order of 100 billion times cheaper

Tsarathustra

250,942 views • 2 years ago

🚨The launch of the day!! Xiaomi just unveiled the Xring O100 (玄戒O100) - first dedicated high-bandwidth AI accelerator for edge large models - Vertically stacks a 6nm NPU compute wafer with dual high-speed DRAM memory wafers (Wafer-on-Wafer 3D stacking). - Uses advanced Hybrid Bonding instead of conventional micro-bumps. •- Bonding pitch shrinks from ~50 μm all the way to 1.4 μm. - Creates 2.58 million high-speed physical pathways (TSV diameter just 0.7 μm). - Face-to-Face metal-layer direct connections open ultra-short “data wells” straight into the compute cores. - Paired with Xiaomi’s self-designed high-bandwidth matrix bus (HB-Matrix) and 14-core NPU, data flows between the NPUs and memory at extreme speed with dynamic topology switching (ring for prefill, broadcast for decode). - 1.22 TB/s near-memory bandwidth — 16× a flagship phone’s LPDDR5X - Up to 330 tokens/s edge inference (3B model in Xiaomi lab tests) - Designed for on-device LLMs for following Applications: - Smartphones - PCs - Robots - Xiaomi cars / intelligent driving systems - AI Cube prototype (paired with Xring O3 + D100), for models up to 120B parameters locally at up to 150W sustained 🗓️ The chip has completed silicon validation and is scheduled for commercial use in 2027. #Xiaomi #XRing #DRAM

🚨The launch of the day!! Xiaomi just unveiled the Xring O100 (玄戒O100) - first dedicated high-bandwidth AI accelerator for edge large models - Vertically stacks a 6nm NPU compute wafer with dual high-speed DRAM memory wafers (Wafer-on-Wafer 3D stacking). - Uses advanced Hybrid Bonding instead of conventional micro-bumps. •- Bonding pitch shrinks from ~50 μm all the way to 1.4 μm. - Creates 2.58 million high-speed physical pathways (TSV diameter just 0.7 μm). - Face-to-Face metal-layer direct connections open ultra-short “data wells” straight into the compute cores. - Paired with Xiaomi’s self-designed high-bandwidth matrix bus (HB-Matrix) and 14-core NPU, data flows between the NPUs and memory at extreme speed with dynamic topology switching (ring for prefill, broadcast for decode). - 1.22 TB/s near-memory bandwidth — 16× a flagship phone’s LPDDR5X - Up to 330 tokens/s edge inference (3B model in Xiaomi lab tests) - Designed for on-device LLMs for following Applications: - Smartphones - PCs - Robots - Xiaomi cars / intelligent driving systems - AI Cube prototype (paired with Xring O3 + D100), for models up to 120B parameters locally at up to 150W sustained 🗓️ The chip has completed silicon validation and is scheduled for commercial use in 2027. #Xiaomi #XRing #DRAM

Neil Shah

69,363 views • 1 month ago

Olam Labs CEO sensho says the only two bottlenecks left to AGI and ASI are compute and data. "For AGI and ASI to come there are only two bottlenecks right now. We have no bottleneck with architecture or agents. It's data and compute." "Compute is the biggest bottleneck." "We're gonna have 10x more compute coming online in the next two years. If you're scaling law-pilled and if you're -pilled in all the ways that these AI labs are, then data is the actual bottleneck."

Olam Labs CEO sensho says the only two bottlenecks left to AGI and ASI are compute and data. "For AGI and ASI to come there are only two bottlenecks right now. We have no bottleneck with architecture or agents. It's data and compute." "Compute is the biggest bottleneck." "We're gonna have 10x more compute coming online in the next two years. If you're scaling law-pilled and if you're -pilled in all the ways that these AI labs are, then data is the actual bottleneck."

TBPN

44,160 views • 13 days ago

Compute is the backbone of the AI-driven future. OptimAI Compute Engine would enable scalable, high-performance workloads across image and video models, supporting: • Text-to-image generation • Image editing and inpainting • 2K / 4K / 8K super-resolution • Brand-aligned visual synthesis • OCR and image intelligence • Video generation and motion synthesis • Frame interpolation and enhancement • Multimodal model inference Distributed compute built for faster inference, shorter training cycles, and production-grade AI execution.

OptimAI Network

18,878 views • 7 months ago

Moonshot AI limiting Kimi K3 subscriptions because of compute constraints is a massive signal because the 2.8 trillion parameter model is already trained and the bottleneck is now emerging as usage scales. Serving that demand requires a ton more accelerators, enormous memory capacity and faster networking to keep data moving efficiently across the cluster. So as AI spending shifts toward serving millions or billions of daily requests then memory and networking become critical bottlenecks giving $MU, $SKHY, $AVGO, $CRDO, $ALAB and $MRVL direct exposure to inference at scale.

Moonshot AI limiting Kimi K3 subscriptions because of compute constraints is a massive signal because the 2.8 trillion parameter model is already trained and the bottleneck is now emerging as usage scales. Serving that demand requires a ton more accelerators, enormous memory capacity and faster networking to keep data moving efficiently across the cluster. So as AI spending shifts toward serving millions or billions of daily requests then memory and networking become critical bottlenecks giving $MU, $SKHY, $AVGO, $CRDO, $ALAB and $MRVL direct exposure to inference at scale.

Shay Boloor

123,791 views • 2 months ago

Teaming up with SanDisk to build the next standard for AI inference. As a full-stack AI memory creator, SK hynix is launching the global standardization of #HBF with Sandisk & Open Compute Project (OCP). HBF is the new memory layer bridging the gap between #HBM and #SSD. #SKhynix

Teaming up with SanDisk to build the next standard for AI inference. As a full-stack AI memory creator, SK hynix is launching the global standardization of #HBF with Sandisk & Open Compute Project (OCP). HBF is the new memory layer bridging the gap between #HBM and #SSD. #SKhynix

SK hynix

194,727 views • 7 months ago

Is compute scarcity masking the real economics of AI? interesting point from Peter Fenton: open-weight models are still complex to run well. For now, the big labs have the bundle: model + compute + reliability + access... and thus pricing power. But if compute supply loosens and open model tooling gets easier, routine AI work can more easily move to cheaper/open models.

Is compute scarcity masking the real economics of AI? interesting point from Peter Fenton: open-weight models are still complex to run well. For now, the big labs have the bundle: model + compute + reliability + access... and thus pricing power. But if compute supply loosens and open model tooling gets easier, routine AI work can more easily move to cheaper/open models.

Deirdre Bosa

50,558 views • 2 months ago

NVIDIA sent us 2 DGX Sparks. For a while we wondered what we would do with them. The memory bandwidth is 273GB/s making it 3x slower than an M3 Ultra (819GB/s) for batch_size=1 inference. But it has 4x more FLOPS (100 TFLOPS compared to 26 TFLOPS). So we thought, what if we could combine the DGX Spark & M3 Ultra, and make use of both the massive compute on the DGX Spark and the massive memory-bandwidth on the M3 Ultra. We came up with a way to split inference across both devices and achieve a speedup of up to 4x for long prompts compared to the M3 Ultra on its own. Full details in the blog post linked below.

NVIDIA sent us 2 DGX Sparks. For a while we wondered what we would do with them. The memory bandwidth is 273GB/s making it 3x slower than an M3 Ultra (819GB/s) for batch_size=1 inference. But it has 4x more FLOPS (100 TFLOPS compared to 26 TFLOPS). So we thought, what if we could combine the DGX Spark & M3 Ultra, and make use of both the massive compute on the DGX Spark and the massive memory-bandwidth on the M3 Ultra. We came up with a way to split inference across both devices and achieve a speedup of up to 4x for long prompts compared to the M3 Ultra on its own. Full details in the blog post linked below.

Alex Cheema

281,225 views • 11 months ago