Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

AI inference is memory-bound more often than compute-bound. High Bandwidth Compute (HBC) flips the traditional model by moving compute to the data, helping unlock dramatic gains in memory bandwidth and efficiency. Learn more:

Qualcomm

441,964 subscribers

11,901 Aufrufe • vor 2 Monaten •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

The first Qualcomm High Bandwidth Compute silicon is here, and it’s validated on real hardware. We’re bringing compute closer to memory to take on the memory wall and lay the foundation for Qualcomm Dragonfly AI250. Learn more:

The first Qualcomm High Bandwidth Compute silicon is here, and it’s validated on real hardware. We’re bringing compute closer to memory to take on the memory wall and lay the foundation for Qualcomm Dragonfly AI250. Learn more:

Qualcomm

16,464 Aufrufe • vor 6 Tagen

AI frameworks prioritize FLOPs and TOPS. But Samsung Semiconductor US President Paul Cho says the bottleneck has shifted. Memory bandwidth now determines AI inference performance and cost. He believes memory-centric compute could define AI’s next decade. Watch:

AI frameworks prioritize FLOPs and TOPS. But Samsung Semiconductor US President Paul Cho says the bottleneck has shifted. Memory bandwidth now determines AI inference performance and cost. He believes memory-centric compute could define AI’s next decade. Watch:

Six Five Media

19,685 Aufrufe • vor 1 Monat

Speaking of spot on. Gavin Gavin Baker was asked in early Jan'25 his prediction for best performing asset class of the year (as @jason references in latest The All-In Podcast): "I think the companies that make high bandwidth memory. Going back to "we're going to run out of compute," it's actually a pretty shocking stat: high bandwidth memory is a bigger part of NVIDIA's COGS on GPUs than Taiwan Semi is. And today there's two companies that can make it, SK Hynix and Micron. We'll see if Samsung gets their act together. But HBM, it's in NVIDIA GPUs, AMD GPUs, Amazon Trainiums. And particularly in a world of test-time compute and inference being so important, high bandwidth memory is arguably more important than it ever was. And it's been sold out for the last two years. So high bandwidth memory would be my pick."

Speaking of spot on. Gavin Gavin Baker was asked in early Jan'25 his prediction for best performing asset class of the year (as @jason references in latest The All-In Podcast): "I think the companies that make high bandwidth memory. Going back to "we're going to run out of compute," it's actually a pretty shocking stat: high bandwidth memory is a bigger part of NVIDIA's COGS on GPUs than Taiwan Semi is. And today there's two companies that can make it, SK Hynix and Micron. We'll see if Samsung gets their act together. But HBM, it's in NVIDIA GPUs, AMD GPUs, Amazon Trainiums. And particularly in a world of test-time compute and inference being so important, high bandwidth memory is arguably more important than it ever was. And it's been sold out for the last two years. So high bandwidth memory would be my pick."

Fireside Alpha

66,227 Aufrufe • vor 4 Monaten

Micron’s new DRAM-based Compute Express Links (CXLs) are high-speed devices that help interconnect processors, accelerators, memory, storage and IO while simultaneously increasing compute efficiency and scalability. Learn More → Mouser Electronics

Micron’s new DRAM-based Compute Express Links (CXLs) are high-speed devices that help interconnect processors, accelerators, memory, storage and IO while simultaneously increasing compute efficiency and scalability. Learn More → Mouser Electronics

Supplyframe

139,526 Aufrufe • vor 1 Jahr

$MU $SKHY $DRAM Holy Shit $AMD's Lisa Su just spent two full minutes at the 2026 AI keynote raving about HBM4 capacity as the thing that separates Helios from the competition. Not compute. Memory. "When you compare Helios to the competition, we're delivering 15% more compute, 50% more HBM4 memory capacity and memory bandwidth, and 50% more scale-out bandwidth." Notice which number is bigger. "What that means is that every Helios can deliver more performance for the largest models, more capacity for longer context, and the bandwidth to scale across thousands of racks.... More than 18,000 CDNA five GPU compute units, over 4,600 Zen 6 CPU cores, and 31 terabytes of HBM4 memory, all in a single rack." "That's what it takes to run agentic AI at scale. Today, I'm excited to announce that Helios is in full production. We have shipments on track to start at the end of the third quarter and ramping into the fourth quarter in the second half of the year. I can tell you, customer demand for Helios is extremely strong, we're extremely proud of the work that we have done across the leading AI labs to adopt Helios."

$MU $SKHY $DRAM Holy Shit $AMD's Lisa Su just spent two full minutes at the 2026 AI keynote raving about HBM4 capacity as the thing that separates Helios from the competition. Not compute. Memory. "When you compare Helios to the competition, we're delivering 15% more compute, 50% more HBM4 memory capacity and memory bandwidth, and 50% more scale-out bandwidth." Notice which number is bigger. "What that means is that every Helios can deliver more performance for the largest models, more capacity for longer context, and the bandwidth to scale across thousands of racks.... More than 18,000 CDNA five GPU compute units, over 4,600 Zen 6 CPU cores, and 31 terabytes of HBM4 memory, all in a single rack." "That's what it takes to run agentic AI at scale. Today, I'm excited to announce that Helios is in full production. We have shipments on track to start at the end of the third quarter and ramping into the fourth quarter in the second half of the year. I can tell you, customer demand for Helios is extremely strong, we're extremely proud of the work that we have done across the leading AI labs to adopt Helios."

Trade Whisperer

180,543 Aufrufe • vor 3 Monaten

Modal engineer who teaches Full-Stack Deep Learning explained everything an AI engineer needs to know about GPUs in 20 minutes - better than $800 CUDA workshops. memory bandwidth -> SM occupancy -> kernel fusion -> arithmetic intensity -> know when you're compute-bound vs memory-bound. That loop is how you stop overpaying for GPUs you're barely using. VRAM math + roofline model + Modal serverless GPUs + kernel profiling - that's the stack. Watch and save it, then check your own GPU utilization tonight.

Modal engineer who teaches Full-Stack Deep Learning explained everything an AI engineer needs to know about GPUs in 20 minutes - better than $800 CUDA workshops. memory bandwidth -> SM occupancy -> kernel fusion -> arithmetic intensity -> know when you're compute-bound vs memory-bound. That loop is how you stop overpaying for GPUs you're barely using. VRAM math + roofline model + Modal serverless GPUs + kernel profiling - that's the stack. Watch and save it, then check your own GPU utilization tonight.

h100envy

18,442 Aufrufe • vor 4 Monaten

Micron is going to $4,000 and once you understand what inference actually is, the number stops sounding crazy (Save this). Dylan Patel just said that by 2030, OpenAI and Anthropic alone will need over 100 gigawatts of compute combined and by 2040, we may not even be measuring AI infrastructure in gigawatts anymore. We may be talking about terawatts. Every single one of those gigawatts needs memory to function. Without it, the compute is worthless. Most people heard that and thought about Nvidia but they should be thinking about Micron. Every AI model generating a response has two phases. The first is prefill, processing your prompt which is compute-heavy and the second is decode generating each word one token at a time and that phase is almost entirely memory-bound, not compute-bound. During decode, the GPU's processing units sit idle more than 95% of the time, waiting for data to arrive from memory. Google confirmed it in a research paper that decode-phase bottlenecks are dominated by memory bandwidth and capacity not raw compute. The GPU is not the bottleneck but the memory feeding the GPU is. This matters because inference is now where all the money lives. Training a model happens once, Inference happens billions of times a day every ChatGPT response, every Claude output, every agentic workflow running in the background and every one of those token streams is a billing event tied directly to memory performance. Adding more GPUs does not fix this because GPUs are already underutilized in inference because they are sitting idle waiting on memory. Adding more memory bandwidth and capacity is what directly reduces token cost, reduces latency, and allows the same cluster to serve dramatically more users simultaneously. Longer context windows compound the problem further, a model running a 1 million token context window requires dramatically more memory per session than a 10,000 token window, and every new model generation pushes context longer. The market treats memory as a downstream beneficiary of Nvidia orders. The correct framework is the opposite, Micron is the upstream constraint on how much value every Nvidia GPU can actually generate at inference scale. Micron guided Q4 to $50 billion in revenue, has HBM4 ramping at twice the pace of the prior generation, and CEO Sanjay Mehrotra has said supply will not catch demand before the end of 2027. At 8x forward earnings on $112 projected FY2027 EPS, Micron is the most undervalued infrastructure company in the entire AI stack. Inference is memory. Memory is Micron and the inference ramp has barely started. Milk Road Pro members are already up massively on this position and we're just getting started. If you want the full breakdown of what we're buying and why, come join us for just a dollar using the link below!

Micron is going to $4,000 and once you understand what inference actually is, the number stops sounding crazy (Save this). Dylan Patel just said that by 2030, OpenAI and Anthropic alone will need over 100 gigawatts of compute combined and by 2040, we may not even be measuring AI infrastructure in gigawatts anymore. We may be talking about terawatts. Every single one of those gigawatts needs memory to function. Without it, the compute is worthless. Most people heard that and thought about Nvidia but they should be thinking about Micron. Every AI model generating a response has two phases. The first is prefill, processing your prompt which is compute-heavy and the second is decode generating each word one token at a time and that phase is almost entirely memory-bound, not compute-bound. During decode, the GPU's processing units sit idle more than 95% of the time, waiting for data to arrive from memory. Google confirmed it in a research paper that decode-phase bottlenecks are dominated by memory bandwidth and capacity not raw compute. The GPU is not the bottleneck but the memory feeding the GPU is. This matters because inference is now where all the money lives. Training a model happens once, Inference happens billions of times a day every ChatGPT response, every Claude output, every agentic workflow running in the background and every one of those token streams is a billing event tied directly to memory performance. Adding more GPUs does not fix this because GPUs are already underutilized in inference because they are sitting idle waiting on memory. Adding more memory bandwidth and capacity is what directly reduces token cost, reduces latency, and allows the same cluster to serve dramatically more users simultaneously. Longer context windows compound the problem further, a model running a 1 million token context window requires dramatically more memory per session than a 10,000 token window, and every new model generation pushes context longer. The market treats memory as a downstream beneficiary of Nvidia orders. The correct framework is the opposite, Micron is the upstream constraint on how much value every Nvidia GPU can actually generate at inference scale. Micron guided Q4 to $50 billion in revenue, has HBM4 ramping at twice the pace of the prior generation, and CEO Sanjay Mehrotra has said supply will not catch demand before the end of 2027. At 8x forward earnings on $112 projected FY2027 EPS, Micron is the most undervalued infrastructure company in the entire AI stack. Inference is memory. Memory is Micron and the inference ramp has barely started. Milk Road Pro members are already up massively on this position and we're just getting started. If you want the full breakdown of what we're buying and why, come join us for just a dollar using the link below!

Milk Road AI

130,756 Aufrufe • vor 4 Monaten

🎙️ NEW EPISODE: Qualcomm's HBC vs HBM, Dragonfly AI 250, and Winning on TCO Austin sits down with Qualcomm's Durga Malladi live in Maui to talk HBC (High Bandwidth Compute). HBC is Qualcomm's differentiated bet to win data center inference workloads. HBM shuttles data across wide buses to a separate accelerator. HBC puts compute directly on the stacked DRAM's logic die. - HBM is fast but power-hungry; HBC drops latency AND power per bit - 18x effective bandwidth on the AI 250 at the SAME 768 GB and SAME 160 kW - A trillion-parameter FP4 model on a single card - Foundry + memory-vendor relationships de-risk multi-megawatt supply at scale Chapters: 0:00 Introduction 2:49 The Memory Wall Problem 8:05 HBC Physical Architecture 9:33 HBC vs. Custom HBM 11:50 Silicon Proof Points Coming 12:08 Managing Thermals 13:34 Dragonfly AI 250 14:48 Trillion-Parameter Model on One Card 16:56 Tokenomics and TCO 18:26 Why TCO is Key 19:26 Manufacturing at Scale Get more of Austin and Vik daily, free! Sign up: Connect with Vik and Austin: Vik's Paid Substack: Austin's Paid Substack: Austin Lyons Vikram Sekar ✈️ Japan Qualcomm Snapdragon

🎙️ NEW EPISODE: Qualcomm's HBC vs HBM, Dragonfly AI 250, and Winning on TCO Austin sits down with Qualcomm's Durga Malladi live in Maui to talk HBC (High Bandwidth Compute). HBC is Qualcomm's differentiated bet to win data center inference workloads. HBM shuttles data across wide buses to a separate accelerator. HBC puts compute directly on the stacked DRAM's logic die. - HBM is fast but power-hungry; HBC drops latency AND power per bit - 18x effective bandwidth on the AI 250 at the SAME 768 GB and SAME 160 kW - A trillion-parameter FP4 model on a single card - Foundry + memory-vendor relationships de-risk multi-megawatt supply at scale Chapters: 0:00 Introduction 2:49 The Memory Wall Problem 8:05 HBC Physical Architecture 9:33 HBC vs. Custom HBM 11:50 Silicon Proof Points Coming 12:08 Managing Thermals 13:34 Dragonfly AI 250 14:48 Trillion-Parameter Model on One Card 16:56 Tokenomics and TCO 18:26 Why TCO is Key 19:26 Manufacturing at Scale Get more of Austin and Vik daily, free! Sign up: Connect with Vik and Austin: Vik's Paid Substack: Austin's Paid Substack: Austin Lyons Vikram Sekar ✈️ Japan Qualcomm Snapdragon

Semi Doped

22,380 Aufrufe • vor 1 Monat

Everyone’s still talking about GPUs. Meanwhile, memory is now the real constraint for AI at scale. From GTC, Patrick Moorhead & Daniel Newman are joined by Paul Cho & Indong Kim from Samsung Semiconductor US to discuss why bandwidth, not compute, is starting to define AI performance.

Everyone’s still talking about GPUs. Meanwhile, memory is now the real constraint for AI at scale. From GTC, Patrick Moorhead & Daniel Newman are joined by Paul Cho & Indong Kim from Samsung Semiconductor US to discuss why bandwidth, not compute, is starting to define AI performance.

Six Five Media

56,637 Aufrufe • vor 7 Monaten

The compute continuum is complete. Introducing Qualcomm Dragonfly. More soon at #QCOMInvestorDay.

The compute continuum is complete. Introducing Qualcomm Dragonfly. More soon at #QCOMInvestorDay.

Qualcomm

32,806 Aufrufe • vor 5 Monaten

Etched is deploying two new technologies in chip design: low-voltage inference and cluster-scale memory. CEO Gavin Uberti says they'll make their chips much more power-efficient and way, way faster than today's leading GPUs. He breaks it down: "We looked at a lot of early research directions, and we realized the key things that models need are way more compute and way faster memory." "If you think about inference, there are two key parts: prefill and decode. For prefill, it's a compute-bound problem. You need to have more FLOPS, more operations per second on each of your chips." "On our GPU, the bottleneck's actually thermals. You can't really run a GPU at more than around 50% of what it could theoretically do, or it'll melt." "So we're using a new technology today called low-voltage inference to try to solve this problem. You bring the voltage of the chip down dramatically, which allows us to have way, way better efficiency in terms of how much power is drawn per unit of math, and thus fit way way more flops onto the chip..." "For decode, it's all about bandwidth. Not just bandwidth on a chip, but bandwidth across your cluster. That's why we have this technology we call cluster-scale memory. It reduces the amount of time it takes to communicate from one chip to another dramatically." "As a result we can go use all of our HBM, HBM bandwidth, SRAM, SRAM bandwidth, and our scale-up domain as a single coherent pool. And that means if you're a user, you can go get much faster tokens per second, while still keeping your costs low."

Etched is deploying two new technologies in chip design: low-voltage inference and cluster-scale memory. CEO Gavin Uberti says they'll make their chips much more power-efficient and way, way faster than today's leading GPUs. He breaks it down: "We looked at a lot of early research directions, and we realized the key things that models need are way more compute and way faster memory." "If you think about inference, there are two key parts: prefill and decode. For prefill, it's a compute-bound problem. You need to have more FLOPS, more operations per second on each of your chips." "On our GPU, the bottleneck's actually thermals. You can't really run a GPU at more than around 50% of what it could theoretically do, or it'll melt." "So we're using a new technology today called low-voltage inference to try to solve this problem. You bring the voltage of the chip down dramatically, which allows us to have way, way better efficiency in terms of how much power is drawn per unit of math, and thus fit way way more flops onto the chip..." "For decode, it's all about bandwidth. Not just bandwidth on a chip, but bandwidth across your cluster. That's why we have this technology we call cluster-scale memory. It reduces the amount of time it takes to communicate from one chip to another dramatically." "As a result we can go use all of our HBM, HBM bandwidth, SRAM, SRAM bandwidth, and our scale-up domain as a single coherent pool. And that means if you're a user, you can go get much faster tokens per second, while still keeping your costs low."

TBPN

20,404 Aufrufe • vor 4 Monaten

$MU CEO Sanjay Mehrotra says AI is turning memory from a commodity into “strategic infrastructure.” AI cannot scale without more HBM DRAM and SSD capacity which is making memory far more valuable across the entire compute stack.

$MU CEO Sanjay Mehrotra says AI is turning memory from a commodity into “strategic infrastructure.” AI cannot scale without more HBM DRAM and SSD capacity which is making memory far more valuable across the entire compute stack.

Shay Boloor

194,460 Aufrufe • vor 2 Monaten

BG2. AI Compute Landscape ‘25-26, scaling intelligence, chip competition, memory tech, inference time reasoning & more. 👊💥Brad Gerstner Bill Gurley feat Dylan Patel

BG2. AI Compute Landscape ‘25-26, scaling intelligence, chip competition, memory tech, inference time reasoning & more. 👊💥Brad Gerstner Bill Gurley feat Dylan Patel

Bg2 Pod

471,290 Aufrufe • vor 1 Jahr

Power. Compute. Bandwidth. Trust. Learn what #AI needs to scale and why smarter networks fuel innovation from Cisco President & CPO Jeetu Patel and Content Creator David Bombal

Power. Compute. Bandwidth. Trust. Learn what #AI needs to scale and why smarter networks fuel innovation from Cisco President & CPO Jeetu Patel and Content Creator David Bombal

Cisco

11,783 Aufrufe • vor 1 Jahr

What comes after the OptimAI Lite & Edge Node? The answer: OptimAI Core Node, and it's going to be a game-changer. Built for the next evolution of Agentic AI, the Core Node brings AI + Edge Computing, real-time data processing, and reinforcement-grade human validation straight to your desktop. ⚙️ Tap into your CPU, GPU, RAM, and bandwidth to not just contribute — but to compute, validate, and train intelligent agents in real-time. 💡 The more contributors, the smarter the system. The more compute, the more rewards flow back to the community.

What comes after the OptimAI Lite & Edge Node? The answer: OptimAI Core Node, and it's going to be a game-changer. Built for the next evolution of Agentic AI, the Core Node brings AI + Edge Computing, real-time data processing, and reinforcement-grade human validation straight to your desktop. ⚙️ Tap into your CPU, GPU, RAM, and bandwidth to not just contribute — but to compute, validate, and train intelligent agents in real-time. 💡 The more contributors, the smarter the system. The more compute, the more rewards flow back to the community.

OptimAI Network

46,196 Aufrufe • vor 1 Jahr

Compute is stateless. Data isn't. The data layer shouldn't decide where your compute runs. Shelby, now in Private Beta, is S3-compatible object storage for AI, agents, and distributed compute. Aptos Labs' avery.apt 🇺🇸 with DevNTell

Compute is stateless. Data isn't. The data layer shouldn't decide where your compute runs. Shelby, now in Private Beta, is S3-compatible object storage for AI, agents, and distributed compute. Aptos Labs' avery.apt 🇺🇸 with DevNTell

Aptos

38,205 Aufrufe • vor 1 Monat

OpenAI's Noam Brown says that while AI model performance scales roughly equivalently with more training or inference compute, the cost of inference is on the order of 100 billion times cheaper

OpenAI's Noam Brown says that while AI model performance scales roughly equivalently with more training or inference compute, the cost of inference is on the order of 100 billion times cheaper

Tsarathustra

250,988 Aufrufe • vor 2 Jahren

🚨The launch of the day!! Xiaomi just unveiled the Xring O100 (玄戒O100) - first dedicated high-bandwidth AI accelerator for edge large models - Vertically stacks a 6nm NPU compute wafer with dual high-speed DRAM memory wafers (Wafer-on-Wafer 3D stacking). - Uses advanced Hybrid Bonding instead of conventional micro-bumps. •- Bonding pitch shrinks from ~50 μm all the way to 1.4 μm. - Creates 2.58 million high-speed physical pathways (TSV diameter just 0.7 μm). - Face-to-Face metal-layer direct connections open ultra-short “data wells” straight into the compute cores. - Paired with Xiaomi’s self-designed high-bandwidth matrix bus (HB-Matrix) and 14-core NPU, data flows between the NPUs and memory at extreme speed with dynamic topology switching (ring for prefill, broadcast for decode). - 1.22 TB/s near-memory bandwidth — 16× a flagship phone’s LPDDR5X - Up to 330 tokens/s edge inference (3B model in Xiaomi lab tests) - Designed for on-device LLMs for following Applications: - Smartphones - PCs - Robots - Xiaomi cars / intelligent driving systems - AI Cube prototype (paired with Xring O3 + D100), for models up to 120B parameters locally at up to 150W sustained 🗓️ The chip has completed silicon validation and is scheduled for commercial use in 2027. #Xiaomi #XRing #DRAM

🚨The launch of the day!! Xiaomi just unveiled the Xring O100 (玄戒O100) - first dedicated high-bandwidth AI accelerator for edge large models - Vertically stacks a 6nm NPU compute wafer with dual high-speed DRAM memory wafers (Wafer-on-Wafer 3D stacking). - Uses advanced Hybrid Bonding instead of conventional micro-bumps. •- Bonding pitch shrinks from ~50 μm all the way to 1.4 μm. - Creates 2.58 million high-speed physical pathways (TSV diameter just 0.7 μm). - Face-to-Face metal-layer direct connections open ultra-short “data wells” straight into the compute cores. - Paired with Xiaomi’s self-designed high-bandwidth matrix bus (HB-Matrix) and 14-core NPU, data flows between the NPUs and memory at extreme speed with dynamic topology switching (ring for prefill, broadcast for decode). - 1.22 TB/s near-memory bandwidth — 16× a flagship phone’s LPDDR5X - Up to 330 tokens/s edge inference (3B model in Xiaomi lab tests) - Designed for on-device LLMs for following Applications: - Smartphones - PCs - Robots - Xiaomi cars / intelligent driving systems - AI Cube prototype (paired with Xring O3 + D100), for models up to 120B parameters locally at up to 150W sustained 🗓️ The chip has completed silicon validation and is scheduled for commercial use in 2027. #Xiaomi #XRing #DRAM

Neil Shah

69,363 Aufrufe • vor 2 Monaten