Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

Discovered Materials' Akash Ramdas argues the next chip scaling frontier isn't smaller transistors, it's how many you can stack on top of each other: HBM is already 12 high DRAM going to 16 high "We're starting to do this thing where we're taking a logic chip and then putting... memory on top of it. That's where materials can make a difference, because now you're actually thermally limited. The thing that's preventing your scaling is no longer how small is your transistor, but how many transistors can you put on top of each other?" "The high bandwidth memory you'd have seen is a stack of dynamic random access memory. It's a 12-high dynamic random access memory stack right now. One of the roadmaps is to make this even higher, 16-high, and this is how you deal with memory bottlenecks in inference or training." "The next stage of that is putting that HBM on top of a logic die. Both these technologies are going to be partially enabled by new materials."show more

MTS

504,293 subscribers

17,520 Aufrufe • vor 1 Monat •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

2 Kommentare

Profilbild von dev flows

dev flowsvor 1 Monat

boron arsinide

Profilbild von elian

elianvor 1 Monat

16 high DRAM and the whole industry still argues about node names like that's the constraint

Ähnliche Videos

SK Hynix's Hoshik Kim admits high bandwidth memory "is not the answer" to the memory-wall problem, and SK is working on different solutions to solve the bottleneck. "No, actually, I kind of agree with Pat. So HBM is not the answer." [SK Hynix is admitting that HBM is a bad memory?] "Nothing is a bad memory. It's a good memory, but it's not the final answer. So actually, HBM will not solve the memory-wall problem. Memory-wall problem is an inherent AI problem, which you cannot avoid." "So we try our best to provide higher bandwidth with this memory, with HBM technology even though it has some limitation, as Pat already mentioned, with thermal, IOs, cost, a lot of stuff. "But that's the only thing we got right now to help ease this memory-wall problem, memory-bottleneck problem. But at the same time, we are working in different directions to solve this memory-wall problem with different solutions."

SK Hynix's Hoshik Kim admits high bandwidth memory "is not the answer" to the memory-wall problem, and SK is working on different solutions to solve the bottleneck. "No, actually, I kind of agree with Pat. So HBM is not the answer." [SK Hynix is admitting that HBM is a bad memory?] "Nothing is a bad memory. It's a good memory, but it's not the final answer. So actually, HBM will not solve the memory-wall problem. Memory-wall problem is an inherent AI problem, which you cannot avoid." "So we try our best to provide higher bandwidth with this memory, with HBM technology even though it has some limitation, as Pat already mentioned, with thermal, IOs, cost, a lot of stuff. "But that's the only thing we got right now to help ease this memory-wall problem, memory-bottleneck problem. But at the same time, we are working in different directions to solve this memory-wall problem with different solutions."

Fireside Alpha

140,571 Aufrufe • vor 1 Monat

Micron's Sanjay Mehrotra on memory: "Memory is no longer a component in a system, memory is the strategic infrastructure for AI. It is no longer a commodity, it is a high value. Without memory, you cannot make AI smarter, cannot make it faster, you cannot scale up AI...AI is driving a whole hierarchy of memory requirements from HBM to DRAM to SSDs." Via CNBC $MU $DRAM $EWY $SNDK

Micron's Sanjay Mehrotra on memory: "Memory is no longer a component in a system, memory is the strategic infrastructure for AI. It is no longer a commodity, it is a high value. Without memory, you cannot make AI smarter, cannot make it faster, you cannot scale up AI...AI is driving a whole hierarchy of memory requirements from HBM to DRAM to SSDs." Via CNBC $MU $DRAM $EWY $SNDK

P Equity Research 📰

50,226 Aufrufe • vor 1 Monat

Speaking of spot on. Gavin Gavin Baker was asked in early Jan'25 his prediction for best performing asset class of the year (as @jason references in latest The All-In Podcast): "I think the companies that make high bandwidth memory. Going back to "we're going to run out of compute," it's actually a pretty shocking stat: high bandwidth memory is a bigger part of NVIDIA's COGS on GPUs than Taiwan Semi is. And today there's two companies that can make it, SK Hynix and Micron. We'll see if Samsung gets their act together. But HBM, it's in NVIDIA GPUs, AMD GPUs, Amazon Trainiums. And particularly in a world of test-time compute and inference being so important, high bandwidth memory is arguably more important than it ever was. And it's been sold out for the last two years. So high bandwidth memory would be my pick."

Speaking of spot on. Gavin Gavin Baker was asked in early Jan'25 his prediction for best performing asset class of the year (as @jason references in latest The All-In Podcast): "I think the companies that make high bandwidth memory. Going back to "we're going to run out of compute," it's actually a pretty shocking stat: high bandwidth memory is a bigger part of NVIDIA's COGS on GPUs than Taiwan Semi is. And today there's two companies that can make it, SK Hynix and Micron. We'll see if Samsung gets their act together. But HBM, it's in NVIDIA GPUs, AMD GPUs, Amazon Trainiums. And particularly in a world of test-time compute and inference being so important, high bandwidth memory is arguably more important than it ever was. And it's been sold out for the last two years. So high bandwidth memory would be my pick."

Fireside Alpha

66,227 Aufrufe • vor 2 Monaten

HBM creator Kim Jung-ho reveals his lab's tHBM flipped stack publicly for the first time saying Samsung's zHBM with memory stacked on a hot GPU simply melts and is likely not the right fundamental direction "So the idea is, let's just put the HBM on top of the GPU. Then you never go outside the building, you only ride the elevator, and the speed jumps." "That's the structure Samsung is now proposing as zHBM. The z is, from the x, y, z axes we learned back in school, the vertical direction, so it means stacked in the z direction." "But unfortunately, I don't see that as the fundamental direction, because if you stack HBM on top of the GPU, the GPU gets hot and the memory melts." "So I suspect it's for very simple computation, the kind of use people call an LPU or an NPU. They showed it in passing as a fun technology, and the really important technology doesn't get shown at places like that." "So my lab researches tHBM. I believe today is the first time we're saying this publicly. We flip it, stack the HBM, and put the GPU on top of that."

HBM creator Kim Jung-ho reveals his lab's tHBM flipped stack publicly for the first time saying Samsung's zHBM with memory stacked on a hot GPU simply melts and is likely not the right fundamental direction "So the idea is, let's just put the HBM on top of the GPU. Then you never go outside the building, you only ride the elevator, and the speed jumps." "That's the structure Samsung is now proposing as zHBM. The z is, from the x, y, z axes we learned back in school, the vertical direction, so it means stacked in the z direction." "But unfortunately, I don't see that as the fundamental direction, because if you stack HBM on top of the GPU, the GPU gets hot and the memory melts." "So I suspect it's for very simple computation, the kind of use people call an LPU or an NPU. They showed it in passing as a fun technology, and the really important technology doesn't get shown at places like that." "So my lab researches tHBM. I believe today is the first time we're saying this publicly. We flip it, stack the HBM, and put the GPU on top of that."

Fireside Alpha

112,725 Aufrufe • vor 1 Monat

The creator of High Bandwidth Memory (HBM) put a number on the AI build that should stop every infra investor cold. A cluster of a million GPUs runs at roughly 10-20% utilization (Save this). Kim Jung-ho spent thirty years building what feeds the GPU, and his claim is that the GPU is barely working. Here is what is actually happening. Every time a model generates output, the data has to be read out of memory, computed, and written back. The read and the write swallow almost the entire cycle. While that data moves, the GPU does nothing. It sits there, fully powered, fully paid for, waiting. By Kim's estimate the memory is doing only about 30 percent of the work it needs to do. The processor idles the rest. So a million installed GPUs run at 10 to 20 percent. You are not compute constrained. You are memory constrained, and the expensive part is standing around. Adding more GPUs does not fix this. It gives you more processors starving for the same data. Here is the part that decides the next decade. Memory can grow. When a cell cannot shrink any further, you stack it into a high-rise, layer on layer. A GPU cannot be stacked. It runs too hot and needs a cooler bolted to its back, so the one move that rescues memory is closed to the processor. The thing that can keep stacking compounds. The thing that cannot plateaus. The marginal dollar in an AI build now buys more by fixing the memory path than by bolting on another idle GPU. Which is why the companies that control memory bandwidth and supply are not suppliers to the AI trade. They are the AI trade.

The creator of High Bandwidth Memory (HBM) put a number on the AI build that should stop every infra investor cold. A cluster of a million GPUs runs at roughly 10-20% utilization (Save this). Kim Jung-ho spent thirty years building what feeds the GPU, and his claim is that the GPU is barely working. Here is what is actually happening. Every time a model generates output, the data has to be read out of memory, computed, and written back. The read and the write swallow almost the entire cycle. While that data moves, the GPU does nothing. It sits there, fully powered, fully paid for, waiting. By Kim's estimate the memory is doing only about 30 percent of the work it needs to do. The processor idles the rest. So a million installed GPUs run at 10 to 20 percent. You are not compute constrained. You are memory constrained, and the expensive part is standing around. Adding more GPUs does not fix this. It gives you more processors starving for the same data. Here is the part that decides the next decade. Memory can grow. When a cell cannot shrink any further, you stack it into a high-rise, layer on layer. A GPU cannot be stacked. It runs too hot and needs a cooler bolted to its back, so the one move that rescues memory is closed to the processor. The thing that can keep stacking compounds. The thing that cannot plateaus. The marginal dollar in an AI build now buys more by fixing the memory path than by bolting on another idle GPU. Which is why the companies that control memory bandwidth and supply are not suppliers to the AI trade. They are the AI trade.

Fireside Alpha

38,370 Aufrufe • vor 2 Monaten

Etched is deploying two new technologies in chip design: low-voltage inference and cluster-scale memory. CEO Gavin Uberti says they'll make their chips much more power-efficient and way, way faster than today's leading GPUs. He breaks it down: "We looked at a lot of early research directions, and we realized the key things that models need are way more compute and way faster memory." "If you think about inference, there are two key parts: prefill and decode. For prefill, it's a compute-bound problem. You need to have more FLOPS, more operations per second on each of your chips." "On our GPU, the bottleneck's actually thermals. You can't really run a GPU at more than around 50% of what it could theoretically do, or it'll melt." "So we're using a new technology today called low-voltage inference to try to solve this problem. You bring the voltage of the chip down dramatically, which allows us to have way, way better efficiency in terms of how much power is drawn per unit of math, and thus fit way way more flops onto the chip..." "For decode, it's all about bandwidth. Not just bandwidth on a chip, but bandwidth across your cluster. That's why we have this technology we call cluster-scale memory. It reduces the amount of time it takes to communicate from one chip to another dramatically." "As a result we can go use all of our HBM, HBM bandwidth, SRAM, SRAM bandwidth, and our scale-up domain as a single coherent pool. And that means if you're a user, you can go get much faster tokens per second, while still keeping your costs low."

Etched is deploying two new technologies in chip design: low-voltage inference and cluster-scale memory. CEO Gavin Uberti says they'll make their chips much more power-efficient and way, way faster than today's leading GPUs. He breaks it down: "We looked at a lot of early research directions, and we realized the key things that models need are way more compute and way faster memory." "If you think about inference, there are two key parts: prefill and decode. For prefill, it's a compute-bound problem. You need to have more FLOPS, more operations per second on each of your chips." "On our GPU, the bottleneck's actually thermals. You can't really run a GPU at more than around 50% of what it could theoretically do, or it'll melt." "So we're using a new technology today called low-voltage inference to try to solve this problem. You bring the voltage of the chip down dramatically, which allows us to have way, way better efficiency in terms of how much power is drawn per unit of math, and thus fit way way more flops onto the chip..." "For decode, it's all about bandwidth. Not just bandwidth on a chip, but bandwidth across your cluster. That's why we have this technology we call cluster-scale memory. It reduces the amount of time it takes to communicate from one chip to another dramatically." "As a result we can go use all of our HBM, HBM bandwidth, SRAM, SRAM bandwidth, and our scale-up domain as a single coherent pool. And that means if you're a user, you can go get much faster tokens per second, while still keeping your costs low."

TBPN

20,404 Aufrufe • vor 2 Monaten

Bill Ackman reveals the exact difference between valuing a stock and valuing a bond that most people never learn "A bond pays a 5% coupon interest rate. You get that every year, and it's very predictable. If it's a US Government Bond, you know you're going to get it. That's a pretty easy thing to value" "A stock is an interest in a business. It's like owning a piece of a company. A profitable business is like a bond in that it generates these coupons or these earnings or cash flow every year" "The difference is that a bond is a contract. You know what you're going to get. With a stock, you have to make predictions about the business" "How many widgets are they going to sell this year? How many next year? What are their costs going to be? How much money do they need to reinvest to keep the business going?" "What we do is try to find businesses where with a very high degree of confidence we know what those cash flows are going to be for a very long time. There are very few businesses you can have a really high degree of certainty about. Many investments are speculations because it's really very difficult to predict the future"

Bill Ackman reveals the exact difference between valuing a stock and valuing a bond that most people never learn "A bond pays a 5% coupon interest rate. You get that every year, and it's very predictable. If it's a US Government Bond, you know you're going to get it. That's a pretty easy thing to value" "A stock is an interest in a business. It's like owning a piece of a company. A profitable business is like a bond in that it generates these coupons or these earnings or cash flow every year" "The difference is that a bond is a contract. You know what you're going to get. With a stock, you have to make predictions about the business" "How many widgets are they going to sell this year? How many next year? What are their costs going to be? How much money do they need to reinvest to keep the business going?" "What we do is try to find businesses where with a very high degree of confidence we know what those cash flows are going to be for a very long time. There are very few businesses you can have a really high degree of certainty about. Many investments are speculations because it's really very difficult to predict the future"

Jaynit

160,132 Aufrufe • vor 2 Monaten

Cathie Wood explains why she doesn't own memory stocks and how the inference chip providers come into play "Someone asked recently, why don't you own any memory stocks?" "And first of all, and this again, maybe experience is hurting me, it is the most cyclical part of a semiconductor food chain, the most commoditized." "And what we're seeing is companies like Cerebras, like Nvidia with Groq, and we owned Groq in our venture fund. Both of those, Cerebras and Groq, do not require high-bandwidth memory." "So, in technology, when prices triple or quadruple or go up tenfold, that is not the normal state for technology. That's actually a negative, but most people think it's a huge positive." "And from a cash flow point of view, I saw a chart a couple of days ago comparing the chip stocks' free cash flow to the hyperscalers' free cash flow. And they're opposite sides to one another. But that's fleeting." "And in technology, I mean, we've seen it many times with Tesla. If there's a supply chain issue, cobalt from the Congo, you know, using slave labor and all of that, Elon engineered it out of the batteries, right? Or mostly out." "And so we're seeing engineering out the need for high bandwidth memory when it comes to inference."

Cathie Wood explains why she doesn't own memory stocks and how the inference chip providers come into play "Someone asked recently, why don't you own any memory stocks?" "And first of all, and this again, maybe experience is hurting me, it is the most cyclical part of a semiconductor food chain, the most commoditized." "And what we're seeing is companies like Cerebras, like Nvidia with Groq, and we owned Groq in our venture fund. Both of those, Cerebras and Groq, do not require high-bandwidth memory." "So, in technology, when prices triple or quadruple or go up tenfold, that is not the normal state for technology. That's actually a negative, but most people think it's a huge positive." "And from a cash flow point of view, I saw a chart a couple of days ago comparing the chip stocks' free cash flow to the hyperscalers' free cash flow. And they're opposite sides to one another. But that's fleeting." "And in technology, I mean, we've seen it many times with Tesla. If there's a supply chain issue, cobalt from the Congo, you know, using slave labor and all of that, Elon engineered it out of the batteries, right? Or mostly out." "And so we're seeing engineering out the need for high bandwidth memory when it comes to inference."

Fireside Alpha

146,876 Aufrufe • vor 1 Monat

$MU $00060 $00530 $TSM Evolution of the AI Memory Hierarchy: HBM, DRAM, & SRAM Overview and Market Analysis The video outlines the strategic shift of modern computing toward a memory-centric architecture driven by the demands of generative AI. It explains that the industry is moving away from a traditional commodity model to a tightly coupled hierarchy of specialized technologies, where High Bandwidth Memory (HBM) and advanced packaging act as primary bottlenecks for performance. Leading manufacturers like SK hynix, Samsung, and Micron are navigating complex transitions to HBM4 and CXL-based expansion to overcome data movement constraints. Beyond the memory producers, the ecosystem relies on integration enablers like TSMC for packaging and specialized equipment providers for high-precision manufacturing. Ultimately, the analysis suggests that memory intensity and power efficiency have become the decisive factors in the success of gigawatt-scale data centers.

$MU $00060 $00530 $TSM Evolution of the AI Memory Hierarchy: HBM, DRAM, & SRAM Overview and Market Analysis The video outlines the strategic shift of modern computing toward a memory-centric architecture driven by the demands of generative AI. It explains that the industry is moving away from a traditional commodity model to a tightly coupled hierarchy of specialized technologies, where High Bandwidth Memory (HBM) and advanced packaging act as primary bottlenecks for performance. Leading manufacturers like SK hynix, Samsung, and Micron are navigating complex transitions to HBM4 and CXL-based expansion to overcome data movement constraints. Beyond the memory producers, the ecosystem relies on integration enablers like TSMC for packaging and specialized equipment providers for high-precision manufacturing. Ultimately, the analysis suggests that memory intensity and power efficiency have become the decisive factors in the success of gigawatt-scale data centers.

TheValueist

23,901 Aufrufe • vor 8 Monaten

Gavin Baker says the piece missing from the compute debate is SRAM accelerators, and disaggregating inference across three chips lifts AI's ROI. "I do think something that is missing from all of this conversation about compute is what is going to happen when you put these SRAM-based accelerators that are not constrained by HBM DRAM and are often made on older nodes that are not competing with the latest greatest GPUs". "When you disaggregate inference, people talk about pre-fill and decode, but decode is two parts, attention and feed-forward network." "The ultimate holy grail is if you could do pre-fill on one chip that probably doesn't have HBM DRAM, do the attention on a super high-powered chip with HBM DRAM, and then do the feed-forward network on one of these SRAM chips." "You just can't beat SRAM in particular for that feed-forward network, and you just almost can't, no matter how much you try and get the ratio of compute to HBM DRAM to SRAM on the chip correct, the workloads are always changing, and there's different workloads." "Being able to disaggregate into these three parts, I think this is going to be really, really positive for the ROI on AI."

Gavin Baker says the piece missing from the compute debate is SRAM accelerators, and disaggregating inference across three chips lifts AI's ROI. "I do think something that is missing from all of this conversation about compute is what is going to happen when you put these SRAM-based accelerators that are not constrained by HBM DRAM and are often made on older nodes that are not competing with the latest greatest GPUs". "When you disaggregate inference, people talk about pre-fill and decode, but decode is two parts, attention and feed-forward network." "The ultimate holy grail is if you could do pre-fill on one chip that probably doesn't have HBM DRAM, do the attention on a super high-powered chip with HBM DRAM, and then do the feed-forward network on one of these SRAM chips." "You just can't beat SRAM in particular for that feed-forward network, and you just almost can't, no matter how much you try and get the ratio of compute to HBM DRAM to SRAM on the chip correct, the workloads are always changing, and there's different workloads." "Being able to disaggregate into these three parts, I think this is going to be really, really positive for the ROI on AI."

Fireside Alpha

15,501 Aufrufe • vor 1 Monat

Ben Thompson says the memory makers are Iran, and by making the Strait of Hormuz mistake are inviting the whole industry to build around them "The other thing, the risk they run, is Apple. Apple's lobbying to get Chinese memory, right?" "And what is the number one focus of algorithmic changes? How can we use less memory?" "I think the memory makers probably screw themselves in the long run by creating such a massive target on their back." "I've analogized memory makers to Iran. The issue with the Strait of Hormuz is it's very effective. It's more effective if you don't use it, cuz then it's always hanging out there as something you could do. Now they did it. Turns out it worked." "But the UAE and Saudi Arabia, they're going to build pipelines. They're going to build new ports. They're not going to let this happen again." "It's very painful right now. But say Iran wants to close the Strait of Hormuz in 2035. It's not going to have any effect, because it will have been built around." "My concern for the memory makers is they might have done the same thing. No one's going to let themselves get in this situation again as far as memory goes."

Ben Thompson says the memory makers are Iran, and by making the Strait of Hormuz mistake are inviting the whole industry to build around them "The other thing, the risk they run, is Apple. Apple's lobbying to get Chinese memory, right?" "And what is the number one focus of algorithmic changes? How can we use less memory?" "I think the memory makers probably screw themselves in the long run by creating such a massive target on their back." "I've analogized memory makers to Iran. The issue with the Strait of Hormuz is it's very effective. It's more effective if you don't use it, cuz then it's always hanging out there as something you could do. Now they did it. Turns out it worked." "But the UAE and Saudi Arabia, they're going to build pipelines. They're going to build new ports. They're not going to let this happen again." "It's very painful right now. But say Iran wants to close the Strait of Hormuz in 2035. It's not going to have any effect, because it will have been built around." "My concern for the memory makers is they might have done the same thing. No one's going to let themselves get in this situation again as far as memory goes."

Fireside Alpha

157,042 Aufrufe • vor 1 Monat

Gavin Baker, CIO of Atreides Management made one of the most important and nuanced calls on memory stocks in recent months (Save this). His argument is that based on every memory cycle of the last 25 years, the setup today, prices elevated, sentiment high, supply ramping is textbook time to sell but he adds a critical exception. The one cycle in modern memory history where selling was catastrophically wrong was the mid-1990s, which Baker calls the last true capacity cycle in memory. In that cycle, demand was structurally exploding as the internet era required entirely new computing infrastructure to be built from scratch, and memory had to scale with it in a way that had never happened before. His point is that AI may be that same kind of cycle and not a normal boom bust but a once in a generation capacity buildout where the underlying demand is structural, not cyclical. The reason this argument holds weight is the fundamental shift in what memory is in the AI era. Traditional DRAM was a pure commodity, identical specs, interchangeable suppliers, price determined entirely by supply and demand swings. HBM is the opposite because it is custom engineered to fit a specific customer's chip, co-designed between the memory maker and the GPU designer, with SK Hynix's Vice President literally describing it as shifting from a commodity to a customer-tailored custom business. A single Blackwell Ultra GPU now requires up to 288GB of HBM3E, a 3.6x increase over the H100 and major suppliers like SK Hynix and Micron have already sold out their entire HBM production capacity through the end of the year. Because HBM requires advanced packaging processes like CoWoS that can't be spun up overnight, the bottleneck isn't just wafer capacity but rather runs across the entire manufacturing stack. Bank of America projects the global HBM market grows 58% this year alone to $54.6 billion, and Nomura expects the broader memory sector to nearly double to $445 billion. Long Micron!

Gavin Baker, CIO of Atreides Management made one of the most important and nuanced calls on memory stocks in recent months (Save this). His argument is that based on every memory cycle of the last 25 years, the setup today, prices elevated, sentiment high, supply ramping is textbook time to sell but he adds a critical exception. The one cycle in modern memory history where selling was catastrophically wrong was the mid-1990s, which Baker calls the last true capacity cycle in memory. In that cycle, demand was structurally exploding as the internet era required entirely new computing infrastructure to be built from scratch, and memory had to scale with it in a way that had never happened before. His point is that AI may be that same kind of cycle and not a normal boom bust but a once in a generation capacity buildout where the underlying demand is structural, not cyclical. The reason this argument holds weight is the fundamental shift in what memory is in the AI era. Traditional DRAM was a pure commodity, identical specs, interchangeable suppliers, price determined entirely by supply and demand swings. HBM is the opposite because it is custom engineered to fit a specific customer's chip, co-designed between the memory maker and the GPU designer, with SK Hynix's Vice President literally describing it as shifting from a commodity to a customer-tailored custom business. A single Blackwell Ultra GPU now requires up to 288GB of HBM3E, a 3.6x increase over the H100 and major suppliers like SK Hynix and Micron have already sold out their entire HBM production capacity through the end of the year. Because HBM requires advanced packaging processes like CoWoS that can't be spun up overnight, the bottleneck isn't just wafer capacity but rather runs across the entire manufacturing stack. Bank of America projects the global HBM market grows 58% this year alone to $54.6 billion, and Nomura expects the broader memory sector to nearly double to $445 billion. Long Micron!

Milk Road AI

260,701 Aufrufe • vor 2 Monaten

This is one of my favorite movie endings... It has a powerful lesson: "At some point, everything is going to go south on you...and you're gonna say 'this is it, this is how I end.' Now you can either accept that, or you can get to work. That's all it is, you just begin. You do the math. You solve one problem. Then you solve the next one. And the next. And if you solve enough problems, you get to come home." This is the beautiful thing about life: No matter where you are today—no matter how deep in the darkness—you are always just one good decision away from being in a better place tomorrow. You can always focus on that one decision—that one problem—in front of you. Don't worry about the hundreds or thousands of decisions that you still have to make to get to where you want to be—just focus on the next decision. Just begin. Solve one problem. Then solve the next one. And the next. "Life is really simple, but we insist on making it complicated." - Confucius

This is one of my favorite movie endings... It has a powerful lesson: "At some point, everything is going to go south on you...and you're gonna say 'this is it, this is how I end.' Now you can either accept that, or you can get to work. That's all it is, you just begin. You do the math. You solve one problem. Then you solve the next one. And the next. And if you solve enough problems, you get to come home." This is the beautiful thing about life: No matter where you are today—no matter how deep in the darkness—you are always just one good decision away from being in a better place tomorrow. You can always focus on that one decision—that one problem—in front of you. Don't worry about the hundreds or thousands of decisions that you still have to make to get to where you want to be—just focus on the next decision. Just begin. Solve one problem. Then solve the next one. And the next. "Life is really simple, but we insist on making it complicated." - Confucius

Sahil Bloom

213,822 Aufrufe • vor 2 Jahren

DeepMind CEO Demis Hassabis says the AI bottleneck is already moving from model ideas to physical supply chains. "How much of the restriction at this point is physical? I am talking memory shortage, GPU shortage, and electricity." Demis: "That is constraining a lot of the deployment for sure. We see so much more demand for our models - Gemini and the other models that we have - than we can possibly serve." Then the less obvious part: "It does constrain a little bit the research, because you need a lot of chips to be able to experiment on new ideas at big enough scale that you can actually see if they are going to work." And on memory: "The whole supply chain is kind of strained... It still in the end actually comes down to a few suppliers of a few of the key components. Anywhere where there is some kind of constraint on the capacity, there is a sort of choke point." Important framing: The next AI race is not just who has the best model architecture. It is who can secure memory, chips, power, suppliers, and enough experimental capacity to find the next breakthrough before everyone else.

DeepMind CEO Demis Hassabis says the AI bottleneck is already moving from model ideas to physical supply chains. "How much of the restriction at this point is physical? I am talking memory shortage, GPU shortage, and electricity." Demis: "That is constraining a lot of the deployment for sure. We see so much more demand for our models - Gemini and the other models that we have - than we can possibly serve." Then the less obvious part: "It does constrain a little bit the research, because you need a lot of chips to be able to experiment on new ideas at big enough scale that you can actually see if they are going to work." And on memory: "The whole supply chain is kind of strained... It still in the end actually comes down to a few suppliers of a few of the key components. Anywhere where there is some kind of constraint on the capacity, there is a sort of choke point." Important framing: The next AI race is not just who has the best model architecture. It is who can secure memory, chips, power, suppliers, and enough experimental capacity to find the next breakthrough before everyone else.

Karl Mehta

25,716 Aufrufe • vor 2 Monaten

Google Ironwood TPU Memory Hierarchy in 9 levels by hand ✍️ 1. Bit – The most basic unit of information, the on–off decision from which every number, tensor, and model state is ultimately constructed. 2. FP8 (1×8 → 8 bits) – Eight bits are grouped to form a floating-point value, typically used for inference, where reduced precision is a deliberate trade-off to maximize throughput and efficiency. 3. BF16 (×2 → 16 bits) – Two FP8-scale chunks are combined to gain more dynamic range and stability, while still staying friendly to high-throughput hardware. 4. Tensor tile (×1024 → 1K) – Data moves through the chip in blocks of 1024 values at a time, defining the granularity at which tensors are fetched and manipulated. 5. Matrix Multiplication Unit (MXU) (×64 → 64K) – A systolic array where matrix multiplication is not abstract but physical, with tensor tiles flowing through fixed hardware to achieve the highest possible throughput. 6. Vector Memory (VMEM) (×2048 → 128M) – On-chip working memory that holds activations, partial results, and intermediates, sized specifically to keep the systolic array busy without stalling. 7. Common Memory (CMEM) (×8 → 1 GB) – A small but critical shared memory sitting between VMEM and HBM, used for staging, accumulation, synchronization, and cross-lane coordination. 8. HBM (×96 → 96 GB) – Off-chip high-bandwidth memory where model weights and large states live, implemented as HBM3e with 16 stacks at 6 GB each, for a total of 96 GB. 9. Dual-Die (x2 → 192GB) – Two tightly coupled compute dies operate as a single logical accelerator, each with its own local HBM, effectively doubling memory capacity and bandwidth while allowing tensors and activations to stream seamlessly across dies as if they lived on one chip. I created this drawing for this week's seminar. I’ll take you through these 9 levels in a beginner-friendly way by hand ✍️. RSVP 👉

Google Ironwood TPU Memory Hierarchy in 9 levels by hand ✍️ 1. Bit – The most basic unit of information, the on–off decision from which every number, tensor, and model state is ultimately constructed. 2. FP8 (1×8 → 8 bits) – Eight bits are grouped to form a floating-point value, typically used for inference, where reduced precision is a deliberate trade-off to maximize throughput and efficiency. 3. BF16 (×2 → 16 bits) – Two FP8-scale chunks are combined to gain more dynamic range and stability, while still staying friendly to high-throughput hardware. 4. Tensor tile (×1024 → 1K) – Data moves through the chip in blocks of 1024 values at a time, defining the granularity at which tensors are fetched and manipulated. 5. Matrix Multiplication Unit (MXU) (×64 → 64K) – A systolic array where matrix multiplication is not abstract but physical, with tensor tiles flowing through fixed hardware to achieve the highest possible throughput. 6. Vector Memory (VMEM) (×2048 → 128M) – On-chip working memory that holds activations, partial results, and intermediates, sized specifically to keep the systolic array busy without stalling. 7. Common Memory (CMEM) (×8 → 1 GB) – A small but critical shared memory sitting between VMEM and HBM, used for staging, accumulation, synchronization, and cross-lane coordination. 8. HBM (×96 → 96 GB) – Off-chip high-bandwidth memory where model weights and large states live, implemented as HBM3e with 16 stacks at 6 GB each, for a total of 96 GB. 9. Dual-Die (x2 → 192GB) – Two tightly coupled compute dies operate as a single logical accelerator, each with its own local HBM, effectively doubling memory capacity and bandwidth while allowing tensors and activations to stream seamlessly across dies as if they lived on one chip. I created this drawing for this week's seminar. I’ll take you through these 9 levels in a beginner-friendly way by hand ✍️. RSVP 👉

Tom Yeh

30,489 Aufrufe • vor 8 Monaten

Micron is going to $4,000 and once you understand what inference actually is, the number stops sounding crazy (Save this). Dylan Patel just said that by 2030, OpenAI and Anthropic alone will need over 100 gigawatts of compute combined and by 2040, we may not even be measuring AI infrastructure in gigawatts anymore. We may be talking about terawatts. Every single one of those gigawatts needs memory to function. Without it, the compute is worthless. Most people heard that and thought about Nvidia but they should be thinking about Micron. Every AI model generating a response has two phases. The first is prefill, processing your prompt which is compute-heavy and the second is decode generating each word one token at a time and that phase is almost entirely memory-bound, not compute-bound. During decode, the GPU's processing units sit idle more than 95% of the time, waiting for data to arrive from memory. Google confirmed it in a research paper that decode-phase bottlenecks are dominated by memory bandwidth and capacity not raw compute. The GPU is not the bottleneck but the memory feeding the GPU is. This matters because inference is now where all the money lives. Training a model happens once, Inference happens billions of times a day every ChatGPT response, every Claude output, every agentic workflow running in the background and every one of those token streams is a billing event tied directly to memory performance. Adding more GPUs does not fix this because GPUs are already underutilized in inference because they are sitting idle waiting on memory. Adding more memory bandwidth and capacity is what directly reduces token cost, reduces latency, and allows the same cluster to serve dramatically more users simultaneously. Longer context windows compound the problem further, a model running a 1 million token context window requires dramatically more memory per session than a 10,000 token window, and every new model generation pushes context longer. The market treats memory as a downstream beneficiary of Nvidia orders. The correct framework is the opposite, Micron is the upstream constraint on how much value every Nvidia GPU can actually generate at inference scale. Micron guided Q4 to $50 billion in revenue, has HBM4 ramping at twice the pace of the prior generation, and CEO Sanjay Mehrotra has said supply will not catch demand before the end of 2027. At 8x forward earnings on $112 projected FY2027 EPS, Micron is the most undervalued infrastructure company in the entire AI stack. Inference is memory. Memory is Micron and the inference ramp has barely started. Milk Road Pro members are already up massively on this position and we're just getting started. If you want the full breakdown of what we're buying and why, come join us for just a dollar using the link below!

Micron is going to $4,000 and once you understand what inference actually is, the number stops sounding crazy (Save this). Dylan Patel just said that by 2030, OpenAI and Anthropic alone will need over 100 gigawatts of compute combined and by 2040, we may not even be measuring AI infrastructure in gigawatts anymore. We may be talking about terawatts. Every single one of those gigawatts needs memory to function. Without it, the compute is worthless. Most people heard that and thought about Nvidia but they should be thinking about Micron. Every AI model generating a response has two phases. The first is prefill, processing your prompt which is compute-heavy and the second is decode generating each word one token at a time and that phase is almost entirely memory-bound, not compute-bound. During decode, the GPU's processing units sit idle more than 95% of the time, waiting for data to arrive from memory. Google confirmed it in a research paper that decode-phase bottlenecks are dominated by memory bandwidth and capacity not raw compute. The GPU is not the bottleneck but the memory feeding the GPU is. This matters because inference is now where all the money lives. Training a model happens once, Inference happens billions of times a day every ChatGPT response, every Claude output, every agentic workflow running in the background and every one of those token streams is a billing event tied directly to memory performance. Adding more GPUs does not fix this because GPUs are already underutilized in inference because they are sitting idle waiting on memory. Adding more memory bandwidth and capacity is what directly reduces token cost, reduces latency, and allows the same cluster to serve dramatically more users simultaneously. Longer context windows compound the problem further, a model running a 1 million token context window requires dramatically more memory per session than a 10,000 token window, and every new model generation pushes context longer. The market treats memory as a downstream beneficiary of Nvidia orders. The correct framework is the opposite, Micron is the upstream constraint on how much value every Nvidia GPU can actually generate at inference scale. Micron guided Q4 to $50 billion in revenue, has HBM4 ramping at twice the pace of the prior generation, and CEO Sanjay Mehrotra has said supply will not catch demand before the end of 2027. At 8x forward earnings on $112 projected FY2027 EPS, Micron is the most undervalued infrastructure company in the entire AI stack. Inference is memory. Memory is Micron and the inference ramp has barely started. Milk Road Pro members are already up massively on this position and we're just getting started. If you want the full breakdown of what we're buying and why, come join us for just a dollar using the link below!

Milk Road AI

130,756 Aufrufe • vor 2 Monaten

The subject of 'owning a slave' is dense. It is something we hear a lot when we are in the FemDom Realm. Is it just fantasy? Can it actually be a lifestyle? How do we navigate this type of dynamic? How do we even get to that level of D/s? In this short clip [Exerpt from SLAVE TRAINING Part 2] I want to already bring to your attention one thing that will define if your desire for a slave (or desire as a slave) is touching more on a fantasy or... how can you actually navigate this in a realistic way. No one person 'can do it all' or should be expected to. If you want your slave to be 'the best' , assign them a specific role in which they can excel... and then build upon that. Once they 'master' your housekeeping (which takes quite a bit of real training), they can move to other levels. And an important note I want to leave here... make them EARN access to certain things in your life that sometimes you just want to delegate because you don't want to manage or don't know how to manage. Entrusting them with serious tasks that can affect your life, your business, your reputation, are on top of the ladder. Are they even qualified for the thing you want them to take off your shoulders? Start small and allow them to grow in their submission, to develop their skills and to learn how to best satisfy you without setting them up for failure by expecting too much, too quick. In the end, if you want this to truly work, you have to approach it from a place that transcends the roles. As this is consensual power exchange. And you both want to be fulfilled in that relationship.

The subject of 'owning a slave' is dense. It is something we hear a lot when we are in the FemDom Realm. Is it just fantasy? Can it actually be a lifestyle? How do we navigate this type of dynamic? How do we even get to that level of D/s? In this short clip [Exerpt from SLAVE TRAINING Part 2] I want to already bring to your attention one thing that will define if your desire for a slave (or desire as a slave) is touching more on a fantasy or... how can you actually navigate this in a realistic way. No one person 'can do it all' or should be expected to. If you want your slave to be 'the best' , assign them a specific role in which they can excel... and then build upon that. Once they 'master' your housekeeping (which takes quite a bit of real training), they can move to other levels. And an important note I want to leave here... make them EARN access to certain things in your life that sometimes you just want to delegate because you don't want to manage or don't know how to manage. Entrusting them with serious tasks that can affect your life, your business, your reputation, are on top of the ladder. Are they even qualified for the thing you want them to take off your shoulders? Start small and allow them to grow in their submission, to develop their skills and to learn how to best satisfy you without setting them up for failure by expecting too much, too quick. In the end, if you want this to truly work, you have to approach it from a place that transcends the roles. As this is consensual power exchange. And you both want to be fulfilled in that relationship.

Ms. Malissia

13,796 Aufrufe • vor 7 Monaten

The AI boom just hit a wall nobody saw coming. And it's not software. It's not regulation. It's not even energy... It's memory chips. Right now, Dell is raising PC prices by 30%. Intel can't ship chips. Nvidia is slashing GPU production by 40%. And almost nobody understands why. Here's the "hidden" crisis the AI industry is trying to hide: AI data centers are hoarding memory. Not GPUs. Not processors. MEMORY. Every AI server needs massive amounts of high-bandwidth memory (HBM) to run those models everyone's hyping. One problem: There are only 3 companies in the world that can make it. Samsung. SK Hynix. Micron. That's it. And all 3 just diverted their entire production capacity away from normal RAM to feed AI data centers. The math that breaks everything: 1 gigabyte of HBM takes 4X the manufacturing capacity of regular DRAM. AI will consume 20% of global DRAM production in 2026. But the thing is, consumer demand for RAM didn't disappear. PCs still need memory. Phones still need memory. Cars still need memory. But there's no capacity left to make it. The price explosion: RAM prices are up 246% in the last 6 months. DDR5 contract prices jumped 100% month-over-month in some cases. Dell's CFO said he's "never witnessed costs escalating at this pace." SK Hynix and Micron? Sold out through all of 2026. Micron straight up EXITED the consumer memory market entirely to focus on AI customers. If you're not building an AI data center, you're not getting memory chips. AI data centers pay 3-5X margins compared to consumer products. So memory manufacturers are rationally choosing: Serve Microsoft and Google's AI buildout, or serve Dell's laptop business? Easy choice. Every wafer allocated to an Nvidia H100 GPU is a wafer DENIED to your next laptop. It's a zero-sum game. And consumers are losing. The dangerous cascade effect: Nvidia is cutting RTX 50-series GPU production by 30-40% because they can't get GDDR7 memory. Dell, Lenovo, HP are all raising PC prices 15-30% in early 2026. Xiaomi and other smartphone makers are cutting shipment targets. Even Intel's crash last week? Partially driven by memory shortages limiting chip production. This is a PERMANENT reallocation of the world's silicon capacity. Not a temporary supply hiccup. For decades, consumer electronics (phones, PCs, laptops) drove memory production. Now? AI data centers are the priority customer. And that priority shift is reshaping the entire tech economy. The timeline Is worse than you think: Industry analysts project shortages lasting through 2027, maybe 2028. Why? Because building new memory fabs takes 3-5 YEARS. Micron's new Idaho fab won't meaningfully impact supply until 2028. Samsung and SK Hynix are too busy ramping up HBM4 production to expand consumer DRAM. So we're stuck. AI companies need memory to scale. But producing that memory DESTROYS the supply chain for everything else. My question here: Everyone's betting on AI scaling infinitely. But what if the AI boom STALLS because there's not enough memory to support it? What if we're not in an "AI supercycle" but a "memory shortage that kills the AI buildout"? Intel crashed 17% because they can't manufacture enough chips. The root cause though? Memory shortages limiting what they can even produce. Nvidia is cutting GPU production by 40%. AMD is struggling to get GDDR6 for Radeon cards. This isn't just a consumer problem. It's an AI infrastructure problem. And if memory doesn't scale, AI doesn't scale. The AI industry sold you on infinite scaling. But they forgot to mention the part where there's only 3 companies making the memory chips that power everything. And all 3 just chose AI data centers over you. Even Nvidia can't make enough GPUs to meet demand. Not because of energy. Not because of regulation... But because the memory supply chain is BROKEN. And it won't be fixed until 2028.

The AI boom just hit a wall nobody saw coming. And it's not software. It's not regulation. It's not even energy... It's memory chips. Right now, Dell is raising PC prices by 30%. Intel can't ship chips. Nvidia is slashing GPU production by 40%. And almost nobody understands why. Here's the "hidden" crisis the AI industry is trying to hide: AI data centers are hoarding memory. Not GPUs. Not processors. MEMORY. Every AI server needs massive amounts of high-bandwidth memory (HBM) to run those models everyone's hyping. One problem: There are only 3 companies in the world that can make it. Samsung. SK Hynix. Micron. That's it. And all 3 just diverted their entire production capacity away from normal RAM to feed AI data centers. The math that breaks everything: 1 gigabyte of HBM takes 4X the manufacturing capacity of regular DRAM. AI will consume 20% of global DRAM production in 2026. But the thing is, consumer demand for RAM didn't disappear. PCs still need memory. Phones still need memory. Cars still need memory. But there's no capacity left to make it. The price explosion: RAM prices are up 246% in the last 6 months. DDR5 contract prices jumped 100% month-over-month in some cases. Dell's CFO said he's "never witnessed costs escalating at this pace." SK Hynix and Micron? Sold out through all of 2026. Micron straight up EXITED the consumer memory market entirely to focus on AI customers. If you're not building an AI data center, you're not getting memory chips. AI data centers pay 3-5X margins compared to consumer products. So memory manufacturers are rationally choosing: Serve Microsoft and Google's AI buildout, or serve Dell's laptop business? Easy choice. Every wafer allocated to an Nvidia H100 GPU is a wafer DENIED to your next laptop. It's a zero-sum game. And consumers are losing. The dangerous cascade effect: Nvidia is cutting RTX 50-series GPU production by 30-40% because they can't get GDDR7 memory. Dell, Lenovo, HP are all raising PC prices 15-30% in early 2026. Xiaomi and other smartphone makers are cutting shipment targets. Even Intel's crash last week? Partially driven by memory shortages limiting chip production. This is a PERMANENT reallocation of the world's silicon capacity. Not a temporary supply hiccup. For decades, consumer electronics (phones, PCs, laptops) drove memory production. Now? AI data centers are the priority customer. And that priority shift is reshaping the entire tech economy. The timeline Is worse than you think: Industry analysts project shortages lasting through 2027, maybe 2028. Why? Because building new memory fabs takes 3-5 YEARS. Micron's new Idaho fab won't meaningfully impact supply until 2028. Samsung and SK Hynix are too busy ramping up HBM4 production to expand consumer DRAM. So we're stuck. AI companies need memory to scale. But producing that memory DESTROYS the supply chain for everything else. My question here: Everyone's betting on AI scaling infinitely. But what if the AI boom STALLS because there's not enough memory to support it? What if we're not in an "AI supercycle" but a "memory shortage that kills the AI buildout"? Intel crashed 17% because they can't manufacture enough chips. The root cause though? Memory shortages limiting what they can even produce. Nvidia is cutting GPU production by 40%. AMD is struggling to get GDDR6 for Radeon cards. This isn't just a consumer problem. It's an AI infrastructure problem. And if memory doesn't scale, AI doesn't scale. The AI industry sold you on infinite scaling. But they forgot to mention the part where there's only 3 companies making the memory chips that power everything. And all 3 just chose AI data centers over you. Even Nvidia can't make enough GPUs to meet demand. Not because of energy. Not because of regulation... But because the memory supply chain is BROKEN. And it won't be fixed until 2028.

Ricardo

594,643 Aufrufe • vor 7 Monaten