Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

Moonshot AI limiting Kimi K3 subscriptions because of compute constraints is a massive signal because the 2.8 trillion parameter model is already trained and the bottleneck is now emerging as usage scales. Serving that demand requires a ton more accelerators, enormous memory capacity and faster networking to keep data... show more

Shay Boloor

446,043 subscribers

123,791 Aufrufe • vor 1 Monat •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

$NVDA networking is now running at a $60B+ annualized pace, growing ~200% YoY and nearly the size of $AVGO total 2025 revenue. As AI clusters scale, the bottleneck becomes how fast GPUs, CPUs, memory and storage move data together because even the best chips underperform if the network cannot keep up. This is why Nvidia is selling the full AI factory stack.

$NVDA networking is now running at a $60B+ annualized pace, growing ~200% YoY and nearly the size of $AVGO total 2025 revenue. As AI clusters scale, the bottleneck becomes how fast GPUs, CPUs, memory and storage move data together because even the best chips underperform if the network cannot keep up. This is why Nvidia is selling the full AI factory stack.

Shay Boloor

61,104 Aufrufe • vor 3 Monaten

$NVDA CEO Jensen Huang said demand has gone parabolic because agentic AI has arrived, AI can now do productive work and tokens are becoming profitable. If tokens are profitable then model builders have a reason to produce more of them which means more inference, more compute and more demand across networking, memory, CPUs, power and data centers. The near-term stock reaction is about expectations but long-term story is still that the AI factory buildout is running through Nvidia which is why I think it has a real path to becoming the world’s first $10T company.

$NVDA CEO Jensen Huang said demand has gone parabolic because agentic AI has arrived, AI can now do productive work and tokens are becoming profitable. If tokens are profitable then model builders have a reason to produce more of them which means more inference, more compute and more demand across networking, memory, CPUs, power and data centers. The near-term stock reaction is about expectations but long-term story is still that the AI factory buildout is running through Nvidia which is why I think it has a real path to becoming the world’s first $10T company.

Shay Boloor

261,814 Aufrufe • vor 3 Monaten

$AVGO helps hyperscalers build the custom chips and networking fabric needed to run massive AI systems. If $GOOGL, $META, OpenAI or Anthropic want to build their own inference chips then Broadcom is one of the first calls because it has the IP, packaging expertise and networking portfolio required to scale custom AI systems. Thats why Broadcom remains one of the strongest infrastructure positions in the AI economy.

$AVGO helps hyperscalers build the custom chips and networking fabric needed to run massive AI systems. If $GOOGL, $META, OpenAI or Anthropic want to build their own inference chips then Broadcom is one of the first calls because it has the IP, packaging expertise and networking portfolio required to scale custom AI systems. Thats why Broadcom remains one of the strongest infrastructure positions in the AI economy.

Shay Boloor

77,765 Aufrufe • vor 2 Monaten

The recent semiconductor selloff has spread worldwide because the supply chain spans U.S. chip design and cloud spending, Taiwanese manufacturing, Korean memory, Japanese materials, European equipment and Southeast Asian packaging. That makes $MSFT earnings today massive as its June fiscal year-end offers the first major look at spending plans for the first 6 months of 2027 across accelerators, CPUs, memory, networking, optics, advanced packaging, power and equipment. If Microsoft confirms AI demand remains above capacity and capex is still growing 30% to 40% then the selloff will look more like a financing and risk-off reset than evidence the semiconductor cycle is breaking.

The recent semiconductor selloff has spread worldwide because the supply chain spans U.S. chip design and cloud spending, Taiwanese manufacturing, Korean memory, Japanese materials, European equipment and Southeast Asian packaging. That makes $MSFT earnings today massive as its June fiscal year-end offers the first major look at spending plans for the first 6 months of 2027 across accelerators, CPUs, memory, networking, optics, advanced packaging, power and equipment. If Microsoft confirms AI demand remains above capacity and capex is still growing 30% to 40% then the selloff will look more like a financing and risk-off reset than evidence the semiconductor cycle is breaking.

Shay Boloor

78,929 Aufrufe • vor 29 Tagen

THE 5 BIGGEST BOTTLENECKS POWERING THE AI ECONOMY The way I'm thinking about AI winners today is that the market is moving beyond the simple question of which mega-cap company spends most on AI and has been rewarding the companies that control the scarce inputs, contracted capacity, data movement, power infrastructure, edge compute and workflow layers that make the AI economy function. These are the five bottlenecks I'm watching most: • Memory | $MU, Samsung, SK Hynix Memory is the clearest scarce input because HBM feeds the accelerator, only a few companies can make it at volume and buyers are locking in supply through long-term agreements that create revenue visibility through the end of the decade. • Connectivity | $AVGO, $MRVL, $ALAB, $CRDO, $AAOI, $ANET Connectivity determines whether AI clusters can move data fast enough because training runs span tens of thousands of chips that need to act like one machine. Once copper runs out of reach that causes the cluster depends on optics, retimers, switches and custom silicon to keep the system moving. • Power | $CEG, $VST, $GEV, $FPS, $VRT, $NVTS, $TLN, $ON Power determines whether new AI data centers can actually come online because the binding constraint is shifting from getting chips to getting megawatts so the value flows to the companies that control generation, grid equipment, power delivery, thermal management and efficiency. • Compute | $NBIS, $CIFR, $IREN, $APLD, $WULF, $CORZ, $CRWV Compute capacity is overflow layer when hyperscalers are sold out. Capital alone doesn't guarantee GPU access which is why buyers are signing multi-year contracts for clusters before they are even fully built. • CPU | $NVDA, $AMD, $INTC, $ARM, $QCOM On-device CPU (edge compute) becomes next bottleneck as AI moves into inference, agents, PCs, phones, vehicles and physical devices.

THE 5 BIGGEST BOTTLENECKS POWERING THE AI ECONOMY The way I'm thinking about AI winners today is that the market is moving beyond the simple question of which mega-cap company spends most on AI and has been rewarding the companies that control the scarce inputs, contracted capacity, data movement, power infrastructure, edge compute and workflow layers that make the AI economy function. These are the five bottlenecks I'm watching most: • Memory | $MU, Samsung, SK Hynix Memory is the clearest scarce input because HBM feeds the accelerator, only a few companies can make it at volume and buyers are locking in supply through long-term agreements that create revenue visibility through the end of the decade. • Connectivity | $AVGO, $MRVL, $ALAB, $CRDO, $AAOI, $ANET Connectivity determines whether AI clusters can move data fast enough because training runs span tens of thousands of chips that need to act like one machine. Once copper runs out of reach that causes the cluster depends on optics, retimers, switches and custom silicon to keep the system moving. • Power | $CEG, $VST, $GEV, $FPS, $VRT, $NVTS, $TLN, $ON Power determines whether new AI data centers can actually come online because the binding constraint is shifting from getting chips to getting megawatts so the value flows to the companies that control generation, grid equipment, power delivery, thermal management and efficiency. • Compute | $NBIS, $CIFR, $IREN, $APLD, $WULF, $CORZ, $CRWV Compute capacity is overflow layer when hyperscalers are sold out. Capital alone doesn't guarantee GPU access which is why buyers are signing multi-year contracts for clusters before they are even fully built. • CPU | $NVDA, $AMD, $INTC, $ARM, $QCOM On-device CPU (edge compute) becomes next bottleneck as AI moves into inference, agents, PCs, phones, vehicles and physical devices.

Shay Boloor

108,973 Aufrufe • vor 2 Monaten

Micron is going to $4,000 and once you understand what inference actually is, the number stops sounding crazy (Save this). Dylan Patel just said that by 2030, OpenAI and Anthropic alone will need over 100 gigawatts of compute combined and by 2040, we may not even be measuring AI infrastructure in gigawatts anymore. We may be talking about terawatts. Every single one of those gigawatts needs memory to function. Without it, the compute is worthless. Most people heard that and thought about Nvidia but they should be thinking about Micron. Every AI model generating a response has two phases. The first is prefill, processing your prompt which is compute-heavy and the second is decode generating each word one token at a time and that phase is almost entirely memory-bound, not compute-bound. During decode, the GPU's processing units sit idle more than 95% of the time, waiting for data to arrive from memory. Google confirmed it in a research paper that decode-phase bottlenecks are dominated by memory bandwidth and capacity not raw compute. The GPU is not the bottleneck but the memory feeding the GPU is. This matters because inference is now where all the money lives. Training a model happens once, Inference happens billions of times a day every ChatGPT response, every Claude output, every agentic workflow running in the background and every one of those token streams is a billing event tied directly to memory performance. Adding more GPUs does not fix this because GPUs are already underutilized in inference because they are sitting idle waiting on memory. Adding more memory bandwidth and capacity is what directly reduces token cost, reduces latency, and allows the same cluster to serve dramatically more users simultaneously. Longer context windows compound the problem further, a model running a 1 million token context window requires dramatically more memory per session than a 10,000 token window, and every new model generation pushes context longer. The market treats memory as a downstream beneficiary of Nvidia orders. The correct framework is the opposite, Micron is the upstream constraint on how much value every Nvidia GPU can actually generate at inference scale. Micron guided Q4 to $50 billion in revenue, has HBM4 ramping at twice the pace of the prior generation, and CEO Sanjay Mehrotra has said supply will not catch demand before the end of 2027. At 8x forward earnings on $112 projected FY2027 EPS, Micron is the most undervalued infrastructure company in the entire AI stack. Inference is memory. Memory is Micron and the inference ramp has barely started. Milk Road Pro members are already up massively on this position and we're just getting started. If you want the full breakdown of what we're buying and why, come join us for just a dollar using the link below!

Micron is going to $4,000 and once you understand what inference actually is, the number stops sounding crazy (Save this). Dylan Patel just said that by 2030, OpenAI and Anthropic alone will need over 100 gigawatts of compute combined and by 2040, we may not even be measuring AI infrastructure in gigawatts anymore. We may be talking about terawatts. Every single one of those gigawatts needs memory to function. Without it, the compute is worthless. Most people heard that and thought about Nvidia but they should be thinking about Micron. Every AI model generating a response has two phases. The first is prefill, processing your prompt which is compute-heavy and the second is decode generating each word one token at a time and that phase is almost entirely memory-bound, not compute-bound. During decode, the GPU's processing units sit idle more than 95% of the time, waiting for data to arrive from memory. Google confirmed it in a research paper that decode-phase bottlenecks are dominated by memory bandwidth and capacity not raw compute. The GPU is not the bottleneck but the memory feeding the GPU is. This matters because inference is now where all the money lives. Training a model happens once, Inference happens billions of times a day every ChatGPT response, every Claude output, every agentic workflow running in the background and every one of those token streams is a billing event tied directly to memory performance. Adding more GPUs does not fix this because GPUs are already underutilized in inference because they are sitting idle waiting on memory. Adding more memory bandwidth and capacity is what directly reduces token cost, reduces latency, and allows the same cluster to serve dramatically more users simultaneously. Longer context windows compound the problem further, a model running a 1 million token context window requires dramatically more memory per session than a 10,000 token window, and every new model generation pushes context longer. The market treats memory as a downstream beneficiary of Nvidia orders. The correct framework is the opposite, Micron is the upstream constraint on how much value every Nvidia GPU can actually generate at inference scale. Micron guided Q4 to $50 billion in revenue, has HBM4 ramping at twice the pace of the prior generation, and CEO Sanjay Mehrotra has said supply will not catch demand before the end of 2027. At 8x forward earnings on $112 projected FY2027 EPS, Micron is the most undervalued infrastructure company in the entire AI stack. Inference is memory. Memory is Micron and the inference ramp has barely started. Milk Road Pro members are already up massively on this position and we're just getting started. If you want the full breakdown of what we're buying and why, come join us for just a dollar using the link below!

Milk Road AI

128,678 Aufrufe • vor 1 Monat

$MU $SKHY $DRAM Holy Shit $AMD's Lisa Su just spent two full minutes at the 2026 AI keynote raving about HBM4 capacity as the thing that separates Helios from the competition. Not compute. Memory. "When you compare Helios to the competition, we're delivering 15% more compute, 50% more HBM4 memory capacity and memory bandwidth, and 50% more scale-out bandwidth." Notice which number is bigger. "What that means is that every Helios can deliver more performance for the largest models, more capacity for longer context, and the bandwidth to scale across thousands of racks.... More than 18,000 CDNA five GPU compute units, over 4,600 Zen 6 CPU cores, and 31 terabytes of HBM4 memory, all in a single rack." "That's what it takes to run agentic AI at scale. Today, I'm excited to announce that Helios is in full production. We have shipments on track to start at the end of the third quarter and ramping into the fourth quarter in the second half of the year. I can tell you, customer demand for Helios is extremely strong, we're extremely proud of the work that we have done across the leading AI labs to adopt Helios."

$MU $SKHY $DRAM Holy Shit $AMD's Lisa Su just spent two full minutes at the 2026 AI keynote raving about HBM4 capacity as the thing that separates Helios from the competition. Not compute. Memory. "When you compare Helios to the competition, we're delivering 15% more compute, 50% more HBM4 memory capacity and memory bandwidth, and 50% more scale-out bandwidth." Notice which number is bigger. "What that means is that every Helios can deliver more performance for the largest models, more capacity for longer context, and the bandwidth to scale across thousands of racks.... More than 18,000 CDNA five GPU compute units, over 4,600 Zen 6 CPU cores, and 31 terabytes of HBM4 memory, all in a single rack." "That's what it takes to run agentic AI at scale. Today, I'm excited to announce that Helios is in full production. We have shipments on track to start at the end of the third quarter and ramping into the fourth quarter in the second half of the year. I can tell you, customer demand for Helios is extremely strong, we're extremely proud of the work that we have done across the leading AI labs to adopt Helios."

Trade Whisperer

180,381 Aufrufe • vor 1 Monat

$GOOGL Ironwood TPU packs ~190GB of HBM3E so every new deployment directly increases premium memory demand across the cluster. That demand compounds as Gemini APIs process ~22B tokens per minute giving $SKHY the clearest near-term upside and $MU more upside if it gains share. Google has both the returns and the incentive to keep expanding memory capacity with Cloud margins at 36% and capex rising another $15B.

$GOOGL Ironwood TPU packs ~190GB of HBM3E so every new deployment directly increases premium memory demand across the cluster. That demand compounds as Gemini APIs process ~22B tokens per minute giving $SKHY the clearest near-term upside and $MU more upside if it gains share. Google has both the returns and the incentive to keep expanding memory capacity with Cloud margins at 36% and capex rising another $15B.

Shay Boloor

122,624 Aufrufe • vor 1 Monat

Building AI infrastructure at yotta-scale requires more than raw performance; it demands an open, modular rack design that can evolve across product generations, combining leadership compute engines with high-speed networking to connect thousands of accelerators into a single, unified system. Take a look at the Helios. Our rack-scale AI platform powered by MI455X GPUs and EPYC “Venice” CPUs and designed for trillion-parameter training.

Building AI infrastructure at yotta-scale requires more than raw performance; it demands an open, modular rack design that can evolve across product generations, combining leadership compute engines with high-speed networking to connect thousands of accelerators into a single, unified system. Take a look at the Helios. Our rack-scale AI platform powered by MI455X GPUs and EPYC “Venice” CPUs and designed for trillion-parameter training.

AMD

20,832 Aufrufe • vor 7 Monaten

If $MU says NAND pricing is still strong, enterprise SSD demand is picking up and AI inference is creating more storage demand then $SNDK should benefit because SanDisk is one of the more direct ways to own that part of the cycle. Hyperscalers are still figuring out the right mix of DRAM, HBM, TLC NAND, QLC NAND and other storage architectures which can keep supply tighter for longer as memory companies are going to be hesitant to add too much capacity before demand becomes clearer. I'm going to be listening closely for Micron’s QLC commentary because QLC NAND for read-heavy inference workloads is the big driver and if Micron calls that out then it would show that the storage side of the AI memory story is still earlier than DRAM.

If $MU says NAND pricing is still strong, enterprise SSD demand is picking up and AI inference is creating more storage demand then $SNDK should benefit because SanDisk is one of the more direct ways to own that part of the cycle. Hyperscalers are still figuring out the right mix of DRAM, HBM, TLC NAND, QLC NAND and other storage architectures which can keep supply tighter for longer as memory companies are going to be hesitant to add too much capacity before demand becomes clearer. I'm going to be listening closely for Micron’s QLC commentary because QLC NAND for read-heavy inference workloads is the big driver and if Micron calls that out then it would show that the storage side of the AI memory story is still earlier than DRAM.

Shay Boloor

108,953 Aufrufe • vor 2 Monaten

$IBM is having its worst day in nearly six decades after its CEO said customers redirected quarterly spending toward servers, storage and memory to secure supply-constrained infrastructure ahead of expected price increases. That comment is huge because it provides one of the clearest real-world examples yet of the AI infrastructure buildout reshaping enterprise technology budgets. Demand for compute, memory, networking, power, cooling and skilled labor is becoming so large that companies are being forced to reallocate capital across the rest of their businesses.

$IBM is having its worst day in nearly six decades after its CEO said customers redirected quarterly spending toward servers, storage and memory to secure supply-constrained infrastructure ahead of expected price increases. That comment is huge because it provides one of the clearest real-world examples yet of the AI infrastructure buildout reshaping enterprise technology budgets. Demand for compute, memory, networking, power, cooling and skilled labor is becoming so large that companies are being forced to reallocate capital across the rest of their businesses.

Shay Boloor

109,549 Aufrufe • vor 1 Monat

Cathie Wood says she avoids memory stocks like $MU and $SKHY because she still sees HBM as the most cyclical and commoditized part of the semiconductor stack. Her bigger point is that inference architectures from $CBRS and $NVDA via Groq can reduce or eliminate the need for HBM which makes today’s memory pricing surge look more temporary than structural. As she puts it when technology costs “triple or quadruple” engineers eventually design around the bottleneck and she believes inference is already starting to do exactly that.

Cathie Wood says she avoids memory stocks like $MU and $SKHY because she still sees HBM as the most cyclical and commoditized part of the semiconductor stack. Her bigger point is that inference architectures from $CBRS and $NVDA via Groq can reduce or eliminate the need for HBM which makes today’s memory pricing surge look more temporary than structural. As she puts it when technology costs “triple or quadruple” engineers eventually design around the bottleneck and she believes inference is already starting to do exactly that.

Shay Boloor

671,262 Aufrufe • vor 9 Tagen

$MU $00060 $00530 $TSM Evolution of the AI Memory Hierarchy: HBM, DRAM, & SRAM Overview and Market Analysis The video outlines the strategic shift of modern computing toward a memory-centric architecture driven by the demands of generative AI. It explains that the industry is moving away from a traditional commodity model to a tightly coupled hierarchy of specialized technologies, where High Bandwidth Memory (HBM) and advanced packaging act as primary bottlenecks for performance. Leading manufacturers like SK hynix, Samsung, and Micron are navigating complex transitions to HBM4 and CXL-based expansion to overcome data movement constraints. Beyond the memory producers, the ecosystem relies on integration enablers like TSMC for packaging and specialized equipment providers for high-precision manufacturing. Ultimately, the analysis suggests that memory intensity and power efficiency have become the decisive factors in the success of gigawatt-scale data centers.

$MU $00060 $00530 $TSM Evolution of the AI Memory Hierarchy: HBM, DRAM, & SRAM Overview and Market Analysis The video outlines the strategic shift of modern computing toward a memory-centric architecture driven by the demands of generative AI. It explains that the industry is moving away from a traditional commodity model to a tightly coupled hierarchy of specialized technologies, where High Bandwidth Memory (HBM) and advanced packaging act as primary bottlenecks for performance. Leading manufacturers like SK hynix, Samsung, and Micron are navigating complex transitions to HBM4 and CXL-based expansion to overcome data movement constraints. Beyond the memory producers, the ecosystem relies on integration enablers like TSMC for packaging and specialized equipment providers for high-precision manufacturing. Ultimately, the analysis suggests that memory intensity and power efficiency have become the decisive factors in the success of gigawatt-scale data centers.

TheValueist

23,901 Aufrufe • vor 8 Monaten

$PLTR is one of the cleanest ways to play the AI cycle because it benefits from the entire infrastructure buildout without financing the data centers, chips, memory, power or networking underneath it. Palantir is also emerging as one of the biggest sovereign AI winners by building the control layer that determines what AI understands, which proprietary data it can access and what decisions or actions it can influence inside an organization. OpenAI and Anthropic create the intelligence while Palantir connects it to permissions, workflows and real operating systems where measurable economic value is produced.

$PLTR is one of the cleanest ways to play the AI cycle because it benefits from the entire infrastructure buildout without financing the data centers, chips, memory, power or networking underneath it. Palantir is also emerging as one of the biggest sovereign AI winners by building the control layer that determines what AI understands, which proprietary data it can access and what decisions or actions it can influence inside an organization. OpenAI and Anthropic create the intelligence while Palantir connects it to permissions, workflows and real operating systems where measurable economic value is produced.

Shay Boloor

200,100 Aufrufe • vor 23 Tagen

$SNDK CEO David Goeckeler says inference is a memory-bound problem and HBF could break through some bottlenecks, so it's an enormous opportunity as inference scales, which they'll talk more at their analyst day next week. "Look, there's a ton of innovation going on right now in inference memory architectures, which we think is fantastic, very healthy." "There's a lot of different ideas. We're going to dive into this a little deeper next week at our analyst day…" "But yes, all these are opportunities for us. We think especially inference is a memory-bound problem. Storage is extraordinarily important to the equation." "We showed some stuff in our FMS keynote just a couple of hours ago about how, when you use HBF, we simulate performance and maybe break through some bottlenecks…" "As we scale inference, we think it provides an enormous opportunity. The way we're thinking about this is staying very close to our customers, because they're going to be the ones that define what the architecture is in the future."

$SNDK CEO David Goeckeler says inference is a memory-bound problem and HBF could break through some bottlenecks, so it's an enormous opportunity as inference scales, which they'll talk more at their analyst day next week. "Look, there's a ton of innovation going on right now in inference memory architectures, which we think is fantastic, very healthy." "There's a lot of different ideas. We're going to dive into this a little deeper next week at our analyst day…" "But yes, all these are opportunities for us. We think especially inference is a memory-bound problem. Storage is extraordinarily important to the equation." "We showed some stuff in our FMS keynote just a couple of hours ago about how, when you use HBF, we simulate performance and maybe break through some bottlenecks…" "As we scale inference, we think it provides an enormous opportunity. The way we're thinking about this is staying very close to our customers, because they're going to be the ones that define what the architecture is in the future."

Fireside Alpha

16,326 Aufrufe • vor 21 Tagen

DeepMind CEO Demis Hassabis says the AI bottleneck is already moving from model ideas to physical supply chains. "How much of the restriction at this point is physical? I am talking memory shortage, GPU shortage, and electricity." Demis: "That is constraining a lot of the deployment for sure. We see so much more demand for our models - Gemini and the other models that we have - than we can possibly serve." Then the less obvious part: "It does constrain a little bit the research, because you need a lot of chips to be able to experiment on new ideas at big enough scale that you can actually see if they are going to work." And on memory: "The whole supply chain is kind of strained... It still in the end actually comes down to a few suppliers of a few of the key components. Anywhere where there is some kind of constraint on the capacity, there is a sort of choke point." Important framing: The next AI race is not just who has the best model architecture. It is who can secure memory, chips, power, suppliers, and enough experimental capacity to find the next breakthrough before everyone else.

DeepMind CEO Demis Hassabis says the AI bottleneck is already moving from model ideas to physical supply chains. "How much of the restriction at this point is physical? I am talking memory shortage, GPU shortage, and electricity." Demis: "That is constraining a lot of the deployment for sure. We see so much more demand for our models - Gemini and the other models that we have - than we can possibly serve." Then the less obvious part: "It does constrain a little bit the research, because you need a lot of chips to be able to experiment on new ideas at big enough scale that you can actually see if they are going to work." And on memory: "The whole supply chain is kind of strained... It still in the end actually comes down to a few suppliers of a few of the key components. Anywhere where there is some kind of constraint on the capacity, there is a sort of choke point." Important framing: The next AI race is not just who has the best model architecture. It is who can secure memory, chips, power, suppliers, and enough experimental capacity to find the next breakthrough before everyone else.

Karl Mehta

25,716 Aufrufe • vor 2 Monaten

Everyone is obsessed with which AI model is smartest. Wrong question. The real question is: whose worldview does the model export? China will deflate the cost of compute, finance data centers across Africa and emerging markets, and distribute open-source models trained to see history through Beijing’s eyes. Kimi K2 and Tiananmen Square is the perfect example. Influence through inference. That is how America loses global dominance without a shot being fired.

Everyone is obsessed with which AI model is smartest. Wrong question. The real question is: whose worldview does the model export? China will deflate the cost of compute, finance data centers across Africa and emerging markets, and distribute open-source models trained to see history through Beijing’s eyes. Kimi K2 and Tiananmen Square is the perfect example. Influence through inference. That is how America loses global dominance without a shot being fired.

Ben Pouladian

27,799 Aufrufe • vor 1 Monat

$SPCX valuing Cursor at $60B proves durable value in AI is moving to the data layer because compute commoditizes as supply comes online while the application that owns daily workflow compounds as usage grows. $PLTR already owns that loop since every dataset, decision and process mapped into its ontology makes the platform more embedded which is the same learning loop as Cursor but applied to enterprise and government instead of code.

$SPCX valuing Cursor at $60B proves durable value in AI is moving to the data layer because compute commoditizes as supply comes online while the application that owns daily workflow compounds as usage grows. $PLTR already owns that loop since every dataset, decision and process mapped into its ontology makes the platform more embedded which is the same learning loop as Cursor but applied to enterprise and government instead of code.

Shay Boloor

61,460 Aufrufe • vor 2 Monaten

New course on serving LLMs efficiently -- how do you serve models to many concurrent users at low latency and reasonable cost? This short course is built with Red Hat and taught by Cedric Clyburn. Efficient LLM serving requires efficient memory management. A 70B-parameter model takes ~140 GB just to load the weights. On top of that, every active request needs its own chunk of GPU memory, the KV cache, to store the token context it has built up so far. In this course, you'll learn to reduce a model's memory footprint with quantization and serve it using vLLM, which handles many concurrent requests efficiently through smart memory management. Skills you'll gain: - Quantize a model and measure the accuracy tradeoff - Serve a model with vLLM and watch it handle concurrent requests efficiently - Benchmark your deployment and make informed tradeoffs between speed, cost, and accuracy Join and learn to serve LLMs efficiently:

New course on serving LLMs efficiently -- how do you serve models to many concurrent users at low latency and reasonable cost? This short course is built with Red Hat and taught by Cedric Clyburn. Efficient LLM serving requires efficient memory management. A 70B-parameter model takes ~140 GB just to load the weights. On top of that, every active request needs its own chunk of GPU memory, the KV cache, to store the token context it has built up so far. In this course, you'll learn to reduce a model's memory footprint with quantization and serve it using vLLM, which handles many concurrent requests efficiently through smart memory management. Skills you'll gain: - Quantize a model and measure the accuracy tradeoff - Serve a model with vLLM and watch it handle concurrent requests efficiently - Benchmark your deployment and make informed tradeoffs between speed, cost, and accuracy Join and learn to serve LLMs efficiently:

Andrew Ng

132,250 Aufrufe • vor 2 Monaten

Andrej Karpathy just made one of the most interesting arguments about AI model design that most people are completely missing. His take is that frontier AI models are not too big because the technology is complex and too big because the training data is garbage. When you or I think of the internet, we picture Wall Street Journal articles, Wikipedia entries, serious writing. That is not what a pretraining dataset looks like. When researchers at frontier labs look at random documents from the actual training corpus, it is stock ticker symbols, broken HTML, spam, gibberish. One estimate puts Llama 3's information compression at just 0.07 bits per token meaning the model has only a hazy recollection of most of what it trained on. So we build trillion parameter models not because we need a trillion parameter brain but because we need a trillion-parameter compression engine to squeeze some intelligence out of a firehose of noise. Most of those parameters are doing memory work, not cognitive work. Karpathy's prediction is separate the two entirely. Build a cognitive core, a model that contains only the algorithms for reasoning and problem-solving, stripped of encyclopedic memorization and pair it with external memory that it can query when it needs facts. He thinks a cognitive core trained on high-quality data could hit genuine intelligence at around one billion parameters. For reference, today's flagship models run between 200 billion and 1.8 trillion parameters with most of that weight dedicated to remembering the internet's slop. The trend is already moving his direction. GPT-4o operates at roughly 200 billion parameters and outperforms the original 1.8 trillion-parameter GPT-4. Inference costs for GPT-3.5-level performance dropped 280-fold between 2022 and 2024 driven almost entirely by smaller, cleaner, better-architected models. The real bottleneck in AI right now is not compute but rather data quality.

Andrej Karpathy just made one of the most interesting arguments about AI model design that most people are completely missing. His take is that frontier AI models are not too big because the technology is complex and too big because the training data is garbage. When you or I think of the internet, we picture Wall Street Journal articles, Wikipedia entries, serious writing. That is not what a pretraining dataset looks like. When researchers at frontier labs look at random documents from the actual training corpus, it is stock ticker symbols, broken HTML, spam, gibberish. One estimate puts Llama 3's information compression at just 0.07 bits per token meaning the model has only a hazy recollection of most of what it trained on. So we build trillion parameter models not because we need a trillion parameter brain but because we need a trillion-parameter compression engine to squeeze some intelligence out of a firehose of noise. Most of those parameters are doing memory work, not cognitive work. Karpathy's prediction is separate the two entirely. Build a cognitive core, a model that contains only the algorithms for reasoning and problem-solving, stripped of encyclopedic memorization and pair it with external memory that it can query when it needs facts. He thinks a cognitive core trained on high-quality data could hit genuine intelligence at around one billion parameters. For reference, today's flagship models run between 200 billion and 1.8 trillion parameters with most of that weight dedicated to remembering the internet's slop. The trend is already moving his direction. GPT-4o operates at roughly 200 billion parameters and outperforms the original 1.8 trillion-parameter GPT-4. Inference costs for GPT-3.5-level performance dropped 280-fold between 2022 and 2024 driven almost entirely by smaller, cleaner, better-architected models. The real bottleneck in AI right now is not compute but rather data quality.

Milk Road AI

200,535 Aufrufe • vor 4 Monaten