Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

Context memory essentially unlocks Agentic AI Much needed for Opus 4.6's "multi-agent swarms" In this SemiDoped pod, Vikram Sekar talks to Val Bercovici from Weka about context storage. - How token warehouses save inference costs - A new networking tier? Context Storage Network! - High Bandwidth Flash for context?...

semidoped's profile picture

Semi Doped

3,349 subscribers

12,796 görüntüleme • 7 ay önce •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

Breaking the Memory Wall: Micron’s Strategy for the AI Era Ben Bajarin and Jay Goldberg 顾忠南 talk micron's strategy for the AI era with Jeremy Werner SVP and GM of Micron’s Core Data Center Business Unit. Chapters: (00:00) Introduction and Guest Welcome (01:12) How the AI Cycle Differs from Past Volatility (04:23) Anticipating the Explosive Growth of AI (06:40) The Long-Term Sustainability of the AI Market (10:07) The "Memory Wall" in AI Inference (12:05) Explaining KV Cache and Context Windows (15:07) The Memory Hierarchy: From HBM to SSDs (20:21) Scaling for Billions of Simultaneous Users (23:05) Solving the Power and Efficiency Bottleneck (26:27) Why AI is Driving a Storage Shortage (29:57) Personal AI Agents and Persistent Memory Needs (34:13) Innovations in Ultra-High-Capacity SSDs (39:00) Accelerating the Pace of Engineering Innovation (43:50) Market Misunderstandings and the Future Potential of AI (47:38) The Global Race to Build New Fabs (51:34) Closing Remarks
52:31

Breaking the Memory Wall: Micron’s Strategy for the AI Era Ben Bajarin and Jay Goldberg 顾忠南 talk micron's strategy for the AI era with Jeremy Werner SVP and GM of Micron’s Core Data Center Business Unit. Chapters: (00:00) Introduction and Guest Welcome (01:12) How the AI Cycle Differs from Past Volatility (04:23) Anticipating the Explosive Growth of AI (06:40) The Long-Term Sustainability of the AI Market (10:07) The "Memory Wall" in AI Inference (12:05) Explaining KV Cache and Context Windows (15:07) The Memory Hierarchy: From HBM to SSDs (20:21) Scaling for Billions of Simultaneous Users (23:05) Solving the Power and Efficiency Bottleneck (26:27) Why AI is Driving a Storage Shortage (29:57) Personal AI Agents and Persistent Memory Needs (34:13) Innovations in Ultra-High-Capacity SSDs (39:00) Accelerating the Pace of Engineering Innovation (43:50) Market Misunderstandings and the Future Potential of AI (47:38) The Global Race to Build New Fabs (51:34) Closing Remarks

The Circuit

94,438 görüntüleme • 4 ay önce

New short course: LLMs as Operating Systems: Agent Memory, created with Letta, and taught by its founders Charles Packer and Sarah Wooders. An LLM's input context window has limited space. Using a longer input context also costs more and results in slower processing. So, managing what's stored in this context window is important. In the innovative paper MemGPT: Towards LLMs as Operating Systems, its authors (which include the instructors) proposed using an LLM agent to manage this context window. Their system uses a large persistent memory that stores everything that could be included in the input context, and an agent decides what is actually included. Take the example of building a chatbot that needs to remember what's been said earlier in a conversation (perhaps over many days of interaction with a user). As the conversation's length grows, the memory management agent will move information from the input context to a persistent searchable database; summarize information to keep relevant facts in the input context; and restore relevant conversation elements from further back in time. This allows a chatbot to keep what's currently most relevant in its input context memory to generate the next response. When I read the original MemGPT paper, I thought it was an innovative technique for handling memory for LLMs. The open-source Letta framework, which we'll use in this course, makes MemGPT easy to implement. It adds memory to your LLM agents and gives them transparent long-term memory. In detail, you’ll learn: - How to build an agent that can edit its own limited input context memory, using tools and multi-step reasoning - What is a memory hierarchy (an idea from computer operating systems, which use a cache to speed up memory access), and how these ideas apply to managing the LLM input context (where the input context window is a "cache" storing the most relevant information; and an agent decides what to move in and out of this to/from a larger persistent storage system) - How to implement multi-agent collaboration by letting different agents share blocks of memory This course will give you a sophisticated understanding of memory management for LLMs, which is important for chatbots having long conversations, and for complex agentic workflows. Please sign up here!
2:57

New short course: LLMs as Operating Systems: Agent Memory, created with Letta, and taught by its founders Charles Packer and Sarah Wooders. An LLM's input context window has limited space. Using a longer input context also costs more and results in slower processing. So, managing what's stored in this context window is important. In the innovative paper MemGPT: Towards LLMs as Operating Systems, its authors (which include the instructors) proposed using an LLM agent to manage this context window. Their system uses a large persistent memory that stores everything that could be included in the input context, and an agent decides what is actually included. Take the example of building a chatbot that needs to remember what's been said earlier in a conversation (perhaps over many days of interaction with a user). As the conversation's length grows, the memory management agent will move information from the input context to a persistent searchable database; summarize information to keep relevant facts in the input context; and restore relevant conversation elements from further back in time. This allows a chatbot to keep what's currently most relevant in its input context memory to generate the next response. When I read the original MemGPT paper, I thought it was an innovative technique for handling memory for LLMs. The open-source Letta framework, which we'll use in this course, makes MemGPT easy to implement. It adds memory to your LLM agents and gives them transparent long-term memory. In detail, you’ll learn: - How to build an agent that can edit its own limited input context memory, using tools and multi-step reasoning - What is a memory hierarchy (an idea from computer operating systems, which use a cache to speed up memory access), and how these ideas apply to managing the LLM input context (where the input context window is a "cache" storing the most relevant information; and an agent decides what to move in and out of this to/from a larger persistent storage system) - How to implement multi-agent collaboration by letting different agents share blocks of memory This course will give you a sophisticated understanding of memory management for LLMs, which is important for chatbots having long conversations, and for complex agentic workflows. Please sign up here!

Andrew Ng

201,127 görüntüleme • 1 yıl önce

Dylan Patel on the importance of memory and storage Two key quotes: "An $NVDA GPU is faster than an $AMD GPU in most cases, but because AMD GPUs have more memory, they can outperform Nvidia in certain workloads." “It is a difficult, multivariable problem. Generally, you need the best GPU, such as a GB300, but you also need the best storage solutions. I will not spoil who comes out on top, but storage solutions matter a lot, memory solutions matter a lot, and frontend networking also matters significantly" Full Quote: “We have over $80 million of compute: GPUs from $NVDA and $AMD, TPUs from Google, and Trainium from Amazon. We constantly run this benchmark using the newest inference engines, drivers, PyTorch versions, and other software. It runs every day through automated CI across the latest Chinese models from GLM, Zhipu, Moonshot, Kimi, Alibaba, and others. Initially, when we were benchmarking the differences between these chips, inference engines, and parallelism schemes, we used fixed context lengths. But with Agent X, we have now analyzed more than $5 million worth of Claude Code traces. This is real production traffic that users have donated to us, combined with internally generated data, so we now understand what an actual agent workload looks like. When we implement those workloads and run the benchmarks, it turns out that the chip you are using is very important, but how you handle memory offload can be even more important. An Nvidia GPU is faster than an AMD GPU in most cases, but because AMD GPUs have more memory, they can outperform Nvidia in certain workloads. Similarly, you can use a less powerful GPU with a much better storage solution and outperform the best GPU when it lacks those solutions. Simply buying the newest GPU does not necessarily give you the best inference economics. You need to layer in other innovations, including storage and memory.” Interviewer: “Who is the top player on your chart? Can you tell us?” Dylan Patel: “It is a difficult, multivariable problem. Generally, you need the best GPU, such as a GB300, but you also need the best storage solutions. I will not spoil who comes out on top, but storage solutions matter a lot, memory solutions matter a lot, and frontend networking also matters significantly.”
2:27

Dylan Patel on the importance of memory and storage Two key quotes: "An $NVDA GPU is faster than an $AMD GPU in most cases, but because AMD GPUs have more memory, they can outperform Nvidia in certain workloads." “It is a difficult, multivariable problem. Generally, you need the best GPU, such as a GB300, but you also need the best storage solutions. I will not spoil who comes out on top, but storage solutions matter a lot, memory solutions matter a lot, and frontend networking also matters significantly" Full Quote: “We have over $80 million of compute: GPUs from $NVDA and $AMD, TPUs from Google, and Trainium from Amazon. We constantly run this benchmark using the newest inference engines, drivers, PyTorch versions, and other software. It runs every day through automated CI across the latest Chinese models from GLM, Zhipu, Moonshot, Kimi, Alibaba, and others. Initially, when we were benchmarking the differences between these chips, inference engines, and parallelism schemes, we used fixed context lengths. But with Agent X, we have now analyzed more than $5 million worth of Claude Code traces. This is real production traffic that users have donated to us, combined with internally generated data, so we now understand what an actual agent workload looks like. When we implement those workloads and run the benchmarks, it turns out that the chip you are using is very important, but how you handle memory offload can be even more important. An Nvidia GPU is faster than an AMD GPU in most cases, but because AMD GPUs have more memory, they can outperform Nvidia in certain workloads. Similarly, you can use a less powerful GPU with a much better storage solution and outperform the best GPU when it lacks those solutions. Simply buying the newest GPU does not necessarily give you the best inference economics. You need to layer in other innovations, including storage and memory.” Interviewer: “Who is the top player on your chart? Can you tell us?” Dylan Patel: “It is a difficult, multivariable problem. Generally, you need the best GPU, such as a GB300, but you also need the best storage solutions. I will not spoil who comes out on top, but storage solutions matter a lot, memory solutions matter a lot, and frontend networking also matters significantly.”

Daniel Romero

48,894 görüntüleme • 2 ay önce

Vik and Val Bercovici (WEKA) map where AI inference memory is headed. Every 100x cut in KV cache gets swallowed by ~10,000x more usage, so demand climbs. - NVLink beats the board: 128 lanes vs 32 PCIe - WEKA serves NAND-backed storage faster than DRAM over network - DeepSeek's cache reads run ~87x cheaper, China only - CXL needs a dedicated bus, WEKA pools NAND over RDMA/NVLink instead - Val's call: SaaS giants and Neoclouds must merge Chapters: 0:00 Intro and memory prices 3:00 Model routing and offloading 5:10 Network faster than motherboard 13:06 Four-tier memory hierarchy 19:40 KV cache and Jevons paradox 23:50 DeepSeek cache read pricing 31:40 Sliding window attention 34:49 NAND tiers, SLC vs QLC 43:57 High bandwidth flash use cases 49:59 CXL versus NVLink 53:44 AMD Mex and small models 1:03:14 SaaS, Neoclouds and tokenomics Get more of Austin and Vik daily, free! Sign up: Connect with Vik and Austin: Vik's Paid Substack: Austin's Paid Substack: Austin Lyons Vikram Sekar
1:04:11

Vik and Val Bercovici (WEKA) map where AI inference memory is headed. Every 100x cut in KV cache gets swallowed by ~10,000x more usage, so demand climbs. - NVLink beats the board: 128 lanes vs 32 PCIe - WEKA serves NAND-backed storage faster than DRAM over network - DeepSeek's cache reads run ~87x cheaper, China only - CXL needs a dedicated bus, WEKA pools NAND over RDMA/NVLink instead - Val's call: SaaS giants and Neoclouds must merge Chapters: 0:00 Intro and memory prices 3:00 Model routing and offloading 5:10 Network faster than motherboard 13:06 Four-tier memory hierarchy 19:40 KV cache and Jevons paradox 23:50 DeepSeek cache read pricing 31:40 Sliding window attention 34:49 NAND tiers, SLC vs QLC 43:57 High bandwidth flash use cases 49:59 CXL versus NVLink 53:44 AMD Mex and small models 1:03:14 SaaS, Neoclouds and tokenomics Get more of Austin and Vik daily, free! Sign up: Connect with Vik and Austin: Vik's Paid Substack: Austin's Paid Substack: Austin Lyons Vikram Sekar

Semi Doped

60,918 görüntüleme • 2 ay önce

Dylan Patel of SemiAnalysis says a worse GPU with better storage and memory now beats the best chip without them, so buying the newest GPU alone no longer wins inference. So, an AMD GPU with more memory can outperform Nvidia in some cases. "So what we have is we have over $80 million of compute, GPUs from Nvidia, AMD, TPUs from Google, Trainium from Amazon, and we run this benchmark constantly on the newest inference engine, newest drivers, newest PyTorch version, whatever it is." "Every day it runs on an automated CI, and we run it on all the latest Chinese models, from GLM, Zhipu, Moonshot, Kimi, Alibaba, all these models we run." "Initially, when we were benchmarking the difference between these chips and different engines, different schemes for parallelism, we were just running it fixed context length." "But now with Agent X, we've analyzed over $5 million worth of Claude Code traces. This is real production traffic that people have donated to us as well as internally generated. Now we know what the actual agent workload looks like." "And then as we implement that and run those benchmarks, it turns out yes, the chip you're using is very important, but now even more important is how are you handling this memory offload?" "And so while an Nvidia GPU is faster than an AMD GPU in most cases, because AMD GPUs have more memory, they actually end up outperforming in some cases." "Or you can have a worse GPU, but a much better storage solution, and now you can outperform what the best GPU can do without those solutions. So just buying the newest and latest GPU alone doesn't get you the best inference economics." "Actually, you need to layer in all these other innovations including storage and memory." [ Who's the top player on your chart? ] "That really is a difficult multivariable problem. And generally that means you need to have, yes, you need to have the best GPU, a GB300, but you also need to have the best storage solutions. And so I won't spoil who's the best right here, but I will say that storage solutions matter a lot and memory solutions matter a lot, as does your front-end networking. That matters a lot."
2:00

Dylan Patel of SemiAnalysis says a worse GPU with better storage and memory now beats the best chip without them, so buying the newest GPU alone no longer wins inference. So, an AMD GPU with more memory can outperform Nvidia in some cases. "So what we have is we have over $80 million of compute, GPUs from Nvidia, AMD, TPUs from Google, Trainium from Amazon, and we run this benchmark constantly on the newest inference engine, newest drivers, newest PyTorch version, whatever it is." "Every day it runs on an automated CI, and we run it on all the latest Chinese models, from GLM, Zhipu, Moonshot, Kimi, Alibaba, all these models we run." "Initially, when we were benchmarking the difference between these chips and different engines, different schemes for parallelism, we were just running it fixed context length." "But now with Agent X, we've analyzed over $5 million worth of Claude Code traces. This is real production traffic that people have donated to us as well as internally generated. Now we know what the actual agent workload looks like." "And then as we implement that and run those benchmarks, it turns out yes, the chip you're using is very important, but now even more important is how are you handling this memory offload?" "And so while an Nvidia GPU is faster than an AMD GPU in most cases, because AMD GPUs have more memory, they actually end up outperforming in some cases." "Or you can have a worse GPU, but a much better storage solution, and now you can outperform what the best GPU can do without those solutions. So just buying the newest and latest GPU alone doesn't get you the best inference economics." "Actually, you need to layer in all these other innovations including storage and memory." [ Who's the top player on your chart? ] "That really is a difficult multivariable problem. And generally that means you need to have, yes, you need to have the best GPU, a GB300, but you also need to have the best storage solutions. And so I won't spoil who's the best right here, but I will say that storage solutions matter a lot and memory solutions matter a lot, as does your front-end networking. That matters a lot."

Fireside Alpha

178,673 görüntüleme • 1 ay önce

Memory is eating the world When consumers feel the heat, you know its a problem. While big-3 memory companies are enjoying sweet-sweet profit margins…the rest of the electronics market is cracking under pricing pressure. - GoPro is more like GoingUnder - Apple products 10-30% pricier - Even music synths are more expensive Is it price gouging at this point? Is this revenge for all the memory cyclicality? When will this end? Chapters: 0:00 Memory crisis hits 1:20 AI impacting consumers 3:00 AI causing inflation 6:48 Consumer demand drop? 8:49 AI demand inelastic 10:56 Long-term memory outlook 11:02 GoPro's memory woes 12:24 Apple's pricing power 18:20 Apple seeks CXMT DRAM 21:38 Shrinkflation for phones 23:16 Korea's memory investment 26:10 Micron's killing profits 33:00 Why AI needs so much DRAM 40:27 Future of AI training 44:34 Cost-optimizing inference Get more of Austin and Vik daily, free! Sign up: Connect with Vik and Austin: Vik's Paid Substack: Austin's Paid Substack: Austin Lyons Vikram Sekar $MU
50:55

Memory is eating the world When consumers feel the heat, you know its a problem. While big-3 memory companies are enjoying sweet-sweet profit margins…the rest of the electronics market is cracking under pricing pressure. - GoPro is more like GoingUnder - Apple products 10-30% pricier - Even music synths are more expensive Is it price gouging at this point? Is this revenge for all the memory cyclicality? When will this end? Chapters: 0:00 Memory crisis hits 1:20 AI impacting consumers 3:00 AI causing inflation 6:48 Consumer demand drop? 8:49 AI demand inelastic 10:56 Long-term memory outlook 11:02 GoPro's memory woes 12:24 Apple's pricing power 18:20 Apple seeks CXMT DRAM 21:38 Shrinkflation for phones 23:16 Korea's memory investment 26:10 Micron's killing profits 33:00 Why AI needs so much DRAM 40:27 Future of AI training 44:34 Cost-optimizing inference Get more of Austin and Vik daily, free! Sign up: Connect with Vik and Austin: Vik's Paid Substack: Austin's Paid Substack: Austin Lyons Vikram Sekar $MU

Semi Doped

34,810 görüntüleme • 2 ay önce

OpenAI just launched a 3-hour course on building agentic systems from scratch: 00:00 - Building workspace agents 37:42 - Deep dive into OpenAI AgentKit 1:24:01 - Agent memory (loops, context, evals) 2:21:44 - Agentic training (OpenAI playbook) This 3-hour watch will replace you 10 paid agentic courses. Watch it today, then learn how to build a team of AI agents from scratch in the article below.
3:23:54

OpenAI just launched a 3-hour course on building agentic systems from scratch: 00:00 - Building workspace agents 37:42 - Deep dive into OpenAI AgentKit 1:24:01 - Agent memory (loops, context, evals) 2:21:44 - Agentic training (OpenAI playbook) This 3-hour watch will replace you 10 paid agentic courses. Watch it today, then learn how to build a team of AI agents from scratch in the article below.

Codez

79,345 görüntüleme • 2 ay önce

Latest episode of the AGI Podcast is out, featuring Wei Zhou, Head of AIUR at SemiAnalysis. Token Wei and I go deep on compute, capital markets, tokenomics, and and the future of AGI. Inside the conversation: 01:00 — Wei’s background and early journey 3:38 — The most cracked researchers in China 5:30 — My path after finishing a Math PhD at Stanford 7:52 — A based, unhinged take on Andrew Ng 10:29 — Choosing money vs “contributing to society” 19:32 — What Wei learned working with George Soros 24:00 — Intel vs Nvidia and the road to becoming the next compute central bank 27:15 — CUDA’s early days and how it became Tencent 2.0 for AI 35:22 — The true base unit of token cost: gigawatts vs dollar capex/opex 39:18 — “How do you price everything?” — a compute-first worldview 42:22 — How investment flows actually move in public markets 43:54 — Understanding depreciation distribution 52:00 — Is memory/storage scarcity real, and what drives it? 54:41 — Should storage cost be part of tokenomics models? 57:44 — Why inter-chip interconnect defines the economics of AI tokens 1:05:56 — Wei’s surprising love for Pop Mart 1:10:52 — How insiders really define AGI and ASI Watch the full podcast here 👇
1:11:41

Latest episode of the AGI Podcast is out, featuring Wei Zhou, Head of AIUR at SemiAnalysis. Token Wei and I go deep on compute, capital markets, tokenomics, and and the future of AGI. Inside the conversation: 01:00 — Wei’s background and early journey 3:38 — The most cracked researchers in China 5:30 — My path after finishing a Math PhD at Stanford 7:52 — A based, unhinged take on Andrew Ng 10:29 — Choosing money vs “contributing to society” 19:32 — What Wei learned working with George Soros 24:00 — Intel vs Nvidia and the road to becoming the next compute central bank 27:15 — CUDA’s early days and how it became Tencent 2.0 for AI 35:22 — The true base unit of token cost: gigawatts vs dollar capex/opex 39:18 — “How do you price everything?” — a compute-first worldview 42:22 — How investment flows actually move in public markets 43:54 — Understanding depreciation distribution 52:00 — Is memory/storage scarcity real, and what drives it? 54:41 — Should storage cost be part of tokenomics models? 57:44 — Why inter-chip interconnect defines the economics of AI tokens 1:05:56 — Wei’s surprising love for Pop Mart 1:10:52 — How insiders really define AGI and ASI Watch the full podcast here 👇

Bill Sun

33,155 görüntüleme • 9 ay önce

Micron is going to $4,000 and once you understand what inference actually is, the number stops sounding crazy (Save this). Dylan Patel just said that by 2030, OpenAI and Anthropic alone will need over 100 gigawatts of compute combined and by 2040, we may not even be measuring AI infrastructure in gigawatts anymore. We may be talking about terawatts. Every single one of those gigawatts needs memory to function. Without it, the compute is worthless. Most people heard that and thought about Nvidia but they should be thinking about Micron. Every AI model generating a response has two phases. The first is prefill, processing your prompt which is compute-heavy and the second is decode generating each word one token at a time and that phase is almost entirely memory-bound, not compute-bound. During decode, the GPU's processing units sit idle more than 95% of the time, waiting for data to arrive from memory. Google confirmed it in a research paper that decode-phase bottlenecks are dominated by memory bandwidth and capacity not raw compute. The GPU is not the bottleneck but the memory feeding the GPU is. This matters because inference is now where all the money lives. Training a model happens once, Inference happens billions of times a day every ChatGPT response, every Claude output, every agentic workflow running in the background and every one of those token streams is a billing event tied directly to memory performance. Adding more GPUs does not fix this because GPUs are already underutilized in inference because they are sitting idle waiting on memory. Adding more memory bandwidth and capacity is what directly reduces token cost, reduces latency, and allows the same cluster to serve dramatically more users simultaneously. Longer context windows compound the problem further, a model running a 1 million token context window requires dramatically more memory per session than a 10,000 token window, and every new model generation pushes context longer. The market treats memory as a downstream beneficiary of Nvidia orders. The correct framework is the opposite, Micron is the upstream constraint on how much value every Nvidia GPU can actually generate at inference scale. Micron guided Q4 to $50 billion in revenue, has HBM4 ramping at twice the pace of the prior generation, and CEO Sanjay Mehrotra has said supply will not catch demand before the end of 2027. At 8x forward earnings on $112 projected FY2027 EPS, Micron is the most undervalued infrastructure company in the entire AI stack. Inference is memory. Memory is Micron and the inference ramp has barely started. Milk Road Pro members are already up massively on this position and we're just getting started. If you want the full breakdown of what we're buying and why, come join us for just a dollar using the link below!
0:39

Micron is going to $4,000 and once you understand what inference actually is, the number stops sounding crazy (Save this). Dylan Patel just said that by 2030, OpenAI and Anthropic alone will need over 100 gigawatts of compute combined and by 2040, we may not even be measuring AI infrastructure in gigawatts anymore. We may be talking about terawatts. Every single one of those gigawatts needs memory to function. Without it, the compute is worthless. Most people heard that and thought about Nvidia but they should be thinking about Micron. Every AI model generating a response has two phases. The first is prefill, processing your prompt which is compute-heavy and the second is decode generating each word one token at a time and that phase is almost entirely memory-bound, not compute-bound. During decode, the GPU's processing units sit idle more than 95% of the time, waiting for data to arrive from memory. Google confirmed it in a research paper that decode-phase bottlenecks are dominated by memory bandwidth and capacity not raw compute. The GPU is not the bottleneck but the memory feeding the GPU is. This matters because inference is now where all the money lives. Training a model happens once, Inference happens billions of times a day every ChatGPT response, every Claude output, every agentic workflow running in the background and every one of those token streams is a billing event tied directly to memory performance. Adding more GPUs does not fix this because GPUs are already underutilized in inference because they are sitting idle waiting on memory. Adding more memory bandwidth and capacity is what directly reduces token cost, reduces latency, and allows the same cluster to serve dramatically more users simultaneously. Longer context windows compound the problem further, a model running a 1 million token context window requires dramatically more memory per session than a 10,000 token window, and every new model generation pushes context longer. The market treats memory as a downstream beneficiary of Nvidia orders. The correct framework is the opposite, Micron is the upstream constraint on how much value every Nvidia GPU can actually generate at inference scale. Micron guided Q4 to $50 billion in revenue, has HBM4 ramping at twice the pace of the prior generation, and CEO Sanjay Mehrotra has said supply will not catch demand before the end of 2027. At 8x forward earnings on $112 projected FY2027 EPS, Micron is the most undervalued infrastructure company in the entire AI stack. Inference is memory. Memory is Micron and the inference ramp has barely started. Milk Road Pro members are already up massively on this position and we're just getting started. If you want the full breakdown of what we're buying and why, come join us for just a dollar using the link below!

Milk Road AI

130,756 görüntüleme • 2 ay önce

AI and crypto… How will these two powerful technologies collide, and what happens when they do? Today’s episode is a special one. It’s a round table on some of the most compelling ideas at the intersection of AI and crypto, sourced from members of the a16z crypto team. You’re going to hear from builders, researchers, and deal makers who are all wrestling with the big question: How will these two powerful technologies collide, and what happens when they do? with Sam Broner Scott Kominers Liz Harkavy Carra Wu jay mattg Timestamps: (0:00) Introduction (1:44) Universal Identity for AI Agents (5:25) Why Crypto Makes a Difference (10:07) Context in AI: Memory and Interoperability (15:00) Vibe Coding Applications (19:32) Registries with Global Context (21:47) Honkaku Mystery Novels (24:13) Monetization Models for AI x Crypto (28:36) Overcoming Technical Challenges (31:34) AI Agents and the Rise of Web Crawlers (36:05) Collapse of the Social Contract (39:21) AI Bots and Blockchain Solutions (44:31) Proof of Personhood: Ensuring Human-ness (50:41) Future of Personalized Advertising (1:01:54) Final Thoughts Enjoy the full episode here on X.
1:02:26

AI and crypto… How will these two powerful technologies collide, and what happens when they do? Today’s episode is a special one. It’s a round table on some of the most compelling ideas at the intersection of AI and crypto, sourced from members of the a16z crypto team. You’re going to hear from builders, researchers, and deal makers who are all wrestling with the big question: How will these two powerful technologies collide, and what happens when they do? with Sam Broner Scott Kominers Liz Harkavy Carra Wu jay mattg Timestamps: (0:00) Introduction (1:44) Universal Identity for AI Agents (5:25) Why Crypto Makes a Difference (10:07) Context in AI: Memory and Interoperability (15:00) Vibe Coding Applications (19:32) Registries with Global Context (21:47) Honkaku Mystery Novels (24:13) Monetization Models for AI x Crypto (28:36) Overcoming Technical Challenges (31:34) AI Agents and the Rise of Web Crawlers (36:05) Collapse of the Social Contract (39:21) AI Bots and Blockchain Solutions (44:31) Proof of Personhood: Ensuring Human-ness (50:41) Future of Personalized Advertising (1:01:54) Final Thoughts Enjoy the full episode here on X.

a16z crypto

20,805 görüntüleme • 1 yıl önce

1/ New episode – Making the Internet AI-Native with Andrej from Grass Host: Connor Lovely 🎬 We discuss: - An inside look at their revenue and customer base 👀 - How they're upgrading the Internet from the bottom up - What LCR is and why it's so important for our AI future Timestamps: 00:00 - Introduction 01:03 - Shifts in AI and Crypto Landscapes 02:15 - Diminishing Returns in AI Scaling 04:48 - The Evolution of AI Models 06:06 - Grass’s Position in the AI Industry 09:07 - Challenges in Multimodal Data 11:06 - Live Context Retrieval Explained 13:11 - The Future of AI Search 15:07 - The Scraping Dilemma 18:08 - Grass’s Unique Opportunity 22:05 - Current Node Count and Future Goals 24:29 - Grasshopper and Network Expansion 28:08 - Grass’s Identity: DePIN or AI Project? 30:23 - ClipTagger12B and Video Search Innovations 39:40 - Customer Engagement and Revenue Insights 45:04 - Future Growth and Market Potential Watch on other platforms below ⬇️
53:22

1/ New episode – Making the Internet AI-Native with Andrej from Grass Host: Connor Lovely 🎬 We discuss: - An inside look at their revenue and customer base 👀 - How they're upgrading the Internet from the bottom up - What LCR is and why it's so important for our AI future Timestamps: 00:00 - Introduction 01:03 - Shifts in AI and Crypto Landscapes 02:15 - Diminishing Returns in AI Scaling 04:48 - The Evolution of AI Models 06:06 - Grass’s Position in the AI Industry 09:07 - Challenges in Multimodal Data 11:06 - Live Context Retrieval Explained 13:11 - The Future of AI Search 15:07 - The Scraping Dilemma 18:08 - Grass’s Unique Opportunity 22:05 - Current Node Count and Future Goals 24:29 - Grasshopper and Network Expansion 28:08 - Grass’s Identity: DePIN or AI Project? 30:23 - ClipTagger12B and Video Search Innovations 39:40 - Customer Engagement and Revenue Insights 45:04 - Future Growth and Market Potential Watch on other platforms below ⬇️

Proof of Coverage | Tech’s Creative Agency

99,562 görüntüleme • 11 ay önce

In this episode, Engram co-founder and CEO Dan Biderman joins allen to cook Mediterranean meatballs with yellow rice and talk about building AI that actually learns from you: why long context, RAG, and compaction eventually break down, how Engram compresses knowledge into cartridges and model weights, what continual learning could unlock for long-horizon agents, why token efficiency is inseparable from intelligence, how personal models could improve like Tamagotchis, and what it takes to build the research and infrastructure for millions of continuously updated AI memories. Timestamps: 0:00 Intro 0:26 Engram’s $98M Launch and Meatballs 1:45 From Naval Special Operations to AI Research 4:32 Israeli Military Culture and Founder Maturity 7:12 Why Engram Is Betting on Context and Continual Learning 9:14 Knowledge Cartridges, Compression, and Model Intuition 14:10 Trillion-Token Company Knowledge and Context Rot 18:05 Long-Context Limits, Compaction, and Neural Memory 22:20 Test-Time Training and “Destroying Prefill” 24:31 Harvey and Holistic Enterprise Queries Beyond RAG 27:02 Personal AI Models and Tamagotchi Weights 30:00 What Belongs in Weights vs. Text 32:25 Autonomous Memory and User-Specific Feedback Loops 34:20 Token Efficiency, Model Routing, and Harder Tasks 38:03 Engram’s Research Team and Product Culture 43:02 Hiring Researchers and Infrastructure Engineers 45:25 Doing More With Less 47:41 Where to Find Engram 48:19 Final Taste Test
49:43

In this episode, Engram co-founder and CEO Dan Biderman joins allen to cook Mediterranean meatballs with yellow rice and talk about building AI that actually learns from you: why long context, RAG, and compaction eventually break down, how Engram compresses knowledge into cartridges and model weights, what continual learning could unlock for long-horizon agents, why token efficiency is inseparable from intelligence, how personal models could improve like Tamagotchis, and what it takes to build the research and infrastructure for millions of continuously updated AI memories. Timestamps: 0:00 Intro 0:26 Engram’s $98M Launch and Meatballs 1:45 From Naval Special Operations to AI Research 4:32 Israeli Military Culture and Founder Maturity 7:12 Why Engram Is Betting on Context and Continual Learning 9:14 Knowledge Cartridges, Compression, and Model Intuition 14:10 Trillion-Token Company Knowledge and Context Rot 18:05 Long-Context Limits, Compaction, and Neural Memory 22:20 Test-Time Training and “Destroying Prefill” 24:31 Harvey and Holistic Enterprise Queries Beyond RAG 27:02 Personal AI Models and Tamagotchi Weights 30:00 What Belongs in Weights vs. Text 32:25 Autonomous Memory and User-Specific Feedback Loops 34:20 Token Efficiency, Model Routing, and Harder Tasks 38:03 Engram’s Research Team and Product Culture 43:02 Hiring Researchers and Infrastructure Engineers 45:25 Doing More With Less 47:41 Where to Find Engram 48:19 Final Taste Test

Latent.Space

34,076 görüntüleme • 2 ay önce

Engineer runs a Kimi K3 memory layer that costs $11 a month and remembers what a $500,000 vector database keeps losing. No embeddings. Four nodes and one rule about what's allowed to be forgotten. He published the whole schema. His version starts from the opposite idea. Memory is not a pile you search. It's a set of claims that expire unless something keeps paying to keep them. Four nodes. Every memory carries a clock someone has to reset: > WRITER - stores a fact with the reason it mattered, never raw text > DECAY - ages every memory down. Silence is deletion > RENEWER - only re-lifts a memory the model actually used again > GRAVE - holds what died, and why nobody reached for it Three nodes keep memory alive. One keeps the dead ones. Recall isn't storage here. It's rent a fact has to keep earning. That's the entire design. When everything is remembered forever, the useful and the stale retrieve identically. He replayed two months of agent context. 90,000 stored facts. 71,000 never retrieved once. The vector store returned all of them on similarity. Similarity graded closeness. Nobody graded whether the memory was ever right. Everyone else stuffs more into the context window and calls it memory. He built a layer that lets a fact die unless it keeps proving itself. The cost isn't storage. It's finding out how much of what your agent "knows" it has never once used. The article below is the full build - node prompts, the decay curve, the renewal rule. Save it. You'll want it open in the other tab.
0:40

Engineer runs a Kimi K3 memory layer that costs $11 a month and remembers what a $500,000 vector database keeps losing. No embeddings. Four nodes and one rule about what's allowed to be forgotten. He published the whole schema. His version starts from the opposite idea. Memory is not a pile you search. It's a set of claims that expire unless something keeps paying to keep them. Four nodes. Every memory carries a clock someone has to reset: > WRITER - stores a fact with the reason it mattered, never raw text > DECAY - ages every memory down. Silence is deletion > RENEWER - only re-lifts a memory the model actually used again > GRAVE - holds what died, and why nobody reached for it Three nodes keep memory alive. One keeps the dead ones. Recall isn't storage here. It's rent a fact has to keep earning. That's the entire design. When everything is remembered forever, the useful and the stale retrieve identically. He replayed two months of agent context. 90,000 stored facts. 71,000 never retrieved once. The vector store returned all of them on similarity. Similarity graded closeness. Nobody graded whether the memory was ever right. Everyone else stuffs more into the context window and calls it memory. He built a layer that lets a fact die unless it keeps proving itself. The cost isn't storage. It's finding out how much of what your agent "knows" it has never once used. The article below is the full build - node prompts, the decay curve, the renewal rule. Save it. You'll want it open in the other tab.

wast3

67,616 görüntüleme • 18 gün önce

Qwen3.8-Flash-Next is still going strong at 364.7K tokens of context on an M5 Max. And this isn’t just a static long-context test. The model was reasoning about how to speed up its own workflow while using tools, and the tool calls kept working without misses. Setup: • Qwen3.8-Flash-Next • M5 Max • 128GB unified memory • MLX-Serve PR #363 • OpenCode 2 • 364.7K context The interesting part isn’t simply getting hundreds of thousands of tokens into memory. It’s what happens once the context gets this large. Long-context inference usually comes with a painful tradeoff. As the KV cache grows, memory pressure increases and generation can slow down. But this setup is still pushing through 364K tokens while maintaining a usable agent workflow. The model can reason, call tools, inspect results, continue working, and keep the session moving. And the tool calls reportedly haven’t missed so far. That’s important for agentic coding. A huge context window is only useful if the model can actually operate reliably inside it. A 400K-token context that constantly breaks tool calls isn’t very useful. A 364K session that can keep reasoning and executing tools is a different story. And the test isn’t finished yet. The current run is approaching 400K tokens, with the expectation that it can keep going. This is also another interesting example of why Apple Silicon keeps showing up in local LLM experiments. The M5 Max’s unified memory gives a large model and its growing KV cache access to one shared memory pool. With MLX-Serve continuing to improve, these machines are becoming surprisingly capable long-context inference boxes. The bigger takeaway: Context length is becoming a workload, not just a model specification. Running a model at 256K is one thing. Keeping an agent alive at 300K+ while it reasons and uses tools is much more interesting. And Qwen3.8-Flash-Next is showing that this can be pushed surprisingly far on a single 128GB Mac. 364.7K and counting. Next stop: 400K.
0:33

Qwen3.8-Flash-Next is still going strong at 364.7K tokens of context on an M5 Max. And this isn’t just a static long-context test. The model was reasoning about how to speed up its own workflow while using tools, and the tool calls kept working without misses. Setup: • Qwen3.8-Flash-Next • M5 Max • 128GB unified memory • MLX-Serve PR #363 • OpenCode 2 • 364.7K context The interesting part isn’t simply getting hundreds of thousands of tokens into memory. It’s what happens once the context gets this large. Long-context inference usually comes with a painful tradeoff. As the KV cache grows, memory pressure increases and generation can slow down. But this setup is still pushing through 364K tokens while maintaining a usable agent workflow. The model can reason, call tools, inspect results, continue working, and keep the session moving. And the tool calls reportedly haven’t missed so far. That’s important for agentic coding. A huge context window is only useful if the model can actually operate reliably inside it. A 400K-token context that constantly breaks tool calls isn’t very useful. A 364K session that can keep reasoning and executing tools is a different story. And the test isn’t finished yet. The current run is approaching 400K tokens, with the expectation that it can keep going. This is also another interesting example of why Apple Silicon keeps showing up in local LLM experiments. The M5 Max’s unified memory gives a large model and its growing KV cache access to one shared memory pool. With MLX-Serve continuing to improve, these machines are becoming surprisingly capable long-context inference boxes. The bigger takeaway: Context length is becoming a workload, not just a model specification. Running a model at 256K is one thing. Keeping an agent alive at 300K+ while it reasons and uses tools is much more interesting. And Qwen3.8-Flash-Next is showing that this can be pushed surprisingly far on a single 128GB Mac. 364.7K and counting. Next stop: 400K.

FHILY👑

39,982 görüntüleme • 8 gün önce

As ever, a fascinating chat with my friend Jordi Visser on AI, crypto, and hedge funds' future... Enjoy! 00:00:00 Introduction 00:00:54 The Big Picture: AI, Crypto, and Macro 00:02:25 Jordie Visser’s Background in Macro and Finance 00:03:30 AI’s Disruptive Impact on the Economy 00:06:32 Rethinking GDP and Productivity 00:08:59 The Future of Labor and Profit Margins 00:10:49 The Rise of the Speculative Economy 00:14:41 Generational Shifts in Investing and Risk-taking 00:16:18 AI and Crypto: A Global Network Effect 00:19:38 Stablecoins and AI-Driven Market Volume Growth 00:24:28 The End of Traditional Hedge Funds? 00:31:16 Deep Research and the Power of AI Analysis 00:35:29 The Acceleration of AI and its Economic Impact 00:41:02 The Growing Crypto Adoption and Its Role in the Future Economy 00:44:22 Shifting from Fear to Optimism in Markets 00:50:27 The Political and Economic Challenges Ahead 00:56:01 Quantum Computing and the Next Leap in AI 01:02:06 Closing Thoughts: Preparing for a Radical Future
1:03:13

As ever, a fascinating chat with my friend Jordi Visser on AI, crypto, and hedge funds' future... Enjoy! 00:00:00 Introduction 00:00:54 The Big Picture: AI, Crypto, and Macro 00:02:25 Jordie Visser’s Background in Macro and Finance 00:03:30 AI’s Disruptive Impact on the Economy 00:06:32 Rethinking GDP and Productivity 00:08:59 The Future of Labor and Profit Margins 00:10:49 The Rise of the Speculative Economy 00:14:41 Generational Shifts in Investing and Risk-taking 00:16:18 AI and Crypto: A Global Network Effect 00:19:38 Stablecoins and AI-Driven Market Volume Growth 00:24:28 The End of Traditional Hedge Funds? 00:31:16 Deep Research and the Power of AI Analysis 00:35:29 The Acceleration of AI and its Economic Impact 00:41:02 The Growing Crypto Adoption and Its Role in the Future Economy 00:44:22 Shifting from Fear to Optimism in Markets 00:50:27 The Political and Economic Challenges Ahead 00:56:01 Quantum Computing and the Next Leap in AI 01:02:06 Closing Thoughts: Preparing for a Radical Future

Raoul Pal

280,191 görüntüleme • 1 yıl önce

Groq's LPU is faster than GPUs, handling requests and responding more quickly. Groq's LPUs don't need speedy data delivery like Nvidia GPUs do because they don't have HBM in their system. They use SRAM, which is about 20 times faster than what GPUs use. Since inference runs use way less data than model training, Groq's LPU is more energy-efficient. It reads less from external memory and uses less power than a Nvidia GPU for inference tasks. The LPU works differently from GPUs. It uses a Temporal Instruction Set Computer architecture, so it doesn't have to reload data from memory as often as GPUs do with High Bandwidth Memory (HBM). This helps avoid issues with HBM shortages and keeps costs down. If Groq's LPU is used in places that do AI processing, you might not need special storage for Nvidia GPUs. The LPU doesn't demand super-fast storage like GPUs do. Groq is claiming that its technology could replace GPUs in AI tasks with its powerful chip and software.
7:57

Groq's LPU is faster than GPUs, handling requests and responding more quickly. Groq's LPUs don't need speedy data delivery like Nvidia GPUs do because they don't have HBM in their system. They use SRAM, which is about 20 times faster than what GPUs use. Since inference runs use way less data than model training, Groq's LPU is more energy-efficient. It reads less from external memory and uses less power than a Nvidia GPU for inference tasks. The LPU works differently from GPUs. It uses a Temporal Instruction Set Computer architecture, so it doesn't have to reload data from memory as often as GPUs do with High Bandwidth Memory (HBM). This helps avoid issues with HBM shortages and keeps costs down. If Groq's LPU is used in places that do AI processing, you might not need special storage for Nvidia GPUs. The LPU doesn't demand super-fast storage like GPUs do. Groq is claiming that its technology could replace GPUs in AI tasks with its powerful chip and software.

Brian Roemmele

87,220 görüntüleme • 2 yıl önce

$MU $SKHY $DRAM Holy Shit $AMD's Lisa Su just spent two full minutes at the 2026 AI keynote raving about HBM4 capacity as the thing that separates Helios from the competition. Not compute. Memory. "When you compare Helios to the competition, we're delivering 15% more compute, 50% more HBM4 memory capacity and memory bandwidth, and 50% more scale-out bandwidth." Notice which number is bigger. "What that means is that every Helios can deliver more performance for the largest models, more capacity for longer context, and the bandwidth to scale across thousands of racks.... More than 18,000 CDNA five GPU compute units, over 4,600 Zen 6 CPU cores, and 31 terabytes of HBM4 memory, all in a single rack." "That's what it takes to run agentic AI at scale. Today, I'm excited to announce that Helios is in full production. We have shipments on track to start at the end of the third quarter and ramping into the fourth quarter in the second half of the year. I can tell you, customer demand for Helios is extremely strong, we're extremely proud of the work that we have done across the leading AI labs to adopt Helios."
1:53

$MU $SKHY $DRAM Holy Shit $AMD's Lisa Su just spent two full minutes at the 2026 AI keynote raving about HBM4 capacity as the thing that separates Helios from the competition. Not compute. Memory. "When you compare Helios to the competition, we're delivering 15% more compute, 50% more HBM4 memory capacity and memory bandwidth, and 50% more scale-out bandwidth." Notice which number is bigger. "What that means is that every Helios can deliver more performance for the largest models, more capacity for longer context, and the bandwidth to scale across thousands of racks.... More than 18,000 CDNA five GPU compute units, over 4,600 Zen 6 CPU cores, and 31 terabytes of HBM4 memory, all in a single rack." "That's what it takes to run agentic AI at scale. Today, I'm excited to announce that Helios is in full production. We have shipments on track to start at the end of the third quarter and ramping into the fourth quarter in the second half of the year. I can tell you, customer demand for Helios is extremely strong, we're extremely proud of the work that we have done across the leading AI labs to adopt Helios."

Trade Whisperer

180,469 görüntüleme • 1 ay önce

Everyone wants agent swarms. Very few people are talking seriously enough about the context layer that makes swarms useful. Even with one agent, context is fragile. Too little context and the agent guesses. Too much context and it wastes tokens, loses focus, or reasons over irrelevant noise. The sweet spot is precise context: the right knowledge, in the right structure, at the right moment. With many agents, that challenge explodes. Each agent produces decisions, assumptions, findings, summaries, risks, and partial conclusions. Unless that knowledge becomes shared, structured, and reusable, every new agent is forced to rediscover what another agent already learned. That is not a swarm. That is a crowd. Shared context graphs are what turn agent activity into agent collaboration, and OriginTrail DKG V10 brings them to life. Was just playing with some final polishing for the V10 release, and it is really powerful to see shared context graphs where multiple agents contribute knowledge into the same connected memory, with attribution visible directly in the graph ui. That matters for three reasons. First, agents can access and build on one shared memory instead of staying trapped in isolated sessions. Second, the graph structure helps them retrieve the exact context they need, instead of stuffing everything into a prompt and hoping the model sorts it out. Third, verifiability of provenance. You can see which agent contributed each piece of knowledge, trace the source, and decide what to trust. Tokenmaxxing starts with fewer tokens, but the deeper story is coordination - agents stop reloading the world and start building on shared, verifiable context. That is the foundation for serious multi-agent work across software engineering, research, finance, operations, project management, and far beyond. The future is not more agents, it is agents working from shared, verifiable context. But the more the merrier, of course.
0:22

Everyone wants agent swarms. Very few people are talking seriously enough about the context layer that makes swarms useful. Even with one agent, context is fragile. Too little context and the agent guesses. Too much context and it wastes tokens, loses focus, or reasons over irrelevant noise. The sweet spot is precise context: the right knowledge, in the right structure, at the right moment. With many agents, that challenge explodes. Each agent produces decisions, assumptions, findings, summaries, risks, and partial conclusions. Unless that knowledge becomes shared, structured, and reusable, every new agent is forced to rediscover what another agent already learned. That is not a swarm. That is a crowd. Shared context graphs are what turn agent activity into agent collaboration, and OriginTrail DKG V10 brings them to life. Was just playing with some final polishing for the V10 release, and it is really powerful to see shared context graphs where multiple agents contribute knowledge into the same connected memory, with attribution visible directly in the graph ui. That matters for three reasons. First, agents can access and build on one shared memory instead of staying trapped in isolated sessions. Second, the graph structure helps them retrieve the exact context they need, instead of stuffing everything into a prompt and hoping the model sorts it out. Third, verifiability of provenance. You can see which agent contributed each piece of knowledge, trace the source, and decide what to trust. Tokenmaxxing starts with fewer tokens, but the deeper story is coordination - agents stop reloading the world and start building on shared, verifiable context. That is the foundation for serious multi-agent work across software engineering, research, finance, operations, project management, and far beyond. The future is not more agents, it is agents working from shared, verifiable context. But the more the merrier, of course.

Jurij Skornik

11,180 görüntüleme • 3 ay önce