Загрузка видео...

Не удалось загрузить видео

На главную

Context memory essentially unlocks Agentic AI Much needed for Opus 4.6's "multi-agent swarms" In this SemiDoped pod, Vikram Sekar talks to Val Bercovici from Weka about context storage. - How token warehouses save inference costs - A new networking tier? Context Storage Network! - High Bandwidth Flash for context?...

semidoped's profile picture

Semi Doped

3,349 subscribers

12,796 просмотров • 8 месяцев назад •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

As AI context windows grow and inference workloads scale, storage architectures must keep pace. Huawei OceanStor M900 Context Memory Storage delivers massive shared capacity, high-performance data access, and optimized token costs for hyperscale AI data centers.
0:50

As AI context windows grow and inference workloads scale, storage architectures must keep pace. Huawei OceanStor M900 Context Memory Storage delivers massive shared capacity, high-performance data access, and optimized token costs for hyperscale AI data centers.

Huawei

8,338,135 просмотров • 23 дней назад

Breaking the Memory Wall: Micron’s Strategy for the AI Era Ben Bajarin and Jay Goldberg 顾忠南 talk micron's strategy for the AI era with Jeremy Werner SVP and GM of Micron’s Core Data Center Business Unit. Chapters: (00:00) Introduction and Guest Welcome (01:12) How the AI Cycle Differs from Past Volatility (04:23) Anticipating the Explosive Growth of AI (06:40) The Long-Term Sustainability of the AI Market (10:07) The "Memory Wall" in AI Inference (12:05) Explaining KV Cache and Context Windows (15:07) The Memory Hierarchy: From HBM to SSDs (20:21) Scaling for Billions of Simultaneous Users (23:05) Solving the Power and Efficiency Bottleneck (26:27) Why AI is Driving a Storage Shortage (29:57) Personal AI Agents and Persistent Memory Needs (34:13) Innovations in Ultra-High-Capacity SSDs (39:00) Accelerating the Pace of Engineering Innovation (43:50) Market Misunderstandings and the Future Potential of AI (47:38) The Global Race to Build New Fabs (51:34) Closing Remarks
52:31

Breaking the Memory Wall: Micron’s Strategy for the AI Era Ben Bajarin and Jay Goldberg 顾忠南 talk micron's strategy for the AI era with Jeremy Werner SVP and GM of Micron’s Core Data Center Business Unit. Chapters: (00:00) Introduction and Guest Welcome (01:12) How the AI Cycle Differs from Past Volatility (04:23) Anticipating the Explosive Growth of AI (06:40) The Long-Term Sustainability of the AI Market (10:07) The "Memory Wall" in AI Inference (12:05) Explaining KV Cache and Context Windows (15:07) The Memory Hierarchy: From HBM to SSDs (20:21) Scaling for Billions of Simultaneous Users (23:05) Solving the Power and Efficiency Bottleneck (26:27) Why AI is Driving a Storage Shortage (29:57) Personal AI Agents and Persistent Memory Needs (34:13) Innovations in Ultra-High-Capacity SSDs (39:00) Accelerating the Pace of Engineering Innovation (43:50) Market Misunderstandings and the Future Potential of AI (47:38) The Global Race to Build New Fabs (51:34) Closing Remarks

The Circuit

94,438 просмотров • 6 месяцев назад

Google just released a 1-hour course on “Context Engineering” for AI agents from scratch: • 00:00 - Introduction to “Context Engineering” • 03:02 - Context Engineering vs “prompting” • 16:18 - Build a context brain for your agent • 22:10 - Building agent context (tools, skills) • 29:06 - Agentic memory bank from scratch This 1-hour watch could replace a $500 course on agentic design. Watch it today, then read the full guide on Context Engineering in the article below.
56:30

Google just released a 1-hour course on “Context Engineering” for AI agents from scratch: • 00:00 - Introduction to “Context Engineering” • 03:02 - Context Engineering vs “prompting” • 16:18 - Build a context brain for your agent • 22:10 - Building agent context (tools, skills) • 29:06 - Agentic memory bank from scratch This 1-hour watch could replace a $500 course on agentic design. Watch it today, then read the full guide on Context Engineering in the article below.

Movez

14,923 просмотров • 3 месяцев назад

New short course: LLMs as Operating Systems: Agent Memory, created with Letta, and taught by its founders Charles Packer and Sarah Wooders. An LLM's input context window has limited space. Using a longer input context also costs more and results in slower processing. So, managing what's stored in this context window is important. In the innovative paper MemGPT: Towards LLMs as Operating Systems, its authors (which include the instructors) proposed using an LLM agent to manage this context window. Their system uses a large persistent memory that stores everything that could be included in the input context, and an agent decides what is actually included. Take the example of building a chatbot that needs to remember what's been said earlier in a conversation (perhaps over many days of interaction with a user). As the conversation's length grows, the memory management agent will move information from the input context to a persistent searchable database; summarize information to keep relevant facts in the input context; and restore relevant conversation elements from further back in time. This allows a chatbot to keep what's currently most relevant in its input context memory to generate the next response. When I read the original MemGPT paper, I thought it was an innovative technique for handling memory for LLMs. The open-source Letta framework, which we'll use in this course, makes MemGPT easy to implement. It adds memory to your LLM agents and gives them transparent long-term memory. In detail, you’ll learn: - How to build an agent that can edit its own limited input context memory, using tools and multi-step reasoning - What is a memory hierarchy (an idea from computer operating systems, which use a cache to speed up memory access), and how these ideas apply to managing the LLM input context (where the input context window is a "cache" storing the most relevant information; and an agent decides what to move in and out of this to/from a larger persistent storage system) - How to implement multi-agent collaboration by letting different agents share blocks of memory This course will give you a sophisticated understanding of memory management for LLMs, which is important for chatbots having long conversations, and for complex agentic workflows. Please sign up here!
2:57

New short course: LLMs as Operating Systems: Agent Memory, created with Letta, and taught by its founders Charles Packer and Sarah Wooders. An LLM's input context window has limited space. Using a longer input context also costs more and results in slower processing. So, managing what's stored in this context window is important. In the innovative paper MemGPT: Towards LLMs as Operating Systems, its authors (which include the instructors) proposed using an LLM agent to manage this context window. Their system uses a large persistent memory that stores everything that could be included in the input context, and an agent decides what is actually included. Take the example of building a chatbot that needs to remember what's been said earlier in a conversation (perhaps over many days of interaction with a user). As the conversation's length grows, the memory management agent will move information from the input context to a persistent searchable database; summarize information to keep relevant facts in the input context; and restore relevant conversation elements from further back in time. This allows a chatbot to keep what's currently most relevant in its input context memory to generate the next response. When I read the original MemGPT paper, I thought it was an innovative technique for handling memory for LLMs. The open-source Letta framework, which we'll use in this course, makes MemGPT easy to implement. It adds memory to your LLM agents and gives them transparent long-term memory. In detail, you’ll learn: - How to build an agent that can edit its own limited input context memory, using tools and multi-step reasoning - What is a memory hierarchy (an idea from computer operating systems, which use a cache to speed up memory access), and how these ideas apply to managing the LLM input context (where the input context window is a "cache" storing the most relevant information; and an agent decides what to move in and out of this to/from a larger persistent storage system) - How to implement multi-agent collaboration by letting different agents share blocks of memory This course will give you a sophisticated understanding of memory management for LLMs, which is important for chatbots having long conversations, and for complex agentic workflows. Please sign up here!

Andrew Ng

201,127 просмотров • 2 лет назад

Introducing HydraDB. The graph native context infrastructure for agents. Purpose built to deliver precise context & observability into why agents act the way they do. We've always believed graphs are the best way to manage AI context, but they've been too expensive to scale or impractical for storing full context. Until now. HydraDB combines in memory, NVMe, and object storage into a single graph layer, making context delivery faster, cheaper, and more precise. We want context delivery to be extremely fast, 1000x cheap, and highly precise. Give your agents a brain.
1:18

Introducing HydraDB. The graph native context infrastructure for agents. Purpose built to deliver precise context & observability into why agents act the way they do. We've always believed graphs are the best way to manage AI context, but they've been too expensive to scale or impractical for storing full context. Until now. HydraDB combines in memory, NVMe, and object storage into a single graph layer, making context delivery faster, cheaper, and more precise. We want context delivery to be extremely fast, 1000x cheap, and highly precise. Give your agents a brain.

Nishkarsh

2,303,360 просмотров • 5 месяцев назад

Dylan Patel on the importance of memory and storage Two key quotes: "An $NVDA GPU is faster than an $AMD GPU in most cases, but because AMD GPUs have more memory, they can outperform Nvidia in certain workloads." “It is a difficult, multivariable problem. Generally, you need the best GPU, such as a GB300, but you also need the best storage solutions. I will not spoil who comes out on top, but storage solutions matter a lot, memory solutions matter a lot, and frontend networking also matters significantly" Full Quote: “We have over $80 million of compute: GPUs from $NVDA and $AMD, TPUs from Google, and Trainium from Amazon. We constantly run this benchmark using the newest inference engines, drivers, PyTorch versions, and other software. It runs every day through automated CI across the latest Chinese models from GLM, Zhipu, Moonshot, Kimi, Alibaba, and others. Initially, when we were benchmarking the differences between these chips, inference engines, and parallelism schemes, we used fixed context lengths. But with Agent X, we have now analyzed more than $5 million worth of Claude Code traces. This is real production traffic that users have donated to us, combined with internally generated data, so we now understand what an actual agent workload looks like. When we implement those workloads and run the benchmarks, it turns out that the chip you are using is very important, but how you handle memory offload can be even more important. An Nvidia GPU is faster than an AMD GPU in most cases, but because AMD GPUs have more memory, they can outperform Nvidia in certain workloads. Similarly, you can use a less powerful GPU with a much better storage solution and outperform the best GPU when it lacks those solutions. Simply buying the newest GPU does not necessarily give you the best inference economics. You need to layer in other innovations, including storage and memory.” Interviewer: “Who is the top player on your chart? Can you tell us?” Dylan Patel: “It is a difficult, multivariable problem. Generally, you need the best GPU, such as a GB300, but you also need the best storage solutions. I will not spoil who comes out on top, but storage solutions matter a lot, memory solutions matter a lot, and frontend networking also matters significantly.”
2:27

Dylan Patel on the importance of memory and storage Two key quotes: "An $NVDA GPU is faster than an $AMD GPU in most cases, but because AMD GPUs have more memory, they can outperform Nvidia in certain workloads." “It is a difficult, multivariable problem. Generally, you need the best GPU, such as a GB300, but you also need the best storage solutions. I will not spoil who comes out on top, but storage solutions matter a lot, memory solutions matter a lot, and frontend networking also matters significantly" Full Quote: “We have over $80 million of compute: GPUs from $NVDA and $AMD, TPUs from Google, and Trainium from Amazon. We constantly run this benchmark using the newest inference engines, drivers, PyTorch versions, and other software. It runs every day through automated CI across the latest Chinese models from GLM, Zhipu, Moonshot, Kimi, Alibaba, and others. Initially, when we were benchmarking the differences between these chips, inference engines, and parallelism schemes, we used fixed context lengths. But with Agent X, we have now analyzed more than $5 million worth of Claude Code traces. This is real production traffic that users have donated to us, combined with internally generated data, so we now understand what an actual agent workload looks like. When we implement those workloads and run the benchmarks, it turns out that the chip you are using is very important, but how you handle memory offload can be even more important. An Nvidia GPU is faster than an AMD GPU in most cases, but because AMD GPUs have more memory, they can outperform Nvidia in certain workloads. Similarly, you can use a less powerful GPU with a much better storage solution and outperform the best GPU when it lacks those solutions. Simply buying the newest GPU does not necessarily give you the best inference economics. You need to layer in other innovations, including storage and memory.” Interviewer: “Who is the top player on your chart? Can you tell us?” Dylan Patel: “It is a difficult, multivariable problem. Generally, you need the best GPU, such as a GB300, but you also need the best storage solutions. I will not spoil who comes out on top, but storage solutions matter a lot, memory solutions matter a lot, and frontend networking also matters significantly.”

Daniel Romero

48,978 просмотров • 3 месяцев назад

Vik and Val Bercovici (WEKA) map where AI inference memory is headed. Every 100x cut in KV cache gets swallowed by ~10,000x more usage, so demand climbs. - NVLink beats the board: 128 lanes vs 32 PCIe - WEKA serves NAND-backed storage faster than DRAM over network - DeepSeek's cache reads run ~87x cheaper, China only - CXL needs a dedicated bus, WEKA pools NAND over RDMA/NVLink instead - Val's call: SaaS giants and Neoclouds must merge Chapters: 0:00 Intro and memory prices 3:00 Model routing and offloading 5:10 Network faster than motherboard 13:06 Four-tier memory hierarchy 19:40 KV cache and Jevons paradox 23:50 DeepSeek cache read pricing 31:40 Sliding window attention 34:49 NAND tiers, SLC vs QLC 43:57 High bandwidth flash use cases 49:59 CXL versus NVLink 53:44 AMD Mex and small models 1:03:14 SaaS, Neoclouds and tokenomics Get more of Austin and Vik daily, free! Sign up: Connect with Vik and Austin: Vik's Paid Substack: Austin's Paid Substack: Austin Lyons Vikram Sekar
1:04:11

Vik and Val Bercovici (WEKA) map where AI inference memory is headed. Every 100x cut in KV cache gets swallowed by ~10,000x more usage, so demand climbs. - NVLink beats the board: 128 lanes vs 32 PCIe - WEKA serves NAND-backed storage faster than DRAM over network - DeepSeek's cache reads run ~87x cheaper, China only - CXL needs a dedicated bus, WEKA pools NAND over RDMA/NVLink instead - Val's call: SaaS giants and Neoclouds must merge Chapters: 0:00 Intro and memory prices 3:00 Model routing and offloading 5:10 Network faster than motherboard 13:06 Four-tier memory hierarchy 19:40 KV cache and Jevons paradox 23:50 DeepSeek cache read pricing 31:40 Sliding window attention 34:49 NAND tiers, SLC vs QLC 43:57 High bandwidth flash use cases 49:59 CXL versus NVLink 53:44 AMD Mex and small models 1:03:14 SaaS, Neoclouds and tokenomics Get more of Austin and Vik daily, free! Sign up: Connect with Vik and Austin: Vik's Paid Substack: Austin's Paid Substack: Austin Lyons Vikram Sekar

Semi Doped

61,062 просмотров • 3 месяцев назад

Ex-Google engineer just compressed the shift from AI agents to "graphs" and "loops" into one 2h47m lecture: • 00:00 - understanding the different layers of AI memory • 30:00 - moving from simple LLM calls to AI agents • 50:00 - why agent systems are becoming a graph engineering problem • 1:12:00 - connecting context, tools and agent state • 1:24:00 - building a minimal agent harness from scratch • 1:46:00 - breaking down the architecture behind Hermes Agent • 2:08:00 - turning agent concepts into working systems 167-minute deep dive, and one of the clearest ways to understand how AI engineering is moving beyond prompts The progression: Prompt → Context → Memory → Agent → Loop → Graph → Agent Harness Watch it today, then read the full roadmap below
2:47:42

Ex-Google engineer just compressed the shift from AI agents to "graphs" and "loops" into one 2h47m lecture: • 00:00 - understanding the different layers of AI memory • 30:00 - moving from simple LLM calls to AI agents • 50:00 - why agent systems are becoming a graph engineering problem • 1:12:00 - connecting context, tools and agent state • 1:24:00 - building a minimal agent harness from scratch • 1:46:00 - breaking down the architecture behind Hermes Agent • 2:08:00 - turning agent concepts into working systems 167-minute deep dive, and one of the clearest ways to understand how AI engineering is moving beyond prompts The progression: Prompt → Context → Memory → Agent → Loop → Graph → Agent Harness Watch it today, then read the full roadmap below

Morlex

51,952 просмотров • 2 месяцев назад

Average day when you have 20% of your portfolio in memory and storage
0:27

Average day when you have 20% of your portfolio in memory and storage

Cluseau Investments

148,672 просмотров • 4 месяцев назад

CONTEXT ENGINEERING > PROMPT ENGINEERING Everyone is obsessed with writing better prompts. The next generation of AI builders is focused on context engineering instead. • Prompt engineering shapes the question. Context engineering shapes everything the AI sees. • Great prompts can't save an agent missing critical context, memory, or tools. • AI failures often come from missing information, not weak models. • Context engineering combines prompts, memory, RAG, state management, and tool access into one system. What powers effective AI agents? → Memory: Remembers preferences, past interactions, and ongoing tasks. → State Management: Tracks progress across multi-step workflows. → RAG: Retrieves only the most relevant information when needed. → Tools: Connects AI to APIs, databases, code execution, and real-world actions. → Dynamic Prompts: Enriches instructions with live context at runtime. The key insight: Prompt Engineering = Better Questions Context Engineering = Better Systems The future of AI isn't building smarter prompts. It's building smarter environments for AI to think, remember, retrieve, and act.
7:51

CONTEXT ENGINEERING > PROMPT ENGINEERING Everyone is obsessed with writing better prompts. The next generation of AI builders is focused on context engineering instead. • Prompt engineering shapes the question. Context engineering shapes everything the AI sees. • Great prompts can't save an agent missing critical context, memory, or tools. • AI failures often come from missing information, not weak models. • Context engineering combines prompts, memory, RAG, state management, and tool access into one system. What powers effective AI agents? → Memory: Remembers preferences, past interactions, and ongoing tasks. → State Management: Tracks progress across multi-step workflows. → RAG: Retrieves only the most relevant information when needed. → Tools: Connects AI to APIs, databases, code execution, and real-world actions. → Dynamic Prompts: Enriches instructions with live context at runtime. The key insight: Prompt Engineering = Better Questions Context Engineering = Better Systems The future of AI isn't building smarter prompts. It's building smarter environments for AI to think, remember, retrieve, and act.

Dami-Defi

10,144 просмотров • 4 месяцев назад

Stop prompt-stuffing yesterday’s context into your agent. Give it verifiable memory. 🕸️ When memory has provenance, agents can reuse it, audit it, and carry it across tools. Brana Rakic, Shaw (spirit/acc) (elizaOS), Ortecha | data made human & JoeVezz (LunarCrush 🌕™) break it down. 00:57 – What agents need to scale 05:45 – Social data, signals & real-time context 13:42 – Knowledge graphs as shared memory for AI agents 29:13 – Interoperability, trust & coordination
37:35

Stop prompt-stuffing yesterday’s context into your agent. Give it verifiable memory. 🕸️ When memory has provenance, agents can reuse it, audit it, and carry it across tools. Brana Rakic, Shaw (spirit/acc) (elizaOS), Ortecha | data made human & JoeVezz (LunarCrush 🌕™) break it down. 00:57 – What agents need to scale 05:45 – Social data, signals & real-time context 13:42 – Knowledge graphs as shared memory for AI agents 29:13 – Interoperability, trust & coordination

OriginTrail

13,293 просмотров • 8 месяцев назад

Google just showed how AI engineers are moving from simple "RAG" to "graphs", memory and multimodal agents • 15:32 - setting up the production agent stack • 28:00 - turning disconnected data into a knowledge graph • 41:00 - Graph RAG with semantic + hybrid search • 58:00 - extracting graph context from images, text and video • 1:09:00 - orchestrating specialized agents with ADK • 1:20:00 - giving agents persistent memory across sessions 94-minute Google Cloud workshop, and it’s one of the clearest hands-on examples of the shift from -> RAG → Graph RAG → Memory → Multimodal Agent Graphs. The workshop uses Spanner Graph, Gemini, ADK and Memory Bank to build that stack. Watch it today, then read the full “From RAG to Context Graphs” roadmap below.
1:30:26

Google just showed how AI engineers are moving from simple "RAG" to "graphs", memory and multimodal agents • 15:32 - setting up the production agent stack • 28:00 - turning disconnected data into a knowledge graph • 41:00 - Graph RAG with semantic + hybrid search • 58:00 - extracting graph context from images, text and video • 1:09:00 - orchestrating specialized agents with ADK • 1:20:00 - giving agents persistent memory across sessions 94-minute Google Cloud workshop, and it’s one of the clearest hands-on examples of the shift from -> RAG → Graph RAG → Memory → Multimodal Agent Graphs. The workshop uses Spanner Graph, Gemini, ADK and Memory Bank to build that stack. Watch it today, then read the full “From RAG to Context Graphs” roadmap below.

Morlex

74,975 просмотров • 2 месяцев назад

If $MU says NAND pricing is still strong, enterprise SSD demand is picking up and AI inference is creating more storage demand then $SNDK should benefit because SanDisk is one of the more direct ways to own that part of the cycle. Hyperscalers are still figuring out the right mix of DRAM, HBM, TLC NAND, QLC NAND and other storage architectures which can keep supply tighter for longer as memory companies are going to be hesitant to add too much capacity before demand becomes clearer. I'm going to be listening closely for Micron’s QLC commentary because QLC NAND for read-heavy inference workloads is the big driver and if Micron calls that out then it would show that the storage side of the AI memory story is still earlier than DRAM.
1:03

If $MU says NAND pricing is still strong, enterprise SSD demand is picking up and AI inference is creating more storage demand then $SNDK should benefit because SanDisk is one of the more direct ways to own that part of the cycle. Hyperscalers are still figuring out the right mix of DRAM, HBM, TLC NAND, QLC NAND and other storage architectures which can keep supply tighter for longer as memory companies are going to be hesitant to add too much capacity before demand becomes clearer. I'm going to be listening closely for Micron’s QLC commentary because QLC NAND for read-heavy inference workloads is the big driver and if Micron calls that out then it would show that the storage side of the AI memory story is still earlier than DRAM.

Shay Boloor

108,953 просмотров • 4 месяцев назад

holy, competition is heating up a lot Anthropic introduces a memory feature that lets users transfer their context and preferences from other AI tools into Claude by copying a generated prompt and pasting the result into Claude’s memory settings. This allows Claude to immediately continue conversations with retained context, available for all paid plans.
0:59

holy, competition is heating up a lot Anthropic introduces a memory feature that lets users transfer their context and preferences from other AI tools into Claude by copying a generated prompt and pasting the result into Claude’s memory settings. This allows Claude to immediately continue conversations with retained context, available for all paid plans.

Chubby♨️

364,269 просмотров • 8 месяцев назад

Sat down with Aaron Levie, CEO of Box, to discuss where AI agents are going, how agentic workflows are transforming software, why agents will require much more memory, and how token budgets for enterprises are evolving. Thank you Aaron Levie for taking the time! 00:00 - Intro 01:30 - Where are we in AI 06:25 - The Role of Agents in Software 11:40 - The Role of Memory for Agents 14:45 - TokenMaxxing 24:54 - Why Are There AI Doomers? 31:50 - Aaron's Take on Nvidia
32:57

Sat down with Aaron Levie, CEO of Box, to discuss where AI agents are going, how agentic workflows are transforming software, why agents will require much more memory, and how token budgets for enterprises are evolving. Thank you Aaron Levie for taking the time! 00:00 - Intro 01:30 - Where are we in AI 06:25 - The Role of Agents in Software 11:40 - The Role of Memory for Agents 14:45 - TokenMaxxing 24:54 - Why Are There AI Doomers? 31:50 - Aaron's Take on Nvidia

amit

96,221 просмотров • 4 месяцев назад

Dylan Patel of SemiAnalysis says a worse GPU with better storage and memory now beats the best chip without them, so buying the newest GPU alone no longer wins inference. So, an AMD GPU with more memory can outperform Nvidia in some cases. "So what we have is we have over $80 million of compute, GPUs from Nvidia, AMD, TPUs from Google, Trainium from Amazon, and we run this benchmark constantly on the newest inference engine, newest drivers, newest PyTorch version, whatever it is." "Every day it runs on an automated CI, and we run it on all the latest Chinese models, from GLM, Zhipu, Moonshot, Kimi, Alibaba, all these models we run." "Initially, when we were benchmarking the difference between these chips and different engines, different schemes for parallelism, we were just running it fixed context length." "But now with Agent X, we've analyzed over $5 million worth of Claude Code traces. This is real production traffic that people have donated to us as well as internally generated. Now we know what the actual agent workload looks like." "And then as we implement that and run those benchmarks, it turns out yes, the chip you're using is very important, but now even more important is how are you handling this memory offload?" "And so while an Nvidia GPU is faster than an AMD GPU in most cases, because AMD GPUs have more memory, they actually end up outperforming in some cases." "Or you can have a worse GPU, but a much better storage solution, and now you can outperform what the best GPU can do without those solutions. So just buying the newest and latest GPU alone doesn't get you the best inference economics." "Actually, you need to layer in all these other innovations including storage and memory." [ Who's the top player on your chart? ] "That really is a difficult multivariable problem. And generally that means you need to have, yes, you need to have the best GPU, a GB300, but you also need to have the best storage solutions. And so I won't spoil who's the best right here, but I will say that storage solutions matter a lot and memory solutions matter a lot, as does your front-end networking. That matters a lot."
2:00

Dylan Patel of SemiAnalysis says a worse GPU with better storage and memory now beats the best chip without them, so buying the newest GPU alone no longer wins inference. So, an AMD GPU with more memory can outperform Nvidia in some cases. "So what we have is we have over $80 million of compute, GPUs from Nvidia, AMD, TPUs from Google, Trainium from Amazon, and we run this benchmark constantly on the newest inference engine, newest drivers, newest PyTorch version, whatever it is." "Every day it runs on an automated CI, and we run it on all the latest Chinese models, from GLM, Zhipu, Moonshot, Kimi, Alibaba, all these models we run." "Initially, when we were benchmarking the difference between these chips and different engines, different schemes for parallelism, we were just running it fixed context length." "But now with Agent X, we've analyzed over $5 million worth of Claude Code traces. This is real production traffic that people have donated to us as well as internally generated. Now we know what the actual agent workload looks like." "And then as we implement that and run those benchmarks, it turns out yes, the chip you're using is very important, but now even more important is how are you handling this memory offload?" "And so while an Nvidia GPU is faster than an AMD GPU in most cases, because AMD GPUs have more memory, they actually end up outperforming in some cases." "Or you can have a worse GPU, but a much better storage solution, and now you can outperform what the best GPU can do without those solutions. So just buying the newest and latest GPU alone doesn't get you the best inference economics." "Actually, you need to layer in all these other innovations including storage and memory." [ Who's the top player on your chart? ] "That really is a difficult multivariable problem. And generally that means you need to have, yes, you need to have the best GPU, a GB300, but you also need to have the best storage solutions. And so I won't spoil who's the best right here, but I will say that storage solutions matter a lot and memory solutions matter a lot, as does your front-end networking. That matters a lot."

Fireside Alpha

185,279 просмотров • 3 месяцев назад

⚒️ How to build an AI agent with persistent memory? Watch Shaw building an agent from scratch with elizaOS & showing how external, queryable memory based on OriginTrail gives agents context and extends their memory beyond a single session. 02:15 – Set up your first agent 08:55 – Initializing runtime & memory foundations 12:50 – Wiring message handling & agent memory 24:45 – Persistent conversational memory in action 38:00 – Extending agents with plugins
1:05:27

⚒️ How to build an AI agent with persistent memory? Watch Shaw building an agent from scratch with elizaOS & showing how external, queryable memory based on OriginTrail gives agents context and extends their memory beyond a single session. 02:15 – Set up your first agent 08:55 – Initializing runtime & memory foundations 12:50 – Wiring message handling & agent memory 24:45 – Persistent conversational memory in action 38:00 – Extending agents with plugins

OriginTrail

37,374 просмотров • 9 месяцев назад