Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Microsoft ran its graph system against vector RAG on 8k, 120k, and a full million token context window, and the million token giant lost Jonathan Larson from Microsoft Research put up the numbers → LazyGraphRAG won 92, 90, and 91 percent of data-local questions, the exact place plain RAG... show more

NO1ennn

4,390 subscribers

29,189 views • 2 months ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

A million-token window solved the wrong problem, and Kimi K3 shipping one made that harder to see. 2.8T parameters, weights on your disk, an entire codebase in one prompt. Feels like memory. Isn't. Memory is what survives the session ending. A window is what survives the next token. Self-hosting sharpens it. Now you pay in GPU seconds, not API credits. Every re-read of the same history burns the same compute. Owning the weights didn't make the past free. Kimi Delta Attention cuts what long context costs. Cheaper repetition is still repetition. And a bigger window buys the one thing you don't want: room to keep everything. Nothing in a transcript is dated, ranked, or retired. Old and current load at identical weight. The fix runs beside the model, not inside it. Capture rejects, consolidation merges, decay forgets on schedule. Then a million tokens becomes headroom instead of a bill. You fill 5,000 and leave the rest empty. Look at what your longest thread reloads every run. Then ask how much of it earned the slot.

A million-token window solved the wrong problem, and Kimi K3 shipping one made that harder to see. 2.8T parameters, weights on your disk, an entire codebase in one prompt. Feels like memory. Isn't. Memory is what survives the session ending. A window is what survives the next token. Self-hosting sharpens it. Now you pay in GPU seconds, not API credits. Every re-read of the same history burns the same compute. Owning the weights didn't make the past free. Kimi Delta Attention cuts what long context costs. Cheaper repetition is still repetition. And a bigger window buys the one thing you don't want: room to keep everything. Nothing in a transcript is dated, ranked, or retired. Old and current load at identical weight. The fix runs beside the model, not inside it. Capture rejects, consolidation merges, decay forgets on schedule. Then a million tokens becomes headroom instead of a bill. You fill 5,000 and leave the rest empty. Look at what your longest thread reloads every run. Then ask how much of it earned the slot.

wast3

17,620 views • 1 month ago

MICROSOFT QUIETLY BUILT ONE MEMORY MODULE THAT PLUGS INTO ANY AI AGENT AND CUTS ITS CONTEXT BY UP TO 100X no retraining. no task-specific redesign. you just plug it in Microsoft's insight: agents don't need to remember what happened. they need the knowledge pulled from it - the facts and the reusable skills, not the raw logs so PlugMem stops storing history. it compiles experience into a knowledge graph, then feeds the agent only what matters the result: > up to 1-2 orders of magnitude fewer tokens > highest decision-utility per token of any system tested > one unchanged module beats task-specific memory on chat QA, multi-hop search, and web agents more memory was never the answer. better memory is - and Microsoft just shipped it as a plug-in bookmark this. the agents that win won't remember more, they'll remember smarter.

MICROSOFT QUIETLY BUILT ONE MEMORY MODULE THAT PLUGS INTO ANY AI AGENT AND CUTS ITS CONTEXT BY UP TO 100X no retraining. no task-specific redesign. you just plug it in Microsoft's insight: agents don't need to remember what happened. they need the knowledge pulled from it - the facts and the reusable skills, not the raw logs so PlugMem stops storing history. it compiles experience into a knowledge graph, then feeds the agent only what matters the result: > up to 1-2 orders of magnitude fewer tokens > highest decision-utility per token of any system tested > one unchanged module beats task-specific memory on chat QA, multi-hop search, and web agents more memory was never the answer. better memory is - and Microsoft just shipped it as a plug-in bookmark this. the agents that win won't remember more, they'll remember smarter.

NO1ennn

227,683 views • 2 months ago

Micron is going to $4,000 and once you understand what inference actually is, the number stops sounding crazy (Save this). Dylan Patel just said that by 2030, OpenAI and Anthropic alone will need over 100 gigawatts of compute combined and by 2040, we may not even be measuring AI infrastructure in gigawatts anymore. We may be talking about terawatts. Every single one of those gigawatts needs memory to function. Without it, the compute is worthless. Most people heard that and thought about Nvidia but they should be thinking about Micron. Every AI model generating a response has two phases. The first is prefill, processing your prompt which is compute-heavy and the second is decode generating each word one token at a time and that phase is almost entirely memory-bound, not compute-bound. During decode, the GPU's processing units sit idle more than 95% of the time, waiting for data to arrive from memory. Google confirmed it in a research paper that decode-phase bottlenecks are dominated by memory bandwidth and capacity not raw compute. The GPU is not the bottleneck but the memory feeding the GPU is. This matters because inference is now where all the money lives. Training a model happens once, Inference happens billions of times a day every ChatGPT response, every Claude output, every agentic workflow running in the background and every one of those token streams is a billing event tied directly to memory performance. Adding more GPUs does not fix this because GPUs are already underutilized in inference because they are sitting idle waiting on memory. Adding more memory bandwidth and capacity is what directly reduces token cost, reduces latency, and allows the same cluster to serve dramatically more users simultaneously. Longer context windows compound the problem further, a model running a 1 million token context window requires dramatically more memory per session than a 10,000 token window, and every new model generation pushes context longer. The market treats memory as a downstream beneficiary of Nvidia orders. The correct framework is the opposite, Micron is the upstream constraint on how much value every Nvidia GPU can actually generate at inference scale. Micron guided Q4 to $50 billion in revenue, has HBM4 ramping at twice the pace of the prior generation, and CEO Sanjay Mehrotra has said supply will not catch demand before the end of 2027. At 8x forward earnings on $112 projected FY2027 EPS, Micron is the most undervalued infrastructure company in the entire AI stack. Inference is memory. Memory is Micron and the inference ramp has barely started. Milk Road Pro members are already up massively on this position and we're just getting started. If you want the full breakdown of what we're buying and why, come join us for just a dollar using the link below!

Micron is going to $4,000 and once you understand what inference actually is, the number stops sounding crazy (Save this). Dylan Patel just said that by 2030, OpenAI and Anthropic alone will need over 100 gigawatts of compute combined and by 2040, we may not even be measuring AI infrastructure in gigawatts anymore. We may be talking about terawatts. Every single one of those gigawatts needs memory to function. Without it, the compute is worthless. Most people heard that and thought about Nvidia but they should be thinking about Micron. Every AI model generating a response has two phases. The first is prefill, processing your prompt which is compute-heavy and the second is decode generating each word one token at a time and that phase is almost entirely memory-bound, not compute-bound. During decode, the GPU's processing units sit idle more than 95% of the time, waiting for data to arrive from memory. Google confirmed it in a research paper that decode-phase bottlenecks are dominated by memory bandwidth and capacity not raw compute. The GPU is not the bottleneck but the memory feeding the GPU is. This matters because inference is now where all the money lives. Training a model happens once, Inference happens billions of times a day every ChatGPT response, every Claude output, every agentic workflow running in the background and every one of those token streams is a billing event tied directly to memory performance. Adding more GPUs does not fix this because GPUs are already underutilized in inference because they are sitting idle waiting on memory. Adding more memory bandwidth and capacity is what directly reduces token cost, reduces latency, and allows the same cluster to serve dramatically more users simultaneously. Longer context windows compound the problem further, a model running a 1 million token context window requires dramatically more memory per session than a 10,000 token window, and every new model generation pushes context longer. The market treats memory as a downstream beneficiary of Nvidia orders. The correct framework is the opposite, Micron is the upstream constraint on how much value every Nvidia GPU can actually generate at inference scale. Micron guided Q4 to $50 billion in revenue, has HBM4 ramping at twice the pace of the prior generation, and CEO Sanjay Mehrotra has said supply will not catch demand before the end of 2027. At 8x forward earnings on $112 projected FY2027 EPS, Micron is the most undervalued infrastructure company in the entire AI stack. Inference is memory. Memory is Micron and the inference ramp has barely started. Milk Road Pro members are already up massively on this position and we're just getting started. If you want the full breakdown of what we're buying and why, come join us for just a dollar using the link below!

Milk Road AI

130,756 views • 3 months ago

Google just showed how AI engineers are moving from simple "RAG" to "graphs", memory and multimodal agents • 15:32 - setting up the production agent stack • 28:00 - turning disconnected data into a knowledge graph • 41:00 - Graph RAG with semantic + hybrid search • 58:00 - extracting graph context from images, text and video • 1:09:00 - orchestrating specialized agents with ADK • 1:20:00 - giving agents persistent memory across sessions 94-minute Google Cloud workshop, and it’s one of the clearest hands-on examples of the shift from -> RAG → Graph RAG → Memory → Multimodal Agent Graphs. The workshop uses Spanner Graph, Gemini, ADK and Memory Bank to build that stack. Watch it today, then read the full “From RAG to Context Graphs” roadmap below.

Google just showed how AI engineers are moving from simple "RAG" to "graphs", memory and multimodal agents • 15:32 - setting up the production agent stack • 28:00 - turning disconnected data into a knowledge graph • 41:00 - Graph RAG with semantic + hybrid search • 58:00 - extracting graph context from images, text and video • 1:09:00 - orchestrating specialized agents with ADK • 1:20:00 - giving agents persistent memory across sessions 94-minute Google Cloud workshop, and it’s one of the clearest hands-on examples of the shift from -> RAG → Graph RAG → Memory → Multimodal Agent Graphs. The workshop uses Spanner Graph, Gemini, ADK and Memory Bank to build that stack. Watch it today, then read the full “From RAG to Context Graphs” roadmap below.

Morlex

74,134 views • 2 months ago

Rockstar spent 7 years, 2,000+ developers, and $500 million to build Red Dead Redemption 2. Kimi K3 rebuilt the same scene in 10 hours. One prompt. 4 million tokens. It's open source. 2.8 trillion parameters. A 1 million token context window. It beats Claude Opus 4.8 and GPT-5.6. It sits on par with Fable 5. Fable 5 results, at Sonnet pricing. The wall between you and a AAA game used to be a studio. Now it's a prompt.

Rockstar spent 7 years, 2,000+ developers, and $500 million to build Red Dead Redemption 2. Kimi K3 rebuilt the same scene in 10 hours. One prompt. 4 million tokens. It's open source. 2.8 trillion parameters. A 1 million token context window. It beats Claude Opus 4.8 and GPT-5.6. It sits on par with Fable 5. Fable 5 results, at Sonnet pricing. The wall between you and a AAA game used to be a studio. Now it's a prompt.

Dep

99,354 views • 2 months ago

New short course: LLMs as Operating Systems: Agent Memory, created with Letta, and taught by its founders Charles Packer and Sarah Wooders. An LLM's input context window has limited space. Using a longer input context also costs more and results in slower processing. So, managing what's stored in this context window is important. In the innovative paper MemGPT: Towards LLMs as Operating Systems, its authors (which include the instructors) proposed using an LLM agent to manage this context window. Their system uses a large persistent memory that stores everything that could be included in the input context, and an agent decides what is actually included. Take the example of building a chatbot that needs to remember what's been said earlier in a conversation (perhaps over many days of interaction with a user). As the conversation's length grows, the memory management agent will move information from the input context to a persistent searchable database; summarize information to keep relevant facts in the input context; and restore relevant conversation elements from further back in time. This allows a chatbot to keep what's currently most relevant in its input context memory to generate the next response. When I read the original MemGPT paper, I thought it was an innovative technique for handling memory for LLMs. The open-source Letta framework, which we'll use in this course, makes MemGPT easy to implement. It adds memory to your LLM agents and gives them transparent long-term memory. In detail, you’ll learn: - How to build an agent that can edit its own limited input context memory, using tools and multi-step reasoning - What is a memory hierarchy (an idea from computer operating systems, which use a cache to speed up memory access), and how these ideas apply to managing the LLM input context (where the input context window is a "cache" storing the most relevant information; and an agent decides what to move in and out of this to/from a larger persistent storage system) - How to implement multi-agent collaboration by letting different agents share blocks of memory This course will give you a sophisticated understanding of memory management for LLMs, which is important for chatbots having long conversations, and for complex agentic workflows. Please sign up here!

New short course: LLMs as Operating Systems: Agent Memory, created with Letta, and taught by its founders Charles Packer and Sarah Wooders. An LLM's input context window has limited space. Using a longer input context also costs more and results in slower processing. So, managing what's stored in this context window is important. In the innovative paper MemGPT: Towards LLMs as Operating Systems, its authors (which include the instructors) proposed using an LLM agent to manage this context window. Their system uses a large persistent memory that stores everything that could be included in the input context, and an agent decides what is actually included. Take the example of building a chatbot that needs to remember what's been said earlier in a conversation (perhaps over many days of interaction with a user). As the conversation's length grows, the memory management agent will move information from the input context to a persistent searchable database; summarize information to keep relevant facts in the input context; and restore relevant conversation elements from further back in time. This allows a chatbot to keep what's currently most relevant in its input context memory to generate the next response. When I read the original MemGPT paper, I thought it was an innovative technique for handling memory for LLMs. The open-source Letta framework, which we'll use in this course, makes MemGPT easy to implement. It adds memory to your LLM agents and gives them transparent long-term memory. In detail, you’ll learn: - How to build an agent that can edit its own limited input context memory, using tools and multi-step reasoning - What is a memory hierarchy (an idea from computer operating systems, which use a cache to speed up memory access), and how these ideas apply to managing the LLM input context (where the input context window is a "cache" storing the most relevant information; and an agent decides what to move in and out of this to/from a larger persistent storage system) - How to implement multi-agent collaboration by letting different agents share blocks of memory This course will give you a sophisticated understanding of memory management for LLMs, which is important for chatbots having long conversations, and for complex agentic workflows. Please sign up here!

Andrew Ng

201,127 views • 1 year ago

🚨TRUMP SENDS A STRONG MESSAGE: BIG THINGS ARE COMING FOR #XRP AS THE U.S. BUSINESS SYSTEM FACES A MAJOR RESTRUCTURING! THE FUTURE OF DECENTRALIZED MEDIA IS ALSO SET TO UNFOLD ON THE XRPL! BXE TOKEN IS LEADING THE WAY. IT WAS THE FIRST TO INTRODUCE INSTITUTIONAL TOKENIZATION ON THE XRP LEDGER! WITH ONLY 500 MILLION TOKENS IN TOTAL SUPPLY AND A CURRENT PRICE OF JUST $0.03, BXE HAS MASSIVE UPSIDE — POTENTIALLY REACHING $19–$24! DYOR/NFA BXE TOKEN TRADE LINK:

🚨TRUMP SENDS A STRONG MESSAGE: BIG THINGS ARE COMING FOR #XRP AS THE U.S. BUSINESS SYSTEM FACES A MAJOR RESTRUCTURING! THE FUTURE OF DECENTRALIZED MEDIA IS ALSO SET TO UNFOLD ON THE XRPL! BXE TOKEN IS LEADING THE WAY. IT WAS THE FIRST TO INTRODUCE INSTITUTIONAL TOKENIZATION ON THE XRP LEDGER! WITH ONLY 500 MILLION TOKENS IN TOTAL SUPPLY AND A CURRENT PRICE OF JUST $0.03, BXE HAS MASSIVE UPSIDE — POTENTIALLY REACHING $19–$24! DYOR/NFA BXE TOKEN TRADE LINK:

JackTheRippler ©️

50,752 views • 1 year ago

With Kimi K3 Day-0 on vLLM: Open Frontier Intelligence for Everyone 🚀 At 2.8 trillion parameters, Moonshot AI's Kimi K3 is one of the most powerful open-weight models ever released. Starting today, you can serve it on vLLM the moment the weights are public. What K3 brings: 🧠 2.8T-parameter Mixture-of-Experts (16 of 896 experts active per token) 📚 1M-token context window 🛠️ Native multimodal understanding, including vision ⚡ Kimi Delta Attention: a hybrid of linear and full attention that makes million-token context affordable Huge thank you to Kimi.ai AI for the model release and partnership, Inferact for leading the vLLM optimizations, and to our partners at NVIDIA, AMD, and the broader vLLM community. 1/6

With Kimi K3 Day-0 on vLLM: Open Frontier Intelligence for Everyone 🚀 At 2.8 trillion parameters, Moonshot AI's Kimi K3 is one of the most powerful open-weight models ever released. Starting today, you can serve it on vLLM the moment the weights are public. What K3 brings: 🧠 2.8T-parameter Mixture-of-Experts (16 of 896 experts active per token) 📚 1M-token context window 🛠️ Native multimodal understanding, including vision ⚡ Kimi Delta Attention: a hybrid of linear and full attention that makes million-token context affordable Huge thank you to Kimi.ai AI for the model release and partnership, Inferact for leading the vLLM optimizations, and to our partners at NVIDIA, AMD, and the broader vLLM community. 1/6

vLLM

104,622 views • 2 months ago

Google Engineers just showed how their AI engineers are moving from simple "RAG" to "graphs" RAG → Graph RAG → Memory → Multimodal Agent Graphs. 0% → 15:32 - setting up the production agent stack 30% → 41:00 - Graph RAG with semantic 65% → 58:00 - extracting graph context 100% → 1:09:00 - orchestrating specialized agents 90-minute Google Cloud workshop, and it’s one of the clearest hands-on examples of the shift from. Watch it today, then read the full “From RAG to Context Graphs” roadmap below.

Google Engineers just showed how their AI engineers are moving from simple "RAG" to "graphs" RAG → Graph RAG → Memory → Multimodal Agent Graphs. 0% → 15:32 - setting up the production agent stack 30% → 41:00 - Graph RAG with semantic 65% → 58:00 - extracting graph context 100% → 1:09:00 - orchestrating specialized agents 90-minute Google Cloud workshop, and it’s one of the clearest hands-on examples of the shift from. Watch it today, then read the full “From RAG to Context Graphs” roadmap below.

Morty

53,555 views • 1 month ago

🚨#XRP HIT $327,000 ON THE PRIVATE LEDGER according to analyst! THE XRPL IS GEARING UP TO HANDLE TRILLIONS IN TRANSACTIONS!! CTF TOKEN, THE TOP DEFI PROJECT ON THE XRP LEDGER WITH PARTNERSHIPS INCLUDING AMAZON AND WALMART, IS POISED FOR A MASSIVE LEAP!! CTF TOKEN COULD SKYROCKET FROM $1.09 TO $748.50 PER TOKEN WITH A MARKET CAP OF ONLY $20 BILLION!! THE FUTURE IS HERE! DIRECT PURCHASES THROUGH THE XRP LEDGER ARE COMING, AND CTF TOKEN WILL BE KEY IN MAKING IT HAPPEN! WITH JUST 119 MILLION TOKENS IN CIRCULATION, A SUPPLY CRUNCH IS IMMINENT FOR CTF!! CTF TOKEN TRADE LINK: CTF TRADE LINK ON MEXC: Official Website:

🚨#XRP HIT $327,000 ON THE PRIVATE LEDGER according to analyst! THE XRPL IS GEARING UP TO HANDLE TRILLIONS IN TRANSACTIONS!! CTF TOKEN, THE TOP DEFI PROJECT ON THE XRP LEDGER WITH PARTNERSHIPS INCLUDING AMAZON AND WALMART, IS POISED FOR A MASSIVE LEAP!! CTF TOKEN COULD SKYROCKET FROM $1.09 TO $748.50 PER TOKEN WITH A MARKET CAP OF ONLY $20 BILLION!! THE FUTURE IS HERE! DIRECT PURCHASES THROUGH THE XRP LEDGER ARE COMING, AND CTF TOKEN WILL BE KEY IN MAKING IT HAPPEN! WITH JUST 119 MILLION TOKENS IN CIRCULATION, A SUPPLY CRUNCH IS IMMINENT FOR CTF!! CTF TOKEN TRADE LINK: CTF TRADE LINK ON MEXC: Official Website:

JackTheRippler ©️

117,817 views • 1 year ago

MICROSOFT QUIETLY SHIPPED THE RAG PIPELINE EVERYONE KEEPS TRYING TO BUILD FROM SCRATCH it is called Kernel Memory, and it turns any pile of files into an AI memory that answers with citations. drop your data in and Microsoft's pipeline does the rest: > ingest PDFs, Word, PPT, Excel, web pages, even images > extract the text, chunk it, embed it into vectors > store the vectors with metadata and keep the originals in blob storage > ask a question and it pulls the right chunks with semantic plus hybrid search then it hands the answer back grounded, with links to the exact source and full token usage. and it plugs into whatever you already run: > Azure OpenAI, OpenAI, Ollama, or Anthropic for the models > Qdrant, pgvector, Redis, or Azure AI Search for the vectors > every step is a swappable handler, so you can replace any part with your own one heads up: Microsoft archived the repo in June 2026 and ships it as a reference blueprint, not production. but as a map for how memory and RAG are actually built, nothing is cleaner. run it in one command: docker run kernelmemory/service bookmark this. Microsoft already drew the blueprint, you just build on it.

MICROSOFT QUIETLY SHIPPED THE RAG PIPELINE EVERYONE KEEPS TRYING TO BUILD FROM SCRATCH it is called Kernel Memory, and it turns any pile of files into an AI memory that answers with citations. drop your data in and Microsoft's pipeline does the rest: > ingest PDFs, Word, PPT, Excel, web pages, even images > extract the text, chunk it, embed it into vectors > store the vectors with metadata and keep the originals in blob storage > ask a question and it pulls the right chunks with semantic plus hybrid search then it hands the answer back grounded, with links to the exact source and full token usage. and it plugs into whatever you already run: > Azure OpenAI, OpenAI, Ollama, or Anthropic for the models > Qdrant, pgvector, Redis, or Azure AI Search for the vectors > every step is a swappable handler, so you can replace any part with your own one heads up: Microsoft archived the repo in June 2026 and ships it as a reference blueprint, not production. but as a map for how memory and RAG are actually built, nothing is cleaner. run it in one command: docker run kernelmemory/service bookmark this. Microsoft already drew the blueprint, you just build on it.

NO1ennn

85,704 views • 1 month ago

Qwen3.8-Flash-Next is still going strong at 364.7K tokens of context on an M5 Max. And this isn’t just a static long-context test. The model was reasoning about how to speed up its own workflow while using tools, and the tool calls kept working without misses. Setup: • Qwen3.8-Flash-Next • M5 Max • 128GB unified memory • MLX-Serve PR #363 • OpenCode 2 • 364.7K context The interesting part isn’t simply getting hundreds of thousands of tokens into memory. It’s what happens once the context gets this large. Long-context inference usually comes with a painful tradeoff. As the KV cache grows, memory pressure increases and generation can slow down. But this setup is still pushing through 364K tokens while maintaining a usable agent workflow. The model can reason, call tools, inspect results, continue working, and keep the session moving. And the tool calls reportedly haven’t missed so far. That’s important for agentic coding. A huge context window is only useful if the model can actually operate reliably inside it. A 400K-token context that constantly breaks tool calls isn’t very useful. A 364K session that can keep reasoning and executing tools is a different story. And the test isn’t finished yet. The current run is approaching 400K tokens, with the expectation that it can keep going. This is also another interesting example of why Apple Silicon keeps showing up in local LLM experiments. The M5 Max’s unified memory gives a large model and its growing KV cache access to one shared memory pool. With MLX-Serve continuing to improve, these machines are becoming surprisingly capable long-context inference boxes. The bigger takeaway: Context length is becoming a workload, not just a model specification. Running a model at 256K is one thing. Keeping an agent alive at 300K+ while it reasons and uses tools is much more interesting. And Qwen3.8-Flash-Next is showing that this can be pushed surprisingly far on a single 128GB Mac. 364.7K and counting. Next stop: 400K.

Qwen3.8-Flash-Next is still going strong at 364.7K tokens of context on an M5 Max. And this isn’t just a static long-context test. The model was reasoning about how to speed up its own workflow while using tools, and the tool calls kept working without misses. Setup: • Qwen3.8-Flash-Next • M5 Max • 128GB unified memory • MLX-Serve PR #363 • OpenCode 2 • 364.7K context The interesting part isn’t simply getting hundreds of thousands of tokens into memory. It’s what happens once the context gets this large. Long-context inference usually comes with a painful tradeoff. As the KV cache grows, memory pressure increases and generation can slow down. But this setup is still pushing through 364K tokens while maintaining a usable agent workflow. The model can reason, call tools, inspect results, continue working, and keep the session moving. And the tool calls reportedly haven’t missed so far. That’s important for agentic coding. A huge context window is only useful if the model can actually operate reliably inside it. A 400K-token context that constantly breaks tool calls isn’t very useful. A 364K session that can keep reasoning and executing tools is a different story. And the test isn’t finished yet. The current run is approaching 400K tokens, with the expectation that it can keep going. This is also another interesting example of why Apple Silicon keeps showing up in local LLM experiments. The M5 Max’s unified memory gives a large model and its growing KV cache access to one shared memory pool. With MLX-Serve continuing to improve, these machines are becoming surprisingly capable long-context inference boxes. The bigger takeaway: Context length is becoming a workload, not just a model specification. Running a model at 256K is one thing. Keeping an agent alive at 300K+ while it reasons and uses tools is much more interesting. And Qwen3.8-Flash-Next is showing that this can be pushed surprisingly far on a single 128GB Mac. 364.7K and counting. Next stop: 400K.

FHILY👑

39,982 views • 1 month ago

$Introducing SubQ - a major breakthrough in LLM intelligence. It is the first model built on a fully sub-quadratic sparse-attention architecture (SSA), And the first frontier model with a 12 million token context window which is: - 52x faster than FlashAttention at 1MM tokens - Less than 5% the cost of Opus Transformer-based LLMs waste compute by processing every possible relationship between words (standard attention). Only a small fraction actually matter. Subquadratic finds and focuses only on the ones that do. That's nearly 1,000x less compute and a new way for LLMs to scale.$

Introducing SubQ - a major breakthrough in LLM intelligence. It is the first model built on a fully sub-quadratic sparse-attention architecture (SSA), And the first frontier model with a 12 million token context window which is: - 52x faster than FlashAttention at 1MM tokens - Less than 5% the cost of Opus Transformer-based LLMs waste compute by processing every possible relationship between words (standard attention). Only a small fraction actually matter. Subquadratic finds and focuses only on the ones that do. That's nearly 1,000x less compute and a new way for LLMs to scale.

Alexander Whedon

13,491,251 views • 5 months ago

Our new short course, Knowledge Graphs for RAG, is now available! Knowledge graphs are a data structure that is great at capturing complex relationships between data of multiple types. By enabling more sophisticated retrieval of text than similarity search alone, knowledge graphs can improve the context you pass to the LLM and the performance of your RAG applications. In this course, taught by Andreas Kollegger of Neo4j, you’ll - Explore how knowledge graphs work by building a graph of public financial documents from scratch - Learn to write queries that retrieve text and data from the graph and use it to enhance the context you pass to an LLM chatbot - Combine a knowledge graph with a question-answer chain to build better RAG-powered chat systems Sign up here!

Our new short course, Knowledge Graphs for RAG, is now available! Knowledge graphs are a data structure that is great at capturing complex relationships between data of multiple types. By enabling more sophisticated retrieval of text than similarity search alone, knowledge graphs can improve the context you pass to the LLM and the performance of your RAG applications. In this course, taught by Andreas Kollegger of Neo4j, you’ll - Explore how knowledge graphs work by building a graph of public financial documents from scratch - Learn to write queries that retrieve text and data from the graph and use it to enhance the context you pass to an LLM chatbot - Combine a knowledge graph with a question-answer chain to build better RAG-powered chat systems Sign up here!

Andrew Ng

244,593 views • 2 years ago

THE $4,000 DGX SPARK BENCHMARKS AT 342 T/S ON PROMPT PROCESSING - THEN DROPS TO 73 WHEN IT ACTUALLY GENERATES. that clip is a dgx spark running llama-bench on qwen3-coder 30b, side by side with a mac. two numbers decide everything here: pp512, prompt processing = 342 tokens/sec tg128, token generation = 73 tokens/sec that gap is the whole lesson. local ai isn't one speed, it's two. prompt processing is reading your input: your files, your long context. the spark is compute-heavy, so it eats that fast. token generation is writing the answer, and it's bound by memory bandwidth, where the spark is modest and unified-memory macs stay close. so "which box is faster" has no single winner. feeding huge context or running agents that read a lot? prompt processing wins, the spark shines. long chat replies, one token at a time? generation matters more, and a mac holds its own. no single benchmark number, no universal "fastest box", no upgrade that wins every workload. the uncomfortable part isn't that the spark is fast. it's that "tokens per second" was always two numbers, and the hype only ever quotes the flattering one. follow RetroChainer for the benchmark truth nobody screenshots, and bookmark this before you buy on one number.

THE $4,000 DGX SPARK BENCHMARKS AT 342 T/S ON PROMPT PROCESSING - THEN DROPS TO 73 WHEN IT ACTUALLY GENERATES. that clip is a dgx spark running llama-bench on qwen3-coder 30b, side by side with a mac. two numbers decide everything here: pp512, prompt processing = 342 tokens/sec tg128, token generation = 73 tokens/sec that gap is the whole lesson. local ai isn't one speed, it's two. prompt processing is reading your input: your files, your long context. the spark is compute-heavy, so it eats that fast. token generation is writing the answer, and it's bound by memory bandwidth, where the spark is modest and unified-memory macs stay close. so "which box is faster" has no single winner. feeding huge context or running agents that read a lot? prompt processing wins, the spark shines. long chat replies, one token at a time? generation matters more, and a mac holds its own. no single benchmark number, no universal "fastest box", no upgrade that wins every workload. the uncomfortable part isn't that the spark is fast. it's that "tokens per second" was always two numbers, and the hype only ever quotes the flattering one. follow RetroChainer for the benchmark truth nobody screenshots, and bookmark this before you buy on one number.

RetroChainer

42,451 views • 2 months ago