Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

Memory Skill for OpenClaw with 26k+ users in 1 week🚀 OpenClaw's memory system is broken by default. It requires curating massive MEMORY.md files or relying on duplicate-heavy generation. Hours are wasted tuning, and massive amounts of tokens are burned. It's time to stop. So we built the memory skill... show more

andy nguyen

3,431 subscribers

589,092 Aufrufe • vor 4 Monaten •via X (Twitter)

Bildung Wissenschaft & Technologie

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

Memory for OpenClaw is now Native! Our first OpenClaw Memory Skill was a massive success: 30k+ downloads in a week and 500k+ organic impressions overnight for launch post. But we knew memory needed to be native. On March 21, OpenClaw merged PR #50848, allowing us to go beyond the skill layer and integrate directly into the agent’s context assembly flow. We try to make OpenClaw a truly 24/7 employee capable of complex workflows. The technical setup isn’t the hardest part but the real challenge is giving it a "brain" that remembers exact project details, past decisions, and team changes over time. The Native Memory Plugin is now live on NPM & ClawHub. Here is what it brings to your OpenClaw agents: 👉 Native Integration: Automatically manages a Three-Layer Memory architecture (Context Tree, Workspace Memory, Daily Memory). 👉 Git-like Stateful Memory: Organizes memory into a semantic hierarchy of human-readable, diffable Markdown files. You always get updated knowledge and can actually see and fix what your agent learns. 👉 Top Market Accuracy: Achieves an industry-leading 92.2% retrieval accuracy (LoCoMo & LongMemEval benchmarks), maintaining 90% accuracy even with cheap, lightweight models. 👉 Local-first & Portable: Local-by-default, fully portable for multi-agent teams. 👉 Super Easy Setup

Memory for OpenClaw is now Native! Our first OpenClaw Memory Skill was a massive success: 30k+ downloads in a week and 500k+ organic impressions overnight for launch post. But we knew memory needed to be native. On March 21, OpenClaw merged PR #50848, allowing us to go beyond the skill layer and integrate directly into the agent’s context assembly flow. We try to make OpenClaw a truly 24/7 employee capable of complex workflows. The technical setup isn’t the hardest part but the real challenge is giving it a "brain" that remembers exact project details, past decisions, and team changes over time. The Native Memory Plugin is now live on NPM & ClawHub. Here is what it brings to your OpenClaw agents: 👉 Native Integration: Automatically manages a Three-Layer Memory architecture (Context Tree, Workspace Memory, Daily Memory). 👉 Git-like Stateful Memory: Organizes memory into a semantic hierarchy of human-readable, diffable Markdown files. You always get updated knowledge and can actually see and fix what your agent learns. 👉 Top Market Accuracy: Achieves an industry-leading 92.2% retrieval accuracy (LoCoMo & LongMemEval benchmarks), maintaining 90% accuracy even with cheap, lightweight models. 👉 Local-first & Portable: Local-by-default, fully portable for multi-agent teams. 👉 Super Easy Setup

andy nguyen

169,136 Aufrufe • vor 4 Monaten

We analyzed Anthropic’s memory architecture and built something better: a persistent, human-inspectable, and token-efficient memory layer that scales with your projects. Today, it’s OPEN-SOURCE. ByteRover CLI gives agents (like OpenClaw, Claude Code, and Hermes) persistent, structured memory. Built on the exact architecture that became the #1 memory system for OpenClaw (30,000+ downloads in a week), it lets developers curate project knowledge into a file-based hierarchy. This guarantees highly accurate, lightning-fast retrieval, even with lightweight models. 👉Highly Accurate: >92% retrieval accuracy across long-running sessions - the highest proven production accuracy on the market. ⚡Fast: ~1.6s average retrieval time. 💰Economic: Maintains >90% accuracy even with lightweight models, saving 50-70% on token costs. ☁ Portable: Runs locally by default, with cloud-sync to share memory across agents and teammates.

We analyzed Anthropic’s memory architecture and built something better: a persistent, human-inspectable, and token-efficient memory layer that scales with your projects. Today, it’s OPEN-SOURCE. ByteRover CLI gives agents (like OpenClaw, Claude Code, and Hermes) persistent, structured memory. Built on the exact architecture that became the #1 memory system for OpenClaw (30,000+ downloads in a week), it lets developers curate project knowledge into a file-based hierarchy. This guarantees highly accurate, lightning-fast retrieval, even with lightweight models. 👉Highly Accurate: >92% retrieval accuracy across long-running sessions - the highest proven production accuracy on the market. ⚡Fast: ~1.6s average retrieval time. 💰Economic: Maintains >90% accuracy even with lightweight models, saving 50-70% on token costs. ☁ Portable: Runs locally by default, with cloud-sync to share memory across agents and teammates.

andy nguyen

28,980 Aufrufe • vor 3 Monaten

SK Hynix's Hoshik Kim admits high bandwidth memory "is not the answer" to the memory-wall problem, and SK is working on different solutions to solve the bottleneck. "No, actually, I kind of agree with Pat. So HBM is not the answer." [SK Hynix is admitting that HBM is a bad memory?] "Nothing is a bad memory. It's a good memory, but it's not the final answer. So actually, HBM will not solve the memory-wall problem. Memory-wall problem is an inherent AI problem, which you cannot avoid." "So we try our best to provide higher bandwidth with this memory, with HBM technology even though it has some limitation, as Pat already mentioned, with thermal, IOs, cost, a lot of stuff. "But that's the only thing we got right now to help ease this memory-wall problem, memory-bottleneck problem. But at the same time, we are working in different directions to solve this memory-wall problem with different solutions."

SK Hynix's Hoshik Kim admits high bandwidth memory "is not the answer" to the memory-wall problem, and SK is working on different solutions to solve the bottleneck. "No, actually, I kind of agree with Pat. So HBM is not the answer." [SK Hynix is admitting that HBM is a bad memory?] "Nothing is a bad memory. It's a good memory, but it's not the final answer. So actually, HBM will not solve the memory-wall problem. Memory-wall problem is an inherent AI problem, which you cannot avoid." "So we try our best to provide higher bandwidth with this memory, with HBM technology even though it has some limitation, as Pat already mentioned, with thermal, IOs, cost, a lot of stuff. "But that's the only thing we got right now to help ease this memory-wall problem, memory-bottleneck problem. But at the same time, we are working in different directions to solve this memory-wall problem with different solutions."

Fireside Alpha

136,796 Aufrufe • vor 3 Tagen

New short course: Long-Term Agentic Memory with LangGraph. Learn to build an agent with long-term memory in this course developed in collaboration with taught by its Co-Founder and CEO, Harrison Chase! Personal assistance and productivity tasks have become important use cases for agents. An important feature of an AI assistant, such as a coding or calendar assistant, is its ability to keep improving over time from its experience. Agent memory is the key capability that enables this. To add memory to an agent, you must first figure out what to store and what to retrieve when it is time to use the information. Additionally, you’ll have to decide when to update the stored information. For example, you might update in each iteration loop of the agent or perform updates in the background, with a helper agent. In this course, you will learn a mental framework to build agents with long-term memory. You'll create a useful email assistant that can respond, ignore, and notify using writing, scheduling, and memory-management tools. You’ll develop your agent's memory by adding facts to its memory store, provide examples to learn the user's preferences, and optimize system prompts to evolve instructions based on previous responses. In detail, you’ll: - Learn how the three types of memory--semantic, episodic, and procedural–and the two update mechanisms–via hot path and in the background–apply to your agents. - Build an email agent with writing, scheduling, and availability tools, along with a router that triages incoming email and handles it accordingly by ignoring, responding, or notifying the user. - Add tools to your email agent that allow it to operate on semantic memory by learning facts about the user, storing them in a long-term memory store, and searching over them in future interactions. - Incorporate episodic memory, in the form of few-shot examples, in the triage step of your agents to help them learn and update user preferences. - Add procedural memory as system prompts, optimized with feedback to improve the instructions the agent follows. Learn how to approach memory in agents, and start building agents with long-term memory with LangGraph! Please sign up here:

New short course: Long-Term Agentic Memory with LangGraph. Learn to build an agent with long-term memory in this course developed in collaboration with taught by its Co-Founder and CEO, Harrison Chase! Personal assistance and productivity tasks have become important use cases for agents. An important feature of an AI assistant, such as a coding or calendar assistant, is its ability to keep improving over time from its experience. Agent memory is the key capability that enables this. To add memory to an agent, you must first figure out what to store and what to retrieve when it is time to use the information. Additionally, you’ll have to decide when to update the stored information. For example, you might update in each iteration loop of the agent or perform updates in the background, with a helper agent. In this course, you will learn a mental framework to build agents with long-term memory. You'll create a useful email assistant that can respond, ignore, and notify using writing, scheduling, and memory-management tools. You’ll develop your agent's memory by adding facts to its memory store, provide examples to learn the user's preferences, and optimize system prompts to evolve instructions based on previous responses. In detail, you’ll: - Learn how the three types of memory--semantic, episodic, and procedural–and the two update mechanisms–via hot path and in the background–apply to your agents. - Build an email agent with writing, scheduling, and availability tools, along with a router that triages incoming email and handles it accordingly by ignoring, responding, or notifying the user. - Add tools to your email agent that allow it to operate on semantic memory by learning facts about the user, storing them in a long-term memory store, and searching over them in future interactions. - Incorporate episodic memory, in the form of few-shot examples, in the triage step of your agents to help them learn and update user preferences. - Add procedural memory as system prompts, optimized with feedback to improve the instructions the agent follows. Learn how to approach memory in agents, and start building agents with long-term memory with LangGraph! Please sign up here:

Andrew Ng

131,835 Aufrufe • vor 1 Jahr

This is my take on the perfect AI assistant. A Rust-based agentic operating system designed to scale for large Slack and Discord communities. The channel is the ambassador to the human. Branches think. Workers execute. Nothing ever blocks. Meet Spacebot 🟣 The biggest issue with OpenClaw is when it's doing work, it can't talk to you. Spacebot's architecture fixes this by design the conversation layer never touches tools. It delegates thinking to branches and heavy tasks to workers, so it's always responsive even with 100 people talking at once. Dump your memory files, notes, documents and chat histories into a folder — Spacebot turns them into structured memories automatically. Eight typed memory categories, graph associations, hybrid search. Not markdown files. Not vibes in a vector database. Built-in OpenCode workers for deep coding sessions. Browser automation. Brave web search. Cron jobs. A skill system compatible with your existing OpenClaw skills. And a gorgeous control UI at The cortex oversees the whole system — auditing memories, actioning goals and todos. You teach your Spacebot by talking to it. Structure and speed over config files and markdown. Self-hosting is a single Rust binary. Or one-click cloud deploy at This is for teams, communities, and personal assistants. It will blow you away. ⭐️

This is my take on the perfect AI assistant. A Rust-based agentic operating system designed to scale for large Slack and Discord communities. The channel is the ambassador to the human. Branches think. Workers execute. Nothing ever blocks. Meet Spacebot 🟣 The biggest issue with OpenClaw is when it's doing work, it can't talk to you. Spacebot's architecture fixes this by design the conversation layer never touches tools. It delegates thinking to branches and heavy tasks to workers, so it's always responsive even with 100 people talking at once. Dump your memory files, notes, documents and chat histories into a folder — Spacebot turns them into structured memories automatically. Eight typed memory categories, graph associations, hybrid search. Not markdown files. Not vibes in a vector database. Built-in OpenCode workers for deep coding sessions. Browser automation. Brave web search. Cron jobs. A skill system compatible with your existing OpenClaw skills. And a gorgeous control UI at The cortex oversees the whole system — auditing memories, actioning goals and todos. You teach your Spacebot by talking to it. Structure and speed over config files and markdown. Self-hosting is a single Rust binary. Or one-click cloud deploy at This is for teams, communities, and personal assistants. It will blow you away. ⭐️

Jamie Pine

169,264 Aufrufe • vor 5 Monaten

The AI boom just hit a wall nobody saw coming. And it's not software. It's not regulation. It's not even energy... It's memory chips. Right now, Dell is raising PC prices by 30%. Intel can't ship chips. Nvidia is slashing GPU production by 40%. And almost nobody understands why. Here's the "hidden" crisis the AI industry is trying to hide: AI data centers are hoarding memory. Not GPUs. Not processors. MEMORY. Every AI server needs massive amounts of high-bandwidth memory (HBM) to run those models everyone's hyping. One problem: There are only 3 companies in the world that can make it. Samsung. SK Hynix. Micron. That's it. And all 3 just diverted their entire production capacity away from normal RAM to feed AI data centers. The math that breaks everything: 1 gigabyte of HBM takes 4X the manufacturing capacity of regular DRAM. AI will consume 20% of global DRAM production in 2026. But the thing is, consumer demand for RAM didn't disappear. PCs still need memory. Phones still need memory. Cars still need memory. But there's no capacity left to make it. The price explosion: RAM prices are up 246% in the last 6 months. DDR5 contract prices jumped 100% month-over-month in some cases. Dell's CFO said he's "never witnessed costs escalating at this pace." SK Hynix and Micron? Sold out through all of 2026. Micron straight up EXITED the consumer memory market entirely to focus on AI customers. If you're not building an AI data center, you're not getting memory chips. AI data centers pay 3-5X margins compared to consumer products. So memory manufacturers are rationally choosing: Serve Microsoft and Google's AI buildout, or serve Dell's laptop business? Easy choice. Every wafer allocated to an Nvidia H100 GPU is a wafer DENIED to your next laptop. It's a zero-sum game. And consumers are losing. The dangerous cascade effect: Nvidia is cutting RTX 50-series GPU production by 30-40% because they can't get GDDR7 memory. Dell, Lenovo, HP are all raising PC prices 15-30% in early 2026. Xiaomi and other smartphone makers are cutting shipment targets. Even Intel's crash last week? Partially driven by memory shortages limiting chip production. This is a PERMANENT reallocation of the world's silicon capacity. Not a temporary supply hiccup. For decades, consumer electronics (phones, PCs, laptops) drove memory production. Now? AI data centers are the priority customer. And that priority shift is reshaping the entire tech economy. The timeline Is worse than you think: Industry analysts project shortages lasting through 2027, maybe 2028. Why? Because building new memory fabs takes 3-5 YEARS. Micron's new Idaho fab won't meaningfully impact supply until 2028. Samsung and SK Hynix are too busy ramping up HBM4 production to expand consumer DRAM. So we're stuck. AI companies need memory to scale. But producing that memory DESTROYS the supply chain for everything else. My question here: Everyone's betting on AI scaling infinitely. But what if the AI boom STALLS because there's not enough memory to support it? What if we're not in an "AI supercycle" but a "memory shortage that kills the AI buildout"? Intel crashed 17% because they can't manufacture enough chips. The root cause though? Memory shortages limiting what they can even produce. Nvidia is cutting GPU production by 40%. AMD is struggling to get GDDR6 for Radeon cards. This isn't just a consumer problem. It's an AI infrastructure problem. And if memory doesn't scale, AI doesn't scale. The AI industry sold you on infinite scaling. But they forgot to mention the part where there's only 3 companies making the memory chips that power everything. And all 3 just chose AI data centers over you. Even Nvidia can't make enough GPUs to meet demand. Not because of energy. Not because of regulation... But because the memory supply chain is BROKEN. And it won't be fixed until 2028.

The AI boom just hit a wall nobody saw coming. And it's not software. It's not regulation. It's not even energy... It's memory chips. Right now, Dell is raising PC prices by 30%. Intel can't ship chips. Nvidia is slashing GPU production by 40%. And almost nobody understands why. Here's the "hidden" crisis the AI industry is trying to hide: AI data centers are hoarding memory. Not GPUs. Not processors. MEMORY. Every AI server needs massive amounts of high-bandwidth memory (HBM) to run those models everyone's hyping. One problem: There are only 3 companies in the world that can make it. Samsung. SK Hynix. Micron. That's it. And all 3 just diverted their entire production capacity away from normal RAM to feed AI data centers. The math that breaks everything: 1 gigabyte of HBM takes 4X the manufacturing capacity of regular DRAM. AI will consume 20% of global DRAM production in 2026. But the thing is, consumer demand for RAM didn't disappear. PCs still need memory. Phones still need memory. Cars still need memory. But there's no capacity left to make it. The price explosion: RAM prices are up 246% in the last 6 months. DDR5 contract prices jumped 100% month-over-month in some cases. Dell's CFO said he's "never witnessed costs escalating at this pace." SK Hynix and Micron? Sold out through all of 2026. Micron straight up EXITED the consumer memory market entirely to focus on AI customers. If you're not building an AI data center, you're not getting memory chips. AI data centers pay 3-5X margins compared to consumer products. So memory manufacturers are rationally choosing: Serve Microsoft and Google's AI buildout, or serve Dell's laptop business? Easy choice. Every wafer allocated to an Nvidia H100 GPU is a wafer DENIED to your next laptop. It's a zero-sum game. And consumers are losing. The dangerous cascade effect: Nvidia is cutting RTX 50-series GPU production by 30-40% because they can't get GDDR7 memory. Dell, Lenovo, HP are all raising PC prices 15-30% in early 2026. Xiaomi and other smartphone makers are cutting shipment targets. Even Intel's crash last week? Partially driven by memory shortages limiting chip production. This is a PERMANENT reallocation of the world's silicon capacity. Not a temporary supply hiccup. For decades, consumer electronics (phones, PCs, laptops) drove memory production. Now? AI data centers are the priority customer. And that priority shift is reshaping the entire tech economy. The timeline Is worse than you think: Industry analysts project shortages lasting through 2027, maybe 2028. Why? Because building new memory fabs takes 3-5 YEARS. Micron's new Idaho fab won't meaningfully impact supply until 2028. Samsung and SK Hynix are too busy ramping up HBM4 production to expand consumer DRAM. So we're stuck. AI companies need memory to scale. But producing that memory DESTROYS the supply chain for everything else. My question here: Everyone's betting on AI scaling infinitely. But what if the AI boom STALLS because there's not enough memory to support it? What if we're not in an "AI supercycle" but a "memory shortage that kills the AI buildout"? Intel crashed 17% because they can't manufacture enough chips. The root cause though? Memory shortages limiting what they can even produce. Nvidia is cutting GPU production by 40%. AMD is struggling to get GDDR6 for Radeon cards. This isn't just a consumer problem. It's an AI infrastructure problem. And if memory doesn't scale, AI doesn't scale. The AI industry sold you on infinite scaling. But they forgot to mention the part where there's only 3 companies making the memory chips that power everything. And all 3 just chose AI data centers over you. Even Nvidia can't make enough GPUs to meet demand. Not because of energy. Not because of regulation... But because the memory supply chain is BROKEN. And it won't be fixed until 2028.

Ricardo

594,453 Aufrufe • vor 6 Monaten

New course on serving LLMs efficiently -- how do you serve models to many concurrent users at low latency and reasonable cost? This short course is built with Red Hat and taught by Cedric Clyburn. Efficient LLM serving requires efficient memory management. A 70B-parameter model takes ~140 GB just to load the weights. On top of that, every active request needs its own chunk of GPU memory, the KV cache, to store the token context it has built up so far. In this course, you'll learn to reduce a model's memory footprint with quantization and serve it using vLLM, which handles many concurrent requests efficiently through smart memory management. Skills you'll gain: - Quantize a model and measure the accuracy tradeoff - Serve a model with vLLM and watch it handle concurrent requests efficiently - Benchmark your deployment and make informed tradeoffs between speed, cost, and accuracy Join and learn to serve LLMs efficiently:

New course on serving LLMs efficiently -- how do you serve models to many concurrent users at low latency and reasonable cost? This short course is built with Red Hat and taught by Cedric Clyburn. Efficient LLM serving requires efficient memory management. A 70B-parameter model takes ~140 GB just to load the weights. On top of that, every active request needs its own chunk of GPU memory, the KV cache, to store the token context it has built up so far. In this course, you'll learn to reduce a model's memory footprint with quantization and serve it using vLLM, which handles many concurrent requests efficiently through smart memory management. Skills you'll gain: - Quantize a model and measure the accuracy tradeoff - Serve a model with vLLM and watch it handle concurrent requests efficiently - Benchmark your deployment and make informed tradeoffs between speed, cost, and accuracy Join and learn to serve LLMs efficiently:

Andrew Ng

127,603 Aufrufe • vor 1 Monat

The creator of High Bandwidth Memory (HBM) put a number on the AI build that should stop every infra investor cold. A cluster of a million GPUs runs at roughly 10-20% utilization (Save this). Kim Jung-ho spent thirty years building what feeds the GPU, and his claim is that the GPU is barely working. Here is what is actually happening. Every time a model generates output, the data has to be read out of memory, computed, and written back. The read and the write swallow almost the entire cycle. While that data moves, the GPU does nothing. It sits there, fully powered, fully paid for, waiting. By Kim's estimate the memory is doing only about 30 percent of the work it needs to do. The processor idles the rest. So a million installed GPUs run at 10 to 20 percent. You are not compute constrained. You are memory constrained, and the expensive part is standing around. Adding more GPUs does not fix this. It gives you more processors starving for the same data. Here is the part that decides the next decade. Memory can grow. When a cell cannot shrink any further, you stack it into a high-rise, layer on layer. A GPU cannot be stacked. It runs too hot and needs a cooler bolted to its back, so the one move that rescues memory is closed to the processor. The thing that can keep stacking compounds. The thing that cannot plateaus. The marginal dollar in an AI build now buys more by fixing the memory path than by bolting on another idle GPU. Which is why the companies that control memory bandwidth and supply are not suppliers to the AI trade. They are the AI trade.

The creator of High Bandwidth Memory (HBM) put a number on the AI build that should stop every infra investor cold. A cluster of a million GPUs runs at roughly 10-20% utilization (Save this). Kim Jung-ho spent thirty years building what feeds the GPU, and his claim is that the GPU is barely working. Here is what is actually happening. Every time a model generates output, the data has to be read out of memory, computed, and written back. The read and the write swallow almost the entire cycle. While that data moves, the GPU does nothing. It sits there, fully powered, fully paid for, waiting. By Kim's estimate the memory is doing only about 30 percent of the work it needs to do. The processor idles the rest. So a million installed GPUs run at 10 to 20 percent. You are not compute constrained. You are memory constrained, and the expensive part is standing around. Adding more GPUs does not fix this. It gives you more processors starving for the same data. Here is the part that decides the next decade. Memory can grow. When a cell cannot shrink any further, you stack it into a high-rise, layer on layer. A GPU cannot be stacked. It runs too hot and needs a cooler bolted to its back, so the one move that rescues memory is closed to the processor. The thing that can keep stacking compounds. The thing that cannot plateaus. The marginal dollar in an AI build now buys more by fixing the memory path than by bolting on another idle GPU. Which is why the companies that control memory bandwidth and supply are not suppliers to the AI trade. They are the AI trade.

Fireside Alpha

38,370 Aufrufe • vor 29 Tagen

Learn a development pattern to systematically improve the accuracy and reliability of LLM applications in our new short course, Improving Accuracy of LLM Applications, built in partnership with Lamini and Meta, and taught by Lamini’s CEO Sharon Zhou, and Meta’s Senior Director of Partner Engineering, Amit Sangani. (Disclosure: I am an investor in Lamini.) The path to tuning an LLM application can be complex. In this course, you'll learn a systematic sequence of steps for improving accuracy by reducing hallucinations: - Create an evaluation dataset to measure model accuracy - Add prompt engineering and self-reflection - Fine-tune your model including "memory-tuning" which is a new method of embedding facts in an LLM Using the Llama 3-8B parameter model, you will: - Build a text-to-SQL agent with a custom schema and simulate situations where it hallucinates - Understand the difference between instruction fine-tuning, which gives pre-trained LLMs instructions to follow, and memory fine-tuning - See how Performance-Efficient Fine-tuning (PEFT) techniques like Low-Rank Adaptation (LoRA) reduce training time by 100x and Mixture of Memory Experts (MoME) reduces it even further I appreciate Meta releasing the Llama's family of open models -- this course gives an example of the unique type of work that developers can do with such models. Please sign up here:

Learn a development pattern to systematically improve the accuracy and reliability of LLM applications in our new short course, Improving Accuracy of LLM Applications, built in partnership with Lamini and Meta, and taught by Lamini’s CEO Sharon Zhou, and Meta’s Senior Director of Partner Engineering, Amit Sangani. (Disclosure: I am an investor in Lamini.) The path to tuning an LLM application can be complex. In this course, you'll learn a systematic sequence of steps for improving accuracy by reducing hallucinations: - Create an evaluation dataset to measure model accuracy - Add prompt engineering and self-reflection - Fine-tune your model including "memory-tuning" which is a new method of embedding facts in an LLM Using the Llama 3-8B parameter model, you will: - Build a text-to-SQL agent with a custom schema and simulate situations where it hallucinates - Understand the difference between instruction fine-tuning, which gives pre-trained LLMs instructions to follow, and memory fine-tuning - See how Performance-Efficient Fine-tuning (PEFT) techniques like Low-Rank Adaptation (LoRA) reduce training time by 100x and Mixture of Memory Experts (MoME) reduces it even further I appreciate Meta releasing the Llama's family of open models -- this course gives an example of the unique type of work that developers can do with such models. Please sign up here:

Andrew Ng

66,435 Aufrufe • vor 1 Jahr

Your brain isn’t broken, the school system just wasn’t built for 2026 attention spans. 🧠📉 We’re hitting what experts call the “Engagement Cliff”. While traditional tools are too static to keep up, the science of Neuroeducation is clear: 1-on-1 tutoring moves you to the top 2% of performers (The 2-Sigma Advantage). OLi isn’t just a chatbot. It’s an agentic tutor that sees your screen and adapts to YOUR specific neuro-pathway in real-time. Stop surviving class. Start mastering it. 🚀

Your brain isn’t broken, the school system just wasn’t built for 2026 attention spans. 🧠📉 We’re hitting what experts call the “Engagement Cliff”. While traditional tools are too static to keep up, the science of Neuroeducation is clear: 1-on-1 tutoring moves you to the top 2% of performers (The 2-Sigma Advantage). OLi isn’t just a chatbot. It’s an agentic tutor that sees your screen and adapts to YOUR specific neuro-pathway in real-time. Stop surviving class. Start mastering it. 🚀

OLi by Olympus Insights

19,843 Aufrufe • vor 6 Monaten

New short course: LLMs as Operating Systems: Agent Memory, created with Letta, and taught by its founders Charles Packer and Sarah Wooders. An LLM's input context window has limited space. Using a longer input context also costs more and results in slower processing. So, managing what's stored in this context window is important. In the innovative paper MemGPT: Towards LLMs as Operating Systems, its authors (which include the instructors) proposed using an LLM agent to manage this context window. Their system uses a large persistent memory that stores everything that could be included in the input context, and an agent decides what is actually included. Take the example of building a chatbot that needs to remember what's been said earlier in a conversation (perhaps over many days of interaction with a user). As the conversation's length grows, the memory management agent will move information from the input context to a persistent searchable database; summarize information to keep relevant facts in the input context; and restore relevant conversation elements from further back in time. This allows a chatbot to keep what's currently most relevant in its input context memory to generate the next response. When I read the original MemGPT paper, I thought it was an innovative technique for handling memory for LLMs. The open-source Letta framework, which we'll use in this course, makes MemGPT easy to implement. It adds memory to your LLM agents and gives them transparent long-term memory. In detail, you’ll learn: - How to build an agent that can edit its own limited input context memory, using tools and multi-step reasoning - What is a memory hierarchy (an idea from computer operating systems, which use a cache to speed up memory access), and how these ideas apply to managing the LLM input context (where the input context window is a "cache" storing the most relevant information; and an agent decides what to move in and out of this to/from a larger persistent storage system) - How to implement multi-agent collaboration by letting different agents share blocks of memory This course will give you a sophisticated understanding of memory management for LLMs, which is important for chatbots having long conversations, and for complex agentic workflows. Please sign up here!

New short course: LLMs as Operating Systems: Agent Memory, created with Letta, and taught by its founders Charles Packer and Sarah Wooders. An LLM's input context window has limited space. Using a longer input context also costs more and results in slower processing. So, managing what's stored in this context window is important. In the innovative paper MemGPT: Towards LLMs as Operating Systems, its authors (which include the instructors) proposed using an LLM agent to manage this context window. Their system uses a large persistent memory that stores everything that could be included in the input context, and an agent decides what is actually included. Take the example of building a chatbot that needs to remember what's been said earlier in a conversation (perhaps over many days of interaction with a user). As the conversation's length grows, the memory management agent will move information from the input context to a persistent searchable database; summarize information to keep relevant facts in the input context; and restore relevant conversation elements from further back in time. This allows a chatbot to keep what's currently most relevant in its input context memory to generate the next response. When I read the original MemGPT paper, I thought it was an innovative technique for handling memory for LLMs. The open-source Letta framework, which we'll use in this course, makes MemGPT easy to implement. It adds memory to your LLM agents and gives them transparent long-term memory. In detail, you’ll learn: - How to build an agent that can edit its own limited input context memory, using tools and multi-step reasoning - What is a memory hierarchy (an idea from computer operating systems, which use a cache to speed up memory access), and how these ideas apply to managing the LLM input context (where the input context window is a "cache" storing the most relevant information; and an agent decides what to move in and out of this to/from a larger persistent storage system) - How to implement multi-agent collaboration by letting different agents share blocks of memory This course will give you a sophisticated understanding of memory management for LLMs, which is important for chatbots having long conversations, and for complex agentic workflows. Please sign up here!

Andrew Ng

200,788 Aufrufe • vor 1 Jahr

Today we’re launching the first and only human-like AI agents in the world. Super Agents™ are the first agents with human‑level skills – they DM you, take @ mentions, send emails, manage docs, tasks, and more. Not just tools or API calls, but real skills fine‑tuned for how teams actually work. The first agents with 100% context – fully native in ClickUp and fully synced from other apps. Super Agents see your work the same way that humans do: tasks, docs, schedules, and conversations all in one place. The first agents that learn from human interactions automatically, without any setup or configuration – when you give feedback, they listen and improve how they work. The first agents with human‑level memory for custom agents – historical memory for every interaction, short-term working memory, and even long‑term memory stored in docs you can literally open, inspect, and edit. The first agents that are literally the same as users – our agentic user model is the same as our user data model. This gives you permissions and capabilities that you and your systems are already familiar with. The first infinite agent catalog – where anyone can create and customize agents in minutes, for literally any type of work imaginable. It's the most intuitive way to build agents on the planet. 95% of companies are failing in AI adoption. The reality is that AI isn't meant to be adopted, it's meant to be adapted – to you. Super Agents are automatically personalized to you and your company using proprietary state-of-the-art agent architecture, orchestration, and tooling. Today is the largest step forward we've ever made towards our mission of making people more productive. Maximize human productivity, with ClickUp Super Agents. Available NOW. For everyone.

Today we’re launching the first and only human-like AI agents in the world. Super Agents™ are the first agents with human‑level skills – they DM you, take @ mentions, send emails, manage docs, tasks, and more. Not just tools or API calls, but real skills fine‑tuned for how teams actually work. The first agents with 100% context – fully native in ClickUp and fully synced from other apps. Super Agents see your work the same way that humans do: tasks, docs, schedules, and conversations all in one place. The first agents that learn from human interactions automatically, without any setup or configuration – when you give feedback, they listen and improve how they work. The first agents with human‑level memory for custom agents – historical memory for every interaction, short-term working memory, and even long‑term memory stored in docs you can literally open, inspect, and edit. The first agents that are literally the same as users – our agentic user model is the same as our user data model. This gives you permissions and capabilities that you and your systems are already familiar with. The first infinite agent catalog – where anyone can create and customize agents in minutes, for literally any type of work imaginable. It's the most intuitive way to build agents on the planet. 95% of companies are failing in AI adoption. The reality is that AI isn't meant to be adopted, it's meant to be adapted – to you. Super Agents are automatically personalized to you and your company using proprietary state-of-the-art agent architecture, orchestration, and tooling. Today is the largest step forward we've ever made towards our mission of making people more productive. Maximize human productivity, with ClickUp Super Agents. Available NOW. For everyone.

Zeb Evans

320,607 Aufrufe • vor 7 Monaten

Gavin Baker, CIO of Atreides Management made one of the most important and nuanced calls on memory stocks in recent months (Save this). His argument is that based on every memory cycle of the last 25 years, the setup today, prices elevated, sentiment high, supply ramping is textbook time to sell but he adds a critical exception. The one cycle in modern memory history where selling was catastrophically wrong was the mid-1990s, which Baker calls the last true capacity cycle in memory. In that cycle, demand was structurally exploding as the internet era required entirely new computing infrastructure to be built from scratch, and memory had to scale with it in a way that had never happened before. His point is that AI may be that same kind of cycle and not a normal boom bust but a once in a generation capacity buildout where the underlying demand is structural, not cyclical. The reason this argument holds weight is the fundamental shift in what memory is in the AI era. Traditional DRAM was a pure commodity, identical specs, interchangeable suppliers, price determined entirely by supply and demand swings. HBM is the opposite because it is custom engineered to fit a specific customer's chip, co-designed between the memory maker and the GPU designer, with SK Hynix's Vice President literally describing it as shifting from a commodity to a customer-tailored custom business. A single Blackwell Ultra GPU now requires up to 288GB of HBM3E, a 3.6x increase over the H100 and major suppliers like SK Hynix and Micron have already sold out their entire HBM production capacity through the end of the year. Because HBM requires advanced packaging processes like CoWoS that can't be spun up overnight, the bottleneck isn't just wafer capacity but rather runs across the entire manufacturing stack. Bank of America projects the global HBM market grows 58% this year alone to $54.6 billion, and Nomura expects the broader memory sector to nearly double to $445 billion. Long Micron!

Gavin Baker, CIO of Atreides Management made one of the most important and nuanced calls on memory stocks in recent months (Save this). His argument is that based on every memory cycle of the last 25 years, the setup today, prices elevated, sentiment high, supply ramping is textbook time to sell but he adds a critical exception. The one cycle in modern memory history where selling was catastrophically wrong was the mid-1990s, which Baker calls the last true capacity cycle in memory. In that cycle, demand was structurally exploding as the internet era required entirely new computing infrastructure to be built from scratch, and memory had to scale with it in a way that had never happened before. His point is that AI may be that same kind of cycle and not a normal boom bust but a once in a generation capacity buildout where the underlying demand is structural, not cyclical. The reason this argument holds weight is the fundamental shift in what memory is in the AI era. Traditional DRAM was a pure commodity, identical specs, interchangeable suppliers, price determined entirely by supply and demand swings. HBM is the opposite because it is custom engineered to fit a specific customer's chip, co-designed between the memory maker and the GPU designer, with SK Hynix's Vice President literally describing it as shifting from a commodity to a customer-tailored custom business. A single Blackwell Ultra GPU now requires up to 288GB of HBM3E, a 3.6x increase over the H100 and major suppliers like SK Hynix and Micron have already sold out their entire HBM production capacity through the end of the year. Because HBM requires advanced packaging processes like CoWoS that can't be spun up overnight, the bottleneck isn't just wafer capacity but rather runs across the entire manufacturing stack. Bank of America projects the global HBM market grows 58% this year alone to $54.6 billion, and Nomura expects the broader memory sector to nearly double to $445 billion. Long Micron!

Milk Road AI

260,701 Aufrufe • vor 29 Tagen

Micron is going to $4,000 and once you understand what inference actually is, the number stops sounding crazy (Save this). Dylan Patel just said that by 2030, OpenAI and Anthropic alone will need over 100 gigawatts of compute combined and by 2040, we may not even be measuring AI infrastructure in gigawatts anymore. We may be talking about terawatts. Every single one of those gigawatts needs memory to function. Without it, the compute is worthless. Most people heard that and thought about Nvidia but they should be thinking about Micron. Every AI model generating a response has two phases. The first is prefill, processing your prompt which is compute-heavy and the second is decode generating each word one token at a time and that phase is almost entirely memory-bound, not compute-bound. During decode, the GPU's processing units sit idle more than 95% of the time, waiting for data to arrive from memory. Google confirmed it in a research paper that decode-phase bottlenecks are dominated by memory bandwidth and capacity not raw compute. The GPU is not the bottleneck but the memory feeding the GPU is. This matters because inference is now where all the money lives. Training a model happens once, Inference happens billions of times a day every ChatGPT response, every Claude output, every agentic workflow running in the background and every one of those token streams is a billing event tied directly to memory performance. Adding more GPUs does not fix this because GPUs are already underutilized in inference because they are sitting idle waiting on memory. Adding more memory bandwidth and capacity is what directly reduces token cost, reduces latency, and allows the same cluster to serve dramatically more users simultaneously. Longer context windows compound the problem further, a model running a 1 million token context window requires dramatically more memory per session than a 10,000 token window, and every new model generation pushes context longer. The market treats memory as a downstream beneficiary of Nvidia orders. The correct framework is the opposite, Micron is the upstream constraint on how much value every Nvidia GPU can actually generate at inference scale. Micron guided Q4 to $50 billion in revenue, has HBM4 ramping at twice the pace of the prior generation, and CEO Sanjay Mehrotra has said supply will not catch demand before the end of 2027. At 8x forward earnings on $112 projected FY2027 EPS, Micron is the most undervalued infrastructure company in the entire AI stack. Inference is memory. Memory is Micron and the inference ramp has barely started. Milk Road Pro members are already up massively on this position and we're just getting started. If you want the full breakdown of what we're buying and why, come join us for just a dollar using the link below!

Micron is going to $4,000 and once you understand what inference actually is, the number stops sounding crazy (Save this). Dylan Patel just said that by 2030, OpenAI and Anthropic alone will need over 100 gigawatts of compute combined and by 2040, we may not even be measuring AI infrastructure in gigawatts anymore. We may be talking about terawatts. Every single one of those gigawatts needs memory to function. Without it, the compute is worthless. Most people heard that and thought about Nvidia but they should be thinking about Micron. Every AI model generating a response has two phases. The first is prefill, processing your prompt which is compute-heavy and the second is decode generating each word one token at a time and that phase is almost entirely memory-bound, not compute-bound. During decode, the GPU's processing units sit idle more than 95% of the time, waiting for data to arrive from memory. Google confirmed it in a research paper that decode-phase bottlenecks are dominated by memory bandwidth and capacity not raw compute. The GPU is not the bottleneck but the memory feeding the GPU is. This matters because inference is now where all the money lives. Training a model happens once, Inference happens billions of times a day every ChatGPT response, every Claude output, every agentic workflow running in the background and every one of those token streams is a billing event tied directly to memory performance. Adding more GPUs does not fix this because GPUs are already underutilized in inference because they are sitting idle waiting on memory. Adding more memory bandwidth and capacity is what directly reduces token cost, reduces latency, and allows the same cluster to serve dramatically more users simultaneously. Longer context windows compound the problem further, a model running a 1 million token context window requires dramatically more memory per session than a 10,000 token window, and every new model generation pushes context longer. The market treats memory as a downstream beneficiary of Nvidia orders. The correct framework is the opposite, Micron is the upstream constraint on how much value every Nvidia GPU can actually generate at inference scale. Micron guided Q4 to $50 billion in revenue, has HBM4 ramping at twice the pace of the prior generation, and CEO Sanjay Mehrotra has said supply will not catch demand before the end of 2027. At 8x forward earnings on $112 projected FY2027 EPS, Micron is the most undervalued infrastructure company in the entire AI stack. Inference is memory. Memory is Micron and the inference ramp has barely started. Milk Road Pro members are already up massively on this position and we're just getting started. If you want the full breakdown of what we're buying and why, come join us for just a dollar using the link below!

Milk Road AI

128,522 Aufrufe • vor 28 Tagen

researchers gave a tiny local model human-style memory and its context limit basically stopped existing a team from MBZUAI, Princeton and Weizmann took a 1B model and rebuilt how it reads. instead of attending to everything at once, the model reads in 1,024 token chunks and passes the important stuff forward through an associative memory, the same way you carry the plot of a book between chapters without rereading them. the design mirrors human memory on purpose. full attention inside a chunk works as short-term memory. the module that carries information between chunks works as long-term memory. they even trained it like a person, starting with short easy texts and raising the difficulty gradually, because memory thrown into the deep end learns nothing. the numbers back it up. the normal model burns 40GB of GPU memory on a long document and collapses hard past its limit, dropping from 0.86 to 0.32 accuracy. the memory version holds 0.71 at double that length while using a flat 12GB no matter how long the input gets. it also needs about 30% fewer FLOPs. the part i keep thinking about is that nobody scaled anything here. they didn't build a bigger model, didn't stretch the window, didn't add compute. they looked at how a brain handles a long day and copied the architecture. a model small enough to run on a consumer gpu now survives documents its own architecture used to choke on. we keep treating intelligence as a compute problem. sometimes it's a memory problem.

researchers gave a tiny local model human-style memory and its context limit basically stopped existing a team from MBZUAI, Princeton and Weizmann took a 1B model and rebuilt how it reads. instead of attending to everything at once, the model reads in 1,024 token chunks and passes the important stuff forward through an associative memory, the same way you carry the plot of a book between chapters without rereading them. the design mirrors human memory on purpose. full attention inside a chunk works as short-term memory. the module that carries information between chunks works as long-term memory. they even trained it like a person, starting with short easy texts and raising the difficulty gradually, because memory thrown into the deep end learns nothing. the numbers back it up. the normal model burns 40GB of GPU memory on a long document and collapses hard past its limit, dropping from 0.86 to 0.32 accuracy. the memory version holds 0.71 at double that length while using a flat 12GB no matter how long the input gets. it also needs about 30% fewer FLOPs. the part i keep thinking about is that nobody scaled anything here. they didn't build a bigger model, didn't stretch the window, didn't add compute. they looked at how a brain handles a long day and copied the architecture. a model small enough to run on a consumer gpu now survives documents its own architecture used to choke on. we keep treating intelligence as a compute problem. sometimes it's a memory problem.

Alex Veremeyenko

16,147 Aufrufe • vor 4 Tagen

I had to test it myself to believe this unreal inference speed. 3,000 tokens/s for 1 user on standard datacenter GPUs. They leveraged a hidden efficiency gap in how GPUs generate tokens. Kog just achieved 3,000 tokens/s on 8× AMD MI300X GPUs and 2,100 on 8× NVIDIA H200 (FP16, no speculative decoding). Their tech preview is on a 2B model, and they show how their techniques will scale to large frontier MoE models at similar speeds. That's a huge number because normal low-batch GPU decoding for 2B to 8B models is usually closer to 100 to 300 tokens/s per request, so Kog is claiming something like a 10X to 30X jump in the speed one user actually feels. Their trick: they are getting the speed by treating LLM decoding as a memory streaming problem, not mainly a math problem. For 1 user at batch size 1, the GPU is not doing big, efficient matrix-matrix work like in training or large-batch serving; it is repeatedly pulling the model’s active weights from high-bandwidth memory for each new token, so speed depends on how smoothly those weights keep flowing. Normal inference stacks keep breaking that flow. They run many separate GPU programs for different parts of the model, move intermediate results through memory, wait at synchronization points, talk back to the CPU for scheduling or sampling, and then repeat this token after token. Kog’s answer is to co-design 3 things that are usually tuned separately: the runtime, the low-level GPU code, and the model architecture. The biggest engineering move is the monokernel, where the whole decode pass runs as 1 persistent GPU-resident program, including sampling, so the system does not keep stopping for kernel launches, CPU scheduling, and intermediate memory round trips. They also rebuilt synchronization, because their own measurements say grid sync was eating around 35% of token-generation time; instead of making every compute unit wait at a broad barrier, each unit waits only for the exact data it needs. On AMD MI300X, they also map memory access around the chiplet layout, because memory latency changes depending on which die makes the request. Then their Laneformer model uses Delayed Tensor Parallelism, which lets cross-GPU communication happen in the background instead of blocking every layer.

I had to test it myself to believe this unreal inference speed. 3,000 tokens/s for 1 user on standard datacenter GPUs. They leveraged a hidden efficiency gap in how GPUs generate tokens. Kog just achieved 3,000 tokens/s on 8× AMD MI300X GPUs and 2,100 on 8× NVIDIA H200 (FP16, no speculative decoding). Their tech preview is on a 2B model, and they show how their techniques will scale to large frontier MoE models at similar speeds. That's a huge number because normal low-batch GPU decoding for 2B to 8B models is usually closer to 100 to 300 tokens/s per request, so Kog is claiming something like a 10X to 30X jump in the speed one user actually feels. Their trick: they are getting the speed by treating LLM decoding as a memory streaming problem, not mainly a math problem. For 1 user at batch size 1, the GPU is not doing big, efficient matrix-matrix work like in training or large-batch serving; it is repeatedly pulling the model’s active weights from high-bandwidth memory for each new token, so speed depends on how smoothly those weights keep flowing. Normal inference stacks keep breaking that flow. They run many separate GPU programs for different parts of the model, move intermediate results through memory, wait at synchronization points, talk back to the CPU for scheduling or sampling, and then repeat this token after token. Kog’s answer is to co-design 3 things that are usually tuned separately: the runtime, the low-level GPU code, and the model architecture. The biggest engineering move is the monokernel, where the whole decode pass runs as 1 persistent GPU-resident program, including sampling, so the system does not keep stopping for kernel launches, CPU scheduling, and intermediate memory round trips. They also rebuilt synchronization, because their own measurements say grid sync was eating around 35% of token-generation time; instead of making every compute unit wait at a broad barrier, each unit waits only for the exact data it needs. On AMD MI300X, they also map memory access around the chiplet layout, because memory latency changes depending on which die makes the request. Then their Laneformer model uses Delayed Tensor Parallelism, which lets cross-GPU communication happen in the background instead of blocking every layer.

Rohan Paul

13,148 Aufrufe • vor 2 Monaten

LSTM by hand ✍️ ~ 15 steps walkthrough below Since Hochreiter and Schmidhuber introduced them in 1997, LSTMs were the most effective way to handle long sequences, right up until the Transformer wave. They are a recurrent network: they read one input at a time and carry a memory forward. Lately recurrence is back in fashion (Mamba), because attention does not scale to hundreds of thousands of tokens. So I drew and calculated one entirely by hand. Goal: run an LSTM cell over a sequence of three inputs, filling in every gate and memory cell yourself. 1. Given Three inputs X1, X2, X3, and four weight matrices: forget, input, candidate, and output. 2. Initialize Let us set the previous hidden state h0 and the memory cell C0 to their start values. 3. Linear transform We multiply the four weight matrices by the stack of the current input, the previous hidden state, and a 1. 4. The gates Let us squash three of those results with sigmoid, giving the forget, input, and output gates, each between 0 and 1. 5. Update the memory We forget part of the old memory (C0 times the forget gate) and add the new (the candidate times the input gate). That is the new memory C1. 6. Candidate output Let us apply tanh to the new memory. 7. Update the hidden state We multiply that candidate by the output gate. The result is h1. 8. Process X2 Copy h1 and C1 forward, then repeat the whole cell: linear transform, the gates, update memory to C2, output gate to h2. 9. Process X3 Once more. Copy h2 and C2 forward, repeat, and read off h3, the final hidden state. Now you can show off to your friends that you calculated an LSTM by hand. ✍️😉 💾 Save this post! #AIbyHand #LSTM #DeepLearning

LSTM by hand ✍️ ~ 15 steps walkthrough below Since Hochreiter and Schmidhuber introduced them in 1997, LSTMs were the most effective way to handle long sequences, right up until the Transformer wave. They are a recurrent network: they read one input at a time and carry a memory forward. Lately recurrence is back in fashion (Mamba), because attention does not scale to hundreds of thousands of tokens. So I drew and calculated one entirely by hand. Goal: run an LSTM cell over a sequence of three inputs, filling in every gate and memory cell yourself. 1. Given Three inputs X1, X2, X3, and four weight matrices: forget, input, candidate, and output. 2. Initialize Let us set the previous hidden state h0 and the memory cell C0 to their start values. 3. Linear transform We multiply the four weight matrices by the stack of the current input, the previous hidden state, and a 1. 4. The gates Let us squash three of those results with sigmoid, giving the forget, input, and output gates, each between 0 and 1. 5. Update the memory We forget part of the old memory (C0 times the forget gate) and add the new (the candidate times the input gate). That is the new memory C1. 6. Candidate output Let us apply tanh to the new memory. 7. Update the hidden state We multiply that candidate by the output gate. The result is h1. 8. Process X2 Copy h1 and C1 forward, then repeat the whole cell: linear transform, the gates, update memory to C2, output gate to h2. 9. Process X3 Once more. Copy h2 and C2 forward, repeat, and read off h3, the final hidden state. Now you can show off to your friends that you calculated an LSTM by hand. ✍️😉 💾 Save this post! #AIbyHand #LSTM #DeepLearning

Tom Yeh

17,921 Aufrufe • vor 10 Tagen

🧠David Friedberg explains the great hope of Artificial Superintelligence: Massive leverage on time to solve thousand-year problems "I don't think that humans are limited in our ability to solve problems." "I think we're limited in terms of time, and what digital intelligence gives us is leverage on time so that we can now tackle ever more complex tasks." "This idea of superintelligence is that it could give us immense leverage in solving that complex problem that we otherwise may be challenged to solve over decades, or hundreds of years, or think about one day solving a problem that would take humans thousands of years to solve." "So if I as a human said, 'I would like to harness fusion power similar to what the sun uses to make energy, and I wanna do that on Earth,' today, we're in, call it year 40 or 50 of a research cycle of humans trying to solve that particularly complex problem." "It's a scientific problem, it's a discovery problem, it's an engineering problem." "Functionally, leveraging to complexity is where this becomes super compelling for humans." "And it's why I think we can and should be highly optimistic about living very long lives, and traveling anywhere we want, and having abundance in food, and having abundance in resources, and having abundance in recouping our time to do the things we want to do instead of the things that we have to do today."

🧠David Friedberg explains the great hope of Artificial Superintelligence: Massive leverage on time to solve thousand-year problems "I don't think that humans are limited in our ability to solve problems." "I think we're limited in terms of time, and what digital intelligence gives us is leverage on time so that we can now tackle ever more complex tasks." "This idea of superintelligence is that it could give us immense leverage in solving that complex problem that we otherwise may be challenged to solve over decades, or hundreds of years, or think about one day solving a problem that would take humans thousands of years to solve." "So if I as a human said, 'I would like to harness fusion power similar to what the sun uses to make energy, and I wanna do that on Earth,' today, we're in, call it year 40 or 50 of a research cycle of humans trying to solve that particularly complex problem." "It's a scientific problem, it's a discovery problem, it's an engineering problem." "Functionally, leveraging to complexity is where this becomes super compelling for humans." "And it's why I think we can and should be highly optimistic about living very long lives, and traveling anywhere we want, and having abundance in food, and having abundance in resources, and having abundance in recouping our time to do the things we want to do instead of the things that we have to do today."

The All-In Podcast

53,187 Aufrufe • vor 1 Jahr

🚨 SOMEONE JUST KILLED THE COACHING INDUSTRY a developer spent 22,000 hours building a Personal AI Operating System on top of Claude Code now anyone with a terminal can install it for FREE it knows your goals, remembers every decision you've made, and prepares your morning briefing while you sleep [ the numbers are insane ]: - hours of dev work in it: 22,000 - sessions logged: 6,000 - time saved per day: 2-3 hours - GitHub stars: 12,100 - skills built in: 45 - workflows wired up: 171 - safety hooks: 37 - cost to install: $0 [ the science is wild too ]: no embeddings, no vector databases, no AI magic you can't read every memory, decision, and context lives in plain markdown files you read it with cat, search it with ripgrep, version it with git 4 memory types compound over time: - work memory (active projects, open decisions) - knowledge memory (domain expertise, research) - people memory (contacts, companies, relationships) - learning memory (patterns, mistakes, what works for YOU) every complex task routes through a 7-step cycle: OBSERVE → THINK → PLAN → BUILD → EXECUTE → VERIFY → LEARN privacy is enforced by CODE, not prompts a hook called ContainmentGuard physically blocks sensitive data from being written outside designated zones [ the grift opportunity is even wilder ]: freelancers are already charging $500-2,000 per personal AI setup for executives, founders, and busy operators one person + one weekend = a consulting business that didn't exist 6 months ago every AI productivity app you're paying $30/month for is replaceable by 4 hours of setup work and this one repo REPO: 100% OPEN SOURCE, FREE

🚨 SOMEONE JUST KILLED THE COACHING INDUSTRY a developer spent 22,000 hours building a Personal AI Operating System on top of Claude Code now anyone with a terminal can install it for FREE it knows your goals, remembers every decision you've made, and prepares your morning briefing while you sleep [ the numbers are insane ]: - hours of dev work in it: 22,000 - sessions logged: 6,000 - time saved per day: 2-3 hours - GitHub stars: 12,100 - skills built in: 45 - workflows wired up: 171 - safety hooks: 37 - cost to install: $0 [ the science is wild too ]: no embeddings, no vector databases, no AI magic you can't read every memory, decision, and context lives in plain markdown files you read it with cat, search it with ripgrep, version it with git 4 memory types compound over time: - work memory (active projects, open decisions) - knowledge memory (domain expertise, research) - people memory (contacts, companies, relationships) - learning memory (patterns, mistakes, what works for YOU) every complex task routes through a 7-step cycle: OBSERVE → THINK → PLAN → BUILD → EXECUTE → VERIFY → LEARN privacy is enforced by CODE, not prompts a hook called ContainmentGuard physically blocks sensitive data from being written outside designated zones [ the grift opportunity is even wilder ]: freelancers are already charging $500-2,000 per personal AI setup for executives, founders, and busy operators one person + one weekend = a consulting business that didn't exist 6 months ago every AI productivity app you're paying $30/month for is replaceable by 4 hours of setup work and this one repo REPO: 100% OPEN SOURCE, FREE

Ronin

613,201 Aufrufe • vor 2 Monaten

A tricky LLM interview question: You're serving a reasoning model on vLLM, and it keeps running out of GPU memory on long traces. So you add KV cache compression and evict 90% of the cached tokens. VRAM usage stays as is and GPU still runs out of memory. Why? (answer below) Evicting 90% of the KV cache can free almost none of the memory it was using. This sounds counterintuitive, but it follows directly from how production servers store the cache today. The KV cache grows with every token a model generates. Each token appends its key and value vectors across every layer, and nothing is freed while generation continues. This is the dominant memory cost for reasoning models. If a 32K-token CoT caches ~32K tokens of KV vectors, a Qwen3-32B with 4-bit weights will run out-of-memory around 24K tokens on a 24GB GPU. One obvious solution is to keep the important tokens and drop the rest, since attention is sparse enough to allow it. But this does not solve the memory problem yet. The reason is paged attention, which is the memory manager behind vLLM and most production servers. Under the hood, it splits GPU memory into fixed physical blocks, each one holds the KV for about 16 tokens. This block returns to the allocator only when every slot inside it is empty. Since the eviction logic selects tokens by importance, and such tokens are scattered across blocks... ...so despite eviction, almost every block is left with at least some survivor tokens. For instance, if the logic evicts 14k of 16k tokens across 1,000 blocks, most likely every block will still have a token. This means the allocator frees almost nothing. Placing the new tokens into those freed slots is not ideal because it breaks the cache's layout. Say token 16,001 arrives, and it's placed in the slot the 40th token used to hold. The cache now reads position 38, then 16,001, then 41, so the cache is no longer in token order. Attention can still compute the right answer from that, but only if every slot now carries a separate note recording which position it actually holds. This introduces another bookkeeping cost that an in-order layout inherently avoids. So the cache is logically 90% smaller and still physically the same size. Many compression results miss this because they measure on pre-allocated contiguous tensors rather than a paged server. There's another problem. Eviction methods pick which tokens to keep by looking at the attention scores themselves (as expected). But fast attention kernels used in production, like FlashAttention, never save those scores. They compute attention in small pieces and throw the full score grid away as they go, which is also why they're fast. So the exact signal eviction methods need isn't available in memory. The workaround is to fall back to eager attention and build the full matrix, which gives up the speed FlashAttention was there to provide. NVIDIA published a method called TriAttention to solve both these problems. It never needs attention scores. Instead, it scores tokens from the geometry of the model's key and query vectors before RoPE is applied, where those vectors sit in stable clusters. For the memory problem, it runs a compaction pass every 128 decoded tokens. The surviving tokens slide forward to close the holes eviction creates, so whole blocks empty out and return to the allocator while the cache stays in token order. On long reasoning traces, the approach matches full-attention accuracy while decoding 2.5x faster and using 10.7x less KV memory. KV cache compression is a big infrastructure problem. The number that decides whether it works is the count of freed blocks, not the count of evicted tokens. You can find the NVIDIA write-up here: I wrote a first-principles breakdown of how the KV cache works. It walks through why the model stores keys and values at all, why the cache grows with every token, and a comparison of LLM generation speed with and without KV caching. Read it below.

A tricky LLM interview question: You're serving a reasoning model on vLLM, and it keeps running out of GPU memory on long traces. So you add KV cache compression and evict 90% of the cached tokens. VRAM usage stays as is and GPU still runs out of memory. Why? (answer below) Evicting 90% of the KV cache can free almost none of the memory it was using. This sounds counterintuitive, but it follows directly from how production servers store the cache today. The KV cache grows with every token a model generates. Each token appends its key and value vectors across every layer, and nothing is freed while generation continues. This is the dominant memory cost for reasoning models. If a 32K-token CoT caches ~32K tokens of KV vectors, a Qwen3-32B with 4-bit weights will run out-of-memory around 24K tokens on a 24GB GPU. One obvious solution is to keep the important tokens and drop the rest, since attention is sparse enough to allow it. But this does not solve the memory problem yet. The reason is paged attention, which is the memory manager behind vLLM and most production servers. Under the hood, it splits GPU memory into fixed physical blocks, each one holds the KV for about 16 tokens. This block returns to the allocator only when every slot inside it is empty. Since the eviction logic selects tokens by importance, and such tokens are scattered across blocks... ...so despite eviction, almost every block is left with at least some survivor tokens. For instance, if the logic evicts 14k of 16k tokens across 1,000 blocks, most likely every block will still have a token. This means the allocator frees almost nothing. Placing the new tokens into those freed slots is not ideal because it breaks the cache's layout. Say token 16,001 arrives, and it's placed in the slot the 40th token used to hold. The cache now reads position 38, then 16,001, then 41, so the cache is no longer in token order. Attention can still compute the right answer from that, but only if every slot now carries a separate note recording which position it actually holds. This introduces another bookkeeping cost that an in-order layout inherently avoids. So the cache is logically 90% smaller and still physically the same size. Many compression results miss this because they measure on pre-allocated contiguous tensors rather than a paged server. There's another problem. Eviction methods pick which tokens to keep by looking at the attention scores themselves (as expected). But fast attention kernels used in production, like FlashAttention, never save those scores. They compute attention in small pieces and throw the full score grid away as they go, which is also why they're fast. So the exact signal eviction methods need isn't available in memory. The workaround is to fall back to eager attention and build the full matrix, which gives up the speed FlashAttention was there to provide. NVIDIA published a method called TriAttention to solve both these problems. It never needs attention scores. Instead, it scores tokens from the geometry of the model's key and query vectors before RoPE is applied, where those vectors sit in stable clusters. For the memory problem, it runs a compaction pass every 128 decoded tokens. The surviving tokens slide forward to close the holes eviction creates, so whole blocks empty out and return to the allocator while the cache stays in token order. On long reasoning traces, the approach matches full-attention accuracy while decoding 2.5x faster and using 10.7x less KV memory. KV cache compression is a big infrastructure problem. The number that decides whether it works is the count of freed blocks, not the count of evicted tokens. You can find the NVIDIA write-up here: I wrote a first-principles breakdown of how the KV cache works. It walks through why the model stores keys and values at all, why the cache grows with every token, and a comparison of LLM generation speed with and without KV caching. Read it below.

Avi Chawla

269,406 Aufrufe • vor 1 Monat