Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

A developer figured out how to give AI permanent memory for $0.40 a year. No complex vector databases. No massive cloud storage. Just a single 4,000-token file that refuses to grow. The secret? True memory isn’t about storing everything. It’s about aggressive deletion. Instead of endlessly appending data, his...

shmidtqq's profile picture

shmidt

11,166 subscribers

17,304 Aufrufe • vor 9 Tagen •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

MICROSOFT QUIETLY BUILT ONE MEMORY MODULE THAT PLUGS INTO ANY AI AGENT AND CUTS ITS CONTEXT BY UP TO 100X no retraining. no task-specific redesign. you just plug it in Microsoft's insight: agents don't need to remember what happened. they need the knowledge pulled from it - the facts and the reusable skills, not the raw logs so PlugMem stops storing history. it compiles experience into a knowledge graph, then feeds the agent only what matters the result: > up to 1-2 orders of magnitude fewer tokens > highest decision-utility per token of any system tested > one unchanged module beats task-specific memory on chat QA, multi-hop search, and web agents more memory was never the answer. better memory is - and Microsoft just shipped it as a plug-in bookmark this. the agents that win won't remember more, they'll remember smarter.
0:42

MICROSOFT QUIETLY BUILT ONE MEMORY MODULE THAT PLUGS INTO ANY AI AGENT AND CUTS ITS CONTEXT BY UP TO 100X no retraining. no task-specific redesign. you just plug it in Microsoft's insight: agents don't need to remember what happened. they need the knowledge pulled from it - the facts and the reusable skills, not the raw logs so PlugMem stops storing history. it compiles experience into a knowledge graph, then feeds the agent only what matters the result: > up to 1-2 orders of magnitude fewer tokens > highest decision-utility per token of any system tested > one unchanged module beats task-specific memory on chat QA, multi-hop search, and web agents more memory was never the answer. better memory is - and Microsoft just shipped it as a plug-in bookmark this. the agents that win won't remember more, they'll remember smarter.

NO1ennn

226,539 Aufrufe • vor 1 Monat

A tricky LLM interview question: You're serving a reasoning model on vLLM, and it keeps running out of GPU memory on long traces. So you add KV cache compression and evict 90% of the cached tokens. VRAM usage stays as is and GPU still runs out of memory. Why? (answer below) Evicting 90% of the KV cache can free almost none of the memory it was using. This sounds counterintuitive, but it follows directly from how production servers store the cache today. The KV cache grows with every token a model generates. Each token appends its key and value vectors across every layer, and nothing is freed while generation continues. This is the dominant memory cost for reasoning models. If a 32K-token CoT caches ~32K tokens of KV vectors, a Qwen3-32B with 4-bit weights will run out-of-memory around 24K tokens on a 24GB GPU. One obvious solution is to keep the important tokens and drop the rest, since attention is sparse enough to allow it. But this does not solve the memory problem yet. The reason is paged attention, which is the memory manager behind vLLM and most production servers. Under the hood, it splits GPU memory into fixed physical blocks, each one holds the KV for about 16 tokens. This block returns to the allocator only when every slot inside it is empty. Since the eviction logic selects tokens by importance, and such tokens are scattered across blocks... ...so despite eviction, almost every block is left with at least some survivor tokens. For instance, if the logic evicts 14k of 16k tokens across 1,000 blocks, most likely every block will still have a token. This means the allocator frees almost nothing. Placing the new tokens into those freed slots is not ideal because it breaks the cache's layout. Say token 16,001 arrives, and it's placed in the slot the 40th token used to hold. The cache now reads position 38, then 16,001, then 41, so the cache is no longer in token order. Attention can still compute the right answer from that, but only if every slot now carries a separate note recording which position it actually holds. This introduces another bookkeeping cost that an in-order layout inherently avoids. So the cache is logically 90% smaller and still physically the same size. Many compression results miss this because they measure on pre-allocated contiguous tensors rather than a paged server. There's another problem. Eviction methods pick which tokens to keep by looking at the attention scores themselves (as expected). But fast attention kernels used in production, like FlashAttention, never save those scores. They compute attention in small pieces and throw the full score grid away as they go, which is also why they're fast. So the exact signal eviction methods need isn't available in memory. The workaround is to fall back to eager attention and build the full matrix, which gives up the speed FlashAttention was there to provide. NVIDIA published a method called TriAttention to solve both these problems. It never needs attention scores. Instead, it scores tokens from the geometry of the model's key and query vectors before RoPE is applied, where those vectors sit in stable clusters. For the memory problem, it runs a compaction pass every 128 decoded tokens. The surviving tokens slide forward to close the holes eviction creates, so whole blocks empty out and return to the allocator while the cache stays in token order. On long reasoning traces, the approach matches full-attention accuracy while decoding 2.5x faster and using 10.7x less KV memory. KV cache compression is a big infrastructure problem. The number that decides whether it works is the count of freed blocks, not the count of evicted tokens. You can find the NVIDIA write-up here: I wrote a first-principles breakdown of how the KV cache works. It walks through why the model stores keys and values at all, why the cache grows with every token, and a comparison of LLM generation speed with and without KV caching. Read it below.
0:23

A tricky LLM interview question: You're serving a reasoning model on vLLM, and it keeps running out of GPU memory on long traces. So you add KV cache compression and evict 90% of the cached tokens. VRAM usage stays as is and GPU still runs out of memory. Why? (answer below) Evicting 90% of the KV cache can free almost none of the memory it was using. This sounds counterintuitive, but it follows directly from how production servers store the cache today. The KV cache grows with every token a model generates. Each token appends its key and value vectors across every layer, and nothing is freed while generation continues. This is the dominant memory cost for reasoning models. If a 32K-token CoT caches ~32K tokens of KV vectors, a Qwen3-32B with 4-bit weights will run out-of-memory around 24K tokens on a 24GB GPU. One obvious solution is to keep the important tokens and drop the rest, since attention is sparse enough to allow it. But this does not solve the memory problem yet. The reason is paged attention, which is the memory manager behind vLLM and most production servers. Under the hood, it splits GPU memory into fixed physical blocks, each one holds the KV for about 16 tokens. This block returns to the allocator only when every slot inside it is empty. Since the eviction logic selects tokens by importance, and such tokens are scattered across blocks... ...so despite eviction, almost every block is left with at least some survivor tokens. For instance, if the logic evicts 14k of 16k tokens across 1,000 blocks, most likely every block will still have a token. This means the allocator frees almost nothing. Placing the new tokens into those freed slots is not ideal because it breaks the cache's layout. Say token 16,001 arrives, and it's placed in the slot the 40th token used to hold. The cache now reads position 38, then 16,001, then 41, so the cache is no longer in token order. Attention can still compute the right answer from that, but only if every slot now carries a separate note recording which position it actually holds. This introduces another bookkeeping cost that an in-order layout inherently avoids. So the cache is logically 90% smaller and still physically the same size. Many compression results miss this because they measure on pre-allocated contiguous tensors rather than a paged server. There's another problem. Eviction methods pick which tokens to keep by looking at the attention scores themselves (as expected). But fast attention kernels used in production, like FlashAttention, never save those scores. They compute attention in small pieces and throw the full score grid away as they go, which is also why they're fast. So the exact signal eviction methods need isn't available in memory. The workaround is to fall back to eager attention and build the full matrix, which gives up the speed FlashAttention was there to provide. NVIDIA published a method called TriAttention to solve both these problems. It never needs attention scores. Instead, it scores tokens from the geometry of the model's key and query vectors before RoPE is applied, where those vectors sit in stable clusters. For the memory problem, it runs a compaction pass every 128 decoded tokens. The surviving tokens slide forward to close the holes eviction creates, so whole blocks empty out and return to the allocator while the cache stays in token order. On long reasoning traces, the approach matches full-attention accuracy while decoding 2.5x faster and using 10.7x less KV memory. KV cache compression is a big infrastructure problem. The number that decides whether it works is the count of freed blocks, not the count of evicted tokens. You can find the NVIDIA write-up here: I wrote a first-principles breakdown of how the KV cache works. It walks through why the model stores keys and values at all, why the cache grows with every token, and a comparison of LLM generation speed with and without KV caching. Read it below.

Avi Chawla

271,839 Aufrufe • vor 2 Monaten

ANTHROPIC LEAKED A SETUP WHERE 6 FILES CUT TOKENS BY 84% AND RAISE ACCURACY BY 39% engineers at Google and Microsoft have been running this for a while - they pay $3 where you pay $20. decisions → contracts → dead ends → state → sources → open questions memory moves outside the window - what enters the context is a pointer to a file, not the history. raw history refills the window to 200k every single turn - six pointers take 8k and leave 96% free. same work, same five turns: 1,000,000 tokens against 40,000. decisions written to a file kill the endless "wait, why did we do it this way". written dead ends kill the second lap around the same circle - the agent never walks in there twice. a stable prefix keeps the cache warm - a cache read costs 0.1x of normal input. that's why 20 turns cost 3.15x instead of 20x - one write and nineteen reads. save this and paste it into Claude Code - and your agent stops paying for what it already knew ↓
0:51

ANTHROPIC LEAKED A SETUP WHERE 6 FILES CUT TOKENS BY 84% AND RAISE ACCURACY BY 39% engineers at Google and Microsoft have been running this for a while - they pay $3 where you pay $20. decisions → contracts → dead ends → state → sources → open questions memory moves outside the window - what enters the context is a pointer to a file, not the history. raw history refills the window to 200k every single turn - six pointers take 8k and leave 96% free. same work, same five turns: 1,000,000 tokens against 40,000. decisions written to a file kill the endless "wait, why did we do it this way". written dead ends kill the second lap around the same circle - the agent never walks in there twice. a stable prefix keeps the cache warm - a cache read costs 0.1x of normal input. that's why 20 turns cost 3.15x instead of 20x - one write and nineteen reads. save this and paste it into Claude Code - and your agent stops paying for what it already knew ↓

Noisy

137,563 Aufrufe • vor 24 Tagen

Nvidia CEO Jensen Huang on why engineers will soon be paid in tokens, not just salary: Jensen lays out a future where compute access becomes part of an engineer's compensation package. "I could totally imagine in the future every single engineer in our company will need an annual token budget," he says. He explains how the math would work: "They're going to make a few hundred thousand a year their base pay. I'm going to give them probably half of that on top of it as tokens so that they could be amplified 10x. Of course, we would." According to Huang, this is already changing how companies compete for talent: "It is now one of the recruiting tools in Silicon Valley. How many tokens comes along with my job?" His reasoning is simple: tokens make engineers more productive. As he puts it, "every engineer that has access to tokens will be more productive and those tokens as you know will be produced by AI factories that all of you and us we partner to build." Huang then zooms out to describe how this reshapes the nature of companies themselves: "Every single enterprise company in today sit on top of file systems and data centers. Every single software company of the future will be agentic and they will be token manufacturers. They be token users for their engineers and they'll be token manufacturers for all of their customers."
1:04

Nvidia CEO Jensen Huang on why engineers will soon be paid in tokens, not just salary: Jensen lays out a future where compute access becomes part of an engineer's compensation package. "I could totally imagine in the future every single engineer in our company will need an annual token budget," he says. He explains how the math would work: "They're going to make a few hundred thousand a year their base pay. I'm going to give them probably half of that on top of it as tokens so that they could be amplified 10x. Of course, we would." According to Huang, this is already changing how companies compete for talent: "It is now one of the recruiting tools in Silicon Valley. How many tokens comes along with my job?" His reasoning is simple: tokens make engineers more productive. As he puts it, "every engineer that has access to tokens will be more productive and those tokens as you know will be produced by AI factories that all of you and us we partner to build." Huang then zooms out to describe how this reshapes the nature of companies themselves: "Every single enterprise company in today sit on top of file systems and data centers. Every single software company of the future will be agentic and they will be token manufacturers. They be token users for their engineers and they'll be token manufacturers for all of their customers."

Big Brain AI

90,171 Aufrufe • vor 3 Monaten

THE $4,000 DGX SPARK BENCHMARKS AT 342 T/S ON PROMPT PROCESSING - THEN DROPS TO 73 WHEN IT ACTUALLY GENERATES. that clip is a dgx spark running llama-bench on qwen3-coder 30b, side by side with a mac. two numbers decide everything here: pp512, prompt processing = 342 tokens/sec tg128, token generation = 73 tokens/sec that gap is the whole lesson. local ai isn't one speed, it's two. prompt processing is reading your input: your files, your long context. the spark is compute-heavy, so it eats that fast. token generation is writing the answer, and it's bound by memory bandwidth, where the spark is modest and unified-memory macs stay close. so "which box is faster" has no single winner. feeding huge context or running agents that read a lot? prompt processing wins, the spark shines. long chat replies, one token at a time? generation matters more, and a mac holds its own. no single benchmark number, no universal "fastest box", no upgrade that wins every workload. the uncomfortable part isn't that the spark is fast. it's that "tokens per second" was always two numbers, and the hype only ever quotes the flattering one. follow RetroChainer for the benchmark truth nobody screenshots, and bookmark this before you buy on one number.
1:14

THE $4,000 DGX SPARK BENCHMARKS AT 342 T/S ON PROMPT PROCESSING - THEN DROPS TO 73 WHEN IT ACTUALLY GENERATES. that clip is a dgx spark running llama-bench on qwen3-coder 30b, side by side with a mac. two numbers decide everything here: pp512, prompt processing = 342 tokens/sec tg128, token generation = 73 tokens/sec that gap is the whole lesson. local ai isn't one speed, it's two. prompt processing is reading your input: your files, your long context. the spark is compute-heavy, so it eats that fast. token generation is writing the answer, and it's bound by memory bandwidth, where the spark is modest and unified-memory macs stay close. so "which box is faster" has no single winner. feeding huge context or running agents that read a lot? prompt processing wins, the spark shines. long chat replies, one token at a time? generation matters more, and a mac holds its own. no single benchmark number, no universal "fastest box", no upgrade that wins every workload. the uncomfortable part isn't that the spark is fast. it's that "tokens per second" was always two numbers, and the hype only ever quotes the flattering one. follow RetroChainer for the benchmark truth nobody screenshots, and bookmark this before you buy on one number.

RetroChainer

42,451 Aufrufe • vor 27 Tagen

Dylan on why access to the most intelligent AI models will determine who wins in every industry: "What really matters is access to the most intelligent tokens and leveraging them at the highest-value tasks. A lot of folks will want tokens, but the shitty SaaS startup in SF who is using Claude is not necessarily creating a ton of value, and therefore they're going to get priced out of tokens. Anthropic only released Mythos to certain companies for cybersecurity. You also don't want people to distill your model so you release them to a fewer and fewer customers. Those customers are now wrestling over the tokens. Token usage and therefore the additional value generated on top of those tokens aggregates among fewer and fewer companies. I don't have Mythos but you know who has Mythos, top banks. There are people like Ken Griffin of Citadel who can just sign a deal with OpenAI or Antrhopic for access to your models. I'll buy the first $10 billion worth of tokens each year. So whenever you release the model, I'll spend the first 10 billion tokens and then everyone else can get the model after that. Now, he's going to crush everyone in the markets. I can envision because I have an enterprise Anthropic contract and Anthropic people like me, they're willing to give us slightly earlier access or higher rate limits, and then my competitor doesn't have that, and I'm able to f*cking crush them."
2:22

Dylan on why access to the most intelligent AI models will determine who wins in every industry: "What really matters is access to the most intelligent tokens and leveraging them at the highest-value tasks. A lot of folks will want tokens, but the shitty SaaS startup in SF who is using Claude is not necessarily creating a ton of value, and therefore they're going to get priced out of tokens. Anthropic only released Mythos to certain companies for cybersecurity. You also don't want people to distill your model so you release them to a fewer and fewer customers. Those customers are now wrestling over the tokens. Token usage and therefore the additional value generated on top of those tokens aggregates among fewer and fewer companies. I don't have Mythos but you know who has Mythos, top banks. There are people like Ken Griffin of Citadel who can just sign a deal with OpenAI or Antrhopic for access to your models. I'll buy the first $10 billion worth of tokens each year. So whenever you release the model, I'll spend the first 10 billion tokens and then everyone else can get the model after that. Now, he's going to crush everyone in the markets. I can envision because I have an enterprise Anthropic contract and Anthropic people like me, they're willing to give us slightly earlier access or higher rate limits, and then my competitor doesn't have that, and I'm able to f*cking crush them."

Patrick OShaughnessy

36,100 Aufrufe • vor 4 Monaten

most agent memory is one file that grows forever and gets re-read every single turn. it overflows the context, overwrites old facts, loses the thread. that's why your agent feels sharp on day one and lost by week three. Sibyl Memory replaces the pile with a structure: → leaner context, lower token cost. a bloated agent file and flat memory get re-read in full every message. Sibyl Memory keeps a light hot layer and retrieves the rest on demand, so each turn carries only the tokens that matter. → relations are first-class. people link to projects, projects to deals, deals to the decisions behind them. ask about one partner and the agent surfaces the whole connected web around them. that relational context is what lets it help run a company. → one source of truth per fact, person, and project. no duplicates, no silent overwrites. works with hermes, claude code, and codex. beta open.
0:34

most agent memory is one file that grows forever and gets re-read every single turn. it overflows the context, overwrites old facts, loses the thread. that's why your agent feels sharp on day one and lost by week three. Sibyl Memory replaces the pile with a structure: → leaner context, lower token cost. a bloated agent file and flat memory get re-read in full every message. Sibyl Memory keeps a light hot layer and retrieves the rest on demand, so each turn carries only the tokens that matter. → relations are first-class. people link to projects, projects to deals, deals to the decisions behind them. ask about one partner and the agent surfaces the whole connected web around them. that relational context is what lets it help run a company. → one source of truth per fact, person, and project. no duplicates, no silent overwrites. works with hermes, claude code, and codex. beta open.

SIBYL

84,702 Aufrufe • vor 3 Monaten

Crypto moves fast, narratives shift overnight, and tokens pump before anyone understands why. By the time most people notice, the opportunity is gone. The market isn’t chaotic, it’s just hard to interpret. Crypto AI Alerts fixes this. Experience a real-time intelligence engine that analyzes everything moving markets: sentiment, news, on-chain activity, liquidity, narrative momentum, and price action. It converts that data into clear bullish, neutral, or bearish signals you can act on. No guessing. No hype. No noise. Here’s what makes it powerful: 🔹 Signal Engine: AI classifies market behavior across tokens and sentiment with explanations behind trends. 🔹 Narrative Discovery: Spot emerging narratives early, track momentum, and find clusters of tokens gaining traction. 🔹 Personalized Watchlists: Follow tokens you care about and get alerts when sentiment changes. Crypto AI Alerts isn’t a signal feed, it’s an intelligence layer for the market: a way to understand what’s happening, why it matters, and where momentum is heading next.
0:37

Crypto moves fast, narratives shift overnight, and tokens pump before anyone understands why. By the time most people notice, the opportunity is gone. The market isn’t chaotic, it’s just hard to interpret. Crypto AI Alerts fixes this. Experience a real-time intelligence engine that analyzes everything moving markets: sentiment, news, on-chain activity, liquidity, narrative momentum, and price action. It converts that data into clear bullish, neutral, or bearish signals you can act on. No guessing. No hype. No noise. Here’s what makes it powerful: 🔹 Signal Engine: AI classifies market behavior across tokens and sentiment with explanations behind trends. 🔹 Narrative Discovery: Spot emerging narratives early, track momentum, and find clusters of tokens gaining traction. 🔹 Personalized Watchlists: Follow tokens you care about and get alerts when sentiment changes. Crypto AI Alerts isn’t a signal feed, it’s an intelligence layer for the market: a way to understand what’s happening, why it matters, and where momentum is heading next.

ChainGPT

54,423 Aufrufe • vor 8 Monaten

THIS GUY AUDITED 926 CLAUDE CODE SESSIONS AND FOUND MOST OF THE TOKEN WASTE WAS ON HIS SIDE everyone is blaming anthropic for the limits, so he decided to actually look at the data 858 sessions, 18,903 turns, and $1,619 estimated spend across 33 days here's what he found: 1\ one default setting was burning 14,000 tokens per turn Claude Code loads the full JSON schema for every tool into context at session start. whether you use them or not. 20,000 tokens of tool definitions sitting there on every single turn. the fix: one line in your settings.json "ENABLE_TOOL_SEARCH": "true" context dropped from 45K to 20K instantly. across 858 sessions that one setting was wasting an estimated 264 million tokens 2\ cache expiry is the single biggest waste 54% of his turns came after a 5+ minute idle gap. every one of those turns re-processed the entire conversation at full price which caused a 10x cost jump you go grab coffee. come back 5 minutes later. type your next message. everything rebuilds from scratch. the context didn't change. you didn't change. the cache just expired. 12.3 million tokens wasted on idle gaps alone 3\ 42 skills loaded. 19 of them used twice or less across 858 sessions. every one of those skill schemas sat in context on every turn eating tokens for nothing. 4\ 1,122 redundant file reads where the same file was read 3+ times one session read the same file 33 times. he ALSO built a full token auditor dashboard that shows you exactly where your waste is coming from 19 charts, opens in your browser, free AND open source
0:20

THIS GUY AUDITED 926 CLAUDE CODE SESSIONS AND FOUND MOST OF THE TOKEN WASTE WAS ON HIS SIDE everyone is blaming anthropic for the limits, so he decided to actually look at the data 858 sessions, 18,903 turns, and $1,619 estimated spend across 33 days here's what he found: 1\ one default setting was burning 14,000 tokens per turn Claude Code loads the full JSON schema for every tool into context at session start. whether you use them or not. 20,000 tokens of tool definitions sitting there on every single turn. the fix: one line in your settings.json "ENABLE_TOOL_SEARCH": "true" context dropped from 45K to 20K instantly. across 858 sessions that one setting was wasting an estimated 264 million tokens 2\ cache expiry is the single biggest waste 54% of his turns came after a 5+ minute idle gap. every one of those turns re-processed the entire conversation at full price which caused a 10x cost jump you go grab coffee. come back 5 minutes later. type your next message. everything rebuilds from scratch. the context didn't change. you didn't change. the cache just expired. 12.3 million tokens wasted on idle gaps alone 3\ 42 skills loaded. 19 of them used twice or less across 858 sessions. every one of those skill schemas sat in context on every turn eating tokens for nothing. 4\ 1,122 redundant file reads where the same file was read 3+ times one session read the same file 33 times. he ALSO built a full token auditor dashboard that shows you exactly where your waste is coming from 19 charts, opens in your browser, free AND open source

Om Patel

299,832 Aufrufe • vor 5 Monaten

Most people use Claude Code like a chatbot. That is the mistake. The real power is not in better prompts. It is in controlling the session. /init gives the project memory. /memory keeps the rules alive. /clear removes old context. /compact saves the important parts. /context shows where the tokens go. /rewind fixes bad turns. /plan stops blind edits. /model matches the brain to the task. Claude Code gets 10x better when you stop letting the conversation run itself. Memory. Context. Boundaries. Rollback. Planning. That is the system. Stop asking Claude Code to “just do it”. Teach it what to remember, what to forget, and when to stop. Bookmark this before your next session turns into 40 messages of context debt.
0:33

Most people use Claude Code like a chatbot. That is the mistake. The real power is not in better prompts. It is in controlling the session. /init gives the project memory. /memory keeps the rules alive. /clear removes old context. /compact saves the important parts. /context shows where the tokens go. /rewind fixes bad turns. /plan stops blind edits. /model matches the brain to the task. Claude Code gets 10x better when you stop letting the conversation run itself. Memory. Context. Boundaries. Rollback. Planning. That is the system. Stop asking Claude Code to “just do it”. Teach it what to remember, what to forget, and when to stop. Bookmark this before your next session turns into 40 messages of context debt.

Fluixo

19,226 Aufrufe • vor 2 Monaten

Sam Altman shared a number that should change how you think about AI. Six and a half years ago, the heaviest AI user on the planet was an OpenAI employee. Using 100,000 tokens a month. That was considered insane at the time. The global average was basically zero. Today the global average is 100,000 tokens a month. What was insane is now normal. And the heaviest user at OpenAI right now? Hundreds of billions of tokens a month. Now here's where it gets wild. If the same pattern repeats, in another six and a half years the average person will use 500 billion tokens a month. That's a 5 million x increase from today. Think about what that means. → AI is not something you open once a day to ask a question anymore. It becomes always on. Running in the background across your work, education, personal life. → Agentic workflows, coding assistants, multi-step reasoning — that's what consumes billions of tokens. Not chatting. → OpenAI is already building for this. Hundreds of billions in infrastructure spend through 2030. Stargate. Partnerships with Microsoft, Oracle, Nvidia. And cost per token keeps falling roughly 10x every 12 months for the same capability level. So usage explodes because it keeps getting cheaper. The pattern is the same one we saw with computing, internet, and mobile. What the power user does today, the average person does in six years. Right now you are the power user. Act like it.
0:46

Sam Altman shared a number that should change how you think about AI. Six and a half years ago, the heaviest AI user on the planet was an OpenAI employee. Using 100,000 tokens a month. That was considered insane at the time. The global average was basically zero. Today the global average is 100,000 tokens a month. What was insane is now normal. And the heaviest user at OpenAI right now? Hundreds of billions of tokens a month. Now here's where it gets wild. If the same pattern repeats, in another six and a half years the average person will use 500 billion tokens a month. That's a 5 million x increase from today. Think about what that means. → AI is not something you open once a day to ask a question anymore. It becomes always on. Running in the background across your work, education, personal life. → Agentic workflows, coding assistants, multi-step reasoning — that's what consumes billions of tokens. Not chatting. → OpenAI is already building for this. Hundreds of billions in infrastructure spend through 2030. Stargate. Partnerships with Microsoft, Oracle, Nvidia. And cost per token keeps falling roughly 10x every 12 months for the same capability level. So usage explodes because it keeps getting cheaper. The pattern is the same one we saw with computing, internet, and mobile. What the power user does today, the average person does in six years. Right now you are the power user. Act like it.

Vaibhav Sisinty

24,323 Aufrufe • vor 1 Monat

Gavin Baker on the one variable that now dominates all others in AI compute: "the more memory you put with flop for a given unit of compute, the more tokens you get out -- it's the single most important thing you could do to increase token output per unit of compute." This is him laying out why the whole AI buildout now turns on memory, why breaking a supply agreement can end your entire franchise, and what Nvidia quietly writes into every equity check it signs. "You might blow up your entire business and your franchise by breaking an LTA." "If you break your LTA and then in the next two or three years for any reason leverage shifts back to the memory guys -- you're out of business." "Essentially every time they haven't taken an equity stake in something, it's been a mistake, they've taken equity stakes in everything essentially except the memory companies." Bookmark & watch ↓
11:45

Gavin Baker on the one variable that now dominates all others in AI compute: "the more memory you put with flop for a given unit of compute, the more tokens you get out -- it's the single most important thing you could do to increase token output per unit of compute." This is him laying out why the whole AI buildout now turns on memory, why breaking a supply agreement can end your entire franchise, and what Nvidia quietly writes into every equity check it signs. "You might blow up your entire business and your franchise by breaking an LTA." "If you break your LTA and then in the next two or three years for any reason leverage shifts back to the memory guys -- you're out of business." "Essentially every time they haven't taken an equity stake in something, it's been a mistake, they've taken equity stakes in everything essentially except the memory companies." Bookmark & watch ↓

Fireside Alpha

18,521 Aufrufe • vor 1 Monat

HERMES AGENT WITHOUT THESE 3 FILES IS A CHATBOT. WITH THEM IT KNOWS WHO IT IS, WHO YOU ARE, AND WHAT IT LEARNED. SOUL.md — who the agent is. first thing in the system prompt. defines personality, voice, values, how it operates, what it can and can't do. structure yours like this: → identity (name, role, relationship to you) → values (what matters, what principles guide decisions) → voice (how it communicates, tone, style) → operations (autonomy level, ground rules) → restrictions (what it never does) → failure protocol (how to operate when things break) lives in ~/.hermes/SOUL.md. auto-seeded on first install. edit anytime. scanned for prompt injection on every load. keep it concise. SOUL.md injects into every turn. a 200-line soul burns tokens on every message. aim for 50-80 lines. one paragraph per section. MEMORY.md — what the agent remembers. persistent facts, insights, preferences. survives across sessions and restarts. capped at ~800 tokens by default: memory: memory_char_limit: 2200 the agent writes to this automatically as it learns about your work. USER.md — who you are. your profile, preferences, context. capped at ~500 tokens by default: memory: user_char_limit: 1375 injected every turn so the agent always knows who it's working for. bonus: AGENTS.md — project-specific instructions. drop one in any project folder. subdirectory AGENTS.md files load lazily during tool calls, not at startup. keeps your system prompt light. prompt stack order on every turn: SOUL.md → tool guidance → MEMORY.md + USER.md → skills index → AGENTS.md → platform hints skills come preloaded. 60+ built-in tools. the agent creates more skills from completed work. you focus on these 3 files. each profile gets its own copy: ~/.hermes/SOUL.md (default profile) ~/.hermes/profiles/researcher/SOUL.md ~/.hermes/profiles/ops/SOUL.md different agent, different soul, different memory, same machine. full breakdown of Hermes as a Personal AI OS in the article 👇
0:54

HERMES AGENT WITHOUT THESE 3 FILES IS A CHATBOT. WITH THEM IT KNOWS WHO IT IS, WHO YOU ARE, AND WHAT IT LEARNED. SOUL.md — who the agent is. first thing in the system prompt. defines personality, voice, values, how it operates, what it can and can't do. structure yours like this: → identity (name, role, relationship to you) → values (what matters, what principles guide decisions) → voice (how it communicates, tone, style) → operations (autonomy level, ground rules) → restrictions (what it never does) → failure protocol (how to operate when things break) lives in ~/.hermes/SOUL.md. auto-seeded on first install. edit anytime. scanned for prompt injection on every load. keep it concise. SOUL.md injects into every turn. a 200-line soul burns tokens on every message. aim for 50-80 lines. one paragraph per section. MEMORY.md — what the agent remembers. persistent facts, insights, preferences. survives across sessions and restarts. capped at ~800 tokens by default: memory: memory_char_limit: 2200 the agent writes to this automatically as it learns about your work. USER.md — who you are. your profile, preferences, context. capped at ~500 tokens by default: memory: user_char_limit: 1375 injected every turn so the agent always knows who it's working for. bonus: AGENTS.md — project-specific instructions. drop one in any project folder. subdirectory AGENTS.md files load lazily during tool calls, not at startup. keeps your system prompt light. prompt stack order on every turn: SOUL.md → tool guidance → MEMORY.md + USER.md → skills index → AGENTS.md → platform hints skills come preloaded. 60+ built-in tools. the agent creates more skills from completed work. you focus on these 3 files. each profile gets its own copy: ~/.hermes/SOUL.md (default profile) ~/.hermes/profiles/researcher/SOUL.md ~/.hermes/profiles/ops/SOUL.md different agent, different soul, different memory, same machine. full breakdown of Hermes as a Personal AI OS in the article 👇

YanXbt

28,924 Aufrufe • vor 2 Monaten

Peter Steinberger Quietly Started A Shift That Makes Claude 35x Cheaper To Run The way Claude Code talks to tools matters more than people think. Most use MCP, and it quietly eats tokens, it loads everything into context every time, even tools you never call. One benchmark: MCP burned 35x more tokens than a CLI on the same task. Peter Steinberger, the OpenClaw guy, got annoyed by this and started building lean CLIs himself. That kicked off a tool called Printing Press. You point Claude Code at any website, even ones with no API like ESPN or Craigslist, and it builds a small command tool for it in about 10 minutes. In one demo, a request pulled 132,000 tokens of raw data, but the tool processed it locally and only handed Claude a 2,000-token summary. The rest never touched the context window. It also comes with ~50 ready-made tools you can grab right away. To start, point Claude Code at the links from and ask it to set it up. Bookmark this.
14:45

Peter Steinberger Quietly Started A Shift That Makes Claude 35x Cheaper To Run The way Claude Code talks to tools matters more than people think. Most use MCP, and it quietly eats tokens, it loads everything into context every time, even tools you never call. One benchmark: MCP burned 35x more tokens than a CLI on the same task. Peter Steinberger, the OpenClaw guy, got annoyed by this and started building lean CLIs himself. That kicked off a tool called Printing Press. You point Claude Code at any website, even ones with no API like ESPN or Craigslist, and it builds a small command tool for it in about 10 minutes. In one demo, a request pulled 132,000 tokens of raw data, but the tool processed it locally and only handed Claude a 2,000-token summary. The rest never touched the context window. It also comes with ~50 ready-made tools you can grab right away. To start, point Claude Code at the links from and ask it to set it up. Bookmark this.

Ridark

32,351 Aufrufe • vor 2 Monaten

We asked James Wang why Cerebras can run large models so quickly. His answer: Inference speed is mostly a memory problem. Instead of constantly pulling weights from external memory, Cerebras splits the model across multiple wafers and pipelines the layers together. “Inference is all about memory bandwidth.” “You don’t want to store weights external to the wafer, because the second it’s outside, it becomes much slower.” “We put the weights on the inside.” “The team built a new software stack that basically lets us split the models by layer and store them layer by layer across multiple wafers.” “That allows us to never have to read from external memory, and that’s what makes it so fast.” “For context, Claude is about 100 tokens per second.”
1:31

We asked James Wang why Cerebras can run large models so quickly. His answer: Inference speed is mostly a memory problem. Instead of constantly pulling weights from external memory, Cerebras splits the model across multiple wafers and pipelines the layers together. “Inference is all about memory bandwidth.” “You don’t want to store weights external to the wafer, because the second it’s outside, it becomes much slower.” “We put the weights on the inside.” “The team built a new software stack that basically lets us split the models by layer and store them layer by layer across multiple wafers.” “That allows us to never have to read from external memory, and that’s what makes it so fast.” “For context, Claude is about 100 tokens per second.”

MTS

27,935 Aufrufe • vor 3 Monaten

This guy plugged a DGX Spark (the $3K Nvidia box) and a Mac Mini M4 together to run AI and what happened next surprised everyone > the Nvidia box handles the hard part - processing your prompt in milliseconds > Mac Mini M4 handles the fast part - generating the response at memory bandwidth speeds nothing else can match > together they hit 84 tokens per second on Llama - 6x faster than the Spark alone > running compute agents locally on this setup means your data never leaves your hardware > two boxes. two different architectures. one AI system that Deepseek runs at data center scale he ran it on his desk save this. the way we build local AI is about to change
18:41

This guy plugged a DGX Spark (the $3K Nvidia box) and a Mac Mini M4 together to run AI and what happened next surprised everyone > the Nvidia box handles the hard part - processing your prompt in milliseconds > Mac Mini M4 handles the fast part - generating the response at memory bandwidth speeds nothing else can match > together they hit 84 tokens per second on Llama - 6x faster than the Spark alone > running compute agents locally on this setup means your data never leaves your hardware > two boxes. two different architectures. one AI system that Deepseek runs at data center scale he ran it on his desk save this. the way we build local AI is about to change

Mr. Buzzoni

153,768 Aufrufe • vor 3 Monaten

MIT PhD student Alex Zhang reveals the scaling result where a model trained on short tasks generalizes to problems 100x longer for free: "If you're very clever about the design of your harness or how you use the language model, you can almost get scaling gains for free." "If you train a model naively, there's no tricks. It's just the same way you train a model on these RL environments. You just roll it out, and then you just get some reward." "If you train it on only short tasks, like only tasks that are 10,000 tokens long, and then you were to run it on a similar domain, but at a million tokens, or 10 million tokens, or 100,000 tokens, it generalizes really, really well. If you look at it compared to even the base transformer, you get way better generalization properties." "When the model uses an RLM (Recursive Language Model) after it's trained on these short tasks, it will see some kind of trajectory of actions that it does. Between these two problems of different lengths, the RLM learns to see them as almost the same problem." "Token for token, they're almost the same. You can describe it in code. In one code setting, maybe the for loop is a little bigger, but it's the same kind of code and it derives the constants from the data. There's no hard coding, so they literally look the same." alex zhang
2:41

MIT PhD student Alex Zhang reveals the scaling result where a model trained on short tasks generalizes to problems 100x longer for free: "If you're very clever about the design of your harness or how you use the language model, you can almost get scaling gains for free." "If you train a model naively, there's no tricks. It's just the same way you train a model on these RL environments. You just roll it out, and then you just get some reward." "If you train it on only short tasks, like only tasks that are 10,000 tokens long, and then you were to run it on a similar domain, but at a million tokens, or 10 million tokens, or 100,000 tokens, it generalizes really, really well. If you look at it compared to even the base transformer, you get way better generalization properties." "When the model uses an RLM (Recursive Language Model) after it's trained on these short tasks, it will see some kind of trajectory of actions that it does. Between these two problems of different lengths, the RLM learns to see them as almost the same problem." "Token for token, they're almost the same. You can describe it in code. In one code setting, maybe the for loop is a little bigger, but it's the same kind of code and it derives the constants from the data. There's no hard coding, so they literally look the same." alex zhang

MTS

100,519 Aufrufe • vor 1 Monat

.Dwarkesh Patel: By 2030, it will be less expensive to monitor every single nook and cranny in America than it is to remodel the White House. “Mass surveillance is, at least in certain forms, already legal. It has just been impractical to enforce so far. Under current law, you have no Fourth Amendment protection against any data you share with a third party. That includes your bank, your ISP, your phone carrier, and your email provider. The government reserves the right to purchase and read this data in bulk without a warrant. What’s been missing is the ability to actually do anything with all of this data — no agency has the manpower to monitor every single camera, read every single message, and cross-reference every single transaction. However, that bottleneck goes away with AI. There are 100 million CCTV cameras in America. You can get pretty good open source multimodal models for 10 cents per million input tokens. So if you process a frame every ten seconds, and each frame is 1,000 tokens, then for 30 billion dollars, you can process every single camera in America. And remember that a given level of AI ability gets 10x cheaper every single year - so a year from now it’ll cost 3 billion, and then a year after 300 million, and by 2030, it’ll be less expensive to monitor every single nook and cranny in this country than it is to remodel the White House. Once the technical capacity for mass surveillance and political suppression exists, the only thing standing between us and an authoritarian state is the political expectation that this is not something we do here.”
1:30

.Dwarkesh Patel: By 2030, it will be less expensive to monitor every single nook and cranny in America than it is to remodel the White House. “Mass surveillance is, at least in certain forms, already legal. It has just been impractical to enforce so far. Under current law, you have no Fourth Amendment protection against any data you share with a third party. That includes your bank, your ISP, your phone carrier, and your email provider. The government reserves the right to purchase and read this data in bulk without a warrant. What’s been missing is the ability to actually do anything with all of this data — no agency has the manpower to monitor every single camera, read every single message, and cross-reference every single transaction. However, that bottleneck goes away with AI. There are 100 million CCTV cameras in America. You can get pretty good open source multimodal models for 10 cents per million input tokens. So if you process a frame every ten seconds, and each frame is 1,000 tokens, then for 30 billion dollars, you can process every single camera in America. And remember that a given level of AI ability gets 10x cheaper every single year - so a year from now it’ll cost 3 billion, and then a year after 300 million, and by 2030, it’ll be less expensive to monitor every single nook and cranny in this country than it is to remodel the White House. Once the technical capacity for mass surveillance and political suppression exists, the only thing standing between us and an authoritarian state is the political expectation that this is not something we do here.”

Arjun Khemani

145,706 Aufrufe • vor 5 Monaten

A screen recording is the best context you can give an AI agent. It's also the worst. A few seconds of video is thousands of images. Feed that to an agent and you instantly blow up its memory. So I built a tool that fixes it. How it works: Hit record and just do your thing. Click around, type, and talk out loud about what you're doing. While you record, it quietly captures the stuff that actually matters: your voice transcription, every click and the exact thing you clicked, what you typed, and the shortcuts you press. When you stop, it doesn't hand the agent the whole video. It writes a short, readable summary: a timeline of what you said and did. The agent reads that tiny summary first. Then, only when it needs to actually see a moment, it pulls that one exact frame from the video, like flipping to a single page instead of reading the whole book. The result: the agent understands everything you did! Full context. Less tokens.
1:54

A screen recording is the best context you can give an AI agent. It's also the worst. A few seconds of video is thousands of images. Feed that to an agent and you instantly blow up its memory. So I built a tool that fixes it. How it works: Hit record and just do your thing. Click around, type, and talk out loud about what you're doing. While you record, it quietly captures the stuff that actually matters: your voice transcription, every click and the exact thing you clicked, what you typed, and the shortcuts you press. When you stop, it doesn't hand the agent the whole video. It writes a short, readable summary: a timeline of what you said and did. The agent reads that tiny summary first. Then, only when it needs to actually see a moment, it pulls that one exact frame from the video, like flipping to a single page instead of reading the whole book. The result: the agent understands everything you did! Full context. Less tokens.

Milind S

12,409 Aufrufe • vor 2 Monaten

A million-token window solved the wrong problem, and Kimi K3 shipping one made that harder to see. 2.8T parameters, weights on your disk, an entire codebase in one prompt. Feels like memory. Isn't. Memory is what survives the session ending. A window is what survives the next token. Self-hosting sharpens it. Now you pay in GPU seconds, not API credits. Every re-read of the same history burns the same compute. Owning the weights didn't make the past free. Kimi Delta Attention cuts what long context costs. Cheaper repetition is still repetition. And a bigger window buys the one thing you don't want: room to keep everything. Nothing in a transcript is dated, ranked, or retired. Old and current load at identical weight. The fix runs beside the model, not inside it. Capture rejects, consolidation merges, decay forgets on schedule. Then a million tokens becomes headroom instead of a bill. You fill 5,000 and leave the rest empty. Look at what your longest thread reloads every run. Then ask how much of it earned the slot.
0:42

A million-token window solved the wrong problem, and Kimi K3 shipping one made that harder to see. 2.8T parameters, weights on your disk, an entire codebase in one prompt. Feels like memory. Isn't. Memory is what survives the session ending. A window is what survives the next token. Self-hosting sharpens it. Now you pay in GPU seconds, not API credits. Every re-read of the same history burns the same compute. Owning the weights didn't make the past free. Kimi Delta Attention cuts what long context costs. Cheaper repetition is still repetition. And a bigger window buys the one thing you don't want: room to keep everything. Nothing in a transcript is dated, ranked, or retired. Old and current load at identical weight. The fix runs beside the model, not inside it. Capture rejects, consolidation merges, decay forgets on schedule. Then a million tokens becomes headroom instead of a bill. You fill 5,000 and leave the rest empty. Look at what your longest thread reloads every run. Then ask how much of it earned the slot.

wast3

17,620 Aufrufe • vor 19 Tagen