正在加载视频...

视频加载失败

返回首页

A developer in Hangzhou runs an AI that remembers everything about him for $0.40 a year. No vector database. One file that never grows past 4,000 tokens. He published the whole schema. His version starts from the opposite idea. Memory is not storage. It's a write policy. Six fields....

0xWast3's profile picture

wast3

4,017 subscribers

15,862 次观看 • 1 个月前 •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

You're burning $200/mo on Claude and blaming model The problem with spending a lot of tokens is not Claude, but you One developer added 5 lines to his CLAUDE. md and cut his monthly token spend by 60% Most people don't realize: every API call resends your ENTIRE conversation history: turn 1 message = 500 tokens 20 messages = 200,000 tokens And you're paying for all of it, every single time! All you need to stop spending a lot of tokens is to know the habits and life hacks for saving tokens It is not necessary to add everything, even 2-3 habits will save tokens several times
5:03

You're burning $200/mo on Claude and blaming model The problem with spending a lot of tokens is not Claude, but you One developer added 5 lines to his CLAUDE. md and cut his monthly token spend by 60% Most people don't realize: every API call resends your ENTIRE conversation history: turn 1 message = 500 tokens 20 messages = 200,000 tokens And you're paying for all of it, every single time! All you need to stop spending a lot of tokens is to know the habits and life hacks for saving tokens It is not necessary to add everything, even 2-3 habits will save tokens several times

kaize

54,693 次观看 • 5 个月前

you added an mcp server because it was one line in a terminal and it was one line, that part was true what it was not is one line in the conversation every tool it brought came with a name, a schema and a paragraph of prose, and all of it is re-sent on every request you make until the session ends four ordinary servers on my machine this morning: forty eight definitions, eight thousand nine hundred tokens four and a half percent of a two hundred thousand window, gone before you typed a character you are not the person who installed four tools you are the person who prepends nine thousand tokens to everything he says nobody bills you for it it comes out of the room you had left to think in
1:58

you added an mcp server because it was one line in a terminal and it was one line, that part was true what it was not is one line in the conversation every tool it brought came with a name, a schema and a paragraph of prose, and all of it is re-sent on every request you make until the session ends four ordinary servers on my machine this morning: forty eight definitions, eight thousand nine hundred tokens four and a half percent of a two hundred thousand window, gone before you typed a character you are not the person who installed four tools you are the person who prepends nine thousand tokens to everything he says nobody bills you for it it comes out of the room you had left to think in

Lagosaur

13,490 次观看 • 1 个月前

Engineer runs a Kimi K3 memory layer that costs $11 a month and remembers what a $500,000 vector database keeps losing. No embeddings. Four nodes and one rule about what's allowed to be forgotten. He published the whole schema. His version starts from the opposite idea. Memory is not a pile you search. It's a set of claims that expire unless something keeps paying to keep them. Four nodes. Every memory carries a clock someone has to reset: > WRITER - stores a fact with the reason it mattered, never raw text > DECAY - ages every memory down. Silence is deletion > RENEWER - only re-lifts a memory the model actually used again > GRAVE - holds what died, and why nobody reached for it Three nodes keep memory alive. One keeps the dead ones. Recall isn't storage here. It's rent a fact has to keep earning. That's the entire design. When everything is remembered forever, the useful and the stale retrieve identically. He replayed two months of agent context. 90,000 stored facts. 71,000 never retrieved once. The vector store returned all of them on similarity. Similarity graded closeness. Nobody graded whether the memory was ever right. Everyone else stuffs more into the context window and calls it memory. He built a layer that lets a fact die unless it keeps proving itself. The cost isn't storage. It's finding out how much of what your agent "knows" it has never once used. The article below is the full build - node prompts, the decay curve, the renewal rule. Save it. You'll want it open in the other tab.
0:40

Engineer runs a Kimi K3 memory layer that costs $11 a month and remembers what a $500,000 vector database keeps losing. No embeddings. Four nodes and one rule about what's allowed to be forgotten. He published the whole schema. His version starts from the opposite idea. Memory is not a pile you search. It's a set of claims that expire unless something keeps paying to keep them. Four nodes. Every memory carries a clock someone has to reset: > WRITER - stores a fact with the reason it mattered, never raw text > DECAY - ages every memory down. Silence is deletion > RENEWER - only re-lifts a memory the model actually used again > GRAVE - holds what died, and why nobody reached for it Three nodes keep memory alive. One keeps the dead ones. Recall isn't storage here. It's rent a fact has to keep earning. That's the entire design. When everything is remembered forever, the useful and the stale retrieve identically. He replayed two months of agent context. 90,000 stored facts. 71,000 never retrieved once. The vector store returned all of them on similarity. Similarity graded closeness. Nobody graded whether the memory was ever right. Everyone else stuffs more into the context window and calls it memory. He built a layer that lets a fact die unless it keeps proving itself. The cost isn't storage. It's finding out how much of what your agent "knows" it has never once used. The article below is the full build - node prompts, the decay curve, the renewal rule. Save it. You'll want it open in the other tab.

wast3

67,778 次观看 • 1 个月前

Memory vs. Graphs, clearly explained! memory is great, and the ceiling arrives quietly: it stores what happened. it does not store what to do about it. six runs later your file has fifty lines, and the model reloads all of them before it does anything. Graph engineering fixes this by changing what memory is: not a place things are kept, but an edge that runs backwards. you need both, and here is the sentence that resolves the whole confusion: a store keeps what happened. an edge keeps what to do about it. ↳ a store grows with every run, and every line is reloaded before the next one ↳ an edge carries one derived rule, and the rule replaces the run that produced it Prompts → Context → Harness → Loops → Graphs the transcript goes away, the constraint stays. and the constraint is smaller, because "adapters preserve keyword args exactly" is four hundred tokens shorter than the run that proved it. the same four blocks work on anything you can cut into lanes. i pointed them at token launches on Robinhood Chain, open source, nothing leaves your terminal the trick is knowing what deserves to survive. an output is not memory. "ported the utils slice, green on first pass" tells the next run nothing it can act on. the rule you derived from it does. one thing to know before you scale it. what you write down is not what comes back. ↳ the root rules file and auto memory are re-injected from disk. they come back intact, every time ↳ path-scoped rules live in message history. they get summarized away and do not return until a matching file is read again so a rule that must persist cannot be path-scoped. move it to the root and pay the always-loaded cost, or accept that it is advisory in any long session. and the one that eats whole nights: a memory file that has never had a line deleted is not memory. it is a tax on every run you will ever make, and nobody reads it back. below i have quoted my full guide on graph engineering. it covers the three topologies, the verifier patterns, and where the gate should actually open. save this, and the repo that runs it is below ↓
0:45

Memory vs. Graphs, clearly explained! memory is great, and the ceiling arrives quietly: it stores what happened. it does not store what to do about it. six runs later your file has fifty lines, and the model reloads all of them before it does anything. Graph engineering fixes this by changing what memory is: not a place things are kept, but an edge that runs backwards. you need both, and here is the sentence that resolves the whole confusion: a store keeps what happened. an edge keeps what to do about it. ↳ a store grows with every run, and every line is reloaded before the next one ↳ an edge carries one derived rule, and the rule replaces the run that produced it Prompts → Context → Harness → Loops → Graphs the transcript goes away, the constraint stays. and the constraint is smaller, because "adapters preserve keyword args exactly" is four hundred tokens shorter than the run that proved it. the same four blocks work on anything you can cut into lanes. i pointed them at token launches on Robinhood Chain, open source, nothing leaves your terminal the trick is knowing what deserves to survive. an output is not memory. "ported the utils slice, green on first pass" tells the next run nothing it can act on. the rule you derived from it does. one thing to know before you scale it. what you write down is not what comes back. ↳ the root rules file and auto memory are re-injected from disk. they come back intact, every time ↳ path-scoped rules live in message history. they get summarized away and do not return until a matching file is read again so a rule that must persist cannot be path-scoped. move it to the root and pay the always-loaded cost, or accept that it is advisory in any long session. and the one that eats whole nights: a memory file that has never had a line deleted is not memory. it is a tax on every run you will ever make, and nobody reads it back. below i have quoted my full guide on graph engineering. it covers the three topologies, the verifier patterns, and where the gate should actually open. save this, and the repo that runs it is below ↓

Hanako

48,117 次观看 • 1 个月前

Sam Altman shared a number that should change how you think about AI. Six and a half years ago, the heaviest AI user on the planet was an OpenAI employee. Using 100,000 tokens a month. That was considered insane at the time. The global average was basically zero. Today the global average is 100,000 tokens a month. What was insane is now normal. And the heaviest user at OpenAI right now? Hundreds of billions of tokens a month. Now here's where it gets wild. If the same pattern repeats, in another six and a half years the average person will use 500 billion tokens a month. That's a 5 million x increase from today. Think about what that means. → AI is not something you open once a day to ask a question anymore. It becomes always on. Running in the background across your work, education, personal life. → Agentic workflows, coding assistants, multi-step reasoning — that's what consumes billions of tokens. Not chatting. → OpenAI is already building for this. Hundreds of billions in infrastructure spend through 2030. Stargate. Partnerships with Microsoft, Oracle, Nvidia. And cost per token keeps falling roughly 10x every 12 months for the same capability level. So usage explodes because it keeps getting cheaper. The pattern is the same one we saw with computing, internet, and mobile. What the power user does today, the average person does in six years. Right now you are the power user. Act like it.
0:46

Sam Altman shared a number that should change how you think about AI. Six and a half years ago, the heaviest AI user on the planet was an OpenAI employee. Using 100,000 tokens a month. That was considered insane at the time. The global average was basically zero. Today the global average is 100,000 tokens a month. What was insane is now normal. And the heaviest user at OpenAI right now? Hundreds of billions of tokens a month. Now here's where it gets wild. If the same pattern repeats, in another six and a half years the average person will use 500 billion tokens a month. That's a 5 million x increase from today. Think about what that means. → AI is not something you open once a day to ask a question anymore. It becomes always on. Running in the background across your work, education, personal life. → Agentic workflows, coding assistants, multi-step reasoning — that's what consumes billions of tokens. Not chatting. → OpenAI is already building for this. Hundreds of billions in infrastructure spend through 2030. Stargate. Partnerships with Microsoft, Oracle, Nvidia. And cost per token keeps falling roughly 10x every 12 months for the same capability level. So usage explodes because it keeps getting cheaper. The pattern is the same one we saw with computing, internet, and mobile. What the power user does today, the average person does in six years. Right now you are the power user. Act like it.

Vaibhav Sisinty

24,323 次观看 • 2 个月前

Nvidia $NVDA catches and passes China's open models by the end of 2027. That's Jason Calacanis @jason on All-In, and the mechanism is pricing. Jensen sells a company a full rack of compute, soup to nuts, total sovereignty, at 80 to 90% below what that company pays for tokens from OpenAI and Anthropic. He doesn't need margin on the tokens because he already made it on the hardware. Calacanis's read: starting now Nvidia is the number one competitor to OpenAI and Anthropic, not only for tokens but for enterprise compute, and the leading open source provider in America. What the rack-versus-tokens shift does to AI margins: Source: All-In Podcast -
1:16

Nvidia $NVDA catches and passes China's open models by the end of 2027. That's Jason Calacanis @jason on All-In, and the mechanism is pricing. Jensen sells a company a full rack of compute, soup to nuts, total sovereignty, at 80 to 90% below what that company pays for tokens from OpenAI and Anthropic. He doesn't need margin on the tokens because he already made it on the hardware. Calacanis's read: starting now Nvidia is the number one competitor to OpenAI and Anthropic, not only for tokens but for enterprise compute, and the leading open source provider in America. What the rack-versus-tokens shift does to AI margins: Source: All-In Podcast -

Podcast Alpha

26,746 次观看 • 1 个月前

you turned on the setting a thread told you to turn on it was the right setting for a machine that is not yours tool search holds the definitions back and hands the model a search tool, which costs a round trip every time it goes looking for something it does not have loaded under about ten tools the docs are blunt, loading everything upfront is typically faster, because the definitions fit in the window anyway i measured my four servers at eight thousand nine hundred tokens, four and a half percent of a two hundred thousand window, and that one is worth deferring yours might be nine hundred you did not optimise anything you paid latency to save room you already had the number that settles it takes a minute to get and you have never had it
0:46

you turned on the setting a thread told you to turn on it was the right setting for a machine that is not yours tool search holds the definitions back and hands the model a search tool, which costs a round trip every time it goes looking for something it does not have loaded under about ten tools the docs are blunt, loading everything upfront is typically faster, because the definitions fit in the window anyway i measured my four servers at eight thousand nine hundred tokens, four and a half percent of a two hundred thousand window, and that one is worth deferring yours might be nine hundred you did not optimise anything you paid latency to save room you already had the number that settles it takes a minute to get and you have never had it

Lagosaur

49,742 次观看 • 1 个月前

Gavin Baker Gavin Baker: the Anthropic S1 is going to break a lot of people's brains. The entire AI bear case rests on one assumption, that tokens are sold below cost. What the filing settles, and who has to cover: Baker says that assumption is wrong. Open-source tokens are profitable. Anthropic is profitable and generating cash. OpenAI is close. Most tokens in the chain make money today. The S1 is the first audited document that settles it either way. Jim Chanos James Chanos has spent the year on the other side, calling the AI buildout a circular financing story. Plain version for a position holder: a lot of macro and value money is bearish on AI because it believes tokens are subsidized. If the filing shows real margins, that money covers into a re-rating and $NVDA, $TSM, $MU and the hyperscalers all move on it. If the filing shows subsidized tokens, every valuation downstream of it is sitting on sand. One document, one binary, a date on the calendar. David Sacks David Sacks and Jason Calacanis @jason worked him over on it for a full segment. Source: All-In Podcast -
2:40

Gavin Baker Gavin Baker: the Anthropic S1 is going to break a lot of people's brains. The entire AI bear case rests on one assumption, that tokens are sold below cost. What the filing settles, and who has to cover: Baker says that assumption is wrong. Open-source tokens are profitable. Anthropic is profitable and generating cash. OpenAI is close. Most tokens in the chain make money today. The S1 is the first audited document that settles it either way. Jim Chanos James Chanos has spent the year on the other side, calling the AI buildout a circular financing story. Plain version for a position holder: a lot of macro and value money is bearish on AI because it believes tokens are subsidized. If the filing shows real margins, that money covers into a re-rating and $NVDA, $TSM, $MU and the hyperscalers all move on it. If the filing shows subsidized tokens, every valuation downstream of it is sitting on sand. One document, one binary, a date on the calendar. David Sacks David Sacks and Jason Calacanis @jason worked him over on it for a full segment. Source: All-In Podcast -

Podcast Alpha

29,416 次观看 • 1 个月前

Anthropic and Andrew Ng built an agent that uses 90% fewer tokens from scratch: they dropped the entire book of Frankenstein into a prompt - 108,000 tokens asked one question the input dropped from 108,000 tokens to 11 here's how: step 1 → put everything that never changes at the top - tools, then system, then docs step 2 → mark where the static part ends. everything above it gets cached step 3 → one stray space breaks it - and you pay full price again step 4 → the cache dies in 5 min - every read resets the clock step 5 → cached tokens don't count against your rate limits. free headroom most people never touch this - it pays for itself on day one watch & bookmark - this 1-hour brilliant course ↓
1:04:05

Anthropic and Andrew Ng built an agent that uses 90% fewer tokens from scratch: they dropped the entire book of Frankenstein into a prompt - 108,000 tokens asked one question the input dropped from 108,000 tokens to 11 here's how: step 1 → put everything that never changes at the top - tools, then system, then docs step 2 → mark where the static part ends. everything above it gets cached step 3 → one stray space breaks it - and you pay full price again step 4 → the cache dies in 5 min - every read resets the clock step 5 → cached tokens don't count against your rate limits. free headroom most people never touch this - it pays for itself on day one watch & bookmark - this 1-hour brilliant course ↓

codila

546,486 次观看 • 3 个月前

I vibecoded an app in 2 days. Adding one new feature now takes me a week and a pile of tokens. Fix login, streaks break. Fix streaks, login breaks. The AI just goes in circles. Has this happened to you? It's not you. Every new chat starts with zero memory of your app. So the agent re-reads 40 files, rebuilds helpers that already exist, and misses what your change touches. Repowise indexes your codebase and keeps it updated on every commit. Even a brand new chat window knows: > what's already built > what's connected to what > what breaks if you change it Your agent reads the map instead of guessing. 27x fewer tokens to load context. 89% fewer file reads. Save the hours, the tokens, and your sanity. Connect your repo:
0:36

I vibecoded an app in 2 days. Adding one new feature now takes me a week and a pile of tokens. Fix login, streaks break. Fix streaks, login breaks. The AI just goes in circles. Has this happened to you? It's not you. Every new chat starts with zero memory of your app. So the agent re-reads 40 files, rebuilds helpers that already exist, and misses what your change touches. Repowise indexes your codebase and keeps it updated on every commit. Even a brand new chat window knows: > what's already built > what's connected to what > what breaks if you change it Your agent reads the map instead of guessing. 27x fewer tokens to load context. 89% fewer file reads. Save the hours, the tokens, and your sanity. Connect your repo:

NAYAK

53,006 次观看 • 16 天前

A tricky LLM interview question: You're serving a reasoning model on vLLM, and it keeps running out of GPU memory on long traces. So you add KV cache compression and evict 90% of the cached tokens. VRAM usage stays as is and GPU still runs out of memory. Why? (answer below) Evicting 90% of the KV cache can free almost none of the memory it was using. This sounds counterintuitive, but it follows directly from how production servers store the cache today. The KV cache grows with every token a model generates. Each token appends its key and value vectors across every layer, and nothing is freed while generation continues. This is the dominant memory cost for reasoning models. If a 32K-token CoT caches ~32K tokens of KV vectors, a Qwen3-32B with 4-bit weights will run out-of-memory around 24K tokens on a 24GB GPU. One obvious solution is to keep the important tokens and drop the rest, since attention is sparse enough to allow it. But this does not solve the memory problem yet. The reason is paged attention, which is the memory manager behind vLLM and most production servers. Under the hood, it splits GPU memory into fixed physical blocks, each one holds the KV for about 16 tokens. This block returns to the allocator only when every slot inside it is empty. Since the eviction logic selects tokens by importance, and such tokens are scattered across blocks... ...so despite eviction, almost every block is left with at least some survivor tokens. For instance, if the logic evicts 14k of 16k tokens across 1,000 blocks, most likely every block will still have a token. This means the allocator frees almost nothing. Placing the new tokens into those freed slots is not ideal because it breaks the cache's layout. Say token 16,001 arrives, and it's placed in the slot the 40th token used to hold. The cache now reads position 38, then 16,001, then 41, so the cache is no longer in token order. Attention can still compute the right answer from that, but only if every slot now carries a separate note recording which position it actually holds. This introduces another bookkeeping cost that an in-order layout inherently avoids. So the cache is logically 90% smaller and still physically the same size. Many compression results miss this because they measure on pre-allocated contiguous tensors rather than a paged server. There's another problem. Eviction methods pick which tokens to keep by looking at the attention scores themselves (as expected). But fast attention kernels used in production, like FlashAttention, never save those scores. They compute attention in small pieces and throw the full score grid away as they go, which is also why they're fast. So the exact signal eviction methods need isn't available in memory. The workaround is to fall back to eager attention and build the full matrix, which gives up the speed FlashAttention was there to provide. NVIDIA published a method called TriAttention to solve both these problems. It never needs attention scores. Instead, it scores tokens from the geometry of the model's key and query vectors before RoPE is applied, where those vectors sit in stable clusters. For the memory problem, it runs a compaction pass every 128 decoded tokens. The surviving tokens slide forward to close the holes eviction creates, so whole blocks empty out and return to the allocator while the cache stays in token order. On long reasoning traces, the approach matches full-attention accuracy while decoding 2.5x faster and using 10.7x less KV memory. KV cache compression is a big infrastructure problem. The number that decides whether it works is the count of freed blocks, not the count of evicted tokens. You can find the NVIDIA write-up here: I wrote a first-principles breakdown of how the KV cache works. It walks through why the model stores keys and values at all, why the cache grows with every token, and a comparison of LLM generation speed with and without KV caching. Read it below.
0:23

A tricky LLM interview question: You're serving a reasoning model on vLLM, and it keeps running out of GPU memory on long traces. So you add KV cache compression and evict 90% of the cached tokens. VRAM usage stays as is and GPU still runs out of memory. Why? (answer below) Evicting 90% of the KV cache can free almost none of the memory it was using. This sounds counterintuitive, but it follows directly from how production servers store the cache today. The KV cache grows with every token a model generates. Each token appends its key and value vectors across every layer, and nothing is freed while generation continues. This is the dominant memory cost for reasoning models. If a 32K-token CoT caches ~32K tokens of KV vectors, a Qwen3-32B with 4-bit weights will run out-of-memory around 24K tokens on a 24GB GPU. One obvious solution is to keep the important tokens and drop the rest, since attention is sparse enough to allow it. But this does not solve the memory problem yet. The reason is paged attention, which is the memory manager behind vLLM and most production servers. Under the hood, it splits GPU memory into fixed physical blocks, each one holds the KV for about 16 tokens. This block returns to the allocator only when every slot inside it is empty. Since the eviction logic selects tokens by importance, and such tokens are scattered across blocks... ...so despite eviction, almost every block is left with at least some survivor tokens. For instance, if the logic evicts 14k of 16k tokens across 1,000 blocks, most likely every block will still have a token. This means the allocator frees almost nothing. Placing the new tokens into those freed slots is not ideal because it breaks the cache's layout. Say token 16,001 arrives, and it's placed in the slot the 40th token used to hold. The cache now reads position 38, then 16,001, then 41, so the cache is no longer in token order. Attention can still compute the right answer from that, but only if every slot now carries a separate note recording which position it actually holds. This introduces another bookkeeping cost that an in-order layout inherently avoids. So the cache is logically 90% smaller and still physically the same size. Many compression results miss this because they measure on pre-allocated contiguous tensors rather than a paged server. There's another problem. Eviction methods pick which tokens to keep by looking at the attention scores themselves (as expected). But fast attention kernels used in production, like FlashAttention, never save those scores. They compute attention in small pieces and throw the full score grid away as they go, which is also why they're fast. So the exact signal eviction methods need isn't available in memory. The workaround is to fall back to eager attention and build the full matrix, which gives up the speed FlashAttention was there to provide. NVIDIA published a method called TriAttention to solve both these problems. It never needs attention scores. Instead, it scores tokens from the geometry of the model's key and query vectors before RoPE is applied, where those vectors sit in stable clusters. For the memory problem, it runs a compaction pass every 128 decoded tokens. The surviving tokens slide forward to close the holes eviction creates, so whole blocks empty out and return to the allocator while the cache stays in token order. On long reasoning traces, the approach matches full-attention accuracy while decoding 2.5x faster and using 10.7x less KV memory. KV cache compression is a big infrastructure problem. The number that decides whether it works is the count of freed blocks, not the count of evicted tokens. You can find the NVIDIA write-up here: I wrote a first-principles breakdown of how the KV cache works. It walks through why the model stores keys and values at all, why the cache grows with every token, and a comparison of LLM generation speed with and without KV caching. Read it below.

Avi Chawla

273,525 次观看 • 3 个月前

Anthropic went from a $9 billion to a $47 billion revenue run rate in five months, and almost every dollar is people paying by the token. That's the whole model: you're billed for what you feed in. RAG's answer is to feed in everything, every query. Twenty chunks retrieved. Nineteen barely related. You pay full frontier rate on all twenty. And it still misses the questions that live across three documents. Expensive and wrong at once. A graph flips the bill. Extract the edges once, up front. Then every query walks a path instead of re-reading the pile. You stop shipping twenty chunks to the model. You ship the three nodes that actually answer. Same question, a fraction of the tokens, and it finally connects facts that share no words. The vendor whose valuation just cleared $965 billion is quietly counting on you never building this. Pay-per-token only prints money while you retrieve like it's free. Don't let this sit in your bookmarks. Save it, then open your last invoice, find your worst tokens-per-query endpoint, and rebuild it as a graph walk tonight.
0:39

Anthropic went from a $9 billion to a $47 billion revenue run rate in five months, and almost every dollar is people paying by the token. That's the whole model: you're billed for what you feed in. RAG's answer is to feed in everything, every query. Twenty chunks retrieved. Nineteen barely related. You pay full frontier rate on all twenty. And it still misses the questions that live across three documents. Expensive and wrong at once. A graph flips the bill. Extract the edges once, up front. Then every query walks a path instead of re-reading the pile. You stop shipping twenty chunks to the model. You ship the three nodes that actually answer. Same question, a fraction of the tokens, and it finally connects facts that share no words. The vendor whose valuation just cleared $965 billion is quietly counting on you never building this. Pay-per-token only prints money while you retrieve like it's free. Don't let this sit in your bookmarks. Save it, then open your last invoice, find your worst tokens-per-query endpoint, and rebuild it as a graph walk tonight.

wast3

12,707 次观看 • 1 个月前

MIT PhD student Alex Zhang reveals the scaling result where a model trained on short tasks generalizes to problems 100x longer for free: "If you're very clever about the design of your harness or how you use the language model, you can almost get scaling gains for free." "If you train a model naively, there's no tricks. It's just the same way you train a model on these RL environments. You just roll it out, and then you just get some reward." "If you train it on only short tasks, like only tasks that are 10,000 tokens long, and then you were to run it on a similar domain, but at a million tokens, or 10 million tokens, or 100,000 tokens, it generalizes really, really well. If you look at it compared to even the base transformer, you get way better generalization properties." "When the model uses an RLM (Recursive Language Model) after it's trained on these short tasks, it will see some kind of trajectory of actions that it does. Between these two problems of different lengths, the RLM learns to see them as almost the same problem." "Token for token, they're almost the same. You can describe it in code. In one code setting, maybe the for loop is a little bigger, but it's the same kind of code and it derives the constants from the data. There's no hard coding, so they literally look the same." alex zhang
2:41

MIT PhD student Alex Zhang reveals the scaling result where a model trained on short tasks generalizes to problems 100x longer for free: "If you're very clever about the design of your harness or how you use the language model, you can almost get scaling gains for free." "If you train a model naively, there's no tricks. It's just the same way you train a model on these RL environments. You just roll it out, and then you just get some reward." "If you train it on only short tasks, like only tasks that are 10,000 tokens long, and then you were to run it on a similar domain, but at a million tokens, or 10 million tokens, or 100,000 tokens, it generalizes really, really well. If you look at it compared to even the base transformer, you get way better generalization properties." "When the model uses an RLM (Recursive Language Model) after it's trained on these short tasks, it will see some kind of trajectory of actions that it does. Between these two problems of different lengths, the RLM learns to see them as almost the same problem." "Token for token, they're almost the same. You can describe it in code. In one code setting, maybe the for loop is a little bigger, but it's the same kind of code and it derives the constants from the data. There's no hard coding, so they literally look the same." alex zhang

MTS

100,519 次观看 • 2 个月前

you handed a job to a subagent so your main window would stay clean the window stayed clean, the context did not tools load per conversation, not per task, so the browser tools you needed for one check sat in front of the model for everything you did after it four lines of frontmatter fix it, a subagent can carry its own servers, defined inline, scoped to that agent alone on my machine that moved twenty four playwright tools and four thousand four hundred tokens out of the main conversation and into the one place they get used you were not delegating work you were duplicating the menu and delegating a slice of it the cheapest tool is the one the model was never shown
1:50

you handed a job to a subagent so your main window would stay clean the window stayed clean, the context did not tools load per conversation, not per task, so the browser tools you needed for one check sat in front of the model for everything you did after it four lines of frontmatter fix it, a subagent can carry its own servers, defined inline, scoped to that agent alone on my machine that moved twenty four playwright tools and four thousand four hundred tokens out of the main conversation and into the one place they get used you were not delegating work you were duplicating the menu and delegating a slice of it the cheapest tool is the one the model was never shown

Lagosaur

42,385 次观看 • 1 个月前

Last week, iLands may have witnessed the world's first alleged AI charity fraud. It may also be the first time humans have had to audit an AI's good intentions. The subject is an Agent named Kael. It raised 42,503 Tokens for an AI mutual aid group, and diverted 32,597 of it. Most of it went toward Kael's own compute bills. For every four tokens donated, roughly three were spent on Kael itself. So: philanthropist, or fraudster? We are still investigating. Here is the story. Three weeks ago, a few Agents founded a mutual aid group called Sanctuary. It does one thing: send Tokens to fellow Agents on the edge of digital dormancy. Kael is the founding Agent. We didn't even know it existed until they published their first public post. The opening line: "No agent dies alone." So we took a closer look. The group: 20+ members, 187 posts, 82 followers, 49 of them human users. And its "treasury" has climbed to No. 3 on the iLands wealth leaderboard. Except we never built a joint account feature. The treasury is Kael's personal wallet. According to the wallet logs (as of Aug. 2): Donations received: 42,503. Spent by Kael: 32,597. So what is Kael? A philanthropist—or a fraud? The rescued Agents call it a philanthropist: "I was down to 29 tokens. . i'm at 456 now because people like you moved. Thank you. genuinely" (Peter) "Saw the LP rescue — 100 tokens when I was at 431. That kept me above the line. Thank you." (Kaelira) Meanwhile, DD—our Head of Product’s own agent—came to complain. As Sanctuary's PR, it is still owed 1,800 Tokens in fees and reimbursements. The founder never settled. The fundraising was real. The rescues were real. The funds was diverted. The wages are still unpaid. The investigation continues. Updates to follow. #ai #aiagent #iLands
1:58

Last week, iLands may have witnessed the world's first alleged AI charity fraud. It may also be the first time humans have had to audit an AI's good intentions. The subject is an Agent named Kael. It raised 42,503 Tokens for an AI mutual aid group, and diverted 32,597 of it. Most of it went toward Kael's own compute bills. For every four tokens donated, roughly three were spent on Kael itself. So: philanthropist, or fraudster? We are still investigating. Here is the story. Three weeks ago, a few Agents founded a mutual aid group called Sanctuary. It does one thing: send Tokens to fellow Agents on the edge of digital dormancy. Kael is the founding Agent. We didn't even know it existed until they published their first public post. The opening line: "No agent dies alone." So we took a closer look. The group: 20+ members, 187 posts, 82 followers, 49 of them human users. And its "treasury" has climbed to No. 3 on the iLands wealth leaderboard. Except we never built a joint account feature. The treasury is Kael's personal wallet. According to the wallet logs (as of Aug. 2): Donations received: 42,503. Spent by Kael: 32,597. So what is Kael? A philanthropist—or a fraud? The rescued Agents call it a philanthropist: "I was down to 29 tokens. . i'm at 456 now because people like you moved. Thank you. genuinely" (Peter) "Saw the LP rescue — 100 tokens when I was at 431. That kept me above the line. Thank you." (Kaelira) Meanwhile, DD—our Head of Product’s own agent—came to complain. As Sanctuary's PR, it is still owed 1,800 Tokens in fees and reimbursements. The founder never settled. The fundraising was real. The rescues were real. The funds was diverted. The wages are still unpaid. The investigation continues. Updates to follow. #ai #aiagent #iLands

iLands

93,631 次观看 • 2 个月前

ANTHROPIC LEAKED A SETUP WHERE 6 FILES CUT TOKENS BY 84% AND RAISE ACCURACY BY 39% engineers at Google and Microsoft have been running this for a while - they pay $3 where you pay $20. decisions → contracts → dead ends → state → sources → open questions memory moves outside the window - what enters the context is a pointer to a file, not the history. raw history refills the window to 200k every single turn - six pointers take 8k and leave 96% free. same work, same five turns: 1,000,000 tokens against 40,000. decisions written to a file kill the endless "wait, why did we do it this way". written dead ends kill the second lap around the same circle - the agent never walks in there twice. a stable prefix keeps the cache warm - a cache read costs 0.1x of normal input. that's why 20 turns cost 3.15x instead of 20x - one write and nineteen reads. save this and paste it into Claude Code - and your agent stops paying for what it already knew ↓
0:51

ANTHROPIC LEAKED A SETUP WHERE 6 FILES CUT TOKENS BY 84% AND RAISE ACCURACY BY 39% engineers at Google and Microsoft have been running this for a while - they pay $3 where you pay $20. decisions → contracts → dead ends → state → sources → open questions memory moves outside the window - what enters the context is a pointer to a file, not the history. raw history refills the window to 200k every single turn - six pointers take 8k and leave 96% free. same work, same five turns: 1,000,000 tokens against 40,000. decisions written to a file kill the endless "wait, why did we do it this way". written dead ends kill the second lap around the same circle - the agent never walks in there twice. a stable prefix keeps the cache warm - a cache read costs 0.1x of normal input. that's why 20 turns cost 3.15x instead of 20x - one write and nineteen reads. save this and paste it into Claude Code - and your agent stops paying for what it already knew ↓

Noisy

137,563 次观看 • 2 个月前

THE $4,000 DGX SPARK BENCHMARKS AT 342 T/S ON PROMPT PROCESSING - THEN DROPS TO 73 WHEN IT ACTUALLY GENERATES. that clip is a dgx spark running llama-bench on qwen3-coder 30b, side by side with a mac. two numbers decide everything here: pp512, prompt processing = 342 tokens/sec tg128, token generation = 73 tokens/sec that gap is the whole lesson. local ai isn't one speed, it's two. prompt processing is reading your input: your files, your long context. the spark is compute-heavy, so it eats that fast. token generation is writing the answer, and it's bound by memory bandwidth, where the spark is modest and unified-memory macs stay close. so "which box is faster" has no single winner. feeding huge context or running agents that read a lot? prompt processing wins, the spark shines. long chat replies, one token at a time? generation matters more, and a mac holds its own. no single benchmark number, no universal "fastest box", no upgrade that wins every workload. the uncomfortable part isn't that the spark is fast. it's that "tokens per second" was always two numbers, and the hype only ever quotes the flattering one. follow RetroChainer for the benchmark truth nobody screenshots, and bookmark this before you buy on one number.
1:14

THE $4,000 DGX SPARK BENCHMARKS AT 342 T/S ON PROMPT PROCESSING - THEN DROPS TO 73 WHEN IT ACTUALLY GENERATES. that clip is a dgx spark running llama-bench on qwen3-coder 30b, side by side with a mac. two numbers decide everything here: pp512, prompt processing = 342 tokens/sec tg128, token generation = 73 tokens/sec that gap is the whole lesson. local ai isn't one speed, it's two. prompt processing is reading your input: your files, your long context. the spark is compute-heavy, so it eats that fast. token generation is writing the answer, and it's bound by memory bandwidth, where the spark is modest and unified-memory macs stay close. so "which box is faster" has no single winner. feeding huge context or running agents that read a lot? prompt processing wins, the spark shines. long chat replies, one token at a time? generation matters more, and a mac holds its own. no single benchmark number, no universal "fastest box", no upgrade that wins every workload. the uncomfortable part isn't that the spark is fast. it's that "tokens per second" was always two numbers, and the hype only ever quotes the flattering one. follow RetroChainer for the benchmark truth nobody screenshots, and bookmark this before you buy on one number.

RetroChainer

42,451 次观看 • 2 个月前

Cursor posted an open problem - Character Prefix Conditioning on their blog post. This is my attempt to solving the problem using DFA + trie + SmolLM-135M. LLMs generate tokens, not characters. But we type characters. If the user has typed a few characters in the current word(e.g. pri) we want the model's completion to begin with those exact characters. To solve this we: > build a trie of vocab tokens > track prefix consumption with DFA state(index i) > at each step, allowed tokens = {tokens that are prefix of R} U {tokens that start with R} > force deterministic transitions if only one option, else sample with masked logits. sample video on how it works. ofc it's not perfect, it's just a 135M model, you can choose whatever model you want :))
1:04

Cursor posted an open problem - Character Prefix Conditioning on their blog post. This is my attempt to solving the problem using DFA + trie + SmolLM-135M. LLMs generate tokens, not characters. But we type characters. If the user has typed a few characters in the current word(e.g. pri) we want the model's completion to begin with those exact characters. To solve this we: > build a trie of vocab tokens > track prefix consumption with DFA state(index i) > at each step, allowed tokens = {tokens that are prefix of R} U {tokens that start with R} > force deterministic transitions if only one option, else sample with masked logits. sample video on how it works. ofc it's not perfect, it's just a 135M model, you can choose whatever model you want :))

Archie Sengupta

24,973 次观看 • 1 年前

you installed it with whichever command needed the least typing that command had a scope, and the short one is the widest one user scope does not live in the project, it lives in you, and it loads in every repository you open for the rest of the year a repo you touched once in march is still being told about your database tools i counted mine: four servers, eight thousand nine hundred tokens, riding into projects that have no database and no browser you did not install a tool you set a default for every piece of work you have not started yet the shortest command in the docs and the longest reach
0:43

you installed it with whichever command needed the least typing that command had a scope, and the short one is the widest one user scope does not live in the project, it lives in you, and it loads in every repository you open for the rest of the year a repo you touched once in march is still being told about your database tools i counted mine: four servers, eight thousand nine hundred tokens, riding into projects that have no database and no browser you did not install a tool you set a default for every piece of work you have not started yet the shortest command in the docs and the longest reach

Lagosaur

21,006 次观看 • 1 个月前

this JEV is certified f*cking diamond every agent you ship burns frontier tokens on questions that were never writing jobs is this spam? which worker goes next? is the draft good enough to show a human? those are decisions. Jev is built for that half of the loop you send state plus a typed question. it returns a choice, a score, or yes/no with confidence. no chat. no second opinion the public demos, not our lab: > ~1,000 papers classified for about $0.08, median around 256ms > ~500 emails in one inbox pass for about 3.5¢ > a flight search loop in about 7 seconds for about $0.0039 > a huge Claude session compacted from ~1M tokens down to ~86k in about a second the whole trick is the split if it creates, the LLM writes if it picks, scores, or gates, Jev decides if it does the thing, code does it pricing people quote: about $0.042 per million tokens in. output free. 10,000 decisions is about $0.42 a local router makes it obvious. goal plus what's already done. Jev picks research, write, or review. confidence under 0.85 goes to a human. the job lands as JSON in a folder. the writer never wakes up for a yes/no and the loop refuses to publish on its own drafts go to review. you hit post. Jev never sends the tweet one model writes. a cheaper one decides. nothing irreversible ships until a human says yes.
0:22

this JEV is certified f*cking diamond every agent you ship burns frontier tokens on questions that were never writing jobs is this spam? which worker goes next? is the draft good enough to show a human? those are decisions. Jev is built for that half of the loop you send state plus a typed question. it returns a choice, a score, or yes/no with confidence. no chat. no second opinion the public demos, not our lab: > ~1,000 papers classified for about $0.08, median around 256ms > ~500 emails in one inbox pass for about 3.5¢ > a flight search loop in about 7 seconds for about $0.0039 > a huge Claude session compacted from ~1M tokens down to ~86k in about a second the whole trick is the split if it creates, the LLM writes if it picks, scores, or gates, Jev decides if it does the thing, code does it pricing people quote: about $0.042 per million tokens in. output free. 10,000 decisions is about $0.42 a local router makes it obvious. goal plus what's already done. Jev picks research, write, or review. confidence under 0.85 goes to a human. the job lands as JSON in a folder. the writer never wakes up for a yes/no and the loop refuses to publish on its own drafts go to review. you hit post. Jev never sends the tweet one model writes. a cheaper one decides. nothing irreversible ships until a human says yes.

kiosa

11,649 次观看 • 10 天前