Загрузка видео...

Не удалось загрузить видео

Возникла проблема при загрузке этого видео. Это может быть связано с временными проблемами сети или видео может быть недоступно.

На главную

OpenHands crushed Codex by 2.3× on token efficiency! We gave both agents the same task on the same model (Qwen3.5 35B): build an 8-bit Space Invaders in 3 iterations (build, fix, polish). Output: • OpenHands: 219K tokens, • Codex: 513K tokens, OpenHands beats Codex in local running. The difference... show more

atomic.chat

15,524 subscribers

29,696 просмотров • 1 месяц назад •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

Комментарии: 9

Фото профиля James

James1 месяц назад

@OpenHandsDev @thsottiaux some ideas for the next iteration?

Фото профиля AI Mastery Guide

AI Mastery Guide1 месяц назад

@OpenHandsDev 2.3x fewer tokens for the same task is a big efficiency gap

Фото профиля unchosen.eth

unchosen.eth1 месяц назад

@OpenHandsDev efficiency will matter more over time

Фото профиля RF北灰

RF北灰1 месяц назад

@OpenHandsDev That’s mean the openhands can save tokens when i use it ?

Фото профиля atomic.chat

atomic.chat1 месяц назад

@OpenHandsDev for multi-step tasks like build/fix/polish loops -yeah, since it reuses context instead of resending it. one-off prompts, less so. depends what you're doing

Фото профиля RF北灰

RF北灰1 месяц назад

@OpenHandsDev Ok, I got it, it’s depends on the type ofmy task, thanks!

Фото профиля zhu

zhu1 месяц назад

@OpenHandsDev @grok explain what is that means "euses the unchanged data across every pass and only pays for new tokens, while Codex re-sends and re-counts it every iteration" and how it affects KV cache

Фото профиля Engel Nyst - open/acc

Engel Nyst - open/acc1 месяц назад

The explanation doesn’t make sense to me. Codex can and does normally send the prefix unchanged for prompt caching. So does OpenHands. The video shows 30 iterations on OpenHands side, 9 for Codex side, so it must be all Codex uncached. I think you guys have a bug of some kind on Codex / Codex config.

Фото профиля Jeff Steve

Jeff Steve1 месяц назад

@OpenHandsDev Hi still here asking for MCP support on atmoc chat on google play store like operit AI

Похожие видео

Agent Zero crushed Codex by 2.6× on token efficiency! We gave both agents the same task on the same local model (Qwen3.6 27B): build a retro Q*bert arcade game. Each agent wrote and rewrote it across iterations, adding enemies and a HUD. Outputs: • Agent Zero: 239K tokens, 53 min • Codex: 627K tokens, 51 min Both agents built a working Q*bert, but GitLawb 's Agent Zero is the cleaner build, with crisper textures and hops that land where they should. Codex refills the context window with its whole history every turn, so the model loses the thread and hallucinates code it never wrote. Agent Zero sends only what's new, so it keeps more of the window open and makes fewer mistakes. Run agents on local AI models in Atomic Chat!

Agent Zero crushed Codex by 2.6× on token efficiency! We gave both agents the same task on the same local model (Qwen3.6 27B): build a retro Qbert arcade game. Each agent wrote and rewrote it across iterations, adding enemies and a HUD. Outputs: • Agent Zero: 239K tokens, 53 min • Codex: 627K tokens, 51 min Both agents built a working Qbert, but GitLawb 's Agent Zero is the cleaner build, with crisper textures and hops that land where they should. Codex refills the context window with its whole history every turn, so the model loses the thread and hallucinates code it never wrote. Agent Zero sends only what's new, so it keeps more of the window open and makes fewer mistakes. Run agents on local AI models in Atomic Chat!

atomic.chat

320,187 просмотров • 1 месяц назад

Live on Cline (v3.67.1) OpenAI 's GPT 5.3 Codex. The speed and token efficiency improvements are real. Here is what's new: > 25% faster than 5.2 Codex > #1 on SWE-Bench Pro (4 different programming languages) > Fewer tokens per task than any prior OpenAI model Runs cost less and finish faster. You can use it in Cline right now without setting up an API key. Select the model and try it on your repo.

Live on Cline (v3.67.1) OpenAI 's GPT 5.3 Codex. The speed and token efficiency improvements are real. Here is what's new: > 25% faster than 5.2 Codex > #1 on SWE-Bench Pro (4 different programming languages) > Fewer tokens per task than any prior OpenAI model Runs cost less and finish faster. You can use it in Cline right now without setting up an API key. Select the model and try it on your repo.

Cline

21,291 просмотров • 6 месяцев назад

I ran the same task on Claude Code and DeepSeek's new agent harness. One cost $150. The other cost $2. Today we're launching (AgentSky), the "OpenRouter for Agents" — one API → Claude Code, Codex, DeepSeek, Kimi, OpenCode, and every major agent in the cloud. And Agent Playground on top: race them on your own task, with your real tools (GitHub, Gmail, more), side by side in a browser: time, cost, tokens burnt. Guess which one was $2.

I ran the same task on Claude Code and DeepSeek's new agent harness. One cost $150. The other cost $2. Today we're launching (AgentSky), the "OpenRouter for Agents" — one API → Claude Code, Codex, DeepSeek, Kimi, OpenCode, and every major agent in the cloud. And Agent Playground on top: race them on your own task, with your real tools (GitHub, Gmail, more), side by side in a browser: time, cost, tokens burnt. Guess which one was $2.

Xiaoyin Qu

2,213,107 просмотров • 19 дней назад

Qwen3.5 runs quite well in mlx-lm. Awesome that we have a frontier-level hybrid model. The context gets longer but the inference speed and memory use barely change. Here's the Q4 generating a space invaders game on an M3 Ultra. Generated 4,120 tokens at 37.6 tok/s.

Qwen3.5 runs quite well in mlx-lm. Awesome that we have a frontier-level hybrid model. The context gets longer but the inference speed and memory use barely change. Here's the Q4 generating a space invaders game on an M3 Ultra. Generated 4,120 tokens at 37.6 tok/s.

Awni Hannun

48,673 просмотров • 6 месяцев назад

Can AI really replace SaaS? We asked Codex Spark to build a simple Salesforce clone. It generated a build plan, then wrote all the code in 29 seconds. And it works! Add contacts, live search, full pipeline dashboard – all unit tests passed. The same task on full Codex took ~5× longer. For rapid prototyping and self-contained work, nothing beats Spark. ✨

Can AI really replace SaaS? We asked Codex Spark to build a simple Salesforce clone. It generated a build plan, then wrote all the code in 29 seconds. And it works! Add contacts, live search, full pipeline dashboard – all unit tests passed. The same task on full Codex took ~5× longer. For rapid prototyping and self-contained work, nothing beats Spark. ✨

Cerebras

16,831 просмотров • 5 месяцев назад

Today, we launched browser use inside Codex to further close the build & verify loop for local development! Now, you can ask Codex to build your front end, and test it like a user would by clicking through the app. Codex sees everything a user sees through vision & checks the network/console logs to help debug & fix any issues that it finds. This change brings us closer to fully autonomous coding agents that delivers high quality and tested changes. Watch Codex test my app in the browser, catch & fix a real bug, and doing that loop again with a brand new feature.

James Sun

226,907 просмотров • 4 месяцев назад

⚡️ JUST IN: MUSK LAUNCHES GROK 4.5, CALLING IT "OPUS-CLASS" AI MODEL Musk just released Grok 4.5 on Grok Build, Cursor, and the SpaceXAI Console, calling it an "Opus-class" AI model. He says it is faster, more token-efficient and lower cost, with EU access expected by mid-July. Built on a 1.5 TRILLION parameter foundation model, it drops the same day as OpenAI's GPT-5.6. API pricing is $2 per million input tokens and $6 per million output tokens.

⚡️ JUST IN: MUSK LAUNCHES GROK 4.5, CALLING IT "OPUS-CLASS" AI MODEL Musk just released Grok 4.5 on Grok Build, Cursor, and the SpaceXAI Console, calling it an "Opus-class" AI model. He says it is faster, more token-efficient and lower cost, with EU access expected by mid-July. Built on a 1.5 TRILLION parameter foundation model, it drops the same day as OpenAI's GPT-5.6. API pricing is $2 per million input tokens and $6 per million output tokens.

Coin Bureau

50,488 просмотров • 2 месяцев назад

A developer figured out how to give AI permanent memory for $0.40 a year. No complex vector databases. No massive cloud storage. Just a single 4,000-token file that refuses to grow. The secret? True memory isn’t about storing everything. It’s about aggressive deletion. Instead of endlessly appending data, his system rewrites just six core fields on every turn: > IDENTITY: Who you are and what you build (300 tokens) > STATE: Current active task and focus (400 tokens) > DECISIONS: Settled choices—zero re-arguing (800 tokens) > CORRECTIONS: Every "no, do it this way" moment (600 tokens) > PEOPLE: Key names, roles, and open dependencies (500 tokens) > GRAVEYARD: Dead attempts, so bad ideas never return (400 tokens) Total size: 3,000 tokens. Absolute hard ceiling: 4,000 tokens. When a category hits its limit, the model compresses it. Data is never stacked—only pruned and replaced. Using Kimi K2.5 caching rates ($0.10/M tokens), running this budget memory costs $0.0004 per turn. That is 2,500 interactive turns for a single dollar. The real breakthrough lies in the CORRECTIONS field. Standard AI models waste compute making the same mistakes twice. This schema forces the assistant to remember its failures instantly. While the rest of the industry burns millions searching through bloated history logs, he pays pennies to keep his context razor-sharp. Your memory system isn't defined by what you save. It’s defined by what you have the discipline to throw away. Bookmark this breakdown. You will need it for your next build.

A developer figured out how to give AI permanent memory for $0.40 a year. No complex vector databases. No massive cloud storage. Just a single 4,000-token file that refuses to grow. The secret? True memory isn’t about storing everything. It’s about aggressive deletion. Instead of endlessly appending data, his system rewrites just six core fields on every turn: > IDENTITY: Who you are and what you build (300 tokens) > STATE: Current active task and focus (400 tokens) > DECISIONS: Settled choices—zero re-arguing (800 tokens) > CORRECTIONS: Every "no, do it this way" moment (600 tokens) > PEOPLE: Key names, roles, and open dependencies (500 tokens) > GRAVEYARD: Dead attempts, so bad ideas never return (400 tokens) Total size: 3,000 tokens. Absolute hard ceiling: 4,000 tokens. When a category hits its limit, the model compresses it. Data is never stacked—only pruned and replaced. Using Kimi K2.5 caching rates ($0.10/M tokens), running this budget memory costs $0.0004 per turn. That is 2,500 interactive turns for a single dollar. The real breakthrough lies in the CORRECTIONS field. Standard AI models waste compute making the same mistakes twice. This schema forces the assistant to remember its failures instantly. While the rest of the industry burns millions searching through bloated history logs, he pays pennies to keep his context razor-sharp. Your memory system isn't defined by what you save. It’s defined by what you have the discipline to throw away. Bookmark this breakdown. You will need it for your next build.

shmidt

17,304 просмотров • 17 дней назад

1-bit Hy3 running locally is 2.2x faster than its API at the same quality! We gave both models the same task and compared one-shot outputs. 1-bit Hy3 295B GGUF (92GB) ran locally on 4x RTX 5090 with 128GB VRAM against the same Hy3 over cloud API Tasks: - Flappy Bird - Arkanoid - Snake Outputs: Hy3 1-bit local: 76.9K tokens, 15.5 min Hy3 cloud API: 75.1K tokens, 34.3 min The 1-bit games look the same as the API ones. Birds fly through the pipes, bricks break, the snake eats and grows. Nothing froze or crashed. Both models even made the same slip: the snake can cross itself and the game does not end. Getting this quality from 1 bit running locally is wild! Run Hy3 GGUF yourself in Atomic Chat in 2 clicks

1-bit Hy3 running locally is 2.2x faster than its API at the same quality! We gave both models the same task and compared one-shot outputs. 1-bit Hy3 295B GGUF (92GB) ran locally on 4x RTX 5090 with 128GB VRAM against the same Hy3 over cloud API Tasks: - Flappy Bird - Arkanoid - Snake Outputs: Hy3 1-bit local: 76.9K tokens, 15.5 min Hy3 cloud API: 75.1K tokens, 34.3 min The 1-bit games look the same as the API ones. Birds fly through the pipes, bricks break, the snake eats and grows. Nothing froze or crashed. Both models even made the same slip: the snake can cross itself and the game does not end. Getting this quality from 1 bit running locally is wild! Run Hy3 GGUF yourself in Atomic Chat in 2 clicks

atomic.chat

91,182 просмотров • 2 месяцев назад

OpenAI's chairman Brett Taylor on the new Chinese model Kimi K3. the timeline thinks it's a frontier killer. he says every CEO he talks to asks the same two things: am I getting value from my token spend, AND what's my moat in the age of AI. on token efficiency, he says the frontier models still finish the same task in far fewer tokens. worth a listen.

OpenAI's chairman Brett Taylor on the new Chinese model Kimi K3. the timeline thinks it's a frontier killer. he says every CEO he talks to asks the same two things: am I getting value from my token spend, AND what's my moat in the age of AI. on token efficiency, he says the frontier models still finish the same task in far fewer tokens. worth a listen.

Damian Player

470,789 просмотров • 1 месяц назад

THE 64 MINUTE OPENAI CODEX MASTERCLASS IS HERE if you've been meaning to learn Codex, this is the episode for you, we cover: 1/ Claude vs Codex (what's better?) 2/How to run Claude Code inside Codex (really cool) 3/Why GPT 5.5 changed everything for browser agents 4/How to create skills (reusable agents you build once and call by name forever) 5/How to connect Notion with surgical permissions (one database, not the whole workspace) 6/How to use Remotion inside Codex to create videos 7/ How to one-shot full mobile apps in Swift 8/The 4 projects to run on day one to get up to speed fast shoutout to Riley Brown for coming onto The Startup Ideas Podcast (SIP) 🧃 for the 5th time. every episode he delivers. this one is the Codex masterclass. if you've been meaning to learn Codex, this is the episode. 100% free i just want to see you build your ideas. watch

THE 64 MINUTE OPENAI CODEX MASTERCLASS IS HERE if you've been meaning to learn Codex, this is the episode for you, we cover: 1/ Claude vs Codex (what's better?) 2/How to run Claude Code inside Codex (really cool) 3/Why GPT 5.5 changed everything for browser agents 4/How to create skills (reusable agents you build once and call by name forever) 5/How to connect Notion with surgical permissions (one database, not the whole workspace) 6/How to use Remotion inside Codex to create videos 7/ How to one-shot full mobile apps in Swift 8/The 4 projects to run on day one to get up to speed fast shoutout to Riley Brown for coming onto The Startup Ideas Podcast (SIP) 🧃 for the 5th time. every episode he delivers. this one is the Codex masterclass. if you've been meaning to learn Codex, this is the episode. 100% free i just want to see you build your ideas. watch

GREG ISENBERG

200,125 просмотров • 4 месяцев назад

Following up on our Intelligence Index v4.1 release yesterday, in the video below, Daniel from our team shares a short overview of what's changed: 1. Three upgraded evaluations: Terminal-Bench 2.1, τ³-Bench Banking and GDPval-AA v2 2. Cost, time, and tokens per task: Understand the cost, time, and tokens of tasks across our Index and for individual evals, and how these trade off against Intelligence 3. Cached input token reporting: We now report the amount of cached tokens a particular model uses and how this influences cost

Following up on our Intelligence Index v4.1 release yesterday, in the video below, Daniel from our team shares a short overview of what's changed: 1. Three upgraded evaluations: Terminal-Bench 2.1, τ³-Bench Banking and GDPval-AA v2 2. Cost, time, and tokens per task: Understand the cost, time, and tokens of tasks across our Index and for individual evals, and how these trade off against Intelligence 3. Cached input token reporting: We now report the amount of cached tokens a particular model uses and how this influences cost

Artificial Analysis

13,214 просмотров • 2 месяцев назад

🚨 OpenAI just launched Codex, a brand-new autonomous coding agent that can build features and fix bugs on its own. We’ve been using it Every 📧 for a few days, and I’m impressed. I invited Alexander Embiricos (ben davies), a member of the product staff responsible for Codex, to demo Codex and talk about it live on a special edition of AI & I: What Codex is and how it works Codex is designed to be used by senior engineers—it performs coding tasks like adding features or fixing bugs autonomously. It's built to allow you to start many sessions at once, so you can have multiple agents working in parallel. Codex is built to have "taste" OpenAI trained Codex to have the taste of a senior software engineer. It knows how big codebases work, how to write a good PR, and uses clean, minimal code. Why an “abundance mindset” is best for interacting with agents Codex is designed to allow users to delegate many tasks at once without getting caught up in the details. This lets you point an abundance of agents at a specific task like a difficult bug—it’s worth it even if only one of them succeeds. How OpenAI is thinking about agents Codex is one piece of a unified super-assistant OpenAI wants to eventually build—an agent that helps users easily get things done by selecting the right tools for them behind the scenes. OpenAI’s vision for the future of programming In the future developers will probably spend less time writing routine code and more time guiding agents, reviewing their work, and making strategy decisions. Programming will become more social, letting teams easily delegate multiple tasks at once, allowing people to focus on ideas and collaboration instead of routine coding. Watch below!

Dan Shipper 📧

145,487 просмотров • 1 год назад

Codex can spawn grok 4.5 or Kimi k3 as subagents now. Not more GPT, other companies' models, running underneath it, in the same session. Wired both into my picker weeks ago. Today they stopped being alternatives and started taking jobs. Each agent gets its own context window and runs on a sub i already pay for. My Codex quota only goes on the turns that need it. Links down below 👇

Codex can spawn grok 4.5 or Kimi k3 as subagents now. Not more GPT, other companies' models, running underneath it, in the same session. Wired both into my picker weeks ago. Today they stopped being alternatives and started taking jobs. Each agent gets its own context window and runs on a sub i already pay for. My Codex quota only goes on the turns that need it. Links down below 👇

Ziwen

46,651 просмотров • 1 месяц назад

New video of Qualcomm CEO Cristiano Amon: AI will require “gazillions” of tokens. Because, Agentic AI will consume dramatically more tokens because it performs autonomous tasks, uses multiple systems, and interacts with tools. AI demand will grow hugely when software starts letting agents act, not just answer. A chatbot spends tokens on language; an agent spends tokens on deciding, checking, calling tools, reading outputs, revising plans, and coordinating with other software. Today a single human-AI exchange may be large, a reasoning task may be much larger, but we are already entering the agentic era, where an autonomous workflow can become exponentially larger still because the model is no longer producing one response. It is running a process. When a SaaS product redesigns itself around human-agent interaction, every task can become a chain of hidden micro-decisions, and each micro-decision consumes context, memory, tool calls, verification, and output tokens. Another point is people hear “more tokens” and think only of bigger data centers, when the deeper change is economic: software usage may stop being measured mainly by clicks, seats, or sessions. It will be measured by how much machine reasoning/tokens gets spent on behalf of each user. ---- From "Reuters" YouTube channel, (link in comment)

New video of Qualcomm CEO Cristiano Amon: AI will require “gazillions” of tokens. Because, Agentic AI will consume dramatically more tokens because it performs autonomous tasks, uses multiple systems, and interacts with tools. AI demand will grow hugely when software starts letting agents act, not just answer. A chatbot spends tokens on language; an agent spends tokens on deciding, checking, calling tools, reading outputs, revising plans, and coordinating with other software. Today a single human-AI exchange may be large, a reasoning task may be much larger, but we are already entering the agentic era, where an autonomous workflow can become exponentially larger still because the model is no longer producing one response. It is running a process. When a SaaS product redesigns itself around human-agent interaction, every task can become a chain of hidden micro-decisions, and each micro-decision consumes context, memory, tool calls, verification, and output tokens. Another point is people hear “more tokens” and think only of bigger data centers, when the deeper change is economic: software usage may stop being measured mainly by clicks, seats, or sessions. It will be measured by how much machine reasoning/tokens gets spent on behalf of each user. ---- From "Reuters" YouTube channel, (link in comment)

Rohan Paul

31,617 просмотров • 3 месяцев назад

A tricky LLM interview question: You're serving a reasoning model on vLLM, and it keeps running out of GPU memory on long traces. So you add KV cache compression and evict 90% of the cached tokens. VRAM usage stays as is and GPU still runs out of memory. Why? (answer below) Evicting 90% of the KV cache can free almost none of the memory it was using. This sounds counterintuitive, but it follows directly from how production servers store the cache today. The KV cache grows with every token a model generates. Each token appends its key and value vectors across every layer, and nothing is freed while generation continues. This is the dominant memory cost for reasoning models. If a 32K-token CoT caches ~32K tokens of KV vectors, a Qwen3-32B with 4-bit weights will run out-of-memory around 24K tokens on a 24GB GPU. One obvious solution is to keep the important tokens and drop the rest, since attention is sparse enough to allow it. But this does not solve the memory problem yet. The reason is paged attention, which is the memory manager behind vLLM and most production servers. Under the hood, it splits GPU memory into fixed physical blocks, each one holds the KV for about 16 tokens. This block returns to the allocator only when every slot inside it is empty. Since the eviction logic selects tokens by importance, and such tokens are scattered across blocks... ...so despite eviction, almost every block is left with at least some survivor tokens. For instance, if the logic evicts 14k of 16k tokens across 1,000 blocks, most likely every block will still have a token. This means the allocator frees almost nothing. Placing the new tokens into those freed slots is not ideal because it breaks the cache's layout. Say token 16,001 arrives, and it's placed in the slot the 40th token used to hold. The cache now reads position 38, then 16,001, then 41, so the cache is no longer in token order. Attention can still compute the right answer from that, but only if every slot now carries a separate note recording which position it actually holds. This introduces another bookkeeping cost that an in-order layout inherently avoids. So the cache is logically 90% smaller and still physically the same size. Many compression results miss this because they measure on pre-allocated contiguous tensors rather than a paged server. There's another problem. Eviction methods pick which tokens to keep by looking at the attention scores themselves (as expected). But fast attention kernels used in production, like FlashAttention, never save those scores. They compute attention in small pieces and throw the full score grid away as they go, which is also why they're fast. So the exact signal eviction methods need isn't available in memory. The workaround is to fall back to eager attention and build the full matrix, which gives up the speed FlashAttention was there to provide. NVIDIA published a method called TriAttention to solve both these problems. It never needs attention scores. Instead, it scores tokens from the geometry of the model's key and query vectors before RoPE is applied, where those vectors sit in stable clusters. For the memory problem, it runs a compaction pass every 128 decoded tokens. The surviving tokens slide forward to close the holes eviction creates, so whole blocks empty out and return to the allocator while the cache stays in token order. On long reasoning traces, the approach matches full-attention accuracy while decoding 2.5x faster and using 10.7x less KV memory. KV cache compression is a big infrastructure problem. The number that decides whether it works is the count of freed blocks, not the count of evicted tokens. You can find the NVIDIA write-up here: I wrote a first-principles breakdown of how the KV cache works. It walks through why the model stores keys and values at all, why the cache grows with every token, and a comparison of LLM generation speed with and without KV caching. Read it below.

A tricky LLM interview question: You're serving a reasoning model on vLLM, and it keeps running out of GPU memory on long traces. So you add KV cache compression and evict 90% of the cached tokens. VRAM usage stays as is and GPU still runs out of memory. Why? (answer below) Evicting 90% of the KV cache can free almost none of the memory it was using. This sounds counterintuitive, but it follows directly from how production servers store the cache today. The KV cache grows with every token a model generates. Each token appends its key and value vectors across every layer, and nothing is freed while generation continues. This is the dominant memory cost for reasoning models. If a 32K-token CoT caches ~32K tokens of KV vectors, a Qwen3-32B with 4-bit weights will run out-of-memory around 24K tokens on a 24GB GPU. One obvious solution is to keep the important tokens and drop the rest, since attention is sparse enough to allow it. But this does not solve the memory problem yet. The reason is paged attention, which is the memory manager behind vLLM and most production servers. Under the hood, it splits GPU memory into fixed physical blocks, each one holds the KV for about 16 tokens. This block returns to the allocator only when every slot inside it is empty. Since the eviction logic selects tokens by importance, and such tokens are scattered across blocks... ...so despite eviction, almost every block is left with at least some survivor tokens. For instance, if the logic evicts 14k of 16k tokens across 1,000 blocks, most likely every block will still have a token. This means the allocator frees almost nothing. Placing the new tokens into those freed slots is not ideal because it breaks the cache's layout. Say token 16,001 arrives, and it's placed in the slot the 40th token used to hold. The cache now reads position 38, then 16,001, then 41, so the cache is no longer in token order. Attention can still compute the right answer from that, but only if every slot now carries a separate note recording which position it actually holds. This introduces another bookkeeping cost that an in-order layout inherently avoids. So the cache is logically 90% smaller and still physically the same size. Many compression results miss this because they measure on pre-allocated contiguous tensors rather than a paged server. There's another problem. Eviction methods pick which tokens to keep by looking at the attention scores themselves (as expected). But fast attention kernels used in production, like FlashAttention, never save those scores. They compute attention in small pieces and throw the full score grid away as they go, which is also why they're fast. So the exact signal eviction methods need isn't available in memory. The workaround is to fall back to eager attention and build the full matrix, which gives up the speed FlashAttention was there to provide. NVIDIA published a method called TriAttention to solve both these problems. It never needs attention scores. Instead, it scores tokens from the geometry of the model's key and query vectors before RoPE is applied, where those vectors sit in stable clusters. For the memory problem, it runs a compaction pass every 128 decoded tokens. The surviving tokens slide forward to close the holes eviction creates, so whole blocks empty out and return to the allocator while the cache stays in token order. On long reasoning traces, the approach matches full-attention accuracy while decoding 2.5x faster and using 10.7x less KV memory. KV cache compression is a big infrastructure problem. The number that decides whether it works is the count of freed blocks, not the count of evicted tokens. You can find the NVIDIA write-up here: I wrote a first-principles breakdown of how the KV cache works. It walks through why the model stores keys and values at all, why the cache grows with every token, and a comparison of LLM generation speed with and without KV caching. Read it below.

Avi Chawla

271,839 просмотров • 2 месяцев назад

We talked to Micky 🐸, the founder of Ribbit Capital and early backer of Coinbase, Robinhood, and Bridge, about what’s coming next. "I’m more excited for the next decade than the last." “Everything is a token. We call it the token factory revolution; tokenizing information, money, and power.” AI, payments, and financial services are converging fast, and tokens are becoming the raw material for decision-making, coordination, and access. Micky sees a new stack forming where every company is either building, supplying, or orchestrating tokens. And he bets on a generation of young founders who are native to this shift: “Their brains are just faster. The best ones are already building access tokens, expert tokens, asset tokens… stablecoins were only the first chapter.” From his appearance on the show in July.

We talked to Micky 🐸, the founder of Ribbit Capital and early backer of Coinbase, Robinhood, and Bridge, about what’s coming next. "I’m more excited for the next decade than the last." “Everything is a token. We call it the token factory revolution; tokenizing information, money, and power.” AI, payments, and financial services are converging fast, and tokens are becoming the raw material for decision-making, coordination, and access. Micky sees a new stack forming where every company is either building, supplying, or orchestrating tokens. And he bets on a generation of young founders who are native to this shift: “Their brains are just faster. The best ones are already building access tokens, expert tokens, asset tokens… stablecoins were only the first chapter.” From his appearance on the show in July.

TBPN

106,752 просмотров • 10 месяцев назад

A developer in Hangzhou runs an AI that remembers everything about him for $0.40 a year. No vector database. One file that never grows past 4,000 tokens. He published the whole schema. His version starts from the opposite idea. Memory is not storage. It's a write policy. Six fields. Rewritten every time, never appended: > IDENTITY - who you are, what you build. 300 tokens. Changes monthly at most > STATE - what you're on right now. 400 tokens. Rewritten daily > DECISIONS - what's already settled, so nothing gets re-argued. 800 tokens > CORRECTIONS - every time you said "no, not like that." 600 tokens > PEOPLE - names, roles, who's waiting on what. 500 tokens > DEAD - tried and abandoned, so it never comes back as a suggestion. 400 tokens Three thousand tokens. Ceiling of four. When a section fills, the model rewrites it shorter. Nothing is ever added. Only replaced. Kimi K2.5 bills $0.10 per million cached input tokens. Four thousand tokens a turn is $0.0004. That's 2,500 turns for a dollar. The free tier hands you 1.5 million tokens a day. 375 turns before you pay anything at all. CORRECTIONS is the field nobody builds, and it's the one that does the work. A model that remembers being wrong stops repeating it. Everyone else is paying to search their own history. He pays to keep it short. The bill stopped growing when the file did. Your memory system isn't defined by what it stores. It's defined by what it agrees to delete. The article below is the full build - schema, rewrite prompts, the compaction rule that keeps it under the cap. Save it. You'll want it open in the other tab.

A developer in Hangzhou runs an AI that remembers everything about him for $0.40 a year. No vector database. One file that never grows past 4,000 tokens. He published the whole schema. His version starts from the opposite idea. Memory is not storage. It's a write policy. Six fields. Rewritten every time, never appended: > IDENTITY - who you are, what you build. 300 tokens. Changes monthly at most > STATE - what you're on right now. 400 tokens. Rewritten daily > DECISIONS - what's already settled, so nothing gets re-argued. 800 tokens > CORRECTIONS - every time you said "no, not like that." 600 tokens > PEOPLE - names, roles, who's waiting on what. 500 tokens > DEAD - tried and abandoned, so it never comes back as a suggestion. 400 tokens Three thousand tokens. Ceiling of four. When a section fills, the model rewrites it shorter. Nothing is ever added. Only replaced. Kimi K2.5 bills $0.10 per million cached input tokens. Four thousand tokens a turn is $0.0004. That's 2,500 turns for a dollar. The free tier hands you 1.5 million tokens a day. 375 turns before you pay anything at all. CORRECTIONS is the field nobody builds, and it's the one that does the work. A model that remembers being wrong stops repeating it. Everyone else is paying to search their own history. He pays to keep it short. The bill stopped growing when the file did. Your memory system isn't defined by what it stores. It's defined by what it agrees to delete. The article below is the full build - schema, rewrite prompts, the compaction rule that keeps it under the cap. Save it. You'll want it open in the other tab.

wast3

15,862 просмотров • 22 дней назад

AI will require “gazillions” of tokens. - Qualcomm CEO Cristiano Amon: Agentic AI will consume dramatically more tokens because it performs autonomous tasks, uses multiple systems, and interacts with tools. A chatbot spends tokens on language; an agent spends tokens on deciding, checking, calling tools, reading outputs, revising plans, and coordinating with other software. So, AI demand will grow hugely when software starts letting agents act, not just answer. Today a single human-AI exchange may be large, a reasoning task may be much larger, but we are already entering the agentic era, where an autonomous workflow can become exponentially larger still because the model is no longer producing one response. It is running a process. When a SaaS product redesigns itself around human-agent interaction, every task can become a chain of hidden micro-decisions, and each micro-decision consumes context, memory, tool calls, verification, and output tokens. Another point is people hear “more tokens” and think only of bigger data centers, when the deeper change is economic: software usage may stop being measured mainly by clicks, seats, or sessions. It will be measured by how much machine reasoning/tokens gets spent on behalf of each user. ---- From "Reuters" YouTube channel, (link in comment)

AI will require “gazillions” of tokens. - Qualcomm CEO Cristiano Amon: Agentic AI will consume dramatically more tokens because it performs autonomous tasks, uses multiple systems, and interacts with tools. A chatbot spends tokens on language; an agent spends tokens on deciding, checking, calling tools, reading outputs, revising plans, and coordinating with other software. So, AI demand will grow hugely when software starts letting agents act, not just answer. Today a single human-AI exchange may be large, a reasoning task may be much larger, but we are already entering the agentic era, where an autonomous workflow can become exponentially larger still because the model is no longer producing one response. It is running a process. When a SaaS product redesigns itself around human-agent interaction, every task can become a chain of hidden micro-decisions, and each micro-decision consumes context, memory, tool calls, verification, and output tokens. Another point is people hear “more tokens” and think only of bigger data centers, when the deeper change is economic: software usage may stop being measured mainly by clicks, seats, or sessions. It will be measured by how much machine reasoning/tokens gets spent on behalf of each user. ---- From "Reuters" YouTube channel, (link in comment)

Rohan Paul

12,664 просмотров • 2 месяцев назад

Continuous batching in LLMs, clearly explained: (a popular LLM interview question; bookmark this) In traditional ML inference, a batch is a matrix. Every input is padded to the same length, one forward pass runs, and every row finishes at the same moment. LLM decoding does not work that way. One forward pass produces one token per sequence, so a request needs as many passes as it has output tokens, and nobody knows that count until the model emits a stop token. Under static batching, membership is fixed when the batch starts. A request that finishes in 30 tokens holds its slot until the slowest request in the same batch finishes at 400. The GPU keeps paying the full weight read for a batch that is mostly empty. Loading model weights out of HBM costs the same whether four slots are producing tokens or one. Continuous batching moves the decision boundary. Instead of scheduling once per batch, the scheduler runs a single forward pass, gets control back, and decides again. A finished request leaves at the next iteration boundary, and a queued request takes its slot right there. No slot stays reserved for work that is already done. Anyscale benchmarked both OPT-13B on a single A100. With uniform generation lengths, the two policies came out about level (as expected), and as output length variance rose, static batching fell to around 81 tokens per second while vLLM reached 23x the throughput of naive Hugging Face serving. Variance drives the entire gap. Production traffic mixes 30-token replies with 400-token ones, which is exactly the condition static batching handles worst. None of this alters the model. vLLM, SGLang, TGI, and TensorRT-LLM all run it by default, and NVIDIA ships the same mechanism under the name in-flight batching. The animation below runs both policies on the same 16 requests and the same 4 slots, stepping in lockstep. The only difference is when a new request is allowed in. To dive deeper into continuous batching specifically, I wrote a full breakdown of the scheduler underneath it. It covers what happens between two forward passes, how tokens get handed out against a fixed budget, why the scheduler needs no separate path for prefill and decode, and what preemption costs you when the KV cache fills up mid-generation. Read it below.

Continuous batching in LLMs, clearly explained: (a popular LLM interview question; bookmark this) In traditional ML inference, a batch is a matrix. Every input is padded to the same length, one forward pass runs, and every row finishes at the same moment. LLM decoding does not work that way. One forward pass produces one token per sequence, so a request needs as many passes as it has output tokens, and nobody knows that count until the model emits a stop token. Under static batching, membership is fixed when the batch starts. A request that finishes in 30 tokens holds its slot until the slowest request in the same batch finishes at 400. The GPU keeps paying the full weight read for a batch that is mostly empty. Loading model weights out of HBM costs the same whether four slots are producing tokens or one. Continuous batching moves the decision boundary. Instead of scheduling once per batch, the scheduler runs a single forward pass, gets control back, and decides again. A finished request leaves at the next iteration boundary, and a queued request takes its slot right there. No slot stays reserved for work that is already done. Anyscale benchmarked both OPT-13B on a single A100. With uniform generation lengths, the two policies came out about level (as expected), and as output length variance rose, static batching fell to around 81 tokens per second while vLLM reached 23x the throughput of naive Hugging Face serving. Variance drives the entire gap. Production traffic mixes 30-token replies with 400-token ones, which is exactly the condition static batching handles worst. None of this alters the model. vLLM, SGLang, TGI, and TensorRT-LLM all run it by default, and NVIDIA ships the same mechanism under the name in-flight batching. The animation below runs both policies on the same 16 requests and the same 4 slots, stepping in lockstep. The only difference is when a new request is allowed in. To dive deeper into continuous batching specifically, I wrote a full breakdown of the scheduler underneath it. It covers what happens between two forward passes, how tokens get handed out against a fixed budget, why the scheduler needs no separate path for prefill and decode, and what preemption costs you when the KV cache fills up mid-generation. Read it below.

Avi Chawla

16,140 просмотров • 1 месяц назад