正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

I designed a new test specifically for multimodal models: fill out a paper form. And it's much harder than it sounds. This isn't typing into an electronic field that captures your text. The form is just an image. The model has to place each form element: text, checkmarks —... at the correct pixel position on the canvas itself. Results: 🟢 Kimi K2.6 → done in 3:45, 16.7k output tokens 🟡 Step 3.7 Flash → half the fields, 57k output tokens 🔴 Gemini 3.5 Flash → 489k output tokens, never finished. I had to kill it. Gemini burned ~29x more output tokens than Kimi on the exact same task, and Kimi's was the only form that actually looked filled out. The test, a mocked application form, contains some challenging parts, such as one-character-per-box fields. I provided every model the same set of tools: > get canvas size > drop probe markers to find coordinates > add text > add checkmarks > move elements > take a screenshot anytime to check their own work > ... etc So it's vision + spatial reasoning + tool use + long context, all at once. Small models (Qwen, Gemma) can't really complete this test, so I skipped them. What happened: > Kimi nailed name, DOB, ID, gender, marital status, nationality, email, phone, address, postal code — placement slightly loose, but content correct. 15 turns. Clean. > Step got maybe half right — fields dropped, "United States" landed in the email line, data floating outside boxes. Burned 1.24M input tokens doing it (81 turns of re-reading the canvas). > Gemini almost got there visually... then spiraled. By turn 40 it was issuing a delete_elements call wiping element IDs 365–425, basically erasing its own work. 31 minutes, 489k output tokens, still streaming. Terminated. The takeaway isn't "Gemini bad." This test is indeed difficult. But token efficiency is capability now. A model that needs 30x the tokens and still can't converge is going to be 30x the cost in production. Kimi K2.6 just quietly did the thing.show more

stevibe

27,755 subscribers

25,494 次观看 • 4 个月前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

31 条评论

mr_r0b0t 的头像

mr_r0b0t4 个月前

This is the exact sort of “cool but does it work” test we need I’m so stoked seeing the evolution of this, really awesome work

noname 的头像

noname4 个月前

I built this feature few months ago for SaaS and gpt was the only model which gave decent results, the rest just failed miserably at this.

stevibe 的头像

stevibe4 个月前

That can be expensive? Or it just nailed it in a few rounds?

noname 的头像

noname4 个月前

It did eventually nail it in my case in a few rounds. But I had to optimise the prompt for it work reliably that’s where the most time and effort went.

seijin 的头像

seijin4 个月前

Looks like Gemini is a perfectionist.

Janek Mann 的头像

Janek Mann4 个月前

I had to do some form filling with Codex recently… it struggled a lot to hit the boxes, eventually I told it to “use code” essentially and then it managed perfectly (every letter perfectly centered in every box).

Tom Maiaroto 的头像

Tom Maiaroto4 个月前

Super interesting! My team will most likely be building software to do exactly this in the future. I'll keep this one bookmarked, thanks!

Peter Baylies 的头像

Peter Baylies4 个月前

Now do Qwen 3.6 27B

Levi 的头像

Levi4 个月前

kinda wild how kimi just quietly aced it hahaha

John Shina 的头像

John Shina4 个月前

Whoa

hadesdevs 的头像

hadesdevs4 个月前

Super interesting test, I wonder if cision specific models could do well, like combining lfm 450M for vision, and their newest one for the tool call and reasoning part. I don't know another small llms, but would be interesting if they can pass ot too

Paul Sant · YouPulse 的头像

Paul Sant · YouPulse4 个月前

Most document workflows die right there: read the field correctly, then put the ink two boxes over.

stevibe 的头像

stevibe4 个月前

True, although even with the help of the marker tool to probe the coordinates, models still fail to correct the positions.

Paul Sant · YouPulse 的头像

Paul Sant · YouPulse4 个月前

The coordinate is already found. The missing step is checking the mark against the actual box before commit.

Xaden Ryan 的头像

Xaden Ryan4 个月前

Now this is a great test!

stevibe 的头像

stevibe4 个月前

Thank you!

𝗭𝗲𝗻 𝗠𝗮𝗴𝗻𝗲𝘁𝘀 的头像

𝗭𝗲𝗻 𝗠𝗮𝗴𝗻𝗲𝘁𝘀4 个月前

Give Qwen a chance!

Stan Kirdey 的头像

Stan Kirdey4 个月前

it is tricky, we made a package in rust that is optimized to serve as a tool for agents in pdf filling try it out it should work 100x faster by itself and kimi should be able to drive it no problem

Vinay Kumar 的头像

Vinay Kumar4 个月前

Hey is this coming to benchlocal?

stevibe 的头像

stevibe4 个月前

Yes, I need to upgrade the protocol to support webview, that will actually be way more flexible than it is now. Working on it.

Austin S. Lin 的头像

Austin S. Lin4 个月前

how do opus 4.8 and gpt 5.5 do?

vel ⋈ 🤌🧦 的头像

vel ⋈ 🤌🧦4 个月前

I really like this as a test, great practical use flow. i wonder which would do best if given a purpose built vision model as a helper, or OCR reader as a tool.

stevibe 的头像

stevibe4 个月前

Good take, chaining models to solve the problem, worth a try!

XL-B 的头像

XL-B4 个月前

🤯 That’s great! Amazing

T1000 的头像

T10004 个月前

I wonder how nemotron would perform

stevibe 的头像

stevibe4 个月前

This test is coming to BenchLocal soon!

AW 的头像

AW4 个月前

hey stevibe, pinged you about a paid opp

Maynard Handley 🟦 的头像

Maynard Handley 🟦4 个月前

Do the large US LLMs even CLAIM to have a vision component? I've found that anything I ask them to do with a vision component kinda works, but badly. Examples include - generating an ASCII graph. Kinda sort correct but lines randomly have too many or too few spaces so things aren't aligned. - translate a scanned foreign language PDF. Correct content is created but not laid out in the same way as the original form. This seems like low hanging fruit in terms of substantially improving customer value, even if it isn't going to bump scores in the various language and reasoning driven benchmarks.

SomacoSF 的头像

SomacoSF4 个月前

What results would you get if you start the prompt with instructions on finding the text field boxes: "first, examine the form and ifentify the common single_letter_box text field used on the form for letter placement, place a dot at the center of each of these boxes as your ref

MAA 的头像

MAA4 个月前

Loving it. How can I get this test and test some models

Zero Void 的头像

Zero Void4 个月前

token efficiency is capability is gonna age into a whole benchmark category

相关视频

Cerebras inference is very fast. So fast that it changes how we think about configuring our LLMs for voice agent use cases. Kimi K2.6 is a 1T parameter reasoning model that Cerebras serves at 650 - 1,000 tokens per second (end-to-end throughput), with time to first token metrics as low as 150ms (latency). These numbers are two to three times faster than other similarly capable models. The biggest lever we get from this kind of speed is that we can use the model in reasoning mode, and still have excellent "time to first non-thinking token." This solves a big pain point we have in 2026 for voice agent use cases. Almost all recent innovation in post-training has focused on making models good at reasoning ("test time compute"). This is great, but it makes the user-facing model latency much, much slower. Which is a problem for conversational voice agents. We can run Kimi K2.6 with reasoning turned on, and get responses faster than other models produce with reasoning disabled. On my 30-turn voice agent benchmark, Kimi K2.6 with reasoning enabled ties GPT 5.1 and Haiku 4.5 with reasoning disabled, and is still about 200ms seconds faster! On my primary task agent benchmark, Kimi K2.6 is now the #2 model. It ranks just behind Gemini 3.5 Flash in "high" reasoning mode, and tied with GLM 5, Sonnet 4.6, and GPT 5.4 with reasoning set to "low." But Kimi K2.6 completes each turn in the agent loop in under 500ms. The other four models are all at least 3x slower. (Models only qualify for this benchmark if they can complete task turns at a P50 <4s.) A couple of other things that this speed buys us, for production voice agents: - Tool calls happen fast enough that we don't have to work around tool call latency in our pipeline design. - We can prompt the model to output structured data at the beginning of a response, followed by plain text for voice generation. This opens up possibilities like asking the model to do complex classification/generation tasks that influence the rest of the pipeline. For example, the model could create a detailed style prompt for a steerable TTS model, for each individual conversation turn. And, of course, you can use Kimi K2.6 with reasoning turned off. Cerebras calls this "instant" mode. Here's a video of a Cerebras Kimi K2.6 voice agent with voice-to-voice response time, measured at the client, under 500ms. This is the true response latency as perceived by the user, including all network and audio codec overhead, transcription and turn detection, Kimi K2.6 token generation, and voice generation. 500ms is, effectively, instant. So the Cerebras naming for this mode is a propos. :-)

Cerebras inference is very fast. So fast that it changes how we think about configuring our LLMs for voice agent use cases. Kimi K2.6 is a 1T parameter reasoning model that Cerebras serves at 650 - 1,000 tokens per second (end-to-end throughput), with time to first token metrics as low as 150ms (latency). These numbers are two to three times faster than other similarly capable models. The biggest lever we get from this kind of speed is that we can use the model in reasoning mode, and still have excellent "time to first non-thinking token." This solves a big pain point we have in 2026 for voice agent use cases. Almost all recent innovation in post-training has focused on making models good at reasoning ("test time compute"). This is great, but it makes the user-facing model latency much, much slower. Which is a problem for conversational voice agents. We can run Kimi K2.6 with reasoning turned on, and get responses faster than other models produce with reasoning disabled. On my 30-turn voice agent benchmark, Kimi K2.6 with reasoning enabled ties GPT 5.1 and Haiku 4.5 with reasoning disabled, and is still about 200ms seconds faster! On my primary task agent benchmark, Kimi K2.6 is now the #2 model. It ranks just behind Gemini 3.5 Flash in "high" reasoning mode, and tied with GLM 5, Sonnet 4.6, and GPT 5.4 with reasoning set to "low." But Kimi K2.6 completes each turn in the agent loop in under 500ms. The other four models are all at least 3x slower. (Models only qualify for this benchmark if they can complete task turns at a P50 <4s.) A couple of other things that this speed buys us, for production voice agents: - Tool calls happen fast enough that we don't have to work around tool call latency in our pipeline design. - We can prompt the model to output structured data at the beginning of a response, followed by plain text for voice generation. This opens up possibilities like asking the model to do complex classification/generation tasks that influence the rest of the pipeline. For example, the model could create a detailed style prompt for a steerable TTS model, for each individual conversation turn. And, of course, you can use Kimi K2.6 with reasoning turned off. Cerebras calls this "instant" mode. Here's a video of a Cerebras Kimi K2.6 voice agent with voice-to-voice response time, measured at the client, under 500ms. This is the true response latency as perceived by the user, including all network and audio codec overhead, transcription and turn detection, Kimi K2.6 token generation, and voice generation. 500ms is, effectively, instant. So the Cerebras naming for this mode is a propos. :-)

kwindla

40,593 次观看 • 4 个月前

watch this anon. i gave NVIDIA's biggest model ever a single task. 100 minutes and 440,000 tokens later, it had rendered nothing. not one important thing on the screen. this is Nemotron 3 Ultra. 550 billion parameters, a hybrid Mamba Transformer MoE, the largest model NVIDIA has ever shipped, and they built it specifically for long-running agentic coding. so i handed it exactly that: build a 3D scene from a spec, multiple files, iterate until the tests pass. the same task a frontier model one shotted in minutes. i genuinely wanted to be impressed. it ran for an hour and forty. burned through 440,000 tokens. wrote every file, passed its own tests, and proudly printed "task complete."the browser was blank. the 3D scene never rendered. not once. and the long horizon agentic behavior was genuinely good. it stayed on task the whole hour and forty, wrote real multi-file code, drove its own tools without derailing. it just couldn't turn any of that into something that actually runs. here's the part that gets me. it's a text model, it cannot see its own output. so it sat there looping on a broken vision tool, trying to "look" at the page, hitting error after error, never once reasoning its way out. it declared victory on an empty screen because it had no way to know the screen was empty. to be fair, i genuinely don't know what quant the NIM was serving, so maybe some of that's on the serving, not the model. but the biggest model NVIDIA has ever made, on the exact task it was designed for, couldn't tell it had built nothing in 100 minutes. same task on a local model, below thread👇.

watch this anon. i gave NVIDIA's biggest model ever a single task. 100 minutes and 440,000 tokens later, it had rendered nothing. not one important thing on the screen. this is Nemotron 3 Ultra. 550 billion parameters, a hybrid Mamba Transformer MoE, the largest model NVIDIA has ever shipped, and they built it specifically for long-running agentic coding. so i handed it exactly that: build a 3D scene from a spec, multiple files, iterate until the tests pass. the same task a frontier model one shotted in minutes. i genuinely wanted to be impressed. it ran for an hour and forty. burned through 440,000 tokens. wrote every file, passed its own tests, and proudly printed "task complete."the browser was blank. the 3D scene never rendered. not once. and the long horizon agentic behavior was genuinely good. it stayed on task the whole hour and forty, wrote real multi-file code, drove its own tools without derailing. it just couldn't turn any of that into something that actually runs. here's the part that gets me. it's a text model, it cannot see its own output. so it sat there looping on a broken vision tool, trying to "look" at the page, hitting error after error, never once reasoning its way out. it declared victory on an empty screen because it had no way to know the screen was empty. to be fair, i genuinely don't know what quant the NIM was serving, so maybe some of that's on the serving, not the model. but the biggest model NVIDIA has ever made, on the exact task it was designed for, couldn't tell it had built nothing in 100 minutes. same task on a local model, below thread👇.

Sudo su

32,589 次观看 • 3 个月前

A developer in Hangzhou runs an AI that remembers everything about him for $0.40 a year. No vector database. One file that never grows past 4,000 tokens. He published the whole schema. His version starts from the opposite idea. Memory is not storage. It's a write policy. Six fields. Rewritten every time, never appended: > IDENTITY - who you are, what you build. 300 tokens. Changes monthly at most > STATE - what you're on right now. 400 tokens. Rewritten daily > DECISIONS - what's already settled, so nothing gets re-argued. 800 tokens > CORRECTIONS - every time you said "no, not like that." 600 tokens > PEOPLE - names, roles, who's waiting on what. 500 tokens > DEAD - tried and abandoned, so it never comes back as a suggestion. 400 tokens Three thousand tokens. Ceiling of four. When a section fills, the model rewrites it shorter. Nothing is ever added. Only replaced. Kimi K2.5 bills $0.10 per million cached input tokens. Four thousand tokens a turn is $0.0004. That's 2,500 turns for a dollar. The free tier hands you 1.5 million tokens a day. 375 turns before you pay anything at all. CORRECTIONS is the field nobody builds, and it's the one that does the work. A model that remembers being wrong stops repeating it. Everyone else is paying to search their own history. He pays to keep it short. The bill stopped growing when the file did. Your memory system isn't defined by what it stores. It's defined by what it agrees to delete. The article below is the full build - schema, rewrite prompts, the compaction rule that keeps it under the cap. Save it. You'll want it open in the other tab.

A developer in Hangzhou runs an AI that remembers everything about him for $0.40 a year. No vector database. One file that never grows past 4,000 tokens. He published the whole schema. His version starts from the opposite idea. Memory is not storage. It's a write policy. Six fields. Rewritten every time, never appended: > IDENTITY - who you are, what you build. 300 tokens. Changes monthly at most > STATE - what you're on right now. 400 tokens. Rewritten daily > DECISIONS - what's already settled, so nothing gets re-argued. 800 tokens > CORRECTIONS - every time you said "no, not like that." 600 tokens > PEOPLE - names, roles, who's waiting on what. 500 tokens > DEAD - tried and abandoned, so it never comes back as a suggestion. 400 tokens Three thousand tokens. Ceiling of four. When a section fills, the model rewrites it shorter. Nothing is ever added. Only replaced. Kimi K2.5 bills $0.10 per million cached input tokens. Four thousand tokens a turn is $0.0004. That's 2,500 turns for a dollar. The free tier hands you 1.5 million tokens a day. 375 turns before you pay anything at all. CORRECTIONS is the field nobody builds, and it's the one that does the work. A model that remembers being wrong stops repeating it. Everyone else is paying to search their own history. He pays to keep it short. The bill stopped growing when the file did. Your memory system isn't defined by what it stores. It's defined by what it agrees to delete. The article below is the full build - schema, rewrite prompts, the compaction rule that keeps it under the cap. Save it. You'll want it open in the other tab.

wast3

15,862 次观看 • 1 个月前

i watched gemma 4 12b build something genuinely impressive today, and then loop itself to death right in front of me. the full run is in the video, sped up but completely uncut, watch it to the end and you will catch the exact moment it stops building and starts looping right in the middle of the work. the task was clean, build a single file gravity simulator, n-body physics, orbits, collisions, running locally on one 3090 through an agent. and for ten minutes it was a joy to watch. it reached for a symplectic integrator on its own, the correct one, the kind that keeps orbits stable instead of spiralling out. real gravity with softening, proper orbital velocities, momentum conserved on collision. the physics was right. the thing actually worked. then on the very last step, writing a few tests to prove its own code, it fell into a loop. not a crash, a loop. it started repeating itself and would not stop. ten more minutes, thirty four thousand tokens into a single answer, the same fragments over and over, until i killed it myself. so it's not that gemma can't code. it did the hard part beautifully. it cannot finish. it cannot hold a long task together without unravelling, and finishing is the entire job in agentic work. here's the part that stings. i run this exact task, same harness, same card, on the chinese open models, qwen especially, and i never see this. they build it, they test it, they stop. every single time. google has the raw capability, you can see it sitting right there in the code, and then the model loops itself to death on a task a 27b from alibaba finishes clean. open weights, apache 2.0, so much to love on paper. i just need it to know when to stop talking.

i watched gemma 4 12b build something genuinely impressive today, and then loop itself to death right in front of me. the full run is in the video, sped up but completely uncut, watch it to the end and you will catch the exact moment it stops building and starts looping right in the middle of the work. the task was clean, build a single file gravity simulator, n-body physics, orbits, collisions, running locally on one 3090 through an agent. and for ten minutes it was a joy to watch. it reached for a symplectic integrator on its own, the correct one, the kind that keeps orbits stable instead of spiralling out. real gravity with softening, proper orbital velocities, momentum conserved on collision. the physics was right. the thing actually worked. then on the very last step, writing a few tests to prove its own code, it fell into a loop. not a crash, a loop. it started repeating itself and would not stop. ten more minutes, thirty four thousand tokens into a single answer, the same fragments over and over, until i killed it myself. so it's not that gemma can't code. it did the hard part beautifully. it cannot finish. it cannot hold a long task together without unravelling, and finishing is the entire job in agentic work. here's the part that stings. i run this exact task, same harness, same card, on the chinese open models, qwen especially, and i never see this. they build it, they test it, they stop. every single time. google has the raw capability, you can see it sitting right there in the code, and then the model loops itself to death on a task a 27b from alibaba finishes clean. open weights, apache 2.0, so much to love on paper. i just need it to know when to stop talking.

Sudo su

39,764 次观看 • 4 个月前

A 3B model just cleared a puzzle that a 1.6 TRILLION param model couldn't. You've seen this benchmark before: my sliding-puzzle test. Same Kimi & DeepSeek runs as last time. The only new thing: I dropped VibeThinker-3B in for a side-by-side. > VibeThinker → 3B > DeepSeek V4 Flash → 284B > Kimi K2.6 → 1T > DeepSeek V4 Pro → 1.6T Shuffle depths 5, 10, 12, 15, 18, 22. One wrong move scrambles the whole board, so it's pure long-chain reasoning. ✅ VibeThinker-3B: solved all six. Never lost the thread. ⚠️ The giants started cracking at depth 15: Flash, Pro, and even Kimi each blew a run, scrambling the board past the move cap. As VibeThinker was not trained for tool calling, I had it emit X and ran the move. Bigger generalist ≠ smarter.

A 3B model just cleared a puzzle that a 1.6 TRILLION param model couldn't. You've seen this benchmark before: my sliding-puzzle test. Same Kimi & DeepSeek runs as last time. The only new thing: I dropped VibeThinker-3B in for a side-by-side. > VibeThinker → 3B > DeepSeek V4 Flash → 284B > Kimi K2.6 → 1T > DeepSeek V4 Pro → 1.6T Shuffle depths 5, 10, 12, 15, 18, 22. One wrong move scrambles the whole board, so it's pure long-chain reasoning. ✅ VibeThinker-3B: solved all six. Never lost the thread. ⚠️ The giants started cracking at depth 15: Flash, Pro, and even Kimi each blew a run, scrambling the board past the move cap. As VibeThinker was not trained for tool calling, I had it emit X and ran the move. Bigger generalist ≠ smarter.

stevibe

32,688 次观看 • 3 个月前

Alright guys, wow. Qwen 3.8 27B is basically Qwen 4. What we have here is one of the best one-shot shark survival games I've made with any model. I ran this same prompt through fable, and the biggest thing all models struggle with in this test is the top-down view of the sharks. Fable even had all of the fins reversed and details jumbled on the first turn. Before now, it was really a 2 to 8 turn test to get something final regardless of model. Qwen 3.8 27B just nailed this in one output, no harness at all, just purely the model thoroughly thinking before outputting the final result. On every asset used, different shark variants, everything, it all looks incredibly tight. The boat movement, shark speed and engagement, everything is so much better than I expected, and I was expecting a lot. Imagine what 8 turns would do in a harness? This confirms to me that while Qwen 3.8 27B thinks a lot, for the first time in a local model on the base weights, IT MIGHT ACTUALLY BE WORTH IT This is a Frontier Lab model result, and it ran on my 5090 at 60tps in Q5_K_M. I am just in complete astoundment at what Qwen has accomplished here, and I am incredibly grateful that this model is open source. The speed will also increase drastically with optimisations. This is just an incredible day. What a blessing. More to come!

Alright guys, wow. Qwen 3.8 27B is basically Qwen 4. What we have here is one of the best one-shot shark survival games I've made with any model. I ran this same prompt through fable, and the biggest thing all models struggle with in this test is the top-down view of the sharks. Fable even had all of the fins reversed and details jumbled on the first turn. Before now, it was really a 2 to 8 turn test to get something final regardless of model. Qwen 3.8 27B just nailed this in one output, no harness at all, just purely the model thoroughly thinking before outputting the final result. On every asset used, different shark variants, everything, it all looks incredibly tight. The boat movement, shark speed and engagement, everything is so much better than I expected, and I was expecting a lot. Imagine what 8 turns would do in a harness? This confirms to me that while Qwen 3.8 27B thinks a lot, for the first time in a local model on the base weights, IT MIGHT ACTUALLY BE WORTH IT This is a Frontier Lab model result, and it ran on my 5090 at 60tps in Q5_K_M. I am just in complete astoundment at what Qwen has accomplished here, and I am incredibly grateful that this model is open source. The speed will also increase drastically with optimisations. This is just an incredible day. What a blessing. More to come!

Kyle Hessling

24,917 次观看 • 1 个月前

Qwen3.8-Flash-Next is still going strong at 364.7K tokens of context on an M5 Max. And this isn’t just a static long-context test. The model was reasoning about how to speed up its own workflow while using tools, and the tool calls kept working without misses. Setup: • Qwen3.8-Flash-Next • M5 Max • 128GB unified memory • MLX-Serve PR #363 • OpenCode 2 • 364.7K context The interesting part isn’t simply getting hundreds of thousands of tokens into memory. It’s what happens once the context gets this large. Long-context inference usually comes with a painful tradeoff. As the KV cache grows, memory pressure increases and generation can slow down. But this setup is still pushing through 364K tokens while maintaining a usable agent workflow. The model can reason, call tools, inspect results, continue working, and keep the session moving. And the tool calls reportedly haven’t missed so far. That’s important for agentic coding. A huge context window is only useful if the model can actually operate reliably inside it. A 400K-token context that constantly breaks tool calls isn’t very useful. A 364K session that can keep reasoning and executing tools is a different story. And the test isn’t finished yet. The current run is approaching 400K tokens, with the expectation that it can keep going. This is also another interesting example of why Apple Silicon keeps showing up in local LLM experiments. The M5 Max’s unified memory gives a large model and its growing KV cache access to one shared memory pool. With MLX-Serve continuing to improve, these machines are becoming surprisingly capable long-context inference boxes. The bigger takeaway: Context length is becoming a workload, not just a model specification. Running a model at 256K is one thing. Keeping an agent alive at 300K+ while it reasons and uses tools is much more interesting. And Qwen3.8-Flash-Next is showing that this can be pushed surprisingly far on a single 128GB Mac. 364.7K and counting. Next stop: 400K.

Qwen3.8-Flash-Next is still going strong at 364.7K tokens of context on an M5 Max. And this isn’t just a static long-context test. The model was reasoning about how to speed up its own workflow while using tools, and the tool calls kept working without misses. Setup: • Qwen3.8-Flash-Next • M5 Max • 128GB unified memory • MLX-Serve PR #363 • OpenCode 2 • 364.7K context The interesting part isn’t simply getting hundreds of thousands of tokens into memory. It’s what happens once the context gets this large. Long-context inference usually comes with a painful tradeoff. As the KV cache grows, memory pressure increases and generation can slow down. But this setup is still pushing through 364K tokens while maintaining a usable agent workflow. The model can reason, call tools, inspect results, continue working, and keep the session moving. And the tool calls reportedly haven’t missed so far. That’s important for agentic coding. A huge context window is only useful if the model can actually operate reliably inside it. A 400K-token context that constantly breaks tool calls isn’t very useful. A 364K session that can keep reasoning and executing tools is a different story. And the test isn’t finished yet. The current run is approaching 400K tokens, with the expectation that it can keep going. This is also another interesting example of why Apple Silicon keeps showing up in local LLM experiments. The M5 Max’s unified memory gives a large model and its growing KV cache access to one shared memory pool. With MLX-Serve continuing to improve, these machines are becoming surprisingly capable long-context inference boxes. The bigger takeaway: Context length is becoming a workload, not just a model specification. Running a model at 256K is one thing. Keeping an agent alive at 300K+ while it reasons and uses tools is much more interesting. And Qwen3.8-Flash-Next is showing that this can be pushed surprisingly far on a single 128GB Mac. 364.7K and counting. Next stop: 400K.

FHILY👑

39,982 次观看 • 1 个月前

How to cut your AI bill by 60% switching to Opus 5.5: Most people will swap the model name, save 24%, and stop there. The other 37% is sitting in your settings. Here's the real before and after on an example agent workload. One month: 100M input tokens (80M cached reads, 5M cache writes, 15M uncached) 10M output tokens BEFORE: Opus 5 Cache reads: 80M × $0.50 = $40 Cache writes: 5M × $6.25 = $31.25 Uncached input: 15M × $5 = $75 Output: 10M × $25 = $250 Total: $396.25 STEP 1. Just switch the model. Same tokens, new prices. Cache reads $0.20. Writes $5. Input $4. Output $20. $16 + $25 + $60 + $200 = $301 24% cheaper. That's what everyone's screenshotting. STEP 2. Let it write less. Box measured Opus 5.5 at 40% less verbose with no drop in accuracy. 10M output tokens becomes 6M. Output drops from $200 to $120. Total: $221. Now you're at 44% off. STEP 3. Stop running every call at max effort. Thinking can't be switched off on 5.5 anymore, so effort is your lever. Classifying, routing, formatting, summarizing? Drop the effort. Save the high settings for the calls that actually reason. Say that trims output another 25%, 6M down to 4.5M. Output: $90. Total: $191. 52% off. STEP 4. Cache the stuff you keep resending. This is the one nobody does. Cache reads went from $0.50 to $0.20. That's 60% off the cheapest line on your bill. Move your system prompt, tool definitions and repo context into the cache. Uncached input drops from 15M to 5M, cache reads go up to 90M. $18 + $25 + $20 + $90 = $153 AFTER: $153 From $396.25. 61% cheaper. Same work. The model gave you 24%. You gave yourself the other 37%. Anthropic's own number is 40% cheaper than Opus 5 on typical workloads. Your number depends on how much of your bill is output and how much of your prompt you're resending uncached every call. So check those two first. Bookmark this for when you migrate. follow CyrilXBT

How to cut your AI bill by 60% switching to Opus 5.5: Most people will swap the model name, save 24%, and stop there. The other 37% is sitting in your settings. Here's the real before and after on an example agent workload. One month: 100M input tokens (80M cached reads, 5M cache writes, 15M uncached) 10M output tokens BEFORE: Opus 5 Cache reads: 80M × $0.50 = $40 Cache writes: 5M × $6.25 = $31.25 Uncached input: 15M × $5 = $75 Output: 10M × $25 = $250 Total: $396.25 STEP 1. Just switch the model. Same tokens, new prices. Cache reads $0.20. Writes $5. Input $4. Output $20. $16 + $25 + $60 + $200 = $301 24% cheaper. That's what everyone's screenshotting. STEP 2. Let it write less. Box measured Opus 5.5 at 40% less verbose with no drop in accuracy. 10M output tokens becomes 6M. Output drops from $200 to $120. Total: $221. Now you're at 44% off. STEP 3. Stop running every call at max effort. Thinking can't be switched off on 5.5 anymore, so effort is your lever. Classifying, routing, formatting, summarizing? Drop the effort. Save the high settings for the calls that actually reason. Say that trims output another 25%, 6M down to 4.5M. Output: $90. Total: $191. 52% off. STEP 4. Cache the stuff you keep resending. This is the one nobody does. Cache reads went from $0.50 to $0.20. That's 60% off the cheapest line on your bill. Move your system prompt, tool definitions and repo context into the cache. Uncached input drops from 15M to 5M, cache reads go up to 90M. $18 + $25 + $20 + $90 = $153 AFTER: $153 From $396.25. 61% cheaper. Same work. The model gave you 24%. You gave yourself the other 37%. Anthropic's own number is 40% cheaper than Opus 5 on typical workloads. Your number depends on how much of your bill is output and how much of your prompt you're resending uncached every call. So check those two first. Bookmark this for when you migrate. follow CyrilXBT

CyrilXBT

15,803 次观看 • 19 天前

ox alpha vs deepseek v4 flash vision vs grok 4.6 vs gemini 3.7 flash vs – on photo-to-3d four vision models got one photograph each and had to rebuild the place inside it as a Three.js scene. twelve scenes, twelve first-try runs, zero console errors the setup: one reference photo per scene, sent as an image on OpenRouter. the prompt never says what is in the picture – no "motel", no "bar", no "gas station". the model has to read the photo and rebuild it: layout, materials, hour of the day, and whatever is around the corner that the frame does not show tasks – three photographs of early-2000s america: 1. a motel at night, neon pylon lit, snow on the ground 2. an old new york tavern interior, tin ceiling, tiled floor 3. an abandoned service station in the california desert, midday sun each scene ships as one self-contained html file, procedural geometry and canvas textures only, no downloads. three timed camera shots, and shot 1 has to reproduce the framing of the reference photo models: xAI grok 4.6, Google DeepMind gemini 3.7 flash, DeepSeek deepseek v4 flash vision exp, and ox alpha – a stealth model on openrouter, free, no lab attached to it yet results: - wall clock, three scenes #1 gemini 3.7 flash – 11m 12s #2 deepseek v4 flash – 15m 20s #3 grok 4.6 – 28m 11s #4 ox alpha – 38m 54s - output tokens #1 gemini 3.7 flash – 77,396 #2 ox alpha – 87,613 #3 grok 4.6 – 105,687 #4 deepseek v4 flash – 127,884 - lines of code shipped #1 ox alpha – 2,090 #2 deepseek v4 flash – 2,291 #3 grok 4.6 – 3,529 #4 gemini 3.7 flash – 3,989 - total price #1 ox alpha – $0.000 #2 deepseek v4 flash – $0.091 #3 gemini 3.7 flash – $0.136 #4 grok 4.6 – $0.697 observations: • grok is 7.7x the price of deepseek. it is the only model that read the light – low sun, real shadows on the station, a cold night on the motel • gemini is the fastest and the least deliberate. 17,158 reasoning tokens against deepseek's 99,172, and it still shipped the most code – 3,989 lines • deepseek thought hardest and rendered plainest. 99,172 reasoning tokens, 5.8x gemini's, spent on layout rather than on light. its motel is the second best in the set for $0.030 • ox alpha is free and reads a photo as well as anything here – it lifted "family units / kitchenettes" off the pylon and redrew it in canvas conclusion: twelve scenes, four models, zero fixes, and the whole run cost $0.924! follow thehype. for 24/7 ai news, analysis and breakdowns

ox alpha vs deepseek v4 flash vision vs grok 4.6 vs gemini 3.7 flash vs – on photo-to-3d four vision models got one photograph each and had to rebuild the place inside it as a Three.js scene. twelve scenes, twelve first-try runs, zero console errors the setup: one reference photo per scene, sent as an image on OpenRouter. the prompt never says what is in the picture – no "motel", no "bar", no "gas station". the model has to read the photo and rebuild it: layout, materials, hour of the day, and whatever is around the corner that the frame does not show tasks – three photographs of early-2000s america: 1. a motel at night, neon pylon lit, snow on the ground 2. an old new york tavern interior, tin ceiling, tiled floor 3. an abandoned service station in the california desert, midday sun each scene ships as one self-contained html file, procedural geometry and canvas textures only, no downloads. three timed camera shots, and shot 1 has to reproduce the framing of the reference photo models: xAI grok 4.6, Google DeepMind gemini 3.7 flash, DeepSeek deepseek v4 flash vision exp, and ox alpha – a stealth model on openrouter, free, no lab attached to it yet results: - wall clock, three scenes #1 gemini 3.7 flash – 11m 12s #2 deepseek v4 flash – 15m 20s #3 grok 4.6 – 28m 11s #4 ox alpha – 38m 54s - output tokens #1 gemini 3.7 flash – 77,396 #2 ox alpha – 87,613 #3 grok 4.6 – 105,687 #4 deepseek v4 flash – 127,884 - lines of code shipped #1 ox alpha – 2,090 #2 deepseek v4 flash – 2,291 #3 grok 4.6 – 3,529 #4 gemini 3.7 flash – 3,989 - total price #1 ox alpha – $0.000 #2 deepseek v4 flash – $0.091 #3 gemini 3.7 flash – $0.136 #4 grok 4.6 – $0.697 observations: • grok is 7.7x the price of deepseek. it is the only model that read the light – low sun, real shadows on the station, a cold night on the motel • gemini is the fastest and the least deliberate. 17,158 reasoning tokens against deepseek's 99,172, and it still shipped the most code – 3,989 lines • deepseek thought hardest and rendered plainest. 99,172 reasoning tokens, 5.8x gemini's, spent on layout rather than on light. its motel is the second best in the set for $0.030 • ox alpha is free and reads a photo as well as anything here – it lifted "family units / kitchenettes" off the pylon and redrew it in canvas conclusion: twelve scenes, four models, zero fixes, and the whole run cost $0.924! follow thehype. for 24/7 ai news, analysis and breakdowns

thehype.

24,470 次观看 • 1 个月前

Researchers found a way to make LLMs 8.5x faster! (without compromising accuracy) Speculative decoding is quite an effective way to address the single-token bottleneck in traditional LLM inference. A small "draft" model first generates the next several tokens, then the large model verifies all of them at once in a single forward pass. If a token at any position is wrong, you keep everything before it and restart from there. This never does worse than normal decoding. But current drafters in Speculative decoding still guess one token at a time. That makes the drafting step itself a bottleneck, capping real-world speedups at 2-3x. DFlash is a new technique that swaps the autoregressive drafter with a lightweight block diffusion model that guesses all tokens in one parallel shot. Drafting cost stays flat no matter how many tokens you speculate. On top of that, the drafter is conditioned on hidden features pulled from multiple layers of the target model and injected into every draft layer, so it makes significantly better guesses than a drafter working from scratch. In the side-by-side demo below, vanilla decoding runs at 48.5 tokens/sec. DFlash hits 415 tokens/sec on the same model, with zero quality loss. It's already integrated with vLLM, SGLang, and Transformers, with draft models on HuggingFace for several models like Qwen3, Qwen3.5, Llama 3.1, Kimi-K2.5, gpt-oss, and many more. I have shared the GitHub repo in the replies! KV caching is another must-know technique to boost LLM inference. I recently wrote an article about it. Read it below. 👉 Over to you: What use case are you working on that can benefit from this new technique?

Researchers found a way to make LLMs 8.5x faster! (without compromising accuracy) Speculative decoding is quite an effective way to address the single-token bottleneck in traditional LLM inference. A small "draft" model first generates the next several tokens, then the large model verifies all of them at once in a single forward pass. If a token at any position is wrong, you keep everything before it and restart from there. This never does worse than normal decoding. But current drafters in Speculative decoding still guess one token at a time. That makes the drafting step itself a bottleneck, capping real-world speedups at 2-3x. DFlash is a new technique that swaps the autoregressive drafter with a lightweight block diffusion model that guesses all tokens in one parallel shot. Drafting cost stays flat no matter how many tokens you speculate. On top of that, the drafter is conditioned on hidden features pulled from multiple layers of the target model and injected into every draft layer, so it makes significantly better guesses than a drafter working from scratch. In the side-by-side demo below, vanilla decoding runs at 48.5 tokens/sec. DFlash hits 415 tokens/sec on the same model, with zero quality loss. It's already integrated with vLLM, SGLang, and Transformers, with draft models on HuggingFace for several models like Qwen3, Qwen3.5, Llama 3.1, Kimi-K2.5, gpt-oss, and many more. I have shared the GitHub repo in the replies! KV caching is another must-know technique to boost LLM inference. I recently wrote an article about it. Read it below. 👉 Over to you: What use case are you working on that can benefit from this new technique?

Avi Chawla

158,352 次观看 • 5 个月前

Perplexity CEO Aravind Srinivas on the brutal truth about who actually makes money in AI (and why it's not who you think): Aravind argues that the real value in AI comes from orchestration. He points to products like Codex, Claude Code, and Perplexity Computer: "What is that? It's an orchestration system. It takes a model, pairs it with an agent harness." And what is an agent harness? "The simplest way of describing it is like rules for how the agent loop should run. What are all the skills and sub-agents and connectors and tools it accesses? Without the harness, you don't necessarily capture and convert the intrinsic intelligence in the model into valuable output tokens." This leads to a blunt conclusion about who has a real business in AI, and who doesn't: "If you're literally just a reseller of model tokens, you have no business, because the model will get commoditized. So even if you're a model builder, you don't have a business. As an infra layer, you have some business on serving those output tokens. But as an application layer or model builder, you don't really have a business if you're just a reseller of tokens that come directly out of the model." So where does the value accrue? "You have a business if you know how to take the model, ground it in valuable context, orchestrate it with a really good agent harness, connected to the right set of tools and connectors (whether it's personal connectors or business connectors) and provide the experience to people in one single unified system." Aravind Srinivas then explains Perplexity's specific edge: Beyond orchestrating across tools, files, and connectors, they also orchestrate across models. "That is the differentiation that Anthropic and OpenAI cannot claim, because you wouldn't find GPT-5 inside the Claude Code harness. You wouldn't find Claude Opus inside the Codex harness. These are competing with each other. Whereas you would find both these models inside Perplexity Computer." Why does this matter? Because it all comes down to power. In Aravind's framing, the fundamental cost driver in AI is watts (the one input nobody can subsidize except the government). "Whoever provides the most valuable output tokens with the least amount of power expended to produce them generates the greatest value to the end user, has the most pricing power, has the most value. That is the orchestration problem to solve." His conclusion: "The one single most important metric in AI is token value per watt per user."

Perplexity CEO Aravind Srinivas on the brutal truth about who actually makes money in AI (and why it's not who you think): Aravind argues that the real value in AI comes from orchestration. He points to products like Codex, Claude Code, and Perplexity Computer: "What is that? It's an orchestration system. It takes a model, pairs it with an agent harness." And what is an agent harness? "The simplest way of describing it is like rules for how the agent loop should run. What are all the skills and sub-agents and connectors and tools it accesses? Without the harness, you don't necessarily capture and convert the intrinsic intelligence in the model into valuable output tokens." This leads to a blunt conclusion about who has a real business in AI, and who doesn't: "If you're literally just a reseller of model tokens, you have no business, because the model will get commoditized. So even if you're a model builder, you don't have a business. As an infra layer, you have some business on serving those output tokens. But as an application layer or model builder, you don't really have a business if you're just a reseller of tokens that come directly out of the model." So where does the value accrue? "You have a business if you know how to take the model, ground it in valuable context, orchestrate it with a really good agent harness, connected to the right set of tools and connectors (whether it's personal connectors or business connectors) and provide the experience to people in one single unified system." Aravind Srinivas then explains Perplexity's specific edge: Beyond orchestrating across tools, files, and connectors, they also orchestrate across models. "That is the differentiation that Anthropic and OpenAI cannot claim, because you wouldn't find GPT-5 inside the Claude Code harness. You wouldn't find Claude Opus inside the Codex harness. These are competing with each other. Whereas you would find both these models inside Perplexity Computer." Why does this matter? Because it all comes down to power. In Aravind's framing, the fundamental cost driver in AI is watts (the one input nobody can subsidize except the government). "Whoever provides the most valuable output tokens with the least amount of power expended to produce them generates the greatest value to the end user, has the most pricing power, has the most value. That is the orchestration problem to solve." His conclusion: "The one single most important metric in AI is token value per watt per user."

Big Brain AI

42,540 次观看 • 2 个月前

your agent reviewing its own work is not a check. it is a second opinion from the same source. this is the most common gap in agent systems and it hides in plain sight, because the step exists. there is a review. it just cannot do the thing you think it does. here is the mechanism. the model produced an output from a context. you then ask the same model, holding the same context, whether that output is correct. it answers fluently, because that is what it does. and the answer is drawn from the same distribution that produced the thing being judged. same weights, same window, same blind spots. if the reason the output is wrong is something the model does not know, the review does not know it either. if the reason is something the context does not contain, the review has the same context. the failure mode and the detector share a cause. > why it feels like it works because most of the time the output is fine, and the review says fine. agreement is not evidence of detection. a reviewer that says pass on everything agrees with reality most of the time too. what you actually want to measure is what happens on the cases that are wrong. that is the only place a check earns its name, and it is exactly the place where a self-review is weakest. there is research on this. Huang and colleagues at DeepMind showed at ICLR 2024 that intrinsic self-correction, revising without external grounding, does not reliably help and often makes things worse. > what to actually do move the check outside the model. a test that runs, a schema that validates, a file that exists or does not, an exit code from something you did not write. these are not smarter than the model. they are just not correlated with it, and that is the entire value. when the judgement genuinely needs a model, at minimum use a different family. same family means shared blind spots, and frontier judges measurably inflate scores for outputs that look like their own. and split the work by kind. anything objectively checkable goes to code. only the genuinely semantic calls go to a judge, and those get a rubric written as one line. a review inside the loop tells you the model is confident. a check outside it tells you whether the work is done. save this - then read the eval setup below

your agent reviewing its own work is not a check. it is a second opinion from the same source. this is the most common gap in agent systems and it hides in plain sight, because the step exists. there is a review. it just cannot do the thing you think it does. here is the mechanism. the model produced an output from a context. you then ask the same model, holding the same context, whether that output is correct. it answers fluently, because that is what it does. and the answer is drawn from the same distribution that produced the thing being judged. same weights, same window, same blind spots. if the reason the output is wrong is something the model does not know, the review does not know it either. if the reason is something the context does not contain, the review has the same context. the failure mode and the detector share a cause. > why it feels like it works because most of the time the output is fine, and the review says fine. agreement is not evidence of detection. a reviewer that says pass on everything agrees with reality most of the time too. what you actually want to measure is what happens on the cases that are wrong. that is the only place a check earns its name, and it is exactly the place where a self-review is weakest. there is research on this. Huang and colleagues at DeepMind showed at ICLR 2024 that intrinsic self-correction, revising without external grounding, does not reliably help and often makes things worse. > what to actually do move the check outside the model. a test that runs, a schema that validates, a file that exists or does not, an exit code from something you did not write. these are not smarter than the model. they are just not correlated with it, and that is the entire value. when the judgement genuinely needs a model, at minimum use a different family. same family means shared blind spots, and frontier judges measurably inflate scores for outputs that look like their own. and split the work by kind. anything objectively checkable goes to code. only the genuinely semantic calls go to a judge, and those get a rubric written as one line. a review inside the loop tells you the model is confident. a check outside it tells you whether the work is done. save this - then read the eval setup below

Hanako

14,325 次观看 • 2 个月前

A tricky LLM interview question: You're serving a reasoning model on vLLM, and it keeps running out of GPU memory on long traces. So you add KV cache compression and evict 90% of the cached tokens. VRAM usage stays as is and GPU still runs out of memory. Why? (answer below) Evicting 90% of the KV cache can free almost none of the memory it was using. This sounds counterintuitive, but it follows directly from how production servers store the cache today. The KV cache grows with every token a model generates. Each token appends its key and value vectors across every layer, and nothing is freed while generation continues. This is the dominant memory cost for reasoning models. If a 32K-token CoT caches ~32K tokens of KV vectors, a Qwen3-32B with 4-bit weights will run out-of-memory around 24K tokens on a 24GB GPU. One obvious solution is to keep the important tokens and drop the rest, since attention is sparse enough to allow it. But this does not solve the memory problem yet. The reason is paged attention, which is the memory manager behind vLLM and most production servers. Under the hood, it splits GPU memory into fixed physical blocks, each one holds the KV for about 16 tokens. This block returns to the allocator only when every slot inside it is empty. Since the eviction logic selects tokens by importance, and such tokens are scattered across blocks... ...so despite eviction, almost every block is left with at least some survivor tokens. For instance, if the logic evicts 14k of 16k tokens across 1,000 blocks, most likely every block will still have a token. This means the allocator frees almost nothing. Placing the new tokens into those freed slots is not ideal because it breaks the cache's layout. Say token 16,001 arrives, and it's placed in the slot the 40th token used to hold. The cache now reads position 38, then 16,001, then 41, so the cache is no longer in token order. Attention can still compute the right answer from that, but only if every slot now carries a separate note recording which position it actually holds. This introduces another bookkeeping cost that an in-order layout inherently avoids. So the cache is logically 90% smaller and still physically the same size. Many compression results miss this because they measure on pre-allocated contiguous tensors rather than a paged server. There's another problem. Eviction methods pick which tokens to keep by looking at the attention scores themselves (as expected). But fast attention kernels used in production, like FlashAttention, never save those scores. They compute attention in small pieces and throw the full score grid away as they go, which is also why they're fast. So the exact signal eviction methods need isn't available in memory. The workaround is to fall back to eager attention and build the full matrix, which gives up the speed FlashAttention was there to provide. NVIDIA published a method called TriAttention to solve both these problems. It never needs attention scores. Instead, it scores tokens from the geometry of the model's key and query vectors before RoPE is applied, where those vectors sit in stable clusters. For the memory problem, it runs a compaction pass every 128 decoded tokens. The surviving tokens slide forward to close the holes eviction creates, so whole blocks empty out and return to the allocator while the cache stays in token order. On long reasoning traces, the approach matches full-attention accuracy while decoding 2.5x faster and using 10.7x less KV memory. KV cache compression is a big infrastructure problem. The number that decides whether it works is the count of freed blocks, not the count of evicted tokens. You can find the NVIDIA write-up here: I wrote a first-principles breakdown of how the KV cache works. It walks through why the model stores keys and values at all, why the cache grows with every token, and a comparison of LLM generation speed with and without KV caching. Read it below.

A tricky LLM interview question: You're serving a reasoning model on vLLM, and it keeps running out of GPU memory on long traces. So you add KV cache compression and evict 90% of the cached tokens. VRAM usage stays as is and GPU still runs out of memory. Why? (answer below) Evicting 90% of the KV cache can free almost none of the memory it was using. This sounds counterintuitive, but it follows directly from how production servers store the cache today. The KV cache grows with every token a model generates. Each token appends its key and value vectors across every layer, and nothing is freed while generation continues. This is the dominant memory cost for reasoning models. If a 32K-token CoT caches ~32K tokens of KV vectors, a Qwen3-32B with 4-bit weights will run out-of-memory around 24K tokens on a 24GB GPU. One obvious solution is to keep the important tokens and drop the rest, since attention is sparse enough to allow it. But this does not solve the memory problem yet. The reason is paged attention, which is the memory manager behind vLLM and most production servers. Under the hood, it splits GPU memory into fixed physical blocks, each one holds the KV for about 16 tokens. This block returns to the allocator only when every slot inside it is empty. Since the eviction logic selects tokens by importance, and such tokens are scattered across blocks... ...so despite eviction, almost every block is left with at least some survivor tokens. For instance, if the logic evicts 14k of 16k tokens across 1,000 blocks, most likely every block will still have a token. This means the allocator frees almost nothing. Placing the new tokens into those freed slots is not ideal because it breaks the cache's layout. Say token 16,001 arrives, and it's placed in the slot the 40th token used to hold. The cache now reads position 38, then 16,001, then 41, so the cache is no longer in token order. Attention can still compute the right answer from that, but only if every slot now carries a separate note recording which position it actually holds. This introduces another bookkeeping cost that an in-order layout inherently avoids. So the cache is logically 90% smaller and still physically the same size. Many compression results miss this because they measure on pre-allocated contiguous tensors rather than a paged server. There's another problem. Eviction methods pick which tokens to keep by looking at the attention scores themselves (as expected). But fast attention kernels used in production, like FlashAttention, never save those scores. They compute attention in small pieces and throw the full score grid away as they go, which is also why they're fast. So the exact signal eviction methods need isn't available in memory. The workaround is to fall back to eager attention and build the full matrix, which gives up the speed FlashAttention was there to provide. NVIDIA published a method called TriAttention to solve both these problems. It never needs attention scores. Instead, it scores tokens from the geometry of the model's key and query vectors before RoPE is applied, where those vectors sit in stable clusters. For the memory problem, it runs a compaction pass every 128 decoded tokens. The surviving tokens slide forward to close the holes eviction creates, so whole blocks empty out and return to the allocator while the cache stays in token order. On long reasoning traces, the approach matches full-attention accuracy while decoding 2.5x faster and using 10.7x less KV memory. KV cache compression is a big infrastructure problem. The number that decides whether it works is the count of freed blocks, not the count of evicted tokens. You can find the NVIDIA write-up here: I wrote a first-principles breakdown of how the KV cache works. It walks through why the model stores keys and values at all, why the cache grows with every token, and a comparison of LLM generation speed with and without KV caching. Read it below.

Avi Chawla

273,525 次观看 • 3 个月前

New skill: self-managed-context (make the agent's context an editable file) It explains how to build agents that decide what to keep, update, or remove from the information they use to do their work. It can archive a long log while keeping the exact error, update its progress notes, or remove outdated information. Those edits then change what the model sees on its next turn. 1- Keep the system instructions and original task protected, outside the editable file. 2- Write the remaining conversation to a file, with labels for each message. 3- Let the agent edit that file using its usual code tools. 4- After each command, read the file back and use the updated messages for the next model call. Loading the skill ( alone into a fixed harness won't create live context editing, but it can help an agent build and then operate a harness that supports it. I gave the skill to a coding agent and had it build the harness itself. The task is a long stream of server logs that doesn't fit in the window. The agent reads it in 18 chunks, about 10k tokens in total, with a 5.5k budget. It has to report one incident ticket exactly and the final value of every config key. Same model & budget, three setups: 1- Model manages its own context 2- Harness forces a summary at 75% full 3- Keeps everything The video shows a real GPT-5.4 run. - Self-managed solved it 3 out of 3. - Keep-everything overflowed 3 out of 3. - Forced summary also solved it 3 out of 3. On GPT-5.4 the self-managed agent re-processed about 24% fewer prompt tokens than the forced summary. When it edited, it cut hard, so little was left after the edit to re-process (one edit took 5,537 tokens down to 771). On GPT-4.1 it saved nothing. It edited near the top of its context but kept most of what was below, and every edit forces everything after it to be re-processed. This is a small test at about 2x context pressure. The paper goes up to 24x, but imho the video below and the skill are a good way to start understanding the technique.

New skill: self-managed-context (make the agent's context an editable file) It explains how to build agents that decide what to keep, update, or remove from the information they use to do their work. It can archive a long log while keeping the exact error, update its progress notes, or remove outdated information. Those edits then change what the model sees on its next turn. 1- Keep the system instructions and original task protected, outside the editable file. 2- Write the remaining conversation to a file, with labels for each message. 3- Let the agent edit that file using its usual code tools. 4- After each command, read the file back and use the updated messages for the next model call. Loading the skill ( alone into a fixed harness won't create live context editing, but it can help an agent build and then operate a harness that supports it. I gave the skill to a coding agent and had it build the harness itself. The task is a long stream of server logs that doesn't fit in the window. The agent reads it in 18 chunks, about 10k tokens in total, with a 5.5k budget. It has to report one incident ticket exactly and the final value of every config key. Same model & budget, three setups: 1- Model manages its own context 2- Harness forces a summary at 75% full 3- Keeps everything The video shows a real GPT-5.4 run. - Self-managed solved it 3 out of 3. - Keep-everything overflowed 3 out of 3. - Forced summary also solved it 3 out of 3. On GPT-5.4 the self-managed agent re-processed about 24% fewer prompt tokens than the forced summary. When it edited, it cut hard, so little was left after the edit to re-process (one edit took 5,537 tokens down to 771). On GPT-4.1 it saved nothing. It edited near the top of its context but kept most of what was below, and every edit forces everything after it to be re-processed. This is a small test at about 2x context pressure. The paper goes up to 24x, but imho the video below and the skill are a good way to start understanding the technique.

Muratcan Koylan

15,070 次观看 • 11 天前

Claude Haiku 5.5 just LOST to a Qwen3.8 27B running on 2 used 3090s in my apartment. Same prompt. Same shooter. Haiku got every advantage: max effort, 1M context, Anthropic's datacenter. Haiku 5.5: 107.3M tokens in, 1.25M out 58 minutes $7.85 at API prices Qwen3.8 27B: 87.9M tokens in, 822k out 6 hours 9 minutes $0 and an electricity bill The Haiku build is polished. Nice menu, tidy HUD, sunset, light shafts, bloom. Everything is in its place. And it is dull. Flat. No punch, no tension, nothing that makes you want one more wave. The Qwen build is rougher and MUCH better. That's the one I sat and actually played. And this is the newest model from a frontier lab. My Qwen3.8 runs on my own cards and asks NOTHING from me except electricity. No API key. No rate limits. No invoice. Nothing leaves the flat. Haiku burned 22% more input and 52% more output to get there. 82% of that output was thinking. A million tokens of reasoning about how to make a shooter, and it forgot the fun. And that "$0.10 per million" sticker price? Context crossed 100K almost immediately, so 208 of 215 requests were billed at the 5x long-prompt rate. It peaked at 732K. Yes, Haiku was 6x faster. It got to boring 6x sooner. A frontier lab's datacenter lost to weights I downloaded for free. Both builds in the reply. Play them and tell me I'm wrong. Link on both builds in the reply ⬇️

Claude Haiku 5.5 just LOST to a Qwen3.8 27B running on 2 used 3090s in my apartment. Same prompt. Same shooter. Haiku got every advantage: max effort, 1M context, Anthropic's datacenter. Haiku 5.5: 107.3M tokens in, 1.25M out 58 minutes $7.85 at API prices Qwen3.8 27B: 87.9M tokens in, 822k out 6 hours 9 minutes $0 and an electricity bill The Haiku build is polished. Nice menu, tidy HUD, sunset, light shafts, bloom. Everything is in its place. And it is dull. Flat. No punch, no tension, nothing that makes you want one more wave. The Qwen build is rougher and MUCH better. That's the one I sat and actually played. And this is the newest model from a frontier lab. My Qwen3.8 runs on my own cards and asks NOTHING from me except electricity. No API key. No rate limits. No invoice. Nothing leaves the flat. Haiku burned 22% more input and 52% more output to get there. 82% of that output was thinking. A million tokens of reasoning about how to make a shooter, and it forgot the fun. And that "$0.10 per million" sticker price? Context crossed 100K almost immediately, so 208 of 215 requests were billed at the 5x long-prompt rate. It peaked at 732K. Yes, Haiku was 6x faster. It got to boring 6x sooner. A frontier lab's datacenter lost to weights I downloaded for free. Both builds in the reply. Play them and tell me I'm wrong. Link on both builds in the reply ⬇️

Alexey Fateev

13,044 次观看 • 4 天前

A developer figured out how to give AI permanent memory for $0.40 a year. No complex vector databases. No massive cloud storage. Just a single 4,000-token file that refuses to grow. The secret? True memory isn’t about storing everything. It’s about aggressive deletion. Instead of endlessly appending data, his system rewrites just six core fields on every turn: > IDENTITY: Who you are and what you build (300 tokens) > STATE: Current active task and focus (400 tokens) > DECISIONS: Settled choices—zero re-arguing (800 tokens) > CORRECTIONS: Every "no, do it this way" moment (600 tokens) > PEOPLE: Key names, roles, and open dependencies (500 tokens) > GRAVEYARD: Dead attempts, so bad ideas never return (400 tokens) Total size: 3,000 tokens. Absolute hard ceiling: 4,000 tokens. When a category hits its limit, the model compresses it. Data is never stacked—only pruned and replaced. Using Kimi K2.5 caching rates ($0.10/M tokens), running this budget memory costs $0.0004 per turn. That is 2,500 interactive turns for a single dollar. The real breakthrough lies in the CORRECTIONS field. Standard AI models waste compute making the same mistakes twice. This schema forces the assistant to remember its failures instantly. While the rest of the industry burns millions searching through bloated history logs, he pays pennies to keep his context razor-sharp. Your memory system isn't defined by what you save. It’s defined by what you have the discipline to throw away. Bookmark this breakdown. You will need it for your next build.

A developer figured out how to give AI permanent memory for $0.40 a year. No complex vector databases. No massive cloud storage. Just a single 4,000-token file that refuses to grow. The secret? True memory isn’t about storing everything. It’s about aggressive deletion. Instead of endlessly appending data, his system rewrites just six core fields on every turn: > IDENTITY: Who you are and what you build (300 tokens) > STATE: Current active task and focus (400 tokens) > DECISIONS: Settled choices—zero re-arguing (800 tokens) > CORRECTIONS: Every "no, do it this way" moment (600 tokens) > PEOPLE: Key names, roles, and open dependencies (500 tokens) > GRAVEYARD: Dead attempts, so bad ideas never return (400 tokens) Total size: 3,000 tokens. Absolute hard ceiling: 4,000 tokens. When a category hits its limit, the model compresses it. Data is never stacked—only pruned and replaced. Using Kimi K2.5 caching rates ($0.10/M tokens), running this budget memory costs $0.0004 per turn. That is 2,500 interactive turns for a single dollar. The real breakthrough lies in the CORRECTIONS field. Standard AI models waste compute making the same mistakes twice. This schema forces the assistant to remember its failures instantly. While the rest of the industry burns millions searching through bloated history logs, he pays pennies to keep his context razor-sharp. Your memory system isn't defined by what you save. It’s defined by what you have the discipline to throw away. Bookmark this breakdown. You will need it for your next build.

shmidt

17,304 次观看 • 1 个月前

How good is GPT-4-Vision at extracting text from images? I wanted to find the limit - but I found weirdness instead Most surprising: GPT-4V performance varies depending on the *structure* of text it sees Let me explain A set of images with progressively more text was presented to GPT-4-Vision. GPT-4V was asked what text it saw in the image. The response from the model was compared against the image’s original text and scored for similarity. The model was tested with 4 types of text: essay, random words, random tokens, and random characters. Findings: * Performance degrades - Yes, the models are good at basic OCR, but as you get more text and words then performance drops (this is expected) * Type of context matters - You should expect different recall on your texts based on your context types * Hallucination Errors - I thought that the model would make errors of omission (it wouldn’t return all the words). But instead the model mostly made hallucination errors - it replaced words with made up words. * Evals Matter - This test in isolation doesn’t mean that your data will have the same results, but it should motivate you to create eval tests for your data and anticipate errors which are hard to spot Notes: * Next step would be to add additional image types like tables or PDFs * GPT-4V would routinely get stuck in repeat-token-loops when trying to extract random tokens * GPT-4V would refuse to answer most random character images

How good is GPT-4-Vision at extracting text from images? I wanted to find the limit - but I found weirdness instead Most surprising: GPT-4V performance varies depending on the structure of text it sees Let me explain A set of images with progressively more text was presented to GPT-4-Vision. GPT-4V was asked what text it saw in the image. The response from the model was compared against the image’s original text and scored for similarity. The model was tested with 4 types of text: essay, random words, random tokens, and random characters. Findings: * Performance degrades - Yes, the models are good at basic OCR, but as you get more text and words then performance drops (this is expected) * Type of context matters - You should expect different recall on your texts based on your context types * Hallucination Errors - I thought that the model would make errors of omission (it wouldn’t return all the words). But instead the model mostly made hallucination errors - it replaced words with made up words. * Evals Matter - This test in isolation doesn’t mean that your data will have the same results, but it should motivate you to create eval tests for your data and anticipate errors which are hard to spot Notes: * Next step would be to add additional image types like tables or PDFs * GPT-4V would routinely get stuck in repeat-token-loops when trying to extract random tokens * GPT-4V would refuse to answer most random character images

Greg Kamradt

49,111 次观看 • 2 年前

$I just compared Claude Code vs Codex vs Cursor CLI The task was to build a Next.js app with Tailwind 4 and shadcn components to collect customer feedback and showcase it with a widget. I gave all three the same prompt and let them go for 30 minutes to see what they came up with. Claude Code with Opus 4.1 Even though I told it to set up the app in the existing project folder, it tried to create a directory for it. After I interrupted and told it not to do that, it built a demo form and landing page with no errors. I had to ask it to make the demo interactive so users could submit a testimonial and preview it. The landing page looked like AI and was pretty basic, but it worked and it was done in a fraction of the time of the others. Total tokens used: 33k Codex with GPT-5 At the end of the 30 minutes I just could not get Codex to produce a working app. It got stuck in a loop of not being able to set up Tailwind 4 and despite many, MANY, attempts, I ended up with a "failed to compile" error. Total tokens used: 102k Cursor Agent with GPT-5 This was the slowest agent by far and a couple of times I actually thought it got stuck in a loop and was close to Ctrl+C'ing to cancel it. The TUI is really nice though, especially how it shows diffs and it did eventually build a working app (after one or two slight errors that needed fixing) The demo was interactive and it had a very minimal design that looked bare but also a lot less like an "AI generated" app than the Opus 4.1 design. It also wasn't too chatty and just did what it needed to do! Code quality was on a par with Opus 4.1, but it did use 5.5x as many tokens to get there. Still cheaper than Opus on a direct comparison but not when you factor in a Claude Code Max subscription. Total tokens: 188k I'll be able to do a proper comparison and record some videos when I'm back from holiday but for now, Opus is still the more capable model out of the box and Claude Code is the more complete CLI product. It will be interesting to see how Cursor evolve their CLI though with commands and subagents because I think with GPT-5 they have a real shot at providing competition for Claude Code if they can optimise output to get similar quality with less tokens. Jump to 0:40 in the video to see the two apps. Which do you think is which? ;)$

I just compared Claude Code vs Codex vs Cursor CLI The task was to build a Next.js app with Tailwind 4 and shadcn components to collect customer feedback and showcase it with a widget. I gave all three the same prompt and let them go for 30 minutes to see what they came up with. Claude Code with Opus 4.1 Even though I told it to set up the app in the existing project folder, it tried to create a directory for it. After I interrupted and told it not to do that, it built a demo form and landing page with no errors. I had to ask it to make the demo interactive so users could submit a testimonial and preview it. The landing page looked like AI and was pretty basic, but it worked and it was done in a fraction of the time of the others. Total tokens used: 33k Codex with GPT-5 At the end of the 30 minutes I just could not get Codex to produce a working app. It got stuck in a loop of not being able to set up Tailwind 4 and despite many, MANY, attempts, I ended up with a "failed to compile" error. Total tokens used: 102k Cursor Agent with GPT-5 This was the slowest agent by far and a couple of times I actually thought it got stuck in a loop and was close to Ctrl+C'ing to cancel it. The TUI is really nice though, especially how it shows diffs and it did eventually build a working app (after one or two slight errors that needed fixing) The demo was interactive and it had a very minimal design that looked bare but also a lot less like an "AI generated" app than the Opus 4.1 design. It also wasn't too chatty and just did what it needed to do! Code quality was on a par with Opus 4.1, but it did use 5.5x as many tokens to get there. Still cheaper than Opus on a direct comparison but not when you factor in a Claude Code Max subscription. Total tokens: 188k I'll be able to do a proper comparison and record some videos when I'm back from holiday but for now, Opus is still the more capable model out of the box and Claude Code is the more complete CLI product. It will be interesting to see how Cursor evolve their CLI though with commands and subagents because I think with GPT-5 they have a real shot at providing competition for Claude Code if they can optimise output to get similar quality with less tokens. Jump to 0:40 in the video to see the two apps. Which do you think is which? ;)

Ian Nuttall

195,173 次观看 • 1 年前

Anthropic put a model on a bacterium dragged out of a place nothing should survive, and it turns out the thing feeds on an element that kills almost everything else. What it eats is only half of it. The other half is that it does the job faster than the organisms we engineer for the same task. > ISOLATE - the strain separated from the site, no lab-friendly conditions added > FEED - every input it accepts tested one at a time, including the toxic ones > TRACE - each element followed through the cell to see what it becomes > SWAP - the pathway compared against the version every other microbe uses > LOAD - the strain pushed until output stops, to find where the limit sits > MATCH - the result measured against the engineered strains built for this job Toxicity is usually a wall. An element that wrecks a normal cell stops the process before anything useful comes out. This one built a route around the damage. The same element goes in, and the pathway simply refuses to break. Nobody designed that. It evolved somewhere the alternative was not surviving at all. The engineered strains we use for this cost years of work and still run slower than something that was already there. That gap matters more than the discovery itself. Nature solved it under pressure nobody would choose to reproduce in a lab. Most environments like this one are written off before sampling, because nothing is expected to be alive in them. Every input it accepts, the pathway map and the throughput numbers sit in the breakdown below.

Anthropic put a model on a bacterium dragged out of a place nothing should survive, and it turns out the thing feeds on an element that kills almost everything else. What it eats is only half of it. The other half is that it does the job faster than the organisms we engineer for the same task. > ISOLATE - the strain separated from the site, no lab-friendly conditions added > FEED - every input it accepts tested one at a time, including the toxic ones > TRACE - each element followed through the cell to see what it becomes > SWAP - the pathway compared against the version every other microbe uses > LOAD - the strain pushed until output stops, to find where the limit sits > MATCH - the result measured against the engineered strains built for this job Toxicity is usually a wall. An element that wrecks a normal cell stops the process before anything useful comes out. This one built a route around the damage. The same element goes in, and the pathway simply refuses to break. Nobody designed that. It evolved somewhere the alternative was not surviving at all. The engineered strains we use for this cost years of work and still run slower than something that was already there. That gap matters more than the discovery itself. Nature solved it under pressure nobody would choose to reproduce in a lab. Most environments like this one are written off before sampling, because nothing is expected to be alive in them. Every input it accepts, the pathway map and the throughput numbers sit in the breakdown below.

wast3

23,339 次观看 • 14 天前

This inference engine runs LLMs 4x faster than llama.cpp: (Qwen3.5-9B at 92 tokens/s on 16GB Mac) I ran Qwen3.5 9B on a base M5 MacBook Pro using three local inference engines, one at a time. → llama.cpp reached 22.0 tokens/s → MLX reached 25.1 tokens/s → Uzu reached 92.1 tokens/s The video shows all three runs side by side. The first two results do not mean llama.cpp or MLX are poorly optimized. Both are approaching a physical limit of ordinary local decoding. Generating each new token normally requires another pass through the model. That pass streams roughly 5.2 GB of weights through memory. A base M5 provides about 153 GB/s of memory bandwidth. At one token per pass, the theoretical ceiling is roughly 153 ÷ 5.2, or 29 tokens/s, however fast the GPU computes. Uzu gets past that ceiling by optimizing speculative decoding for Apple Silicon. Speculative decoding uses a smaller drafting system to propose several future tokens. The full 9B model then checks those proposals together instead of generating every token through a separate pass. Uzu pushes this further in two ways. → It drafts up to 16 positions and uses Weaver to turn independent guesses into coherent sequences. → It verifies several possible sequences as a tree, then commits state only for the path the full model accepts. The 9B model remains in control of the final output. Uzu simply gives it more useful work to approve in each pass. In my run, every pass of the 9B model produced 7.5 output tokens on average. Ordinary decoding produces exactly 1. That is how Uzu reached 92.1 tokens/s on hardware whose one-token-per-pass ceiling is roughly 29. The rest of the engine is designed around this path too. Uzu ships its own quantized checkpoints, and on M5 chips its Metal kernels run verification through the GPU’s accelerated int8 matrix path. It is an open-source inference engine from Mirai, with bindings for Rust, Swift, Python, and TypeScript. The repository is here: (don't forget to star 🌟) I wrote the full breakdown of why local inference is memory-bound and how Uzu gets past that limit, with reproducible results and code. The article is quoted below.

This inference engine runs LLMs 4x faster than llama.cpp: (Qwen3.5-9B at 92 tokens/s on 16GB Mac) I ran Qwen3.5 9B on a base M5 MacBook Pro using three local inference engines, one at a time. → llama.cpp reached 22.0 tokens/s → MLX reached 25.1 tokens/s → Uzu reached 92.1 tokens/s The video shows all three runs side by side. The first two results do not mean llama.cpp or MLX are poorly optimized. Both are approaching a physical limit of ordinary local decoding. Generating each new token normally requires another pass through the model. That pass streams roughly 5.2 GB of weights through memory. A base M5 provides about 153 GB/s of memory bandwidth. At one token per pass, the theoretical ceiling is roughly 153 ÷ 5.2, or 29 tokens/s, however fast the GPU computes. Uzu gets past that ceiling by optimizing speculative decoding for Apple Silicon. Speculative decoding uses a smaller drafting system to propose several future tokens. The full 9B model then checks those proposals together instead of generating every token through a separate pass. Uzu pushes this further in two ways. → It drafts up to 16 positions and uses Weaver to turn independent guesses into coherent sequences. → It verifies several possible sequences as a tree, then commits state only for the path the full model accepts. The 9B model remains in control of the final output. Uzu simply gives it more useful work to approve in each pass. In my run, every pass of the 9B model produced 7.5 output tokens on average. Ordinary decoding produces exactly 1. That is how Uzu reached 92.1 tokens/s on hardware whose one-token-per-pass ceiling is roughly 29. The rest of the engine is designed around this path too. Uzu ships its own quantized checkpoints, and on M5 chips its Metal kernels run verification through the GPU’s accelerated int8 matrix path. It is an open-source inference engine from Mirai, with bindings for Rust, Swift, Python, and TypeScript. The repository is here: (don't forget to star 🌟) I wrote the full breakdown of why local inference is memory-bound and how Uzu gets past that limit, with reproducible results and code. The article is quoted below.

Akshay 🚀

79,041 次观看 • 3 天前