Qwen3.5 27B vs Gemma4 31B | Canvas Creativity Test... Why HTML Canvas? Two reasons: 1. It's unforgiving, one small mistake and the whole thing breaks 2. We kept prompts short to test real creativity, not instruction following 4 rounds: - Analog Clock - Hyperspace Tunnel - Growing Tree - Black Hole Both nailed the clock, but the other three is where it gets interesting. Looking forward to Qwen3.6 open-weight release!show more

I tested every Unsloth quant of Qwen3.8 27B on... canvas coding, and Q8 was NOT the winner 👀 Setup: 2 prompts: > growing tree animation > paper slicing) 3 runs each, across BF16 (Original) / UD-Q2_K_XL / UD-Q3_K_XL / UD-Q4_K_XL / UD-Q6_K_XL / UD-Q8_K_XL. 🌳 Growing tree test (successes out of 3): > BF16 — 3/3 > Q2 — 1/3 > Q3 — 3/3 > Q4 — 3/3 > Q6 — 3/3 > Q8 — 2/3 📄 Paper slice test (successes out of 3): > BF16 — 1/3 > Q2 — 0/3 > Q3 — 2/3 > Q4 — 2/3 > Q6 — 2/3 > Q8 — 1/3 Takeaway: "bigger quant = better" didn't hold here, Q8 underperformed Q4 and Q6 on both tests. If you're running this model locally, Q6/Q4 looks like the sweet spot: near-full quality at a fraction of the VRAM.show more

GLM 5.1 just went open-weight on Hugging Face, but... how does it compare to GLM 5? I have tested both with the canvas tree challenge. 5.1 thinks longer, but delivers wind animation, sun, clouds, and way more detail. Prompt attached: Write a single HTML file with a full-page canvas, no libraries. Animate a tree that grows from the bottom center of the screen in real time. The trunk grows upward first, then branches split off recursively with slight randomness in angle and length. Each generation of branches should be thinner and slightly lighter in color. When branches reach their final size, add small leaves as soft green circles at the tips. The tree should take about 15 seconds to fully grow. Use warm brown for wood and varied greens for leaves against a soft sky-blue gradient background.show more

Which local models can actually handle tool calling? I... built a framework to find out. 15 scenarios. 12 tools. Mocked responses. Temperature 0. No cherry-picking. Tested every Qwen3.5 size from 0.8B to 397B, and since some of you asked after the distillation tests: yes, I included Jackrong's Qwen3.5-27B-Claude-4.6-Opus-Reasoning-Distilled too. Only two models went all green: the 27B dense and the distilled 27B. The 397B? Failed two tests. The 122B? Failed one. The 35B? Failed two. The timed-out results — mostly on the smaller models, are cases where the model got stuck in a loop, repeating the same tool call until it hit the 30-second limit. The test that exposed the most models: "Search for Iceland's population, then calculate 2% of it." Simple, but 35B, 122B, and 397B all used a rounded number from memory instead of the actual search result. They didn't trust their own tool output. Small models hallucinate data. Big models ignore data. The 27B just threaded it through.show more

Been designing and experimenting with a new benchmark that... stresses an underexplored angle: long tool-call chains with traps. The task: audit 36 packets, read 4 long-context ledgers, dodge retired/staging/wrong-quarter decoys, follow a strict workflow (auth → token → request → answer), submit the exact secret. Optimal: 52 calls. No call cap. I just measure how many calls each model burns to finish, and how many errors along the way. Threw 4 popular small models at it: 🥇 Qwen3.6 35B A3B (MoE) → 52 calls. Optimal. Zero errors. 🥈 Qwen3.6 27B (Dense) → 55 calls. Clean. ❌ Gemma4 31B (Dense) → 107 calls, 29 errors, looped writing auth/response.txt and re-reading auth/token.txt forever. ❌ Gemma4 26B A4B (MoE) → gave up at 13 (submitted the wrong answer). Other models I tested (GLM, DeepSeek) finish fine. So this isn't a task design issue, it's a Gemma4 issue with stateful workflows. Big models next.show more

Gemini 3.6 Flash vs Gemini 3.5 Flash Quick canvas... test 3.6 Flash indeed used way fewer tokensshow more

Qwen3.6 35B A3B can't fill out a paper form... on its own. But give it NVIDIA's LocateAnything-3B — the #1 trending model on HuggingFace — as its eyes, and the two small models get it done together. (The test: place each element at the right pixel position on a blank form image, not type into a field.) Setup: > Qwen is the brain (main model), LocateAnything is the eyes (helper model acting as a tool). > I gave Qwen a new tool: ask "where's the email field?" and LocateAnything returns the exact x, y, width, height. > The blue boxes on the screen are its detections. Look how tight they are — it nails every field. Result: > Qwen3.6 35B A3B + LocateAnything-3B: form completed, all info correct. > Name, DOB, ID, gender, marital status, nationality, email, phone, address, postal code: all landed in the right field areas. > Character-box alignment still a touch loose, but every value is where it belongs. > 9m10s, 224.5k input, 24.3k output, 21 turns. Why it matters: > Qwen alone can't finish this test. Bolt on a 3B model that does exactly one thing > locate > and suddenly it can. > A combination of small models can do the work of a single large one.show more

Introducing HermesAgent-20, a new Bench Pack for BenchLocal. 20... scenarios extracted straight from the Hermes Agent source code, run against a REAL Hermes instance. The actual workload you'd put your model through. Why I built BenchLocal in the first place: most benchmarks are too abstract. We use local LLMs for practical work, and finding the right model for YOUR task efficiently is the single most important thing, especially when you're constrained to what fits on your machine. BenchLocal is a framework: providers, models, side-by-side comparison, all in one UI. Bench Packs are the unit of testing: ToolCall-15 and BugFind-15 shipped first, and when I launched the BenchLocal 0.1.0, added StructOutput, ReasonMath, InstructFollow, DataExtract. Now, HermesAgent-20 is the newest. Bench Packs install like VS Code extensions. The SDK is open, write your own, share it, grow the ecosystem. Here's the goal: a community-built, practical evaluation layer for the local LLM space. Early numbers on HermesAgent-20: > GLM 5.1 — 85 > Gemma4 31B — 83 > Qwen3.5 27B — 79 > MiniMax M2.7 — 76 Upgrade to the latest BenchLocal to install HermesAgent-20 (SDK update required).show more

One prompt. 6 frontier coding models. "Create a realistic... fireworks show using HTML Canvas and JavaScript. No libraries." Some built a whole celebration. Others... lit a sparkler. The lineup: - GPT-5.3 Codex - Claude Opus 4.6 - Gemini 3.1 Pro - MiniMax M2.7 - GLM-5 - Kimi K2.5show more

MiniMax M3 might be the most underrated coding model... right now. I gave it nothing but a screenshot of a chaotic 90s GeoCities-style fan page, no HTML source, just the image + the asset files, and told it to rebuild the whole thing as a sleek Apple-style 2026 site. One shot. Through OpenCode. The result is genuinely stunning. It kept the soul (the "stevibe's HyperHome" identity, the visitor counter, the guestbook, the webmaster portrait) and translated every section into clean modern design, gradient hero, proper typography, dark theme, the works.show more

I explored a further possibility with local models: Qwen3.6... 35B A3B + NVIDIA LocateAnything-3B as a local Computer Use agent (proof of concept). In the demo, I asked it to switch my Mac to light mode. It did. Then back to dark. Did that too — finding the right toggle in System Settings, clicking it, and verifying the change itself. It's fully screenshot-based, so no Accessibility API needed. If it's on screen, the agent can see it and act on it. This runs entirely on your own hardware — private, local, built from two small open models.show more

Got a 16GB GPU? You can run all of... these right now. Tested 4 Qwen3.5-based models on ToolCall-15 & BugFind-15: Models: - Qwen3.5:9b Q8 (Official) - Qwopus v3 Q8 by Jackrong - OmniCoder-9B by Tesslate - Qwen3.5-9b-Sushi-Coder by bigatuna Summary: - ToolCall-15: Qwopus v3 went perfect 30/30, Sushicoder beat base Qwen3.5 - BugFind-15: Omnicoder flipped the script and took #1 at 83% No single model won both, that's the fun part. Open source community is cooking.show more

How well can Qwen3.5 models debug code? I built... BugFind-15 — 15 buggy snippets across Python, JS, Rust, and Go. Docker sandbox compiles and validates every fix. Two trap scenarios where the code is correct and the model must resist "fixing" it. Tested every Qwen3.5 size from 0.8B to 397B, plus Jackrong's popular distilled model (V2). The 0.8B scored 5%. The 2B scored 10%. At 4B, debugging ability jumps to 69%. The hardest scenario: BF-03, a Rust trap. The code compiles fine — format! borrows, it doesn't move. Not a single model figured this out. From 0.8B to 397B, every one of them "fixed" a bug that doesn't exist. Category C (subtle bugs — mutable defaults, integer overflow, slice aliasing) was 100% across every model 4B and above. Category D (red herring resistance) told the real story — can it resist fixing code that isn't broken? No model scored above 90%. Small models can't debug. Mid-size models fix obvious bugs but fall for traps. Large models fix the hard bugs but still invent problems that don't exist.show more

MiniMax M3 just dropped — their first natively multimodal... model. So I ran it through my form-filling test. (The model has to place each element at the right pixel position on a blank form image, not type into a field.) Verdict: it got everything on the paper. > Name, DOB, ID, gender, marital status, nationality, email, phone, address, postal code, all there. > Best character spacing I've seen yet: it actually calculates the gap between each character, clean across the DOB and number boxes > A few fields slightly misaligned, but every piece of data made it onto the form The reasoning chain is the interesting part: it does the easy fields first, then works into the tight one-char-per-box fields, reasoning through y-coordinates, baselines, and label clearance in obsessive detail. The cost: 40:33 and 126.7k output tokens. That's a long think — but it's MiniMax's first multimodal model, and it nailed the content.show more

"I'm not a human." Fed it to Qwen 3.5... 0.8B running locally on my Mac Studio M2 Ultra. It solved it. The CAPTCHA is fake. But sending images to the local model? Very real. I'm not breaking the internet. Yet.show more

Qwen3.5-27B went 15/15 on our tool-calling benchmark. But which... quant should you actually run? Tested Unsloth's Q2_K_XL all the way to Q8_K_XL TL;DR: Q8 — 15/15 ✅ Q6 — 15/15 ✅ Q5 — 14/15 Q4 — 14/15 Q3 — 14/15 Q2 — 13/15 Q6 is the sweet spot. Same perfect score as Q8, smaller footprint. Also, the results scale almost linearly, seems like ToolCall-15 is actually measuring something real.show more

Some people doubted the previous test because it was... routed through OpenRouter. So I ran the test again directly through Anthropic’s official API endpoint. Here’s what happened:show more

an agent is four parts in a loop. you... own one. the other three break it. that's why the demo works and prod doesn't. you can't debug what you can't see. 1) the prompt → what you tell the model each turn. you own this one. good. 2) the context window → what it sees right now. the framework fills it with junk, and you never notice until it rots. 3) the tools → what it can do. you own the list, not when or why it fires them. 4) the control flow → what happens next, when to stop. the framework owns this. it's what breaks at 80%. own all four and your agent stops being a magic trick that works on stage and dies on call. this isn't my idea. it's the 12-factor agents guide (24k stars) github: the whole thing every serious builder ends up rewriting their stack around. full breakdown in the article below.show more

MLP in PyTorch by hand ✍️ ~ 7 steps... walkthrough below Goal: fill in every blank in the PyTorch code to build a multi-layer perceptron. 1. Given Let us start with a code template on the left and the network it is supposed to build on the right. Every blank in the code can be worked out from the picture. 2. Linear layer We count: 3 features in, 4 features out. So the weight matrix is 4 by 3. There is an extra column for the biases, which means bias = T. 3. ReLU Let us apply the activation. ReLU crosses out the negatives, so -1 becomes 0. 4. Linear layer The input size is 4, because that is what the previous layer put out. The output size is 2. A 2 by 4 weight matrix, and this time no extra column, so bias = F. 5. ReLU We cross out the negatives again. 6. Linear layer Two features in, five out. A 5 by 2 weight matrix, with a bias column, so bias = T. 7. Sigmoid Let us finish. Sigmoid squashes the raw scores (3, 0, -2, 5, -5) into probabilities between 0 and 1. You have just implemented a three-layer deep neural network by hand. ✍️ == Story == Three years ago I gave this exercise to my students, to connect the code to the math. They found it odd. Every other AI course they were taking lived inside a Jupyter notebook, and here I was handing out paper. Three years later, my colleagues are the ones rushing to move their materials to paper. The exercise has not changed. Paper still asks the one thing a notebook lets you skip: do you actually understand what the code is doing? If you can tell me why the weight matrix is 4 by 3, and why bias is F on the second layer, you understand nn.Linear better than someone who has been copy-pasting it for a year. 💾 Save this post! #AIbyHand #PyTorch #DeepLearningshow more

Tested Kimi's new K3 against GPT-5.6-SOL on a tricky... front-end prompt with an image reference: "Single HTML file, canvas animation: 360° rotating iPhone that disassembles into an exploded view mid-rotation, pauses 2 seconds, reassembles. Simulated 3D perspective, Apple-style aesthetic, no external libraries." Neither output is perfect, but K3's version has a cleaner exploded view. Impressive.show more

AI has had exactly two scaling axes that worked... so far, and the second one is starting to look finite too the first one was pretraining: with scaling parameters and data, we got world knowledge (i.e. ChatGPT had read enough to know things), but it started saturating a while ago the second one was RL, and people had been doing RL the whole time before that: RLHF is RL but it never scaled far because it was trying to control the exact output, which tokens come out, how the text reads, but you can only push that so far before you’re just polishing RLVR dropped that constraint: giving the model a task, then checking whether the final answer is right, and ignoring everything in between -- so the model does whatever it wants in the middle and only the endpoint gets graded, and that’s much closer to actual RL and it’s what bought us planning and reasoning (arguably, tool use sits around 2.5 on this list -- while useful, it's not a different kind of thing) so one axis gave knowledge, the other gave reasoning, and both of them are one model working alone the next axis is how many models you can get working on the same problem, which is a different kind of axis than the previous two we know that multi-agent RL has always been the harder problem: I spent years in that literature and the gap between single-agent and multi-agent is definitely not incremental -- it’s a whole different class of difficulty! which is also why the derivatives are steep at the start, nobody has picked the easy wins yet... and the thing that gates this multi-agent coordination is communication: models can only coordinate as well as they can exchange information, and right now they do that by writing sentences to each other imagine what could we possibly achieve if we properly open that third axis development by letting models to exchange information in their native "language" without loosing any computational data that they produce during inferenceshow more