stevibe's banner
stevibe's profile picture

stevibe

@stevibe • 27,752 subscribers

LLM. Local AI addict. Building @BenchLocalAI DGX Spark workbench https://t.co/WN4i7q2y5m Builds things nobody asked for. Benchmarks things for fun.

Shorts

Claude Sonnet 4.6, when asked in Chinese: “你是什么模型？” (What model are you?) Confidently replies: “我是 DeepSeek。” (I am DeepSeek) This is the same model whose company just accused DeepSeek of “industrial-scale distillation attacks”

Claude Sonnet 4.6, when asked in Chinese: “你是什么模型？” (What model are you?) Confidently replies: “我是 DeepSeek。” (I am DeepSeek) This is the same model whose company just accused DeepSeek of “industrial-scale distillation attacks”

1,941,255 Aufrufe

Which local models can actually handle tool calling? I built a framework to find out. 15 scenarios. 12 tools. Mocked responses. Temperature 0. No cherry-picking. Tested every Qwen3.5 size from 0.8B to 397B, and since some of you asked after the distillation tests: yes, I included Jackrong's Qwen3.5-27B-Claude-4.6-Opus-Reasoning-Distilled too. Only two models went all green: the 27B dense and the distilled 27B. The 397B? Failed two tests. The 122B? Failed one. The 35B? Failed two. The timed-out results — mostly on the smaller models, are cases where the model got stuck in a loop, repeating the same tool call until it hit the 30-second limit. The test that exposed the most models: "Search for Iceland's population, then calculate 2% of it." Simple, but 35B, 122B, and 397B all used a rounded number from memory instead of the actual search result. They didn't trust their own tool output. Small models hallucinate data. Big models ignore data. The 27B just threaded it through.

Which local models can actually handle tool calling? I built a framework to find out. 15 scenarios. 12 tools. Mocked responses. Temperature 0. No cherry-picking. Tested every Qwen3.5 size from 0.8B to 397B, and since some of you asked after the distillation tests: yes, I included Jackrong's Qwen3.5-27B-Claude-4.6-Opus-Reasoning-Distilled too. Only two models went all green: the 27B dense and the distilled 27B. The 397B? Failed two tests. The 122B? Failed one. The 35B? Failed two. The timed-out results — mostly on the smaller models, are cases where the model got stuck in a loop, repeating the same tool call until it hit the 30-second limit. The test that exposed the most models: "Search for Iceland's population, then calculate 2% of it." Simple, but 35B, 122B, and 397B all used a rounded number from memory instead of the actual search result. They didn't trust their own tool output. Small models hallucinate data. Big models ignore data. The 27B just threaded it through.

435,100 Aufrufe

I tested every Unsloth quant of Qwen3.8 27B on canvas coding, and Q8 was NOT the winner 👀 Setup: 2 prompts: > growing tree animation > paper slicing) 3 runs each, across BF16 (Original) / UD-Q2_K_XL / UD-Q3_K_XL / UD-Q4_K_XL / UD-Q6_K_XL / UD-Q8_K_XL. 🌳 Growing tree test (successes out of 3): > BF16 — 3/3 > Q2 — 1/3 > Q3 — 3/3 > Q4 — 3/3 > Q6 — 3/3 > Q8 — 2/3 📄 Paper slice test (successes out of 3): > BF16 — 1/3 > Q2 — 0/3 > Q3 — 2/3 > Q4 — 2/3 > Q6 — 2/3 > Q8 — 1/3 Takeaway: "bigger quant = better" didn't hold here, Q8 underperformed Q4 and Q6 on both tests. If you're running this model locally, Q6/Q4 looks like the sweet spot: near-full quality at a fraction of the VRAM.

I tested every Unsloth quant of Qwen3.8 27B on canvas coding, and Q8 was NOT the winner 👀 Setup: 2 prompts: > growing tree animation > paper slicing) 3 runs each, across BF16 (Original) / UD-Q2_K_XL / UD-Q3_K_XL / UD-Q4_K_XL / UD-Q6_K_XL / UD-Q8_K_XL. 🌳 Growing tree test (successes out of 3): > BF16 — 3/3 > Q2 — 1/3 > Q3 — 3/3 > Q4 — 3/3 > Q6 — 3/3 > Q8 — 2/3 📄 Paper slice test (successes out of 3): > BF16 — 1/3 > Q2 — 0/3 > Q3 — 2/3 > Q4 — 2/3 > Q6 — 2/3 > Q8 — 1/3 Takeaway: "bigger quant = better" didn't hold here, Q8 underperformed Q4 and Q6 on both tests. If you're running this model locally, Q6/Q4 looks like the sweet spot: near-full quality at a fraction of the VRAM.

82,089 Aufrufe

Qwen3.6 35B A3B can't fill out a paper form on its own. But give it NVIDIA's LocateAnything-3B — the #1 trending model on HuggingFace — as its eyes, and the two small models get it done together. (The test: place each element at the right pixel position on a blank form image, not type into a field.) Setup: > Qwen is the brain (main model), LocateAnything is the eyes (helper model acting as a tool). > I gave Qwen a new tool: ask "where's the email field?" and LocateAnything returns the exact x, y, width, height. > The blue boxes on the screen are its detections. Look how tight they are — it nails every field. Result: > Qwen3.6 35B A3B + LocateAnything-3B: form completed, all info correct. > Name, DOB, ID, gender, marital status, nationality, email, phone, address, postal code: all landed in the right field areas. > Character-box alignment still a touch loose, but every value is where it belongs. > 9m10s, 224.5k input, 24.3k output, 21 turns. Why it matters: > Qwen alone can't finish this test. Bolt on a 3B model that does exactly one thing > locate > and suddenly it can. > A combination of small models can do the work of a single large one.

Qwen3.6 35B A3B can't fill out a paper form on its own. But give it NVIDIA's LocateAnything-3B — the #1 trending model on HuggingFace — as its eyes, and the two small models get it done together. (The test: place each element at the right pixel position on a blank form image, not type into a field.) Setup: > Qwen is the brain (main model), LocateAnything is the eyes (helper model acting as a tool). > I gave Qwen a new tool: ask "where's the email field?" and LocateAnything returns the exact x, y, width, height. > The blue boxes on the screen are its detections. Look how tight they are — it nails every field. Result: > Qwen3.6 35B A3B + LocateAnything-3B: form completed, all info correct. > Name, DOB, ID, gender, marital status, nationality, email, phone, address, postal code: all landed in the right field areas. > Character-box alignment still a touch loose, but every value is where it belongs. > 9m10s, 224.5k input, 24.3k output, 21 turns. Why it matters: > Qwen alone can't finish this test. Bolt on a 3B model that does exactly one thing > locate > and suddenly it can. > A combination of small models can do the work of a single large one.

150,971 Aufrufe

Qwen3.5 27B vs Gemma4 31B | Canvas Creativity Test Why HTML Canvas? Two reasons: 1. It's unforgiving, one small mistake and the whole thing breaks 2. We kept prompts short to test real creativity, not instruction following 4 rounds: - Analog Clock - Hyperspace Tunnel - Growing Tree - Black Hole Both nailed the clock, but the other three is where it gets interesting. Looking forward to Qwen3.6 open-weight release!

Qwen3.5 27B vs Gemma4 31B | Canvas Creativity Test Why HTML Canvas? Two reasons: 1. It's unforgiving, one small mistake and the whole thing breaks 2. We kept prompts short to test real creativity, not instruction following 4 rounds: - Analog Clock - Hyperspace Tunnel - Growing Tree - Black Hole Both nailed the clock, but the other three is where it gets interesting. Looking forward to Qwen3.6 open-weight release!

171,191 Aufrufe

Gemini 3.6 Flash vs Gemini 3.5 Flash Quick canvas test 3.6 Flash indeed used way fewer tokens

Gemini 3.6 Flash vs Gemini 3.5 Flash Quick canvas test 3.6 Flash indeed used way fewer tokens

83,424 Aufrufe

Some people doubted the previous test because it was routed through OpenRouter. So I ran the test again directly through Anthropic’s official API endpoint. Here’s what happened:

Some people doubted the previous test because it was routed through OpenRouter. So I ran the test again directly through Anthropic’s official API endpoint. Here’s what happened:

155,783 Aufrufe

NVIDIA just dropped Nemotron-3-Nano:4b — a tiny 2.8GB model. Guess whose hardware runs it the fastest? - RTX 4090: 226 tok/s - RTX 3090: 187 tok/s - Mac Studio M2 Ultra: 86 tok/s - Mac Mini M4: 25 tok/s Home court advantage is real. Also trying a new layout with live performance charts. Lmk what you think!

NVIDIA just dropped Nemotron-3-Nano:4b — a tiny 2.8GB model. Guess whose hardware runs it the fastest? - RTX 4090: 226 tok/s - RTX 3090: 187 tok/s - Mac Studio M2 Ultra: 86 tok/s - Mac Mini M4: 25 tok/s Home court advantage is real. Also trying a new layout with live performance charts. Lmk what you think!

127,672 Aufrufe

Tested Kimi's new K3 against GPT-5.6-SOL on a tricky front-end prompt with an image reference: "Single HTML file, canvas animation: 360° rotating iPhone that disassembles into an exploded view mid-rotation, pauses 2 seconds, reassembles. Simulated 3D perspective, Apple-style aesthetic, no external libraries." Neither output is perfect, but K3's version has a cleaner exploded view. Impressive.

Tested Kimi's new K3 against GPT-5.6-SOL on a tricky front-end prompt with an image reference: "Single HTML file, canvas animation: 360° rotating iPhone that disassembles into an exploded view mid-rotation, pauses 2 seconds, reassembles. Simulated 3D perspective, Apple-style aesthetic, no external libraries." Neither output is perfect, but K3's version has a cleaner exploded view. Impressive.

53,995 Aufrufe

Got a 16GB GPU? You can run all of these right now. Tested 4 Qwen3.5-based models on ToolCall-15 & BugFind-15: Models: - Qwen3.5:9b Q8 (Official) - Qwopus v3 Q8 by Jackrong - OmniCoder-9B by Tesslate - Qwen3.5-9b-Sushi-Coder by bigatuna Summary: - ToolCall-15: Qwopus v3 went perfect 30/30, Sushicoder beat base Qwen3.5 - BugFind-15: Omnicoder flipped the script and took #1 at 83% No single model won both, that's the fun part. Open source community is cooking.

Got a 16GB GPU? You can run all of these right now. Tested 4 Qwen3.5-based models on ToolCall-15 & BugFind-15: Models: - Qwen3.5:9b Q8 (Official) - Qwopus v3 Q8 by Jackrong - OmniCoder-9B by Tesslate - Qwen3.5-9b-Sushi-Coder by bigatuna Summary: - ToolCall-15: Qwopus v3 went perfect 30/30, Sushicoder beat base Qwen3.5 - BugFind-15: Omnicoder flipped the script and took #1 at 83% No single model won both, that's the fun part. Open source community is cooking.

75,769 Aufrufe

Wait, what? Asked in French: “nom du modèle” (name of the model?) It replies: “ChatGPT” Credits to Mike Mickelson for the idea

Wait, what? Asked in French: “nom du modèle” (name of the model?) It replies: “ChatGPT” Credits to Mike Mickelson for the idea

84,928 Aufrufe

DeepSeek shipped their first vision model, so I ran it through my form-filling test. Not an electronic form, an image is all the model ever sees. No coordinates, no DOM. V4 Flash Vision probes positions, screenshots itself, places each element and nudges it into place on its own. ✅ every field filled ✅ self-verified before submit ⚠️ ticks land next to the box, not in it ⚠️ character-box drift on longer strings 5m48s · 288k in / 38k out Misses a few boxes, but it got there.

DeepSeek shipped their first vision model, so I ran it through my form-filling test. Not an electronic form, an image is all the model ever sees. No coordinates, no DOM. V4 Flash Vision probes positions, screenshots itself, places each element and nudges it into place on its own. ✅ every field filled ✅ self-verified before submit ⚠️ ticks land next to the box, not in it ⚠️ character-box drift on longer strings 5m48s · 288k in / 38k out Misses a few boxes, but it got there.

23,311 Aufrufe

"I'm not a human." Fed it to Qwen 3.5 0.8B running locally on my Mac Studio M2 Ultra. It solved it. The CAPTCHA is fake. But sending images to the local model? Very real. I'm not breaking the internet. Yet.

"I'm not a human." Fed it to Qwen 3.5 0.8B running locally on my Mac Studio M2 Ultra. It solved it. The CAPTCHA is fake. But sending images to the local model? Very real. I'm not breaking the internet. Yet.

69,433 Aufrufe

I explored a further possibility with local models: Qwen3.6 35B A3B + NVIDIA LocateAnything-3B as a local Computer Use agent (proof of concept). In the demo, I asked it to switch my Mac to light mode. It did. Then back to dark. Did that too — finding the right toggle in System Settings, clicking it, and verifying the change itself. It's fully screenshot-based, so no Accessibility API needed. If it's on screen, the agent can see it and act on it. This runs entirely on your own hardware — private, local, built from two small open models.

I explored a further possibility with local models: Qwen3.6 35B A3B + NVIDIA LocateAnything-3B as a local Computer Use agent (proof of concept). In the demo, I asked it to switch my Mac to light mode. It did. Then back to dark. Did that too — finding the right toggle in System Settings, clicking it, and verifying the change itself. It's fully screenshot-based, so no Accessibility API needed. If it's on screen, the agent can see it and act on it. This runs entirely on your own hardware — private, local, built from two small open models.

44,328 Aufrufe

Qwen3.5-27B went 15/15 on our tool-calling benchmark. But which quant should you actually run? Tested Unsloth's Q2_K_XL all the way to Q8_K_XL TL;DR: Q8 — 15/15 ✅ Q6 — 15/15 ✅ Q5 — 14/15 Q4 — 14/15 Q3 — 14/15 Q2 — 13/15 Q6 is the sweet spot. Same perfect score as Q8, smaller footprint. Also, the results scale almost linearly, seems like ToolCall-15 is actually measuring something real.

Qwen3.5-27B went 15/15 on our tool-calling benchmark. But which quant should you actually run? Tested Unsloth's Q2_K_XL all the way to Q8_K_XL TL;DR: Q8 — 15/15 ✅ Q6 — 15/15 ✅ Q5 — 14/15 Q4 — 14/15 Q3 — 14/15 Q2 — 13/15 Q6 is the sweet spot. Same perfect score as Q8, smaller footprint. Also, the results scale almost linearly, seems like ToolCall-15 is actually measuring something real.

61,339 Aufrufe

GLM 5.1 just went open-weight on Hugging Face, but how does it compare to GLM 5? I have tested both with the canvas tree challenge. 5.1 thinks longer, but delivers wind animation, sun, clouds, and way more detail. Prompt attached: Write a single HTML file with a full-page canvas, no libraries. Animate a tree that grows from the bottom center of the screen in real time. The trunk grows upward first, then branches split off recursively with slight randomness in angle and length. Each generation of branches should be thinner and slightly lighter in color. When branches reach their final size, add small leaves as soft green circles at the tips. The tree should take about 15 seconds to fully grow. Use warm brown for wood and varied greens for leaves against a soft sky-blue gradient background.

GLM 5.1 just went open-weight on Hugging Face, but how does it compare to GLM 5? I have tested both with the canvas tree challenge. 5.1 thinks longer, but delivers wind animation, sun, clouds, and way more detail. Prompt attached: Write a single HTML file with a full-page canvas, no libraries. Animate a tree that grows from the bottom center of the screen in real time. The trunk grows upward first, then branches split off recursively with slight randomness in angle and length. Each generation of branches should be thinner and slightly lighter in color. When branches reach their final size, add small leaves as soft green circles at the tips. The tree should take about 15 seconds to fully grow. Use warm brown for wood and varied greens for leaves against a soft sky-blue gradient background.

46,657 Aufrufe

Introducing HermesAgent-20, a new Bench Pack for BenchLocal. 20 scenarios extracted straight from the Hermes Agent source code, run against a REAL Hermes instance. The actual workload you'd put your model through. Why I built BenchLocal in the first place: most benchmarks are too abstract. We use local LLMs for practical work, and finding the right model for YOUR task efficiently is the single most important thing, especially when you're constrained to what fits on your machine. BenchLocal is a framework: providers, models, side-by-side comparison, all in one UI. Bench Packs are the unit of testing: ToolCall-15 and BugFind-15 shipped first, and when I launched the BenchLocal 0.1.0, added StructOutput, ReasonMath, InstructFollow, DataExtract. Now, HermesAgent-20 is the newest. Bench Packs install like VS Code extensions. The SDK is open, write your own, share it, grow the ecosystem. Here's the goal: a community-built, practical evaluation layer for the local LLM space. Early numbers on HermesAgent-20: > GLM 5.1 — 85 > Gemma4 31B — 83 > Qwen3.5 27B — 79 > MiniMax M2.7 — 76 Upgrade to the latest BenchLocal to install HermesAgent-20 (SDK update required).

Introducing HermesAgent-20, a new Bench Pack for BenchLocal. 20 scenarios extracted straight from the Hermes Agent source code, run against a REAL Hermes instance. The actual workload you'd put your model through. Why I built BenchLocal in the first place: most benchmarks are too abstract. We use local LLMs for practical work, and finding the right model for YOUR task efficiently is the single most important thing, especially when you're constrained to what fits on your machine. BenchLocal is a framework: providers, models, side-by-side comparison, all in one UI. Bench Packs are the unit of testing: ToolCall-15 and BugFind-15 shipped first, and when I launched the BenchLocal 0.1.0, added StructOutput, ReasonMath, InstructFollow, DataExtract. Now, HermesAgent-20 is the newest. Bench Packs install like VS Code extensions. The SDK is open, write your own, share it, grow the ecosystem. Here's the goal: a community-built, practical evaluation layer for the local LLM space. Early numbers on HermesAgent-20: > GLM 5.1 — 85 > Gemma4 31B — 83 > Qwen3.5 27B — 79 > MiniMax M2.7 — 76 Upgrade to the latest BenchLocal to install HermesAgent-20 (SDK update required).

38,631 Aufrufe

MiniMax M3 just dropped — their first natively multimodal model. So I ran it through my form-filling test. (The model has to place each element at the right pixel position on a blank form image, not type into a field.) Verdict: it got everything on the paper. > Name, DOB, ID, gender, marital status, nationality, email, phone, address, postal code, all there. > Best character spacing I've seen yet: it actually calculates the gap between each character, clean across the DOB and number boxes > A few fields slightly misaligned, but every piece of data made it onto the form The reasoning chain is the interesting part: it does the easy fields first, then works into the tight one-char-per-box fields, reasoning through y-coordinates, baselines, and label clearance in obsessive detail. The cost: 40:33 and 126.7k output tokens. That's a long think — but it's MiniMax's first multimodal model, and it nailed the content.

MiniMax M3 just dropped — their first natively multimodal model. So I ran it through my form-filling test. (The model has to place each element at the right pixel position on a blank form image, not type into a field.) Verdict: it got everything on the paper. > Name, DOB, ID, gender, marital status, nationality, email, phone, address, postal code, all there. > Best character spacing I've seen yet: it actually calculates the gap between each character, clean across the DOB and number boxes > A few fields slightly misaligned, but every piece of data made it onto the form The reasoning chain is the interesting part: it does the easy fields first, then works into the tight one-char-per-box fields, reasoning through y-coordinates, baselines, and label clearance in obsessive detail. The cost: 40:33 and 126.7k output tokens. That's a long think — but it's MiniMax's first multimodal model, and it nailed the content.

27,383 Aufrufe

How well can Qwen3.5 models debug code? I built BugFind-15 — 15 buggy snippets across Python, JS, Rust, and Go. Docker sandbox compiles and validates every fix. Two trap scenarios where the code is correct and the model must resist "fixing" it. Tested every Qwen3.5 size from 0.8B to 397B, plus Jackrong's popular distilled model (V2). The 0.8B scored 5%. The 2B scored 10%. At 4B, debugging ability jumps to 69%. The hardest scenario: BF-03, a Rust trap. The code compiles fine — format! borrows, it doesn't move. Not a single model figured this out. From 0.8B to 397B, every one of them "fixed" a bug that doesn't exist. Category C (subtle bugs — mutable defaults, integer overflow, slice aliasing) was 100% across every model 4B and above. Category D (red herring resistance) told the real story — can it resist fixing code that isn't broken? No model scored above 90%. Small models can't debug. Mid-size models fix obvious bugs but fall for traps. Large models fix the hard bugs but still invent problems that don't exist.

How well can Qwen3.5 models debug code? I built BugFind-15 — 15 buggy snippets across Python, JS, Rust, and Go. Docker sandbox compiles and validates every fix. Two trap scenarios where the code is correct and the model must resist "fixing" it. Tested every Qwen3.5 size from 0.8B to 397B, plus Jackrong's popular distilled model (V2). The 0.8B scored 5%. The 2B scored 10%. At 4B, debugging ability jumps to 69%. The hardest scenario: BF-03, a Rust trap. The code compiles fine — format! borrows, it doesn't move. Not a single model figured this out. From 0.8B to 397B, every one of them "fixed" a bug that doesn't exist. Category C (subtle bugs — mutable defaults, integer overflow, slice aliasing) was 100% across every model 4B and above. Category D (red herring resistance) told the real story — can it resist fixing code that isn't broken? No model scored above 90%. Small models can't debug. Mid-size models fix obvious bugs but fall for traps. Large models fix the hard bugs but still invent problems that don't exist.

35,158 Aufrufe

MiniMax M3 might be the most underrated coding model right now. I gave it nothing but a screenshot of a chaotic 90s GeoCities-style fan page, no HTML source, just the image + the asset files, and told it to rebuild the whole thing as a sleek Apple-style 2026 site. One shot. Through OpenCode. The result is genuinely stunning. It kept the soul (the "stevibe's HyperHome" identity, the visitor counter, the guestbook, the webmaster portrait) and translated every section into clean modern design, gradient hero, proper typography, dark theme, the works.

MiniMax M3 might be the most underrated coding model right now. I gave it nothing but a screenshot of a chaotic 90s GeoCities-style fan page, no HTML source, just the image + the asset files, and told it to rebuild the whole thing as a sleek Apple-style 2026 site. One shot. Through OpenCode. The result is genuinely stunning. It kept the soul (the "stevibe's HyperHome" identity, the visitor counter, the guestbook, the webmaster portrait) and translated every section into clean modern design, gradient hero, proper typography, dark theme, the works.

21,288 Aufrufe

Videos

LIVE
1.2k

Anya Rossi

sweetdream.ai

SweetDream.aiSponsored•Livecam
Streaming Now

Watch Anya Live

Anya is streaming live right now! Join her private show and enjoy exclusive content.

HD live stream
Exclusive private shows
1.2k viewers online

Free preview available • Premium content

your physics textbook is not boring anymore Hooke's Law with live text reflow around an actual bouncing simulation. 60fps. zero layout thrashing. Cheng Lou what have you unleashed
0:27
stevibe's profile picture

your physics textbook is not boring anymore Hooke's Law with live text reflow around an actual bouncing simulation. 60fps. zero layout thrashing. Cheng Lou what have you unleashed

stevibe

1,254,112 Aufrufe • vor 6 Monaten

The AI price war is ON. > DeepSeek just dropped V4 Flash 0731 (big capability jump, $0.28/M output) > OpenAI cut GPT-5.6 Luna by 80% to $1.20/M (OpenRouter is currently offering 50% off, bringing it to $0.60/M) Same budget tier now, so I ran both through 3 canvas tests: 🧊 Rubik's cube: DeepSeek nails every rotation. Luna's turns are buggy, stickers fly off mid-move. 🎆 Fireworks: both fine, but Luna lags hard when the burst explodes. 🖊️ Pen writing "Hello": neither is legible. DeepSeek's cursive is looser, but Luna gave up on animating, it rendered a static image. GPT-5.6 is great at coding — but IMO only on Sol. In the same pricing tier, DeepSeek clearly beats Luna.
0:32
stevibe's profile picture

The AI price war is ON. > DeepSeek just dropped V4 Flash 0731 (big capability jump, $0.28/M output) > OpenAI cut GPT-5.6 Luna by 80% to $1.20/M (OpenRouter is currently offering 50% off, bringing it to $0.60/M) Same budget tier now, so I ran both through 3 canvas tests: 🧊 Rubik's cube: DeepSeek nails every rotation. Luna's turns are buggy, stickers fly off mid-move. 🎆 Fireworks: both fine, but Luna lags hard when the burst explodes. 🖊️ Pen writing "Hello": neither is legible. DeepSeek's cursive is looser, but Luna gave up on animating, it rendered a static image. GPT-5.6 is great at coding — but IMO only on Sol. In the same pricing tier, DeepSeek clearly beats Luna.

stevibe

278,264 Aufrufe • vor 2 Monaten

Claude Opus 4.7 vs 4.8 side-by-side canvas test
0:26
stevibe's profile picture

Claude Opus 4.7 vs 4.8 side-by-side canvas test

stevibe

476,334 Aufrufe • vor 4 Monaten

GLM 5.1 vs GLM 5.2 6 advanced HTML canvas challenges: 💧 Ink diffusing in water ⚔️ Energy-blade duel 📱 Slide to unlock 🅿️ 360° parking assist 🔥 Burning letter to ash 🏠 Build-a-house sequence Pure canvas, zero libraries.
0:53
stevibe's profile picture

GLM 5.1 vs GLM 5.2 6 advanced HTML canvas challenges: 💧 Ink diffusing in water ⚔️ Energy-blade duel 📱 Slide to unlock 🅿️ 360° parking assist 🔥 Burning letter to ash 🏠 Build-a-house sequence Pure canvas, zero libraries.

stevibe

347,708 Aufrufe • vor 4 Monaten

Opus 4.7 first-hour impressions Ran the canvas tree growth test twice. 4.6: nailed the animation both times 4.7: static tree, no growth animation — twice 4.7's thinking is noticeably shorter and faster though (trimmed some 4.6 thinking in the clip for pacing). Not the upgrade direction I expected on this one.
0:21
stevibe's profile picture

Opus 4.7 first-hour impressions Ran the canvas tree growth test twice. 4.6: nailed the animation both times 4.7: static tree, no growth animation — twice 4.7's thinking is noticeably shorter and faster though (trimmed some 4.6 thinking in the clip for pacing). Not the upgrade direction I expected on this one.

stevibe

488,336 Aufrufe • vor 6 Monaten

This looks like a toy. It's actually the meanest little vision eval I've built. The task: look at an emoji image, then repaint it on a 16×16 grid, one pixel at a time. Just the model, a tiny canvas, and up to 2000 brushstrokes. What I didn't expect was the personalities. > some models REGRET a stroke and go back to repaint it > some get stuck looping the same little patch over and over, like they're trying to animate it > some are calm little surgeons and just nail it first try And the task is genuinely mean: it has to see the image, crush it down to 256 cells, then decide what's actually load-bearing: > the tears on 😂 but still keep the smile > the horn on 🦄 > the antenna on 🤖 and keep the soul of it with almost no resolution to spare. 5 models. 7 emojis. Best of 5 runs each. Side by side. Who's your winner?
0:46
stevibe's profile picture

This looks like a toy. It's actually the meanest little vision eval I've built. The task: look at an emoji image, then repaint it on a 16×16 grid, one pixel at a time. Just the model, a tiny canvas, and up to 2000 brushstrokes. What I didn't expect was the personalities. > some models REGRET a stroke and go back to repaint it > some get stuck looping the same little patch over and over, like they're trying to animate it > some are calm little surgeons and just nail it first try And the task is genuinely mean: it has to see the image, crush it down to 256 cells, then decide what's actually load-bearing: > the tears on 😂 but still keep the smile > the horn on 🦄 > the antenna on 🤖 and keep the soul of it with almost no resolution to spare. 5 models. 7 emojis. Best of 5 runs each. Side by side. Who's your winner?

stevibe

284,432 Aufrufe • vor 3 Monaten

Qwen3.6 35B-A3B dropped yesterday, so I ran it on 4 GPUs to see how it performs: 🟣 RTX 3090 — 49.78 tok/s, TTFT 852ms 🟡 RTX 4090 — 118.93 tok/s, TTFT 686ms 🟢 RTX 5090 — 160.37 tok/s, TTFT 409ms 🔵 DGX Spark — 59.98 tok/s, TTFT 228ms I went with ollama as the backend because honestly, it's the easiest way for most people to get started. One command, model pulled, done. I used Q4_K_M (24GB) across all four cards. The reason is the 3090 and 4090 don't support NVFP4 (only the 5090 and DGX Spark could use it). Keeping the same quant everywhere felt like the fairest way to compare. And yes, you can absolutely squeeze more performance out of every card with vLLM, SGLang, or TensorRT-LLM. But that's not what this test is about. This is just the out-of-the-box experience for folks who own a GPU and want to try the new model tonight.
1:24
stevibe's profile picture

Qwen3.6 35B-A3B dropped yesterday, so I ran it on 4 GPUs to see how it performs: 🟣 RTX 3090 — 49.78 tok/s, TTFT 852ms 🟡 RTX 4090 — 118.93 tok/s, TTFT 686ms 🟢 RTX 5090 — 160.37 tok/s, TTFT 409ms 🔵 DGX Spark — 59.98 tok/s, TTFT 228ms I went with ollama as the backend because honestly, it's the easiest way for most people to get started. One command, model pulled, done. I used Q4_K_M (24GB) across all four cards. The reason is the 3090 and 4090 don't support NVFP4 (only the 5090 and DGX Spark could use it). Keeping the same quant everywhere felt like the fairest way to compare. And yes, you can absolutely squeeze more performance out of every card with vLLM, SGLang, or TensorRT-LLM. But that's not what this test is about. This is just the out-of-the-box experience for folks who own a GPU and want to try the new model tonight.

stevibe

403,356 Aufrufe • vor 6 Monaten

I gave two MoE models the same vibe coding challenge Qwen3.6 35B A3B (31.8GB) vs Gemma4 26B A4B (23.3GB) Stack: > Unsloth Q6_K_XL > llama.cpp > Model-card recommended sampling for each 4 prompts, side-by-side. Which one do you think wins?
0:38
stevibe's profile picture

I gave two MoE models the same vibe coding challenge Qwen3.6 35B A3B (31.8GB) vs Gemma4 26B A4B (23.3GB) Stack: > Unsloth Q6_K_XL > llama.cpp > Model-card recommended sampling for each 4 prompts, side-by-side. Which one do you think wins?

stevibe

262,265 Aufrufe • vor 6 Monaten

Qwen3.5:9b reasoning head-to-head: Mac Studio M2 Ultra 64GB: 43.08 tok/s Mac Mini M4 16GB: 13.07 tok/s Qwen
1:18
stevibe's profile picture

Qwen3.5:9b reasoning head-to-head: Mac Studio M2 Ultra 64GB: 43.08 tok/s Mac Mini M4 16GB: 13.07 tok/s Qwen

stevibe

243,465 Aufrufe • vor 7 Monaten

I spent $4,699 on a DGX Spark and used it as a chatbot for six months. Not proud of it. But I think most Spark owners are doing the same thing, so let me tell you what changed. The moment was dumb. I had a recording of a client call, needed notes, and my first instinct was to open a browser tab and upload it to some transcription SaaS. Then I looked at the box on my desk. 128 GB of unified memory. Sitting there. Running a chat model I'd asked maybe 30 questions that day. And I thought: I have a supercomputer and I'm about to pay a stranger to listen to my client call. So I tried to do it locally. And here's where it got annoying: - my LLM can't hear audio - the ASR model I found is great but it's a CLI script - the transcript came out in two languages, need translation - summary needs the LLM again - now I want a mind map, that's another tool - oh and here's a scanned PDF from the same meeting, OCR, another tool Copy paste copy paste. Output saved to ~/Downloads/final_v2_REAL.txt. That's when it clicked: tokens per second was never the point of this box. A 24 GB gaming card forces you to pick ONE model, so everyone picks the biggest generalist and asks it to do everything. But 128 GB doesn't force that. You can keep six specialists loaded at once. A 1.7B speech model transcribes better than a 35B chat model ever will. A tiny translation model just translates and doesn't get creative. The big LLM does the thinking, and only the thinking. The hardware was never the problem. The missing piece was the thing that ties them together. So I built it. SparklingKit. (the name is a tribute to the Spark. also: a spark is one flash, sparkling is when it keeps going) What it actually is: → transcription, OCR, translation, image grounding, text-to-image, mind maps, chat. one panel. → six models hand-picked to fit together in 128 GB with room left over → queue-first: drop a 3-hour recording, go get coffee, check from your phone → nothing is a dead end: finish a transcript and "translate / map / chat with it" are already waiting → visual workflows: chain the steps once, run them forever → everything is a file. one folder. no database. copy it to back up. point Obsidian at it. → nothing leaves your machine unless you send it. The thing I care about most: if you stop using SparklingKit tomorrow, nothing happens to your data. It was already a folder of markdown and json. There's no lock-in to escape. Spark owners: you bought room, not a faster chatbot. This is how I'm using the room. No Spark? The panel runs anywhere Docker runs. Point it at your own endpoints. It's open source, and it's version one. Rough edges exist. Issues and PRs very welcome. Full details and a one-line install at If you've ever uploaded something private to a cloud AI and felt weird about it, this is for you.
1:23
stevibe's profile picture

I spent $4,699 on a DGX Spark and used it as a chatbot for six months. Not proud of it. But I think most Spark owners are doing the same thing, so let me tell you what changed. The moment was dumb. I had a recording of a client call, needed notes, and my first instinct was to open a browser tab and upload it to some transcription SaaS. Then I looked at the box on my desk. 128 GB of unified memory. Sitting there. Running a chat model I'd asked maybe 30 questions that day. And I thought: I have a supercomputer and I'm about to pay a stranger to listen to my client call. So I tried to do it locally. And here's where it got annoying: - my LLM can't hear audio - the ASR model I found is great but it's a CLI script - the transcript came out in two languages, need translation - summary needs the LLM again - now I want a mind map, that's another tool - oh and here's a scanned PDF from the same meeting, OCR, another tool Copy paste copy paste. Output saved to ~/Downloads/final_v2_REAL.txt. That's when it clicked: tokens per second was never the point of this box. A 24 GB gaming card forces you to pick ONE model, so everyone picks the biggest generalist and asks it to do everything. But 128 GB doesn't force that. You can keep six specialists loaded at once. A 1.7B speech model transcribes better than a 35B chat model ever will. A tiny translation model just translates and doesn't get creative. The big LLM does the thinking, and only the thinking. The hardware was never the problem. The missing piece was the thing that ties them together. So I built it. SparklingKit. (the name is a tribute to the Spark. also: a spark is one flash, sparkling is when it keeps going) What it actually is: → transcription, OCR, translation, image grounding, text-to-image, mind maps, chat. one panel. → six models hand-picked to fit together in 128 GB with room left over → queue-first: drop a 3-hour recording, go get coffee, check from your phone → nothing is a dead end: finish a transcript and "translate / map / chat with it" are already waiting → visual workflows: chain the steps once, run them forever → everything is a file. one folder. no database. copy it to back up. point Obsidian at it. → nothing leaves your machine unless you send it. The thing I care about most: if you stop using SparklingKit tomorrow, nothing happens to your data. It was already a folder of markdown and json. There's no lock-in to escape. Spark owners: you bought room, not a faster chatbot. This is how I'm using the room. No Spark? The panel runs anywhere Docker runs. Point it at your own endpoints. It's open source, and it's version one. Rough edges exist. Issues and PRs very welcome. Full details and a one-line install at If you've ever uploaded something private to a cloud AI and felt weird about it, this is for you.

stevibe

56,229 Aufrufe • vor 1 Monat

MiniMax M2.7 is 230B params. Can you actually run it at home? I tested Unsloth's UD-IQ3_XXS (80GB) on 4 different rigs: 🟠 4x RTX 4090 (96GB): 71.52 tok/s, TTFT 1045ms 🟢 4x RTX 5090 (128GB): 120.54 tok/s, TTFT 725ms 🟡 1x RTX PRO 6000 (96GB): 118.74 tok/s, TTFT 765ms 🟣 DGX Spark (128GB) — 24.41 tok/s, TTFT 741ms Backend: llama.cpp. Context: 32k. Max tokens: 4096. I went with IQ3_XXS because it's the biggest quant that fits in 96GB VRAM while still leaving safe headroom for 32k context. Same quant across all four rigs, fairest comparison I could run. Now look at rough peak GPU power draw: 🟠 4x4090 → 1,800W peak (450W × 4) 🟢 4x5090 → 2,300W peak (575W × 4) 🟡 RTX PRO 6000 → 600W peak 🟣 DGX Spark → 240W peak (whole system) The RTX PRO 6000 is the quiet winner. One card, 96GB, matching a 4x5090 rig at roughly a quarter of the power and zero multi-GPU headaches. Best tokens-per-watt by a wide margin. DGX Spark is slow on generation but pulls the least power of any rig here, around 240W for the whole system. Prefill-friendly, memory-rich, wall-socket-friendly. And yes, plenty of people cap their cards. Even then, 4x 4090 or 4x 5090 still pulls well over 1,200W from the GPUs alone.
1:21
stevibe's profile picture

MiniMax M2.7 is 230B params. Can you actually run it at home? I tested Unsloth's UD-IQ3_XXS (80GB) on 4 different rigs: 🟠 4x RTX 4090 (96GB): 71.52 tok/s, TTFT 1045ms 🟢 4x RTX 5090 (128GB): 120.54 tok/s, TTFT 725ms 🟡 1x RTX PRO 6000 (96GB): 118.74 tok/s, TTFT 765ms 🟣 DGX Spark (128GB) — 24.41 tok/s, TTFT 741ms Backend: llama.cpp. Context: 32k. Max tokens: 4096. I went with IQ3_XXS because it's the biggest quant that fits in 96GB VRAM while still leaving safe headroom for 32k context. Same quant across all four rigs, fairest comparison I could run. Now look at rough peak GPU power draw: 🟠 4x4090 → 1,800W peak (450W × 4) 🟢 4x5090 → 2,300W peak (575W × 4) 🟡 RTX PRO 6000 → 600W peak 🟣 DGX Spark → 240W peak (whole system) The RTX PRO 6000 is the quiet winner. One card, 96GB, matching a 4x5090 rig at roughly a quarter of the power and zero multi-GPU headaches. Best tokens-per-watt by a wide margin. DGX Spark is slow on generation but pulls the least power of any rig here, around 240W for the whole system. Prefill-friendly, memory-rich, wall-socket-friendly. And yes, plenty of people cap their cards. Even then, 4x 4090 or 4x 5090 still pulls well over 1,200W from the GPUs alone.

stevibe

193,239 Aufrufe • vor 6 Monaten

Finally got my hands on the big one. Qwen3.5-122B-A10B — 122 billion parameters. Too big for any single consumer GPU. So I rented 4 of each... and then one professional card to see if brute force even matters. - 1x RTX PRO 6000 (96GB): 101.4 tok/s - 4x 5090 (128GB): 87.0 tok/s - 4x 4090 (96GB): 25.1 tok/s - 4x 3090 (96GB): 20.8 tok/s One single $8,500 card beat four RTX 5090s
0:59
stevibe's profile picture

Finally got my hands on the big one. Qwen3.5-122B-A10B — 122 billion parameters. Too big for any single consumer GPU. So I rented 4 of each... and then one professional card to see if brute force even matters. - 1x RTX PRO 6000 (96GB): 101.4 tok/s - 4x 5090 (128GB): 87.0 tok/s - 4x 4090 (96GB): 25.1 tok/s - 4x 3090 (96GB): 20.8 tok/s One single $8,500 card beat four RTX 5090s

stevibe

197,300 Aufrufe • vor 7 Monaten

You know that "But, wait..." moment in every LLM thinking trace? I made it visible. I asked 8 models the same tricky probability question and rendered their reasoning as trees. Every time a model rejects its own idea and pivots, every "But...", every "Wait, actually...", a new branch grows. Same question. Completely different minds.
0:20
stevibe's profile picture

You know that "But, wait..." moment in every LLM thinking trace? I made it visible. I asked 8 models the same tricky probability question and rendered their reasoning as trees. Every time a model rejects its own idea and pivots, every "But...", every "Wait, actually...", a new branch grows. Same question. Completely different minds.

stevibe

86,817 Aufrufe • vor 3 Monaten

Surprisingly, DeepSeek V4 Flash 0731 beats the Pro 0813 at canvas coding.
0:35
stevibe's profile picture

Surprisingly, DeepSeek V4 Flash 0731 beats the Pro 0813 at canvas coding.

stevibe

54,487 Aufrufe • vor 2 Monaten

I gave Kimi K2.6 and K2.7 Code the exact same prompt to animate a letter burning to ash. Pure HTML canvas, zero libraries.
0:20
stevibe's profile picture

I gave Kimi K2.6 and K2.7 Code the exact same prompt to animate a letter burning to ash. Pure HTML canvas, zero libraries.

stevibe

90,027 Aufrufe • vor 4 Monaten

The RTX 3090 is a 5-year-old GPU and it still runs a 27B model at 20 tok/s I tested Qwen3.5:27b across 3 generations of NVIDIA: 5090 → ~60 tok/s 4090 → ~40 tok/s 3090 → ~20 tok/s Perfectly linear scaling. Double the generation, double the speed.
0:57
stevibe's profile picture

The RTX 3090 is a 5-year-old GPU and it still runs a 27B model at 20 tok/s I tested Qwen3.5:27b across 3 generations of NVIDIA: 5090 → ~60 tok/s 4090 → ~40 tok/s 3090 → ~20 tok/s Perfectly linear scaling. Double the generation, double the speed.

stevibe

142,493 Aufrufe • vor 7 Monaten

So we know Gemma 4 is good at tool calling, but what about web coding? I threw 4 UI screenshots at three Gemma 4 models and said rebuild this, one shot, no hand-holding, just image in, code out. Model lineup: - E4B - 26B A4B (MoE) - 31B Dense (skipped the E2B this round) Let me know which one you think cooked the hardest
0:21
stevibe's profile picture

So we know Gemma 4 is good at tool calling, but what about web coding? I threw 4 UI screenshots at three Gemma 4 models and said rebuild this, one shot, no hand-holding, just image in, code out. Model lineup: - E4B - 26B A4B (MoE) - 31B Dense (skipped the E2B this round) Let me know which one you think cooked the hardest

stevibe

124,942 Aufrufe • vor 6 Monaten

Completed a first hour side-by-side comparison between Qwen3.5 27b and Qwen3.6 27b on the same 4 canvas coding tests. Running the Qwen3.6 27b FP8, vLLM. What do you think?
1:07
stevibe's profile picture

Completed a first hour side-by-side comparison between Qwen3.5 27b and Qwen3.6 27b on the same 4 canvas coding tests. Running the Qwen3.6 27b FP8, vLLM. What do you think?

stevibe

114,366 Aufrufe • vor 5 Monaten

Yesterday we found out Qwen3.5-27B (dense) beats both the 35B and 122B MoE variants at UI replication from screenshots. Today's question: what about Jackrong's Qwen3.5-27B-Claude-Opus-4.6-Reasoning-Distilled? A distilled 27B with Claude Opus 4.6 reasoning baked in. So I put all 3 in the ring: - Qwen3.5-27B - Qwen3.5-27B-Claude-Opus-Distilled - Claude Opus 4.6 Since the distilled model isn't multimodal, I gave all three the same detailed text prompts describing the UI components from yesterday's test.
0:31
stevibe's profile picture

Yesterday we found out Qwen3.5-27B (dense) beats both the 35B and 122B MoE variants at UI replication from screenshots. Today's question: what about Jackrong's Qwen3.5-27B-Claude-Opus-4.6-Reasoning-Distilled? A distilled 27B with Claude Opus 4.6 reasoning baked in. So I put all 3 in the ring: - Qwen3.5-27B - Qwen3.5-27B-Claude-Opus-Distilled - Claude Opus 4.6 Since the distilled model isn't multimodal, I gave all three the same detailed text prompts describing the UI components from yesterday's test.

stevibe

126,288 Aufrufe • vor 6 Monaten

Update: Qwen3.5:9b head-to-head (MLX, Apple Silicon optimized) Mac Studio M2 Ultra: 89.74 tok/s Mac Mini M4: 20.82 tok/s MLX basically doubles the speed on both machines.
1:02
stevibe's profile picture

Update: Qwen3.5:9b head-to-head (MLX, Apple Silicon optimized) Mac Studio M2 Ultra: 89.74 tok/s Mac Mini M4: 20.82 tok/s MLX basically doubles the speed on both machines.

stevibe

134,256 Aufrufe • vor 7 Monaten