Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

GLM 5.1 vs GLM 5.2 6 advanced HTML canvas challenges: 💧 Ink diffusing in water ⚔️ Energy-blade duel 📱 Slide to unlock 🅿️ 360° parking assist 🔥 Burning letter to ash 🏠 Build-a-house sequence Pure canvas, zero libraries.

stevibe

27,592 subscribers

347,364 Aufrufe • vor 1 Monat •via X (Twitter)

Bildung Wissenschaft & Technologie

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

I gave Kimi K2.6 and K2.7 Code the exact same prompt to animate a letter burning to ash. Pure HTML canvas, zero libraries.

I gave Kimi K2.6 and K2.7 Code the exact same prompt to animate a letter burning to ash. Pure HTML canvas, zero libraries.

stevibe

89,828 Aufrufe • vor 1 Monat

Meituan's LongCat-2.0 reportedly lands near GPT-5.5 on SWE-bench. So I threw 5 HTML canvas animation prompts at both. 🥷 Paper sliced fruit-ninja style. 💧 An ink drop diffusing in water. 🔥 A letter burning. 🗑️ Paper crumpling into a ball. ✂️ A strip-cut shredder. Here's how they did 👇

Meituan's LongCat-2.0 reportedly lands near GPT-5.5 on SWE-bench. So I threw 5 HTML canvas animation prompts at both. 🥷 Paper sliced fruit-ninja style. 💧 An ink drop diffusing in water. 🔥 A letter burning. 🗑️ Paper crumpling into a ball. ✂️ A strip-cut shredder. Here's how they did 👇

stevibe

34,963 Aufrufe • vor 1 Monat

Got a 24GB Graphics Card? These 6 coding models all fit on it (Q4): - qwen3.5:27b (17GB) - qwen3.5:35b (24GB) - glm-4.7-flash (19GB) - nemotron-3-nano:30b (24GB) - nemotron-cascade-2:30b (24GB) - gpt-oss:20b (14GB) I gave them the same challenge: draw a campfire with HTML Canvas. Why Canvas? HTML/CSS forgives bad syntax — things still render. JavaScript + Canvas doesn't — one mistake and the screen goes black.

Got a 24GB Graphics Card? These 6 coding models all fit on it (Q4): - qwen3.5:27b (17GB) - qwen3.5:35b (24GB) - glm-4.7-flash (19GB) - nemotron-3-nano:30b (24GB) - nemotron-cascade-2:30b (24GB) - gpt-oss:20b (14GB) I gave them the same challenge: draw a campfire with HTML Canvas. Why Canvas? HTML/CSS forgives bad syntax — things still render. JavaScript + Canvas doesn't — one mistake and the screen goes black.

stevibe

75,939 Aufrufe • vor 4 Monaten

Round 2: Dense edition Qwen3.5 27B vs Gemma4 31B Same 4 HTML canvas challenges, this time no MoE. Setup: > Quant: Unsloth Q6_K_XL (both) > Each running on an RTX 5090 > Single HTML file, no libraries, real-time animation My read per prompt: 🌳 Tree — Gemma, branches are beautifully structured ✨ Night sky — close, but Qwen's stars blink better 🐟 Fish boids — Qwen, detailed fish and look at that motion, so realistic 🏠 House build — neither really nailed it Agree? Does dense actually beat MoE here?

Round 2: Dense edition Qwen3.5 27B vs Gemma4 31B Same 4 HTML canvas challenges, this time no MoE. Setup: > Quant: Unsloth Q6_K_XL (both) > Each running on an RTX 5090 > Single HTML file, no libraries, real-time animation My read per prompt: 🌳 Tree — Gemma, branches are beautifully structured ✨ Night sky — close, but Qwen's stars blink better 🐟 Fish boids — Qwen, detailed fish and look at that motion, so realistic 🏠 House build — neither really nailed it Agree? Does dense actually beat MoE here?

stevibe

33,304 Aufrufe • vor 3 Monaten

Claude Opus 4.7 vs 4.8 side-by-side canvas test

Claude Opus 4.7 vs 4.8 side-by-side canvas test

stevibe

476,059 Aufrufe • vor 2 Monaten

DeepSeek V4 Flash vs V4 Pro, 4 canvas tests run. I'm a dev, and honestly DeepSeek was never on my list for coding tasks. After seeing these results, I might need to reconsider. 🌳 Tree: Flash actually beat Pro: denser canopy, better leaf distribution 🌌 Night sky: Pro wins, shooting stars, constellations, Milky Way band all there 🐟 Fish boids: Pro is stunning: proper caustics, tight schooling, real underwater feel 🏠 House builder: first model we've tested that knows how to actually build a rooftop DeepSeek coding might be underrated.

DeepSeek V4 Flash vs V4 Pro, 4 canvas tests run. I'm a dev, and honestly DeepSeek was never on my list for coding tasks. After seeing these results, I might need to reconsider. 🌳 Tree: Flash actually beat Pro: denser canopy, better leaf distribution 🌌 Night sky: Pro wins, shooting stars, constellations, Milky Way band all there 🐟 Fish boids: Pro is stunning: proper caustics, tight schooling, real underwater feel 🏠 House builder: first model we've tested that knows how to actually build a rooftop DeepSeek coding might be underrated.

stevibe

56,270 Aufrufe • vor 3 Monaten

3 ways to destroy a piece of paper. Qwen 3.5 35B A3B vs. Ornith 1.0 35B, side-by-side canvas test. (Why 3.5 not 3.6? Ornith is post-trained on Qwen 3.5 and Gemma 4, so this shows what the post-training adds.) Same 3 challenges: 🔪 Slice: three blade swipes, fruit-game style 📄 Shredder: desktop strip-cut 🗑️ Crumple: balled up and tossed Winner: not close. Ornith, decisively. The post-training quality is REAL.

3 ways to destroy a piece of paper. Qwen 3.5 35B A3B vs. Ornith 1.0 35B, side-by-side canvas test. (Why 3.5 not 3.6? Ornith is post-trained on Qwen 3.5 and Gemma 4, so this shows what the post-training adds.) Same 3 challenges: 🔪 Slice: three blade swipes, fruit-game style 📄 Shredder: desktop strip-cut 🗑️ Crumple: balled up and tossed Winner: not close. Ornith, decisively. The post-training quality is REAL.

stevibe

47,792 Aufrufe • vor 1 Monat

"Its (Sonnet 5) performance is close to Opus 4.8, at lower prices." So I ran 4 canvas test through both. > Opus 4.8, 4/4 actually animating. > Sonnet 5, 2/4 came back as static images. And "lower price"? On the paper shredder task, Sonnet 5 spent $0.36 for a static image. Opus 4.8 spent $0.18 and it actually animated. The 4 tests: > Win 98 drag-to-BSOD > Self-typing keyboard + CRT > Letter burning > Paper shredder

"Its (Sonnet 5) performance is close to Opus 4.8, at lower prices." So I ran 4 canvas test through both. > Opus 4.8, 4/4 actually animating. > Sonnet 5, 2/4 came back as static images. And "lower price"? On the paper shredder task, Sonnet 5 spent $0.36 for a static image. Opus 4.8 spent $0.18 and it actually animated. The 4 tests: > Win 98 drag-to-BSOD > Self-typing keyboard + CRT > Letter burning > Paper shredder

stevibe

60,788 Aufrufe • vor 1 Monat

Six open-source LLMs. One sliding puzzle. A brutal test of long-horizon reasoning and tool calling. Five of them broke. One didn't. I gave each model a move_tile tool and a scrambled 3×3 board, then asked it to solve the puzzle through pure turn-by-turn reasoning. The deeper the scramble, the more brutal the search. Five runs per depth, best run kept. A model fails the round if it exceeds 6x the optimal move count. > Depth 5: Everyone solves it. Yawn. > Depth 10: GLM 5.1 melts down. 43 moves. Cut. > Depth 12: Gemma4 26B loses the plot, shuffling tiles in circles. Gone. > Depth 15: The wall. DeepSeek V4 Flash, out. DeepSeek V4 Pro, out. Gemma4, out again. GLM 5.1, out. Two survivors: Qwen3.6 35B-A3B, and Kimi K2.6 with an 11-move solve that looked like cheating. > Depth 18: Same two. Everyone else hallucinating tiles that weren't there. > Depth 22: Final boss. Kimi, flawless for five rounds, finally cracks. 81 moves. Still scrambled. DeepSeek V4 Pro limps home at 90. Qwen3.6 35B-A3B solves it in 36. The smallest model in the room. ~3B active params. Fits on a single 3090. It beat everything. Kimi was elegant. Qwen3.6 was unstoppable.

Six open-source LLMs. One sliding puzzle. A brutal test of long-horizon reasoning and tool calling. Five of them broke. One didn't. I gave each model a move_tile tool and a scrambled 3×3 board, then asked it to solve the puzzle through pure turn-by-turn reasoning. The deeper the scramble, the more brutal the search. Five runs per depth, best run kept. A model fails the round if it exceeds 6x the optimal move count. > Depth 5: Everyone solves it. Yawn. > Depth 10: GLM 5.1 melts down. 43 moves. Cut. > Depth 12: Gemma4 26B loses the plot, shuffling tiles in circles. Gone. > Depth 15: The wall. DeepSeek V4 Flash, out. DeepSeek V4 Pro, out. Gemma4, out again. GLM 5.1, out. Two survivors: Qwen3.6 35B-A3B, and Kimi K2.6 with an 11-move solve that looked like cheating. > Depth 18: Same two. Everyone else hallucinating tiles that weren't there. > Depth 22: Final boss. Kimi, flawless for five rounds, finally cracks. 81 moves. Still scrambled. DeepSeek V4 Pro limps home at 90. Qwen3.6 35B-A3B solves it in 36. The smallest model in the room. ~3B active params. Fits on a single 3090. It beat everything. Kimi was elegant. Qwen3.6 was unstoppable.

stevibe

16,434 Aufrufe • vor 3 Monaten

Completed a first hour side-by-side comparison between Qwen3.5 27b and Qwen3.6 27b on the same 4 canvas coding tests. Running the Qwen3.6 27b FP8, vLLM. What do you think?

Completed a first hour side-by-side comparison between Qwen3.5 27b and Qwen3.6 27b on the same 4 canvas coding tests. Running the Qwen3.6 27b FP8, vLLM. What do you think?

stevibe

114,191 Aufrufe • vor 3 Monaten

Introducing HermesAgent-20, a new Bench Pack for BenchLocal. 20 scenarios extracted straight from the Hermes Agent source code, run against a REAL Hermes instance. The actual workload you'd put your model through. Why I built BenchLocal in the first place: most benchmarks are too abstract. We use local LLMs for practical work, and finding the right model for YOUR task efficiently is the single most important thing, especially when you're constrained to what fits on your machine. BenchLocal is a framework: providers, models, side-by-side comparison, all in one UI. Bench Packs are the unit of testing: ToolCall-15 and BugFind-15 shipped first, and when I launched the BenchLocal 0.1.0, added StructOutput, ReasonMath, InstructFollow, DataExtract. Now, HermesAgent-20 is the newest. Bench Packs install like VS Code extensions. The SDK is open, write your own, share it, grow the ecosystem. Here's the goal: a community-built, practical evaluation layer for the local LLM space. Early numbers on HermesAgent-20: > GLM 5.1 — 85 > Gemma4 31B — 83 > Qwen3.5 27B — 79 > MiniMax M2.7 — 76 Upgrade to the latest BenchLocal to install HermesAgent-20 (SDK update required).

Introducing HermesAgent-20, a new Bench Pack for BenchLocal. 20 scenarios extracted straight from the Hermes Agent source code, run against a REAL Hermes instance. The actual workload you'd put your model through. Why I built BenchLocal in the first place: most benchmarks are too abstract. We use local LLMs for practical work, and finding the right model for YOUR task efficiently is the single most important thing, especially when you're constrained to what fits on your machine. BenchLocal is a framework: providers, models, side-by-side comparison, all in one UI. Bench Packs are the unit of testing: ToolCall-15 and BugFind-15 shipped first, and when I launched the BenchLocal 0.1.0, added StructOutput, ReasonMath, InstructFollow, DataExtract. Now, HermesAgent-20 is the newest. Bench Packs install like VS Code extensions. The SDK is open, write your own, share it, grow the ecosystem. Here's the goal: a community-built, practical evaluation layer for the local LLM space. Early numbers on HermesAgent-20: > GLM 5.1 — 85 > Gemma4 31B — 83 > Qwen3.5 27B — 79 > MiniMax M2.7 — 76 Upgrade to the latest BenchLocal to install HermesAgent-20 (SDK update required).

stevibe

38,631 Aufrufe • vor 3 Monaten

Opus 4.7 first-hour impressions Ran the canvas tree growth test twice. 4.6: nailed the animation both times 4.7: static tree, no growth animation — twice 4.7's thinking is noticeably shorter and faster though (trimmed some 4.6 thinking in the clip for pacing). Not the upgrade direction I expected on this one.

Opus 4.7 first-hour impressions Ran the canvas tree growth test twice. 4.6: nailed the animation both times 4.7: static tree, no growth animation — twice 4.7's thinking is noticeably shorter and faster though (trimmed some 4.6 thinking in the clip for pacing). Not the upgrade direction I expected on this one.

stevibe

488,060 Aufrufe • vor 3 Monaten

I designed a new test specifically for multimodal models: fill out a paper form. And it's much harder than it sounds. This isn't typing into an electronic field that captures your text. The form is just an image. The model has to place each form element: text, checkmarks — at the correct pixel position on the canvas itself. Results: 🟢 Kimi K2.6 → done in 3:45, 16.7k output tokens 🟡 Step 3.7 Flash → half the fields, 57k output tokens 🔴 Gemini 3.5 Flash → 489k output tokens, never finished. I had to kill it. Gemini burned ~29x more output tokens than Kimi on the exact same task, and Kimi's was the only form that actually looked filled out. The test, a mocked application form, contains some challenging parts, such as one-character-per-box fields. I provided every model the same set of tools: > get canvas size > drop probe markers to find coordinates > add text > add checkmarks > move elements > take a screenshot anytime to check their own work > ... etc So it's vision + spatial reasoning + tool use + long context, all at once. Small models (Qwen, Gemma) can't really complete this test, so I skipped them. What happened: > Kimi nailed name, DOB, ID, gender, marital status, nationality, email, phone, address, postal code — placement slightly loose, but content correct. 15 turns. Clean. > Step got maybe half right — fields dropped, "United States" landed in the email line, data floating outside boxes. Burned 1.24M input tokens doing it (81 turns of re-reading the canvas). > Gemini almost got there visually... then spiraled. By turn 40 it was issuing a delete_elements call wiping element IDs 365–425, basically erasing its own work. 31 minutes, 489k output tokens, still streaming. Terminated. The takeaway isn't "Gemini bad." This test is indeed difficult. But token efficiency is capability now. A model that needs 30x the tokens and still can't converge is going to be 30x the cost in production. Kimi K2.6 just quietly did the thing.

I designed a new test specifically for multimodal models: fill out a paper form. And it's much harder than it sounds. This isn't typing into an electronic field that captures your text. The form is just an image. The model has to place each form element: text, checkmarks — at the correct pixel position on the canvas itself. Results: 🟢 Kimi K2.6 → done in 3:45, 16.7k output tokens 🟡 Step 3.7 Flash → half the fields, 57k output tokens 🔴 Gemini 3.5 Flash → 489k output tokens, never finished. I had to kill it. Gemini burned ~29x more output tokens than Kimi on the exact same task, and Kimi's was the only form that actually looked filled out. The test, a mocked application form, contains some challenging parts, such as one-character-per-box fields. I provided every model the same set of tools: > get canvas size > drop probe markers to find coordinates > add text > add checkmarks > move elements > take a screenshot anytime to check their own work > ... etc So it's vision + spatial reasoning + tool use + long context, all at once. Small models (Qwen, Gemma) can't really complete this test, so I skipped them. What happened: > Kimi nailed name, DOB, ID, gender, marital status, nationality, email, phone, address, postal code — placement slightly loose, but content correct. 15 turns. Clean. > Step got maybe half right — fields dropped, "United States" landed in the email line, data floating outside boxes. Burned 1.24M input tokens doing it (81 turns of re-reading the canvas). > Gemini almost got there visually... then spiraled. By turn 40 it was issuing a delete_elements call wiping element IDs 365–425, basically erasing its own work. 31 minutes, 489k output tokens, still streaming. Terminated. The takeaway isn't "Gemini bad." This test is indeed difficult. But token efficiency is capability now. A model that needs 30x the tokens and still can't converge is going to be 30x the cost in production. Kimi K2.6 just quietly did the thing.

stevibe

25,446 Aufrufe • vor 2 Monaten

This looks like a toy. It's actually the meanest little vision eval I've built. The task: look at an emoji image, then repaint it on a 16×16 grid, one pixel at a time. Just the model, a tiny canvas, and up to 2000 brushstrokes. What I didn't expect was the personalities. > some models REGRET a stroke and go back to repaint it > some get stuck looping the same little patch over and over, like they're trying to animate it > some are calm little surgeons and just nail it first try And the task is genuinely mean: it has to see the image, crush it down to 256 cells, then decide what's actually load-bearing: > the tears on 😂 but still keep the smile > the horn on 🦄 > the antenna on 🤖 and keep the soul of it with almost no resolution to spare. 5 models. 7 emojis. Best of 5 runs each. Side by side. Who's your winner?

This looks like a toy. It's actually the meanest little vision eval I've built. The task: look at an emoji image, then repaint it on a 16×16 grid, one pixel at a time. Just the model, a tiny canvas, and up to 2000 brushstrokes. What I didn't expect was the personalities. > some models REGRET a stroke and go back to repaint it > some get stuck looping the same little patch over and over, like they're trying to animate it > some are calm little surgeons and just nail it first try And the task is genuinely mean: it has to see the image, crush it down to 256 cells, then decide what's actually load-bearing: > the tears on 😂 but still keep the smile > the horn on 🦄 > the antenna on 🤖 and keep the soul of it with almost no resolution to spare. 5 models. 7 emojis. Best of 5 runs each. Side by side. Who's your winner?

stevibe

284,178 Aufrufe • vor 1 Monat

sonnet 5 vs sonnet 4.6 vs opus 4.8 vs glm 5.2 – frontend tasks dropped sonnet 5 into a quick test today. same three prompts to all four models, single-shot html/canvas, no edits: • objects falling on a trampoline • rockets playing tennis • a slingshot breaking bottles ranked by speed (total across the 3 tasks): 1. opus 4.8 – 15m 09s 2. sonnet 5 – 16m 05s 3. glm 5.2 – 27m 18s 4. sonnet 4.6 – 35m 06s ranked by code shortness (total loc): 1. sonnet 5 – 1794 2. opus 4.8 – 2063 3. sonnet 4.6 – 2182 4. glm 5.2 – 3285 sonnet 5 came out on top here – leanest code overall and a near-tie for fastest it was also the most creative. in every task it added something none of the others did: – kept the trampoline vibrating after the objects landed – drew a +1 next to the rocket that scored the point – turned the slingshot to face the next bottle before each shot opus 4.8 evaluated the code sonnet 5 produced. four things stood out: • the sphere is a fake, and that's the smart move. the cube and star are real 3d meshes with proper culling and shading, but the ball is just a flat shaded circle. a lit sphere looks identical from every angle, so building it in 3d would burn compute for zero visible payoff. knowing where not to bother is its own kind of skill • weight actually means something on the trampoline. the star is heavy, so it barely bounces and dents the mat hard. the ball is light, so it's lively and leaves a shallow dip. the three objects aren't just different shapes – they have different temperaments, and the physics is what gives them that • the slingshot is framed like a shot, not just drawn. the handle is anchored below the bottom of the screen and runs off-frame, so it reads as something you're holding rather than a sprite parked in the scene. that's a staging instinct, not a rendering one • the paddle ai forward-simulates the ball to predict where it'll land, then adds a deliberate error bias (roughly 1 in 5 shots is a real miss). that's why scoring looks natural instead of robotic – plus four distinct fault types with a catch-all so a rally never hangs without a result bottom line: sonnet 5 does more with less. fastest tier, leanest code, and the only one that added small touches nobody asked for follow thehype. for 24/7 ai news, analysis and breakdowns

sonnet 5 vs sonnet 4.6 vs opus 4.8 vs glm 5.2 – frontend tasks dropped sonnet 5 into a quick test today. same three prompts to all four models, single-shot html/canvas, no edits: • objects falling on a trampoline • rockets playing tennis • a slingshot breaking bottles ranked by speed (total across the 3 tasks): 1. opus 4.8 – 15m 09s 2. sonnet 5 – 16m 05s 3. glm 5.2 – 27m 18s 4. sonnet 4.6 – 35m 06s ranked by code shortness (total loc): 1. sonnet 5 – 1794 2. opus 4.8 – 2063 3. sonnet 4.6 – 2182 4. glm 5.2 – 3285 sonnet 5 came out on top here – leanest code overall and a near-tie for fastest it was also the most creative. in every task it added something none of the others did: – kept the trampoline vibrating after the objects landed – drew a +1 next to the rocket that scored the point – turned the slingshot to face the next bottle before each shot opus 4.8 evaluated the code sonnet 5 produced. four things stood out: • the sphere is a fake, and that's the smart move. the cube and star are real 3d meshes with proper culling and shading, but the ball is just a flat shaded circle. a lit sphere looks identical from every angle, so building it in 3d would burn compute for zero visible payoff. knowing where not to bother is its own kind of skill • weight actually means something on the trampoline. the star is heavy, so it barely bounces and dents the mat hard. the ball is light, so it's lively and leaves a shallow dip. the three objects aren't just different shapes – they have different temperaments, and the physics is what gives them that • the slingshot is framed like a shot, not just drawn. the handle is anchored below the bottom of the screen and runs off-frame, so it reads as something you're holding rather than a sprite parked in the scene. that's a staging instinct, not a rendering one • the paddle ai forward-simulates the ball to predict where it'll land, then adds a deliberate error bias (roughly 1 in 5 shots is a real miss). that's why scoring looks natural instead of robotic – plus four distinct fault types with a catch-all so a rally never hangs without a result bottom line: sonnet 5 does more with less. fastest tier, leanest code, and the only one that added small touches nobody asked for follow thehype. for 24/7 ai news, analysis and breakdowns

thehype.

14,518 Aufrufe • vor 1 Monat

The AI price war is ON. > DeepSeek just dropped V4 Flash 0731 (big capability jump, $0.28/M output) > OpenAI cut GPT-5.6 Luna by 80% to $1.20/M (OpenRouter is currently offering 50% off, bringing it to $0.60/M) Same budget tier now, so I ran both through 3 canvas tests: 🧊 Rubik's cube: DeepSeek nails every rotation. Luna's turns are buggy, stickers fly off mid-move. 🎆 Fireworks: both fine, but Luna lags hard when the burst explodes. 🖊️ Pen writing "Hello": neither is legible. DeepSeek's cursive is looser, but Luna gave up on animating, it rendered a static image. GPT-5.6 is great at coding — but IMO only on Sol. In the same pricing tier, DeepSeek clearly beats Luna.

The AI price war is ON. > DeepSeek just dropped V4 Flash 0731 (big capability jump, $0.28/M output) > OpenAI cut GPT-5.6 Luna by 80% to $1.20/M (OpenRouter is currently offering 50% off, bringing it to $0.60/M) Same budget tier now, so I ran both through 3 canvas tests: 🧊 Rubik's cube: DeepSeek nails every rotation. Luna's turns are buggy, stickers fly off mid-move. 🎆 Fireworks: both fine, but Luna lags hard when the burst explodes. 🖊️ Pen writing "Hello": neither is legible. DeepSeek's cursive is looser, but Luna gave up on animating, it rendered a static image. GPT-5.6 is great at coding — but IMO only on Sol. In the same pricing tier, DeepSeek clearly beats Luna.

stevibe

268,581 Aufrufe • vor 10 Tagen

I built a macOS app for benchmarking local LLMs. 6 test suites. Multiple providers. One workspace. Open source. There are hundreds of local models now. New ones every week. How do you actually pick one? Leaderboards test for general ability. But if you're building an agent that chains tool calls, or a pipeline that extracts structured data, or a code assistant that needs to debug Rust, you need to know if the model handles that specific thing. Not in theory. On your hardware. With your prompts. The benchmarks that exist are either locked behind papers, too abstract to map to real failures, or impossible to extend. You can't add your own test cases. You can't test what matters to your use case. That's what BenchLocal is for. It's a benchmark platform where every test is practical, deterministic, and built around real-world tasks. And you can build your own tests. It ships with 6 Bench Packs TODAY: → ToolCall-15 — tool-use accuracy → BugFind-15 — debugging capabilities → DataExtract-15 — structured data extraction → InstructFollow-15 — constraint-heavy instruction following → ReasonMath-15 — practical reasoning and math → StructOutput-15 — validator-backed structured output Every pack has 15 fixed scenarios. Every score is deterministic and verifiable. Some of you saw ToolCall-15 and BugFind-15 — the individual test packs I open-sourced over the past few weeks. People ran them, filed issues, sent PRs. But managing separate repos, separate scripts, separate results doesn't scale. BenchLocal puts everything in one place. What the app does: > Workspace with tabs — run BugFind-15 in one tab, ToolCall-15 in another. > Any provider — Ollama, llama.cpp, OpenRouter, any OpenAI-compatible endpoint. Local and cloud, same interface. > Run modes — serial, batch per model, batch per test case, or fully parallel. > Test histories — every run saved. Compare any previous session. But the part I'm most excited about isn't the app. It's the ecosystem. BenchLocal is a platform. Each Bench Pack is a plugin. I'm shipping an SDK so anyone can build their own — test what matters to you, package it, share it. Install and uninstall packs right inside the app, same way you'd manage extensions in VS Code. The registry is GitHub-based, fully public. I built 6 packs. I want the community to build the next 60. Theme system built in too — because if I'm staring at benchmark results for hours, it should at least look good. v0.1.0 is macOS only. Windows and Linux are coming. MIT licensed. Everything — the app, the bench packs, the SDK — is open. PRs welcome. Bench Packs even more welcome.

I built a macOS app for benchmarking local LLMs. 6 test suites. Multiple providers. One workspace. Open source. There are hundreds of local models now. New ones every week. How do you actually pick one? Leaderboards test for general ability. But if you're building an agent that chains tool calls, or a pipeline that extracts structured data, or a code assistant that needs to debug Rust, you need to know if the model handles that specific thing. Not in theory. On your hardware. With your prompts. The benchmarks that exist are either locked behind papers, too abstract to map to real failures, or impossible to extend. You can't add your own test cases. You can't test what matters to your use case. That's what BenchLocal is for. It's a benchmark platform where every test is practical, deterministic, and built around real-world tasks. And you can build your own tests. It ships with 6 Bench Packs TODAY: → ToolCall-15 — tool-use accuracy → BugFind-15 — debugging capabilities → DataExtract-15 — structured data extraction → InstructFollow-15 — constraint-heavy instruction following → ReasonMath-15 — practical reasoning and math → StructOutput-15 — validator-backed structured output Every pack has 15 fixed scenarios. Every score is deterministic and verifiable. Some of you saw ToolCall-15 and BugFind-15 — the individual test packs I open-sourced over the past few weeks. People ran them, filed issues, sent PRs. But managing separate repos, separate scripts, separate results doesn't scale. BenchLocal puts everything in one place. What the app does: > Workspace with tabs — run BugFind-15 in one tab, ToolCall-15 in another. > Any provider — Ollama, llama.cpp, OpenRouter, any OpenAI-compatible endpoint. Local and cloud, same interface. > Run modes — serial, batch per model, batch per test case, or fully parallel. > Test histories — every run saved. Compare any previous session. But the part I'm most excited about isn't the app. It's the ecosystem. BenchLocal is a platform. Each Bench Pack is a plugin. I'm shipping an SDK so anyone can build their own — test what matters to you, package it, share it. Install and uninstall packs right inside the app, same way you'd manage extensions in VS Code. The registry is GitHub-based, fully public. I built 6 packs. I want the community to build the next 60. Theme system built in too — because if I'm staring at benchmark results for hours, it should at least look good. v0.1.0 is macOS only. Windows and Linux are coming. MIT licensed. Everything — the app, the bench packs, the SDK — is open. PRs welcome. Bench Packs even more welcome.

stevibe

50,584 Aufrufe • vor 3 Monaten

Which LLMs actually love to think? Tested 7 models on 5 math problems, measured reasoning length. The think winners: both Qwen3.5 models (27B and 35B A3B) — massive overthinkers, up to 10k+ tokens on a single question. Plot twists: > Kimi K2.6 feels verbose, actually one of the leanest > Gemma4 26B A4B solved 2 with ZERO thinking

Which LLMs actually love to think? Tested 7 models on 5 math problems, measured reasoning length. The think winners: both Qwen3.5 models (27B and 35B A3B) — massive overthinkers, up to 10k+ tokens on a single question. Plot twists: > Kimi K2.6 feels verbose, actually one of the leanest > Gemma4 26B A4B solved 2 with ZERO thinking

stevibe

97,403 Aufrufe • vor 3 Monaten

Qwen 3.8 Max vs Opus 5, side-by-side. 🌀 The Inception top, spin, wobble, fall 💡 A hand-crank generator lighting a bulb Qwen 3.8 Max vs Claude Opus 5. Both rendered. One costs 8x more. Worth it? Watch and decide.

Qwen 3.8 Max vs Opus 5, side-by-side. 🌀 The Inception top, spin, wobble, fall 💡 A hand-crank generator lighting a bulb Qwen 3.8 Max vs Claude Opus 5. Both rendered. One costs 8x more. Worth it? Watch and decide.

stevibe

49,166 Aufrufe • vor 7 Tagen

🔥Sakana Fugu-ultra 🟩OpenAI GPT 5.5 🟩GLM 5.2 🟩Opus 4.8 TASK Create a high-quality single HTML file simulation of a Rube Goldberg / chain-reaction machine. The simulation should run automatically from start to finish without user input.The goal is to demonstrate whether the coding model understands real-world physics well enough to create believable cause-and-effect behavior - Rube Goldberg / chain-reaction machine. The scene should show a small physical world where objects interact through realistic mechanics: gravity, collisions, momentum transfer, ramps, pulleys, levers, springs, falling objects, rolling balls, dominoes, and water or particles if possible. The simulation should include: * A ball rolling down a ramp * Dominoes falling one after another * A lever or seesaw transferring force * A spring launcher * A pendulum or swinging weight * A pulley or elevator mechanism * Falling objects affected by gravity * Objects with different mass, friction, and bounce * Clear visual labels explaining what physical law is being demonstrated * A progress timeline showing the current stage of the chain reaction * Automatic reset/replay after the sequence ends The simulation should visually demonstrate: * Gravity * Momentum transfer * Conservation of energy * Friction * Torque * Elastic potential energy * Collision response * Cause and effect The model should not just animate objects on fixed paths. Objects should appear to interact physically. For example, the ball should knock over dominoes, dominoes should push a lever, the lever should launch another object, and so on. Evaluation criteria 1. Does the simulation run automatically without user input? 2. Do objects interact through believable physical cause and effect? 3. Are physical laws represented correctly? 4. Are labels and UI helpful? 5. Is the animation smooth and stable? 6. Does the code avoid fake-looking hardcoded movement? 7. Does the scene reset cleanly?

🔥Sakana Fugu-ultra 🟩OpenAI GPT 5.5 🟩GLM 5.2 🟩Opus 4.8 TASK Create a high-quality single HTML file simulation of a Rube Goldberg / chain-reaction machine. The simulation should run automatically from start to finish without user input.The goal is to demonstrate whether the coding model understands real-world physics well enough to create believable cause-and-effect behavior - Rube Goldberg / chain-reaction machine. The scene should show a small physical world where objects interact through realistic mechanics: gravity, collisions, momentum transfer, ramps, pulleys, levers, springs, falling objects, rolling balls, dominoes, and water or particles if possible. The simulation should include: * A ball rolling down a ramp * Dominoes falling one after another * A lever or seesaw transferring force * A spring launcher * A pendulum or swinging weight * A pulley or elevator mechanism * Falling objects affected by gravity * Objects with different mass, friction, and bounce * Clear visual labels explaining what physical law is being demonstrated * A progress timeline showing the current stage of the chain reaction * Automatic reset/replay after the sequence ends The simulation should visually demonstrate: * Gravity * Momentum transfer * Conservation of energy * Friction * Torque * Elastic potential energy * Collision response * Cause and effect The model should not just animate objects on fixed paths. Objects should appear to interact physically. For example, the ball should knock over dominoes, dominoes should push a lever, the lever should launch another object, and so on. Evaluation criteria 1. Does the simulation run automatically without user input? 2. Do objects interact through believable physical cause and effect? 3. Are physical laws represented correctly? 4. Are labels and UI helpful? 5. Is the animation smooth and stable? 6. Does the code avoid fake-looking hardcoded movement? 7. Does the scene reset cleanly?

Remek Kinas

14,146 Aufrufe • vor 1 Monat