Loading video...

Video Failed to Load

Go Home

How well can Qwen3.5 models debug code? I built BugFind-15 — 15 buggy snippets across Python, JS, Rust, and Go. Docker sandbox compiles and validates every fix. Two trap scenarios where the code is correct and the model must resist "fixing" it. Tested every Qwen3.5 size from 0.8B to...

stevibe's profile picture

stevibe

27,376 subscribers

35,158 views • 6 months ago •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

Got a 16GB GPU? You can run all of these right now. Tested 4 Qwen3.5-based models on ToolCall-15 & BugFind-15: Models: - Qwen3.5:9b Q8 (Official) - Qwopus v3 Q8 by Jackrong - OmniCoder-9B by Tesslate - Qwen3.5-9b-Sushi-Coder by bigatuna Summary: - ToolCall-15: Qwopus v3 went perfect 30/30, Sushicoder beat base Qwen3.5 - BugFind-15: Omnicoder flipped the script and took #1 at 83% No single model won both, that's the fun part. Open source community is cooking.
0:14

Got a 16GB GPU? You can run all of these right now. Tested 4 Qwen3.5-based models on ToolCall-15 & BugFind-15: Models: - Qwen3.5:9b Q8 (Official) - Qwopus v3 Q8 by Jackrong - OmniCoder-9B by Tesslate - Qwen3.5-9b-Sushi-Coder by bigatuna Summary: - ToolCall-15: Qwopus v3 went perfect 30/30, Sushicoder beat base Qwen3.5 - BugFind-15: Omnicoder flipped the script and took #1 at 83% No single model won both, that's the fun part. Open source community is cooking.

stevibe

75,769 views • 6 months ago

The good news: you can mid-train yourself to the frontier! At a fraction of the cost! Without distillation! What we did: We took Qwen3.5 14B and mid- and post-trained it to beat every other open model and perform as well as the very best frontier models. This is no simple toy task or format fixing, we are training the model to understand and reason over DNA sequences. Bio is the next frontier for AI and universal access will be crucial to make progress. The fun part: because we train the model on both natural language and DNA sequences you can actually 'chat to your DNA': Ask it questions, ponder the impact of mutations, or ask it to generate a small sequence. We'll soon see the frontier labs move away from math and cyber towards bio. It already started, and we believe researchers should have access to capable models. We are releasing the full recipe, so you can do it, too! Model: Blog:
0:15

The good news: you can mid-train yourself to the frontier! At a fraction of the cost! Without distillation! What we did: We took Qwen3.5 14B and mid- and post-trained it to beat every other open model and perform as well as the very best frontier models. This is no simple toy task or format fixing, we are training the model to understand and reason over DNA sequences. Bio is the next frontier for AI and universal access will be crucial to make progress. The fun part: because we train the model on both natural language and DNA sequences you can actually 'chat to your DNA': Ask it questions, ponder the impact of mutations, or ask it to generate a small sequence. We'll soon see the frontier labs move away from math and cyber towards bio. It already started, and we believe researchers should have access to capable models. We are releasing the full recipe, so you can do it, too! Model: Blog:

Leandro von Werra

78,274 views • 2 days ago

Qwen3.6 35B A3B can't fill out a paper form on its own. But give it NVIDIA's LocateAnything-3B — the #1 trending model on HuggingFace — as its eyes, and the two small models get it done together. (The test: place each element at the right pixel position on a blank form image, not type into a field.) Setup: > Qwen is the brain (main model), LocateAnything is the eyes (helper model acting as a tool). > I gave Qwen a new tool: ask "where's the email field?" and LocateAnything returns the exact x, y, width, height. > The blue boxes on the screen are its detections. Look how tight they are — it nails every field. Result: > Qwen3.6 35B A3B + LocateAnything-3B: form completed, all info correct. > Name, DOB, ID, gender, marital status, nationality, email, phone, address, postal code: all landed in the right field areas. > Character-box alignment still a touch loose, but every value is where it belongs. > 9m10s, 224.5k input, 24.3k output, 21 turns. Why it matters: > Qwen alone can't finish this test. Bolt on a 3B model that does exactly one thing > locate > and suddenly it can. > A combination of small models can do the work of a single large one.
0:14

Qwen3.6 35B A3B can't fill out a paper form on its own. But give it NVIDIA's LocateAnything-3B — the #1 trending model on HuggingFace — as its eyes, and the two small models get it done together. (The test: place each element at the right pixel position on a blank form image, not type into a field.) Setup: > Qwen is the brain (main model), LocateAnything is the eyes (helper model acting as a tool). > I gave Qwen a new tool: ask "where's the email field?" and LocateAnything returns the exact x, y, width, height. > The blue boxes on the screen are its detections. Look how tight they are — it nails every field. Result: > Qwen3.6 35B A3B + LocateAnything-3B: form completed, all info correct. > Name, DOB, ID, gender, marital status, nationality, email, phone, address, postal code: all landed in the right field areas. > Character-box alignment still a touch loose, but every value is where it belongs. > 9m10s, 224.5k input, 24.3k output, 21 turns. Why it matters: > Qwen alone can't finish this test. Bolt on a 3B model that does exactly one thing > locate > and suddenly it can. > A combination of small models can do the work of a single large one.

stevibe

151,000 views • 4 months ago

"I'm not a human." Fed it to Qwen 3.5 0.8B running locally on my Mac Studio M2 Ultra. It solved it. The CAPTCHA is fake. But sending images to the local model? Very real. I'm not breaking the internet. Yet.
0:12

"I'm not a human." Fed it to Qwen 3.5 0.8B running locally on my Mac Studio M2 Ultra. It solved it. The CAPTCHA is fake. But sending images to the local model? Very real. I'm not breaking the internet. Yet.

stevibe

69,433 views • 7 months ago

✨ I revived my first AI startup from 6 years ago with Claude Code [ 💡 ] Back then it used GPT-3 (this was 2 years before ChatGPT existed!) to generate new startup ideas which then people can vote on And the best startup ideas rise to the top! Back then I made it because people complained they didn't have any ideas to build a startup This week I moved it to its own VPS and installed Claude Code and told it to fix everything, the DB had become big and there was stupid write operations on every page load that it made it very slow Claude Code is excellent at fixing all those small bugs from old projects and quickly fixing them As Garry Tan says "boil the oceans" as in before I'd not have the time to fix these kinds of projects, it wouldn't be worth it, I mean IdeasAI doesn't even make money, but now it takes me an hour to do this and it works again! I also upgraded GPT-3 to xAI's Grok 4.2 for new startup ideas
0:15

✨ I revived my first AI startup from 6 years ago with Claude Code [ 💡 ] Back then it used GPT-3 (this was 2 years before ChatGPT existed!) to generate new startup ideas which then people can vote on And the best startup ideas rise to the top! Back then I made it because people complained they didn't have any ideas to build a startup This week I moved it to its own VPS and installed Claude Code and told it to fix everything, the DB had become big and there was stupid write operations on every page load that it made it very slow Claude Code is excellent at fixing all those small bugs from old projects and quickly fixing them As Garry Tan says "boil the oceans" as in before I'd not have the time to fix these kinds of projects, it wouldn't be worth it, I mean IdeasAI doesn't even make money, but now it takes me an hour to do this and it works again! I also upgraded GPT-3 to xAI's Grok 4.2 for new startup ideas

@levelsio

180,039 views • 7 months ago

🚨Anthropic just gave Claude Code eyes and hands. read that again. it can now open apps on mac, click around macOS, find bugs visually, screenshot them, fix the code, rebuild, and verify the fix. one prompt. absolute zero human input. full autonomy. seems like Claude Code is becoming a super app. i’m so here for it.
0:16

🚨Anthropic just gave Claude Code eyes and hands. read that again. it can now open apps on mac, click around macOS, find bugs visually, screenshot them, fix the code, rebuild, and verify the fix. one prompt. absolute zero human input. full autonomy. seems like Claude Code is becoming a super app. i’m so here for it.

sui ☄️

27,357 views • 6 months ago

My first PhD paper!🎉We learn *diffusion* models for code generation that learn to directly *edit* syntax trees of programs. The result is a system that can incrementally write code, see the execution output, and debug it. 🧵1/n
0:12

My first PhD paper!🎉We learn *diffusion* models for code generation that learn to directly *edit* syntax trees of programs. The result is a system that can incrementally write code, see the execution output, and debug it. 🧵1/n

Shreyas Kapur

742,826 views • 2 years ago

Vibe coders: let's fix bugs by adding more code . Software developers: let's fix the Architecture that caused the bugs .
0:10

Vibe coders: let's fix bugs by adding more code . Software developers: let's fix the Architecture that caused the bugs .

Raj kr

102,299 views • 6 days ago

The newest GTA 6 gameplay trailer dropped yesterday. we ran our subnet vision models on it. Person, car, gun, segmented and tracked across every scene. night, water, motion blur, freefall over the city. footage the model had never seen. The part that matters is that this isn't a huge model. it's tiny, and it runs near real time. Vision that works in the real world...and the fake one.
0:16

The newest GTA 6 gameplay trailer dropped yesterday. we ran our subnet vision models on it. Person, car, gun, segmented and tracked across every scene. night, water, motion blur, freefall over the city. footage the model had never seen. The part that matters is that this isn't a huge model. it's tiny, and it runs near real time. Vision that works in the real world...and the fake one.

Score

681,581 views • 1 month ago

here’s how to make game models with claude opus 5.5 + blender... opus writes python and builds every model in blender, part by part. the trick is treating every model as code, but there’s a smart way to set it up: - step 1: connect opus 5.5 to blender through blender mcp - step 2: send 2-3 refs + a short brief: “copper helmet, navy suit, glass tank. keep the hose and gloves separate” - step 3: opus builds it part by part: boots > suit > helmet > gloves > tank - step 4: rig it in a separate script, with extra bones for props (a hose, a rod, a tank) - step 5: bake materials into one 1024 texture atlas - step 6: animate the clips: idle, walk, capture - step 7: export one fbx with the rig, the mesh and every clip - step 8: test it in the engine: “keep the backpack stable and the hose attached while he captures a ghost” and the rule that saves the project... pick a style that fits code i like to describe how a model should feel, not how to build it
0:15

here’s how to make game models with claude opus 5.5 + blender... opus writes python and builds every model in blender, part by part. the trick is treating every model as code, but there’s a smart way to set it up: - step 1: connect opus 5.5 to blender through blender mcp - step 2: send 2-3 refs + a short brief: “copper helmet, navy suit, glass tank. keep the hose and gloves separate” - step 3: opus builds it part by part: boots > suit > helmet > gloves > tank - step 4: rig it in a separate script, with extra bones for props (a hose, a rod, a tank) - step 5: bake materials into one 1024 texture atlas - step 6: animate the clips: idle, walk, capture - step 7: export one fbx with the rig, the mesh and every clip - step 8: test it in the engine: “keep the backpack stable and the hose attached while he captures a ghost” and the rule that saves the project... pick a style that fits code i like to describe how a model should feel, not how to build it

rewind

71,498 views • 7 days ago

Seedance 2.0 has a fix for character drift almost nobody uses. Every new text description is a fresh interpretation of appearance. That's where the drift comes from: a slightly different face, a slightly different outfit, proportions that shift from scene to scene. The @ Image tag solves this directly — attach a photo once, and it becomes the fixed source of truth for face, outfit, and proportions across the entire multi-shot sequence. The @ Video tag works the same way for motion and camera — tag a reference instead of describing it, and the model copies that exact style instead of an approximation. You're not making the model remember. You're just stopping yourself from reminding it differently every single time.
0:10

Seedance 2.0 has a fix for character drift almost nobody uses. Every new text description is a fresh interpretation of appearance. That's where the drift comes from: a slightly different face, a slightly different outfit, proportions that shift from scene to scene. The @ Image tag solves this directly — attach a photo once, and it becomes the fixed source of truth for face, outfit, and proportions across the entire multi-shot sequence. The @ Video tag works the same way for motion and camera — tag a reference instead of describing it, and the model copies that exact style instead of an approximation. You're not making the model remember. You're just stopping yourself from reminding it differently every single time.

Zentrix⌚️

89,142 views • 3 months ago

let me save you 3 hours of head scratching. if you're running local models like Qwen3.5-35B-A3B through Claude Code via llama.cpp's Anthropic endpoint, the chain will break every 3 to 5 minutes. tool call fails. flow stops. you reprompt. it recovers. 2 minutes later it stops again. the model is fine. the harness chokes on local inference latency. switch to OpenCode. same localhost endpoint. same model. same GPU. the chain doesn't break. the tradeoff: OpenCode sometimes loops. the model forgets what it already read and repeats the same tool call. but a loop you can interrupt. a broken chain kills your momentum and you start over. watch both side by side. proprietary agent vs open source agent. same 3B model. different failure modes. pick your poison.
0:19

let me save you 3 hours of head scratching. if you're running local models like Qwen3.5-35B-A3B through Claude Code via llama.cpp's Anthropic endpoint, the chain will break every 3 to 5 minutes. tool call fails. flow stops. you reprompt. it recovers. 2 minutes later it stops again. the model is fine. the harness chokes on local inference latency. switch to OpenCode. same localhost endpoint. same model. same GPU. the chain doesn't break. the tradeoff: OpenCode sometimes loops. the model forgets what it already read and repeats the same tool call. but a loop you can interrupt. a broken chain kills your momentum and you start over. watch both side by side. proprietary agent vs open source agent. same 3B model. different failure modes. pick your poison.

Sudo su

72,554 views • 7 months ago

MiniMax is the James Bond of AI agents. It uses the world's first open-weight model (MiniMax-M1), and it squeezes every bit of power from it. The agent takes a prompt and does more than any other agent in the market right now: 1. It can do Deep Research 2. It can write code 3. It can design web pages 4. It can build 3D models I built 5 different experiences using MiniMax and recorded them for you:
0:13

MiniMax is the James Bond of AI agents. It uses the world's first open-weight model (MiniMax-M1), and it squeezes every bit of power from it. The agent takes a prompt and does more than any other agent in the market right now: 1. It can do Deep Research 2. It can write code 3. It can design web pages 4. It can build 3D models I built 5 different experiences using MiniMax and recorded them for you:

Santiago

45,328 views • 1 year ago

Qwen3.5-35B-A3B is now in Jan 🔥 It surpasses previous Qwen3 models more than 6× its size. Get the latest Jan at Thanks to Qwen for the base model and Georgi Gerganov for llama.cpp 💛
0:13

Qwen3.5-35B-A3B is now in Jan 🔥 It surpasses previous Qwen3 models more than 6× its size. Get the latest Jan at Thanks to Qwen for the base model and Georgi Gerganov for llama.cpp 💛

👋 Jan

34,433 views • 8 months ago

My project has 39,205 lines of code, and Cursor can't answer questions about it. Cursor's context seems to be capped at around 10,000 tokens. Unfortunately, this is not enough for any decent-sized project. If you have a large codebase, check out Augment Code. This thing is faaaast! I'm currently using their Visual Studio Code plugin, but you can also use them on JetBrains, Neovim, and even Vim. (I'm a Neovim fan, but Copilot's implementation for Neovim is nowhere as good as Augment Code.) Augment Code was gracious enough to sponsor this post. After you install their extension and run it for the first time, it will index your entire codebase. This is why it can answer questions as fast as it does, regardless of the size of your codebase. Augment Code supports chat and completions like every other AI coding assistant, but its killer feature is "Next Edit." When you make a change, two things happen: 1. The model analyzes the change to determine the ripple effects across your *entire* codebase. 2. The model suggests everything you need to update to ensure everything works correctly. This is pretty wild!
0:19

My project has 39,205 lines of code, and Cursor can't answer questions about it. Cursor's context seems to be capped at around 10,000 tokens. Unfortunately, this is not enough for any decent-sized project. If you have a large codebase, check out Augment Code. This thing is faaaast! I'm currently using their Visual Studio Code plugin, but you can also use them on JetBrains, Neovim, and even Vim. (I'm a Neovim fan, but Copilot's implementation for Neovim is nowhere as good as Augment Code.) Augment Code was gracious enough to sponsor this post. After you install their extension and run it for the first time, it will index your entire codebase. This is why it can answer questions as fast as it does, regardless of the size of your codebase. Augment Code supports chat and completions like every other AI coding assistant, but its killer feature is "Next Edit." When you make a change, two things happen: 1. The model analyzes the change to determine the ripple effects across your *entire* codebase. 2. The model suggests everything you need to update to ensure everything works correctly. This is pretty wild!

Santiago

247,888 views • 1 year ago

Don't train the model, evolve the harness. I read a brilliant blog post from Hugging Face where they took a frozen open model scoring 0% on a hard legal agent benchmark, left its weights alone, and let an automated loop rewrite only the code around it. That code layer is the harness, the runtime wrapper that feeds the model context, runs its tool calls, and decides when a run ends. By the time the loop finished, the system had essentially matched Sonnet 4.6 on the benchmark's headline metric, at roughly 7x lower cost per task. Zero weights changed. The gain existed because of where the model was failing. The judge only grades files saved in the right place under the exact requested filename, and the model kept doing the legal analysis correctly, then saving it under the wrong name, dropping it in a scratch folder, or never writing it at all. So the 0% was never measuring legal reasoning. It was measuring the harness. Hand-tuning that layer is slow and model-specific, so they automated it. A Claude proposer adds exactly one mechanism per iteration, and an outer loop keeps it only if it clearly beats the current best, so accepted mechanisms compound. What the loop discovered says a lot about where agents actually fail. → The biggest single gain was file handling, not intelligence. An automatic step that lands the deliverable exactly where the judge expects it beat every prompt change, with zero extra model tokens. → Code fixes transferred across models, prompt playbooks did not. The same harness lifted a smaller model from the same family by 14 points, but the tuned prompts hurt a different model family on tasks it could already finish. → The harness mattered more than anything else. Same model, same judge, same tasks, and five different harnesses scored anywhere between 3.5% and 80.1%. The gains do eventually flatten, and the remaining misses look like real capability gaps. At some point the wrapper runs out of tricks and the model has to carry the work. But the lesson holds. A benchmark score measures the model and its harness together, and until the harness is fixed, it's impossible to know which one failed. I highly recommend reading this: I also wrote a deep dive on agent harness engineering a while back, covering the orchestration loop, tools, memory, context management, and everything that turns a stateless LLM into a capable agent. The article is quoted below.
0:18

Don't train the model, evolve the harness. I read a brilliant blog post from Hugging Face where they took a frozen open model scoring 0% on a hard legal agent benchmark, left its weights alone, and let an automated loop rewrite only the code around it. That code layer is the harness, the runtime wrapper that feeds the model context, runs its tool calls, and decides when a run ends. By the time the loop finished, the system had essentially matched Sonnet 4.6 on the benchmark's headline metric, at roughly 7x lower cost per task. Zero weights changed. The gain existed because of where the model was failing. The judge only grades files saved in the right place under the exact requested filename, and the model kept doing the legal analysis correctly, then saving it under the wrong name, dropping it in a scratch folder, or never writing it at all. So the 0% was never measuring legal reasoning. It was measuring the harness. Hand-tuning that layer is slow and model-specific, so they automated it. A Claude proposer adds exactly one mechanism per iteration, and an outer loop keeps it only if it clearly beats the current best, so accepted mechanisms compound. What the loop discovered says a lot about where agents actually fail. → The biggest single gain was file handling, not intelligence. An automatic step that lands the deliverable exactly where the judge expects it beat every prompt change, with zero extra model tokens. → Code fixes transferred across models, prompt playbooks did not. The same harness lifted a smaller model from the same family by 14 points, but the tuned prompts hurt a different model family on tasks it could already finish. → The harness mattered more than anything else. Same model, same judge, same tasks, and five different harnesses scored anywhere between 3.5% and 80.1%. The gains do eventually flatten, and the remaining misses look like real capability gaps. At some point the wrapper runs out of tricks and the model has to carry the work. But the lesson holds. A benchmark score measures the model and its harness together, and until the harness is fixed, it's impossible to know which one failed. I highly recommend reading this: I also wrote a deep dive on agent harness engineering a while back, covering the orchestration loop, tools, memory, context management, and everything that turns a stateless LLM into a capable agent. The article is quoted below.

Akshay 🚀

245,663 views • 3 months ago

Fable 5 comes back！It can now build playable game prototypes. I think it is actually a signal for where AI coding is going. Making a game is not just “write some code.” Even a small browser game needs: game loop；character movement；collision logic；scoring system；UI states；physics tuning；visual feedback；bug fixing；playtesting This is why game prototyping is a great test for AI models. A model cannot fake it with a pretty answer. Either the game runs, or it does not. What impressed me about Fable 5 is that it is useful for the messy middle: turning an idea into mechanics, turning mechanics into code, debugging broken interactions, and iterating until the prototype feels playable. But here is the practical part: I would not use the strongest model for every step. For game building, I would split the workflow: 1. Fable 5 for game design + architecture 2. a fast coding model for routine implementation 3. a vision-capable model for screenshot/UI feedback 4. a cheaper model for docs, test cases, and small fixes 5. fallback when latency, cost, or output quality becomes a problem That is the real AI coding stack. Not “one magic model does everything.” More like: the right model, for the right task, at the right cost, with fallback when things break. This is why I’ve been looking at ZenMux ZenMux. ZenMux gives developers one gateway to access multiple leading AI models, with OpenAI / Anthropic / Google Vertex compatible APIs, cost tracking, quality benchmarks, auto-routing, and compensation when output quality, latency, or throughput falls short. If AI can now make games, the next question is not just “which model is strongest?” It is:how do we manage the whole model workflow Fable 5 shows the creative ceiling. ZenMux is closer to the infrastructure layer you need when AI coding becomes a real production habit.
0:12

Fable 5 comes back！It can now build playable game prototypes. I think it is actually a signal for where AI coding is going. Making a game is not just “write some code.” Even a small browser game needs: game loop；character movement；collision logic；scoring system；UI states；physics tuning；visual feedback；bug fixing；playtesting This is why game prototyping is a great test for AI models. A model cannot fake it with a pretty answer. Either the game runs, or it does not. What impressed me about Fable 5 is that it is useful for the messy middle: turning an idea into mechanics, turning mechanics into code, debugging broken interactions, and iterating until the prototype feels playable. But here is the practical part: I would not use the strongest model for every step. For game building, I would split the workflow: 1. Fable 5 for game design + architecture 2. a fast coding model for routine implementation 3. a vision-capable model for screenshot/UI feedback 4. a cheaper model for docs, test cases, and small fixes 5. fallback when latency, cost, or output quality becomes a problem That is the real AI coding stack. Not “one magic model does everything.” More like: the right model, for the right task, at the right cost, with fallback when things break. This is why I’ve been looking at ZenMux ZenMux. ZenMux gives developers one gateway to access multiple leading AI models, with OpenAI / Anthropic / Google Vertex compatible APIs, cost tracking, quality benchmarks, auto-routing, and compensation when output quality, latency, or throughput falls short. If AI can now make games, the next question is not just “which model is strongest?” It is:how do we manage the whole model workflow Fable 5 shows the creative ceiling. ZenMux is closer to the infrastructure layer you need when AI coding becomes a real production habit.

Rachel🥥

61,994 views • 3 months ago

Introducing DeepSeek Coder! - SOTA large coding models with params ranging from 1.3B to 33B. - Building games, testing code, fixing bugs, and analyzing data... You dream it, we make it. - Free for commercial use and fully open-source. Try it out now at
0:19

Introducing DeepSeek Coder! - SOTA large coding models with params ranging from 1.3B to 33B. - Building games, testing code, fixing bugs, and analyzing data... You dream it, we make it. - Free for commercial use and fully open-source. Try it out now at

DeepSeek

162,557 views • 3 years ago

Introducing HermesAgent-20, a new Bench Pack for BenchLocal. 20 scenarios extracted straight from the Hermes Agent source code, run against a REAL Hermes instance. The actual workload you'd put your model through. Why I built BenchLocal in the first place: most benchmarks are too abstract. We use local LLMs for practical work, and finding the right model for YOUR task efficiently is the single most important thing, especially when you're constrained to what fits on your machine. BenchLocal is a framework: providers, models, side-by-side comparison, all in one UI. Bench Packs are the unit of testing: ToolCall-15 and BugFind-15 shipped first, and when I launched the BenchLocal 0.1.0, added StructOutput, ReasonMath, InstructFollow, DataExtract. Now, HermesAgent-20 is the newest. Bench Packs install like VS Code extensions. The SDK is open, write your own, share it, grow the ecosystem. Here's the goal: a community-built, practical evaluation layer for the local LLM space. Early numbers on HermesAgent-20: > GLM 5.1 — 85 > Gemma4 31B — 83 > Qwen3.5 27B — 79 > MiniMax M2.7 — 76 Upgrade to the latest BenchLocal to install HermesAgent-20 (SDK update required).
0:20

Introducing HermesAgent-20, a new Bench Pack for BenchLocal. 20 scenarios extracted straight from the Hermes Agent source code, run against a REAL Hermes instance. The actual workload you'd put your model through. Why I built BenchLocal in the first place: most benchmarks are too abstract. We use local LLMs for practical work, and finding the right model for YOUR task efficiently is the single most important thing, especially when you're constrained to what fits on your machine. BenchLocal is a framework: providers, models, side-by-side comparison, all in one UI. Bench Packs are the unit of testing: ToolCall-15 and BugFind-15 shipped first, and when I launched the BenchLocal 0.1.0, added StructOutput, ReasonMath, InstructFollow, DataExtract. Now, HermesAgent-20 is the newest. Bench Packs install like VS Code extensions. The SDK is open, write your own, share it, grow the ecosystem. Here's the goal: a community-built, practical evaluation layer for the local LLM space. Early numbers on HermesAgent-20: > GLM 5.1 — 85 > Gemma4 31B — 83 > Qwen3.5 27B — 79 > MiniMax M2.7 — 76 Upgrade to the latest BenchLocal to install HermesAgent-20 (SDK update required).

stevibe

38,631 views • 6 months ago