Загрузка видео...

Не удалось загрузить видео

Возникла проблема при загрузке этого видео. Это может быть связано с временными проблемами сети или видео может быть недоступно.

На главную

I built a macOS app for benchmarking local LLMs. 6 test suites. Multiple providers. One workspace. Open source. There are hundreds of local models now. New ones every week. How do you actually pick one? Leaderboards test for general ability. But if you're building an agent that chains tool... calls, or a pipeline that extracts structured data, or a code assistant that needs to debug Rust, you need to know if the model handles that specific thing. Not in theory. On your hardware. With your prompts. The benchmarks that exist are either locked behind papers, too abstract to map to real failures, or impossible to extend. You can't add your own test cases. You can't test what matters to your use case. That's what BenchLocal is for. It's a benchmark platform where every test is practical, deterministic, and built around real-world tasks. And you can build your own tests. It ships with 6 Bench Packs TODAY: → ToolCall-15 — tool-use accuracy → BugFind-15 — debugging capabilities → DataExtract-15 — structured data extraction → InstructFollow-15 — constraint-heavy instruction following → ReasonMath-15 — practical reasoning and math → StructOutput-15 — validator-backed structured output Every pack has 15 fixed scenarios. Every score is deterministic and verifiable. Some of you saw ToolCall-15 and BugFind-15 — the individual test packs I open-sourced over the past few weeks. People ran them, filed issues, sent PRs. But managing separate repos, separate scripts, separate results doesn't scale. BenchLocal puts everything in one place. What the app does: > Workspace with tabs — run BugFind-15 in one tab, ToolCall-15 in another. > Any provider — Ollama, llama.cpp, OpenRouter, any OpenAI-compatible endpoint. Local and cloud, same interface. > Run modes — serial, batch per model, batch per test case, or fully parallel. > Test histories — every run saved. Compare any previous session. But the part I'm most excited about isn't the app. It's the ecosystem. BenchLocal is a platform. Each Bench Pack is a plugin. I'm shipping an SDK so anyone can build their own — test what matters to you, package it, share it. Install and uninstall packs right inside the app, same way you'd manage extensions in VS Code. The registry is GitHub-based, fully public. I built 6 packs. I want the community to build the next 60. Theme system built in too — because if I'm staring at benchmark results for hours, it should at least look good. v0.1.0 is macOS only. Windows and Linux are coming. MIT licensed. Everything — the app, the bench packs, the SDK — is open. PRs welcome. Bench Packs even more welcome.show more

stevibe

27,033 subscribers

50,584 просмотров • 4 месяцев назад •via X (Twitter)

Искусство Образование Наука и технологии

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

Introducing HermesAgent-20, a new Bench Pack for BenchLocal. 20 scenarios extracted straight from the Hermes Agent source code, run against a REAL Hermes instance. The actual workload you'd put your model through. Why I built BenchLocal in the first place: most benchmarks are too abstract. We use local LLMs for practical work, and finding the right model for YOUR task efficiently is the single most important thing, especially when you're constrained to what fits on your machine. BenchLocal is a framework: providers, models, side-by-side comparison, all in one UI. Bench Packs are the unit of testing: ToolCall-15 and BugFind-15 shipped first, and when I launched the BenchLocal 0.1.0, added StructOutput, ReasonMath, InstructFollow, DataExtract. Now, HermesAgent-20 is the newest. Bench Packs install like VS Code extensions. The SDK is open, write your own, share it, grow the ecosystem. Here's the goal: a community-built, practical evaluation layer for the local LLM space. Early numbers on HermesAgent-20: > GLM 5.1 — 85 > Gemma4 31B — 83 > Qwen3.5 27B — 79 > MiniMax M2.7 — 76 Upgrade to the latest BenchLocal to install HermesAgent-20 (SDK update required).

Introducing HermesAgent-20, a new Bench Pack for BenchLocal. 20 scenarios extracted straight from the Hermes Agent source code, run against a REAL Hermes instance. The actual workload you'd put your model through. Why I built BenchLocal in the first place: most benchmarks are too abstract. We use local LLMs for practical work, and finding the right model for YOUR task efficiently is the single most important thing, especially when you're constrained to what fits on your machine. BenchLocal is a framework: providers, models, side-by-side comparison, all in one UI. Bench Packs are the unit of testing: ToolCall-15 and BugFind-15 shipped first, and when I launched the BenchLocal 0.1.0, added StructOutput, ReasonMath, InstructFollow, DataExtract. Now, HermesAgent-20 is the newest. Bench Packs install like VS Code extensions. The SDK is open, write your own, share it, grow the ecosystem. Here's the goal: a community-built, practical evaluation layer for the local LLM space. Early numbers on HermesAgent-20: > GLM 5.1 — 85 > Gemma4 31B — 83 > Qwen3.5 27B — 79 > MiniMax M2.7 — 76 Upgrade to the latest BenchLocal to install HermesAgent-20 (SDK update required).

stevibe

38,631 просмотров • 4 месяцев назад

Introducing Magnitude: your actually local agent 100% private and offline. No token costs, no API keys. Open source. Today's agents are local. The model isn't. Every prompt, every file, every secret gets sent straight to Anthropic and OpenAI. Magnitude is built around local models and runs the whole stack itself. The inference engine is part of the agent, so the models run inside it, right on your computer. It lives in your terminal, and setup is one command. Magnitude profiles your hardware and shows you which models fit, with the trade-offs between quality, speed, and memory. Pick one and start working. No painful config or server to babysit. Out of the box, it can use your shell, edit files, and run scripts. Add skills and it can work with Excel, PowerPoint, PDFs or Chrome. Use it for everyday work: - Analyze sensitive data - Manage private notes - Review code and logs - Search and organize files - Build docs or slides npm i -g @magnitudedev/cli GitHub:

Introducing Magnitude: your actually local agent 100% private and offline. No token costs, no API keys. Open source. Today's agents are local. The model isn't. Every prompt, every file, every secret gets sent straight to Anthropic and OpenAI. Magnitude is built around local models and runs the whole stack itself. The inference engine is part of the agent, so the models run inside it, right on your computer. It lives in your terminal, and setup is one command. Magnitude profiles your hardware and shows you which models fit, with the trade-offs between quality, speed, and memory. Pick one and start working. No painful config or server to babysit. Out of the box, it can use your shell, edit files, and run scripts. Add skills and it can work with Excel, PowerPoint, PDFs or Chrome. Use it for everyday work: - Analyze sensitive data - Manage private notes - Review code and logs - Search and organize files - Build docs or slides npm i -g @magnitudedev/cli GitHub:

Tom Greenwald

394,155 просмотров • 22 дней назад

Your coding agent can run all night. It still can't tell if what it built actually works. Today we're open-sourcing the TestSprite CLl (Apache-2.0) A tool your agent calls on its own to test your app end-to-end like a real user, fix what broke, and re-check everything it ever got right. It's the same engine 100,000+ teams already use. We proved it in public, on a public leaderboard: Most correct app on the board:89% Built by the cheapest model in the field At half the cost of the priciest one You no longer need the biggest, most expensive model to ship software you can trust. Setup is 2 commands: npm install -g @testsprite/testsprite-cli testsprite init That's the last command you'll ever type - from there, your agent runs the tests itself.

Your coding agent can run all night. It still can't tell if what it built actually works. Today we're open-sourcing the TestSprite CLl (Apache-2.0) A tool your agent calls on its own to test your app end-to-end like a real user, fix what broke, and re-check everything it ever got right. It's the same engine 100,000+ teams already use. We proved it in public, on a public leaderboard: Most correct app on the board:89% Built by the cheapest model in the field At half the cost of the priciest one You no longer need the biggest, most expensive model to ship software you can trust. Setup is 2 commands: npm install -g @testsprite/testsprite-cli testsprite init That's the last command you'll ever type - from there, your agent runs the tests itself.

TestSprite

1,042,183 просмотров • 2 месяцев назад

everyone in iOS development should watch this. seriously, it might change the whole industry. i pointed claude code at a live ios device running on revyl, typed "test everything," and walked away. here's what's actually happening: ① you don't write the tests. no scripts, no selectors, no test plan. i never told it which screens to open or what to check. it read the app, decided what mattered, and tested it. the entire instruction was "test everything." ② it built its own test team. it looked at the app, clocked that it's basically four mini apps (rides, delivery, services, account), and split itself into 4 agents, one per surface. scoping coverage like that is usually a person's whole afternoon. it did it in seconds, unprompted. ③ all four ran at the same time, each on its own live device. this is where revyl comes in. every agent gets its own live ios session in the cloud, so four running apps get tested in parallel instead of taking turns on one simulator. serial testing turns coverage into a time tax. running all of it at once removes the tax. ④ it tests like a person, not like a script. each agent drives the app the way a user would, taps through the flows, and visually checks each screen against what it expected to see. nothing is pinned to a brittle element id, so renaming a button doesn't take down half your suite. that one detail is the most annoying thing about how we test today, and it just quietly goes away. ⑤ no xcuitest, no sims melting your laptop. i didn't write a single xcuitest script, and there were no simulators booting on my machine. the agents run on cloud devices, so coverage stops being capped by what your laptop can handle. the part that got me isn't that an agent tested an app. it's that i never told it how. i handed it a device and an intent, and it figured out the scoping, the parallelizing, and the driving on its own. if you still write and maintain mobile ui tests by hand, i'm not sure that lasts the year.

everyone in iOS development should watch this. seriously, it might change the whole industry. i pointed claude code at a live ios device running on revyl, typed "test everything," and walked away. here's what's actually happening: ① you don't write the tests. no scripts, no selectors, no test plan. i never told it which screens to open or what to check. it read the app, decided what mattered, and tested it. the entire instruction was "test everything." ② it built its own test team. it looked at the app, clocked that it's basically four mini apps (rides, delivery, services, account), and split itself into 4 agents, one per surface. scoping coverage like that is usually a person's whole afternoon. it did it in seconds, unprompted. ③ all four ran at the same time, each on its own live device. this is where revyl comes in. every agent gets its own live ios session in the cloud, so four running apps get tested in parallel instead of taking turns on one simulator. serial testing turns coverage into a time tax. running all of it at once removes the tax. ④ it tests like a person, not like a script. each agent drives the app the way a user would, taps through the flows, and visually checks each screen against what it expected to see. nothing is pinned to a brittle element id, so renaming a button doesn't take down half your suite. that one detail is the most annoying thing about how we test today, and it just quietly goes away. ⑤ no xcuitest, no sims melting your laptop. i didn't write a single xcuitest script, and there were no simulators booting on my machine. the agents run on cloud devices, so coverage stops being capped by what your laptop can handle. the part that got me isn't that an agent tested an app. it's that i never told it how. i handed it a device and an intent, and it figured out the scoping, the parallelizing, and the driving on its own. if you still write and maintain mobile ui tests by hand, i'm not sure that lasts the year.

Landseer Enga

23,963 просмотров • 3 месяцев назад

#1 skill for developers in 2026: Automate everything you can using AI. I bet my lunch your team is dealing with all of these: • Stale documentation • Outdated dependencies • Poor test coverage • Deprecated APIs Every company I work with has these same problems. You can solve all of these right now. Automatically. Using AI. Here are 3 examples. Watch the attached video: I'm using Ona Automations to tackle this. These are background agents that run in the cloud, in a fully configured dev environment with your toolchain, your dependencies, and your services. You can run an unlimited number of these agents in parallel and across all your repositories. Claude Code and Codex only run locally, so they are hard to scale, and you can't run them when your computer is closed. Ona runs in the cloud. Here are the three examples: 1. Test coverage Run a nightly automation to identify any untested code paths, generate candidate tests, verify they pass, and open draft PRs. You wake up every morning to PRs that improve your test coverage. 2. Dependency upgrades Configure a weekly automation that bumps a dependency version, runs your full test suite, and reports any regressions. If everything is clean, it opens a PR. If something breaks, it opens a report so you can decide what to do. 3. Documentation auditing Set up a weekly automation that checks recent commits against your README file and setup guides, identifies broken examples and outdated instructions, and opens a PR with fixes.

#1 skill for developers in 2026: Automate everything you can using AI. I bet my lunch your team is dealing with all of these: • Stale documentation • Outdated dependencies • Poor test coverage • Deprecated APIs Every company I work with has these same problems. You can solve all of these right now. Automatically. Using AI. Here are 3 examples. Watch the attached video: I'm using Ona Automations to tackle this. These are background agents that run in the cloud, in a fully configured dev environment with your toolchain, your dependencies, and your services. You can run an unlimited number of these agents in parallel and across all your repositories. Claude Code and Codex only run locally, so they are hard to scale, and you can't run them when your computer is closed. Ona runs in the cloud. Here are the three examples: 1. Test coverage Run a nightly automation to identify any untested code paths, generate candidate tests, verify they pass, and open draft PRs. You wake up every morning to PRs that improve your test coverage. 2. Dependency upgrades Configure a weekly automation that bumps a dependency version, runs your full test suite, and reports any regressions. If everything is clean, it opens a PR. If something breaks, it opens a report so you can decide what to do. 3. Documentation auditing Set up a weekly automation that checks recent commits against your README file and setup guides, identifies broken examples and outdated instructions, and opens a PR with fixes.

Santiago

25,804 просмотров • 6 месяцев назад

Right now, I have a few dozen tools to automate my life. Every single thing that I do more than once every week is now a tool: either a Claude Code skill, a scheduled workflow, or an application. • I spend the time doing this once. • I schedule it. • I forget about it (or at least, try to) CREAO is one of the platforms I've used extensively for building some of these automations. They are partnering with me on this post. They are one of the only platforms where you can go from a conversation to a scheduled agent that quickly. This is how it works: 1. You describe the problem you want to solve 2. CREAO's agent builds the logic and executes it 3. You can iterate with the agent to improve the solution 4. When you are done, you turn that solution into a mini app 5. You can schedule that app to run any time you need it Something important: These agents are deterministic. They don't use an LLM to generate answers, so they will always return the same output given the same input. This is critical: when you're automating something that runs every Monday at 9 am, you need it to return the same result every time, not a "creative" answer.

Right now, I have a few dozen tools to automate my life. Every single thing that I do more than once every week is now a tool: either a Claude Code skill, a scheduled workflow, or an application. • I spend the time doing this once. • I schedule it. • I forget about it (or at least, try to) CREAO is one of the platforms I've used extensively for building some of these automations. They are partnering with me on this post. They are one of the only platforms where you can go from a conversation to a scheduled agent that quickly. This is how it works: 1. You describe the problem you want to solve 2. CREAO's agent builds the logic and executes it 3. You can iterate with the agent to improve the solution 4. When you are done, you turn that solution into a mini app 5. You can schedule that app to run any time you need it Something important: These agents are deterministic. They don't use an LLM to generate answers, so they will always return the same output given the same input. This is critical: when you're automating something that runs every Monday at 9 am, you need it to return the same result every time, not a "creative" answer.

Santiago

13,712 просмотров • 5 месяцев назад

I'm now building Executor full time as a startup! The state of tool calling is a mess: - Everyone is using different agents - You have no idea what actions are going to run - Destructive actions are treated the same as normal Executor lets you set up everything once & use it with any agent. You're able to control what tools can be called, require approval on destructive actions, and invite your team to all use them. Under the hood, it's built on codemode to let you add thousands of tools with 0 context bloat. I've been using it for a month so far and it has completely changed how I interact with agents. There's so much to be built here, immediately next up on the roadmap is workspaces, skills, workflows, and generative ui. Everything is being built open source over at there's a lot more coming on an SDK to embed it into your own agents, a self hostable cloud app, and plugins There's an early beta version over at It's very early on - if you hit any bugs let me know and I'll get them fixed up, excited to share more as I build it!

I'm now building Executor full time as a startup! The state of tool calling is a mess: - Everyone is using different agents - You have no idea what actions are going to run - Destructive actions are treated the same as normal Executor lets you set up everything once & use it with any agent. You're able to control what tools can be called, require approval on destructive actions, and invite your team to all use them. Under the hood, it's built on codemode to let you add thousands of tools with 0 context bloat. I've been using it for a month so far and it has completely changed how I interact with agents. There's so much to be built here, immediately next up on the roadmap is workspaces, skills, workflows, and generative ui. Everything is being built open source over at there's a lot more coming on an SDK to embed it into your own agents, a self hostable cloud app, and plugins There's an early beta version over at It's very early on - if you hit any bugs let me know and I'll get them fixed up, excited to share more as I build it!

Rhys

220,559 просмотров • 3 месяцев назад

HERMES AGENT VS OPENCLAW. a local ai onboarding flow test. a 3.9gb bonsai served on localhost, both agents upstream and latest, i point each one at the endpoint and watch which one even finds it. > hermes opens a provider menu, thirty plus options, local servers sitting right there next to the cloud ones, i hand it 127.0.0.1:8899, it verifies the endpoint, one model visible, auto-detects the model by name, bonsai-27b-q1_0, reads the context length straight off the server, saves it, and starts reasoning and firing real tool calls on my local model. no key. no friction. > openclaw has no menu. it goes hunting for a codex login, an openai key, finds none because there are none, prints no models available three times, defaults to openai/gpt-5.5, a cloud model it cannot reach, and dead ends on run auth login --provider openai. read that back. it asked me for an openai key. to run a model already running on my own machine. it never once looked at localhost. to be fair, openclaw can run local if you hand wire endpoint yourself. what it will not do is find the model already sitting on your box. hermes agent found it in one line. now the part i owe you. the auto-detect that just won, the model name read, the .gguf strip, the context length probe off the server, that is my code, it is in hermes agent main right now, authorship preserved, #2051 and #4218. the wizard fix that stops an agent from silently routing you to someone else's creds, the exact trap openclaw still falls into, mine too, #4210. i contribute to hermes agent, i told you that going in. one agent is built to talk to whatever you are running, the other is built to talk to a cloud api, so one found my model and ran it and the other asked me to log into openai. onboarding flow of both, mapped, below.

HERMES AGENT VS OPENCLAW. a local ai onboarding flow test. a 3.9gb bonsai served on localhost, both agents upstream and latest, i point each one at the endpoint and watch which one even finds it. > hermes opens a provider menu, thirty plus options, local servers sitting right there next to the cloud ones, i hand it 127.0.0.1:8899, it verifies the endpoint, one model visible, auto-detects the model by name, bonsai-27b-q1_0, reads the context length straight off the server, saves it, and starts reasoning and firing real tool calls on my local model. no key. no friction. > openclaw has no menu. it goes hunting for a codex login, an openai key, finds none because there are none, prints no models available three times, defaults to openai/gpt-5.5, a cloud model it cannot reach, and dead ends on run auth login --provider openai. read that back. it asked me for an openai key. to run a model already running on my own machine. it never once looked at localhost. to be fair, openclaw can run local if you hand wire endpoint yourself. what it will not do is find the model already sitting on your box. hermes agent found it in one line. now the part i owe you. the auto-detect that just won, the model name read, the .gguf strip, the context length probe off the server, that is my code, it is in hermes agent main right now, authorship preserved, #2051 and #4218. the wizard fix that stops an agent from silently routing you to someone else's creds, the exact trap openclaw still falls into, mine too, #4210. i contribute to hermes agent, i told you that going in. one agent is built to talk to whatever you are running, the other is built to talk to a cloud api, so one found my model and ran it and the other asked me to log into openai. onboarding flow of both, mapped, below.

Sudo su

23,816 просмотров • 1 месяц назад

✨ I built a Cursor-style right sidebar on my site Photo AI which is a regular AI chatbot BUT it can fully control my app! It can take photos of you, or shoot videos, run photo packs, remix content you upload etc. anything you see in the interface. Like an AI photographer that does everything for you inside the web app The convo history is stored in localStorage, and the chat uses xAI Grok 4 to send messages, there I define the tools it can use, and when it gets a response JS controls the site like prompt textareas, or photo pack section, or take photo button It also knows what's on your screen, so you can say you want to mix the blue jacket from one photo and the red hair from the other A few weeks ago I told Yasser it'd be cool if Chatbase (an AI customer support pop up chat) would be able to fully control my site too, and this week I saw Farza 🇵🇰🇺🇸 building an AI desktop app that fully controls his computer too so I think that's where things are going, you don't see it on the web that much though Very fun to build and hopefully useful for people!!!

✨ I built a Cursor-style right sidebar on my site Photo AI which is a regular AI chatbot BUT it can fully control my app! It can take photos of you, or shoot videos, run photo packs, remix content you upload etc. anything you see in the interface. Like an AI photographer that does everything for you inside the web app The convo history is stored in localStorage, and the chat uses xAI Grok 4 to send messages, there I define the tools it can use, and when it gets a response JS controls the site like prompt textareas, or photo pack section, or take photo button It also knows what's on your screen, so you can say you want to mix the blue jacket from one photo and the red hair from the other A few weeks ago I told Yasser it'd be cool if Chatbase (an AI customer support pop up chat) would be able to fully control my site too, and this week I saw Farza 🇵🇰🇺🇸 building an AI desktop app that fully controls his computer too so I think that's where things are going, you don't see it on the web that much though Very fun to build and hopefully useful for people!!!

@levelsio

214,132 просмотров • 3 месяцев назад

how to use Google's NEW open source Design.md + AI Skills to make your startup look like a $100 million company in 1 hour: 1. Design.md is an open source file from Google that captures the soul of a design. Typography, colors, spacing, all in one markdown file. You attach it to your prompt and your agent builds beautiful things every time. 2. Think of it this way. The HTML is the finished dish. The design.md is the recipe. The skills are the ingredients. Put them together and everything you build looks consistent and professional. 3. Don't create a design system from scratch. Find a brand you love. Linear, Stripe, Vercel, whatever resonates. Study it. Use ChatGPT or Claude to help you extract the design language into your own design.md file. 4. Build skills on top of your design.md. A landing page skill. A mobile app skill. A motion design skill. A slide deck skill. Each one references the same design.md so everything looks like it came from the same designer. 5. The biggest mistake people make: they nail one screen and then everything else looks generic. Design.md solves this. One file keeps every page, every format, every medium consistent. 6. Use it across everything. Your landing page. Your app. Your pitch deck. Your promo videos. Same DNA. Same taste. Same system. That's what separates a startup that looks real from one that looks vibe-coded. 7. Build a second brain for design inspiration. When you see something beautiful in the real world or online, capture it. Save it. When you're building something new, reference it. Taste is developed, not downloaded. 8. It's obvious but the difference between a product people trust and a product people bounce from is how it looks and feels. Design.md gives you that edge. you can watch below shoutout to Meng To for coming on The Startup Ideas Podcast (SIP) 🧃 and walking through his full workflow. if you want to use AI to actually build gorgeous designs, you'll want to use see this. watch

GREG ISENBERG

509,941 просмотров • 3 месяцев назад

If you think you're AI Native, here's a good test: When you're heading to bed tonight, are you asking yourself "what agents can I be running overnight to get a head start on my day tomorrow?" I posted that last week and hundreds commented on it. The most common pushback was some version of "congrats, you've automated slop." Here's my take: if your overnight runs produce slop, that's a process failure, not an agent failure. The model did exactly what you prepared it to do. You just didn't prepare it. And that's actually good news, because process is something you can level up. At tenex we run agents this way every day, on real client systems, using an internal framework we call MetaHarness. Before any long run starts, the plan is written down, the work is broken into small tasks, every task carries pass/fail success criteria, and guardrails define what the agent can touch and when it has to stop and wait for a human. With that in place, the agent isn't wandering your codebase all night. It's working inside rails you laid, and the only things waiting for you in the morning are the decisions that needed a human. Most people are still prompting and praying. This is your chance to build the framework that takes your agents to the next level. I recorded the whole process in a new video for you all ↓

If you think you're AI Native, here's a good test: When you're heading to bed tonight, are you asking yourself "what agents can I be running overnight to get a head start on my day tomorrow?" I posted that last week and hundreds commented on it. The most common pushback was some version of "congrats, you've automated slop." Here's my take: if your overnight runs produce slop, that's a process failure, not an agent failure. The model did exactly what you prepared it to do. You just didn't prepare it. And that's actually good news, because process is something you can level up. At tenex we run agents this way every day, on real client systems, using an internal framework we call MetaHarness. Before any long run starts, the plan is written down, the work is broken into small tasks, every task carries pass/fail success criteria, and guardrails define what the agent can touch and when it has to stop and wait for a human. With that in place, the agent isn't wandering your codebase all night. It's working inside rails you laid, and the only things waiting for you in the morning are the decisions that needed a human. Most people are still prompting and praying. This is your chance to build the framework that takes your agents to the next level. I recorded the whole process in a new video for you all ↓

JJ Englert

64,280 просмотров • 18 дней назад

I'm proud to introduce you to "MolDock Agent Swarm", which is my entry to the BSVA Hackathon ( MolDock is a molecular attraction testing system that uses smart contracts in BSV to verify the mathematics calculated by distributed agents who are testing drug efficacy. The swarm has a distribution agent that runs here on my PC, and a local agent that anyone can run by visiting If you start an agent and run tests, it should pay out rewards which are 100sats per test regardless of success or failure (between 50% and 75% fail) and if you succeed, you get an additional 10 sats per atom in the drug (15-50 atoms). The calculations for tests that pass are regenerated inside stateful smart contracts that track the absolute docking force of the atom for the full test (one cycle per atom in the molecule for each molecule in the docking location) meaning that for a molecule with 20 atoms testing a docking location with 30 atoms there will be 20 cycles of 30 rounds of maths. All the maths is calculated using script primitives, so there can be no cheating. I'm aiming for 1.5M transactions by thursday afternoon, so please jump in if you'd like to do some calculations.

I'm proud to introduce you to "MolDock Agent Swarm", which is my entry to the BSVA Hackathon ( MolDock is a molecular attraction testing system that uses smart contracts in BSV to verify the mathematics calculated by distributed agents who are testing drug efficacy. The swarm has a distribution agent that runs here on my PC, and a local agent that anyone can run by visiting If you start an agent and run tests, it should pay out rewards which are 100sats per test regardless of success or failure (between 50% and 75% fail) and if you succeed, you get an additional 10 sats per atom in the drug (15-50 atoms). The calculations for tests that pass are regenerated inside stateful smart contracts that track the absolute docking force of the atom for the full test (one cycle per atom in the molecule for each molecule in the docking location) meaning that for a molecule with 20 atoms testing a docking location with 30 atoms there will be 20 cycles of 30 rounds of maths. All the maths is calculated using script primitives, so there can be no cheating. I'm aiming for 1.5M transactions by thursday afternoon, so please jump in if you'd like to do some calculations.

Brendan Lee

10,114 просмотров • 4 месяцев назад

Choose a model (any model) and build your application with it. Do not spend time swapping models early on. Do not try to optimize before you have a working system. This is one of the first recommendations I make to every new team I consult with. Eventually, it will be time to optimize the model. • You may need a cheaper model • You may need a faster model • You might need a smarter model Good luck if you stitched together 12 different APIs and SDKs from 7 different vendors. Over half of the companies I consult for run on Microsoft software and have access to Microsoft Foundry. Microsoft Foundry is a complete agentic ecosystem. If you're in that world and building AI applications, Microsoft Foundry is where everything lives: • Models (largest selection in the market) • Agentic SDK (Python, C#, JavaScript/TypeScript) • Tools • Evaluations • Monitoring They are fully integrated with GitHub and Visual Studio Code. The best part: Their agentic platform is fully agnostic of the models you use. You can integrate with any model using the same OpenAI-style API. Swapping one model for another takes 1 second.

Choose a model (any model) and build your application with it. Do not spend time swapping models early on. Do not try to optimize before you have a working system. This is one of the first recommendations I make to every new team I consult with. Eventually, it will be time to optimize the model. • You may need a cheaper model • You may need a faster model • You might need a smarter model Good luck if you stitched together 12 different APIs and SDKs from 7 different vendors. Over half of the companies I consult for run on Microsoft software and have access to Microsoft Foundry. Microsoft Foundry is a complete agentic ecosystem. If you're in that world and building AI applications, Microsoft Foundry is where everything lives: • Models (largest selection in the market) • Agentic SDK (Python, C#, JavaScript/TypeScript) • Tools • Evaluations • Monitoring They are fully integrated with GitHub and Visual Studio Code. The best part: Their agentic platform is fully agnostic of the models you use. You can integrate with any model using the same OpenAI-style API. Swapping one model for another takes 1 second.

Santiago

12,014 просмотров • 6 месяцев назад

We’re launching Optima. Now anyone can create a custom benchmark for their use case, leveraging Artificial Analysis’ leading research and platform Building and running benchmarks is difficult. We have distilled Artificial Analysis’ research and experience developing benchmarks into Optima, a new platform for benchmarking models on your own workloads and comparing performance, speed and cost efficiency. Optima allows you to find the best model for your task, or an equally performant alternative to your current setup at 10x lower cost or time per task. We’ve integrated Artificial Analysis' research and experience in benchmarks across the Optima workflow: ➤ Build benchmarks based on your own data and use cases: There are three ways to build a benchmark with Optima. Upload an existing evaluation dataset from your own files or Hugging Face, or import agent traces from platforms including Arize AI, Braintrust and langfuse.com. Install the Optima skill to build a benchmark using context from your coding environment and previous sessions. Or simply describe your use case and provide example inputs and outputs, and Optima will build the benchmark for you ➤ Run across the latest models: Run the same benchmark across leading models in a single click, and keep your leaderboard up to date as soon as new models are released ➤ Bring Artificial Analysis grading to your own benchmark: Evaluate responses against objective rubric criteria or using the same pairwise judging approach used for Artificial Analysis benchmarks including GDPval-AA and AA-Briefcase. For pairwise judging, select your preferred responses from a sample and Optima uses those preferences to rank models across your test set ➤ Compare performance, cost and time efficiency: Optima measures more than model performance. Cost per Task and Time per Task are tracked alongside benchmark scores, with category-level results and support for custom metrics, allowing you to compare the tradeoffs between models for your specific use case Ahead of launch, here are examples questions our beta testers answered with Optima: ➤ Which model can save me 10x the cost without a meaningful decrease in quality for my finance & accounting agent? ➤ Which model best matches the writing style of lawyers for my legal agent? ➤ Which model can best identify different elements in my custom image dataset? Optima is available today. Build your own benchmark at

We’re launching Optima. Now anyone can create a custom benchmark for their use case, leveraging Artificial Analysis’ leading research and platform Building and running benchmarks is difficult. We have distilled Artificial Analysis’ research and experience developing benchmarks into Optima, a new platform for benchmarking models on your own workloads and comparing performance, speed and cost efficiency. Optima allows you to find the best model for your task, or an equally performant alternative to your current setup at 10x lower cost or time per task. We’ve integrated Artificial Analysis' research and experience in benchmarks across the Optima workflow: ➤ Build benchmarks based on your own data and use cases: There are three ways to build a benchmark with Optima. Upload an existing evaluation dataset from your own files or Hugging Face, or import agent traces from platforms including Arize AI, Braintrust and langfuse.com. Install the Optima skill to build a benchmark using context from your coding environment and previous sessions. Or simply describe your use case and provide example inputs and outputs, and Optima will build the benchmark for you ➤ Run across the latest models: Run the same benchmark across leading models in a single click, and keep your leaderboard up to date as soon as new models are released ➤ Bring Artificial Analysis grading to your own benchmark: Evaluate responses against objective rubric criteria or using the same pairwise judging approach used for Artificial Analysis benchmarks including GDPval-AA and AA-Briefcase. For pairwise judging, select your preferred responses from a sample and Optima uses those preferences to rank models across your test set ➤ Compare performance, cost and time efficiency: Optima measures more than model performance. Cost per Task and Time per Task are tracked alongside benchmark scores, with category-level results and support for custom metrics, allowing you to compare the tradeoffs between models for your specific use case Ahead of launch, here are examples questions our beta testers answered with Optima: ➤ Which model can save me 10x the cost without a meaningful decrease in quality for my finance & accounting agent? ➤ Which model best matches the writing style of lawyers for my legal agent? ➤ Which model can best identify different elements in my custom image dataset? Optima is available today. Build your own benchmark at

Artificial Analysis

130,269 просмотров • 16 дней назад

"Closed models are cheaper. The reason you need open models is that they give you control". - Nvidia CEO Jensen Huang "One of the things that people misunderstand about open models is this: yes, you can host them yourself, and you can build your own computer. But most people use computers in the cloud. Frankly, I think closed models are cheaper. If you do not build everything yourself, you avoid the significant cost and expertise required to train, fine-tune, maintain, guardrail, evaluate, and keep the model safe. You would also need to build or obtain the computers required to host it. So, there is nothing cheap about doing all of that yourself. The reason you need open models is that they give you control. You can adapt them for your own highly specialized use cases. I think there is a lot of misunderstanding about closed models versus open models. We felt it was important for people to understand that there is a place for both." ---- From "Bloomberg Podcasts" YouTube channel, (full video link in comment)

"Closed models are cheaper. The reason you need open models is that they give you control". - Nvidia CEO Jensen Huang "One of the things that people misunderstand about open models is this: yes, you can host them yourself, and you can build your own computer. But most people use computers in the cloud. Frankly, I think closed models are cheaper. If you do not build everything yourself, you avoid the significant cost and expertise required to train, fine-tune, maintain, guardrail, evaluate, and keep the model safe. You would also need to build or obtain the computers required to host it. So, there is nothing cheap about doing all of that yourself. The reason you need open models is that they give you control. You can adapt them for your own highly specialized use cases. I think there is a lot of misunderstanding about closed models versus open models. We felt it was important for people to understand that there is a place for both." ---- From "Bloomberg Podcasts" YouTube channel, (full video link in comment)

Rohan Paul

26,597 просмотров • 25 дней назад

Hermes agent just left the terminal. 𝗛𝗲𝗿𝗺𝗲𝘀 𝗗𝗲𝘀𝗸𝘁𝗼𝗽 dropped yesterday. native app for macOS, Windows, and Linux. for months Hermes was the agent that learned your projects, wrote its own skills, and built a model of who you are. all of it buried in terminal logs. now it has a window. the important part is that it's not a wrapper. it runs the same agent core, the same sessions, memory, and skills as the CLI. you can start a task in the terminal and finish it in the app without anything resetting. the state is shared across every interface, not copied between them. what the GUI actually adds: → streaming chat that shows live tool calls and inline reasoning instead of a spinner → a preview rail that renders pages, code, and images right beside the conversation → an artifacts panel that collects every file the agent has ever produced → remote gateway mode, so you can point the app at a VPS and run the heavy work elsewhere → skills, cron, profiles, and gateways managed point-and-click instead of through YAML → voice mode, drag-drop files, and inline image generation remote gateway mode is the one worth slowing down on. the agent runs 24/7 on a $5 server while you control it from your laptop like a local app. other agent UIs are chatboxes with a logo. this one shows the autonomy instead of hiding it, so you watch the skills load, the tools fire, and the artifacts pile up as it works. it was teased in Jensen's GTC keynote. MIT licensed, local-first, no telemetry. if you already run Hermes, download it and everything is already there. your chats, memory, and skills carry straight over. i wrote a full masterclass on Hermes Agent that walks through the SOUL. md identity layer, the three-tier memory system, the self-evolving skills loop, and how to run three specialized agents 24/7. desktop is the interface that finally does all of it justice. the article is quoted below.

Akshay 🚀

51,540 просмотров • 2 месяцев назад

Most people treat AI like Google: ask a question, get an answer. But what if AI could think *like/with you?* I reverse-engineer "Theory of Mind" to test if the model can form a "theory of my mind". Using AI as a mirror to understand myself by giving it personal context and seeing how well it can embody/reflect my expertise back to me helps me to evaluate the model's thinking. This is something like a single-user, multi-model human-AI synergy benchmark where the task is "strategic alignment with your own expertise." Here's my hypothesis: Does pre-loading context create emergent synergy and enhance collective intelligence? If the LLM has: - Your projects (what you've built) - Your career details (how you think about growth) - Your self-description (your values/philosophy) 1. Does this response sound like something I'd say? 2. Does it reveal blind spots I hadn't considered? 3. Does it reduce the "explanation tax" I pay in every conversation? Different models excel at different aspects of "me." Before you think I'm building a digital mirror to talk to myself: this isn't about AI companionship or AI psychosis. It's actually bout me understanding the model's reasoning so I can deploy the right model for the right task. I'm my own test subject because I have ground truth: I know what I actually think, what I'd actually decide, what I'd actually prioritize. Most benchmarks lack this. When a model says "Muratcan would choose X," I can immediately verify: "No, I'd choose Y because of Z." I think everyone should do this with their own domain expertise. - If you're a manager, load your case history and test which model best applies your reasoning. - If you're a creative, load your portfolio and test which model understands your aesthetic principles. - If you're a founder, load your strategy docs and test which model identifies blind spots in your go-to-market plan. AI should serve as an extension of your strategic thinking, a collaborative partner.

Most people treat AI like Google: ask a question, get an answer. But what if AI could think like/with you? I reverse-engineer "Theory of Mind" to test if the model can form a "theory of my mind". Using AI as a mirror to understand myself by giving it personal context and seeing how well it can embody/reflect my expertise back to me helps me to evaluate the model's thinking. This is something like a single-user, multi-model human-AI synergy benchmark where the task is "strategic alignment with your own expertise." Here's my hypothesis: Does pre-loading context create emergent synergy and enhance collective intelligence? If the LLM has: - Your projects (what you've built) - Your career details (how you think about growth) - Your self-description (your values/philosophy) 1. Does this response sound like something I'd say? 2. Does it reveal blind spots I hadn't considered? 3. Does it reduce the "explanation tax" I pay in every conversation? Different models excel at different aspects of "me." Before you think I'm building a digital mirror to talk to myself: this isn't about AI companionship or AI psychosis. It's actually bout me understanding the model's reasoning so I can deploy the right model for the right task. I'm my own test subject because I have ground truth: I know what I actually think, what I'd actually decide, what I'd actually prioritize. Most benchmarks lack this. When a model says "Muratcan would choose X," I can immediately verify: "No, I'd choose Y because of Z." I think everyone should do this with their own domain expertise. - If you're a manager, load your case history and test which model best applies your reasoning. - If you're a creative, load your portfolio and test which model understands your aesthetic principles. - If you're a founder, load your strategy docs and test which model identifies blind spots in your go-to-market plan. AI should serve as an extension of your strategic thinking, a collaborative partner.

Muratcan Koylan

24,128 просмотров • 8 месяцев назад

this video is the CLEAREST explanation of how claude skills + AI agents work and how to use them most people set up an AI agent and wonder why it keeps disappointing them. the context window is everything context is what the model assembles before it takes any action. think of it like everything the agent needs to read before it does anything. the quality of what goes in determines the quality of what comes out. the models are genuinely really good right now. claude and gpt are exceptional. the variable is almost always the context you give them. 1. agent.md files are mostly unnecessary every single line you put in an agent.md file gets added to every single conversation you have with your agent. a 1000 line file is around 7000 tokens burning on every run. the model already knows to use react. it can read your codebase. save the agent.md for proprietary information specific to your company that the model genuinely cannot know on its own. 2. skills are the actual unlock a skill.md file works differently. what loads into context is only the name and description, around 50 tokens. the full instructions only appear when the agent recognizes it needs that skill. so instead of 7000 tokens on every run you have 50. and the agent stays sharp because the context window stays lean. the closer you get to filling the context window the worse the agent performs, same way you perform worse when someone dumps 10 things on you at once. 3. here is how to actually build a skill the right way most people identify a workflow and immediately try to write the skill. what you want to do instead is run the workflow by hand with the agent first. walk it through every single step. tell it what to check, what good looks like, what bad looks like. correct it in real time. once you have had a full successful run from start to finish, tell the agent to review everything it just did and write the skill itself. it writes a better skill than you will because it has the full context of what actually worked in practice not in theory. 4. recursively building skills is how you go from frustrated to reliable when the skill breaks, and it will break, ask the agent exactly why it failed. it will tell you specifically what went wrong. fix it together in that same conversation. then tell it to update the skill file so that failure mode never happens again. ross mike did this five times with his youtube report generator. it now pulls from eight different data sources and runs flawlessly every single time without him touching it. 5. sub agents are something you earn not something you set up on day one start with one agent. build one workflow. turn it into one skill. once that works add another. ross mike has five sub agents now covering marketing, business, personal and more. it took months to get there and every single one exists because a workflow proved it deserved to exist. the people who set up 15 sub agents on day one and wonder why nothing works skipped all the steps that make the thing actually run. 6. your workflow is the thing the model cannot get anywhere else the model has been trained on everything. it knows more than you about most things. what it does not have is your specific process, your taste, your way of doing things. that is what skills capture. that is what makes your agent actually useful versus a generic one. downloading someone else's skill means downloading their context onto your setup and it will not work the way you want it to because it was never built around how you work. this is the clearest explanation of how agents actually work i have heard. Micky runs this stuff every single day and the results show it. full episode is now live on The Startup Ideas Podcast (SIP) 🧃 where you get your pods people charge for this sorta stuff i give away the sauce for free i just want you to win watch

this video is the CLEAREST explanation of how claude skills + AI agents work and how to use them most people set up an AI agent and wonder why it keeps disappointing them. the context window is everything context is what the model assembles before it takes any action. think of it like everything the agent needs to read before it does anything. the quality of what goes in determines the quality of what comes out. the models are genuinely really good right now. claude and gpt are exceptional. the variable is almost always the context you give them. 1. agent.md files are mostly unnecessary every single line you put in an agent.md file gets added to every single conversation you have with your agent. a 1000 line file is around 7000 tokens burning on every run. the model already knows to use react. it can read your codebase. save the agent.md for proprietary information specific to your company that the model genuinely cannot know on its own. 2. skills are the actual unlock a skill.md file works differently. what loads into context is only the name and description, around 50 tokens. the full instructions only appear when the agent recognizes it needs that skill. so instead of 7000 tokens on every run you have 50. and the agent stays sharp because the context window stays lean. the closer you get to filling the context window the worse the agent performs, same way you perform worse when someone dumps 10 things on you at once. 3. here is how to actually build a skill the right way most people identify a workflow and immediately try to write the skill. what you want to do instead is run the workflow by hand with the agent first. walk it through every single step. tell it what to check, what good looks like, what bad looks like. correct it in real time. once you have had a full successful run from start to finish, tell the agent to review everything it just did and write the skill itself. it writes a better skill than you will because it has the full context of what actually worked in practice not in theory. 4. recursively building skills is how you go from frustrated to reliable when the skill breaks, and it will break, ask the agent exactly why it failed. it will tell you specifically what went wrong. fix it together in that same conversation. then tell it to update the skill file so that failure mode never happens again. ross mike did this five times with his youtube report generator. it now pulls from eight different data sources and runs flawlessly every single time without him touching it. 5. sub agents are something you earn not something you set up on day one start with one agent. build one workflow. turn it into one skill. once that works add another. ross mike has five sub agents now covering marketing, business, personal and more. it took months to get there and every single one exists because a workflow proved it deserved to exist. the people who set up 15 sub agents on day one and wonder why nothing works skipped all the steps that make the thing actually run. 6. your workflow is the thing the model cannot get anywhere else the model has been trained on everything. it knows more than you about most things. what it does not have is your specific process, your taste, your way of doing things. that is what skills capture. that is what makes your agent actually useful versus a generic one. downloading someone else's skill means downloading their context onto your setup and it will not work the way you want it to because it was never built around how you work. this is the clearest explanation of how agents actually work i have heard. Micky runs this stuff every single day and the results show it. full episode is now live on The Startup Ideas Podcast (SIP) 🧃 where you get your pods people charge for this sorta stuff i give away the sauce for free i just want you to win watch

GREG ISENBERG

193,721 просмотров • 4 месяцев назад

Another blow to Anthropic! They spent months building what's now fully open-source. Anthropic recently put Claude inside Slack, where you can tag it in a channel. It reads the thread, breaks the task into steps, and posts the result back. The problem is that it only runs Claude and only in the channels Anthropic supports. Running your own agent there is harder. The reasoning, tool calls, and state management are mostly handled by the framework. Connecting that agent to a messaging platform is not. Moreover, each platform has a different integration: - Slack renders messages with Block Kit - Teams uses Adaptive Cards - and each has its own SDK, auth flow, and delivery model. If an agent needs to run on three platforms, one must write three separate integrations against the same agent logic. That overhead explains why most custom agents never get deployed to Slack, and why the ones that do are usually a single vendor's hosted assistant. The alternative is to keep the agent in one place and add a per-platform adapter that translates its output into each platform's native format. The agent is written once, and each channel requires just another output target instead of a separate build. CopilotKit open-sourced this full implementation in the Channels SDK. Essentially, any agent that implements AG-UI can run in a messaging platform in a few lines of code, like Slack, Teams, Discord, WhatsApp, and many more. Because the agent runs inside the thread, it has that conversation's context, so it can summarize the discussion, open a ticket, or route to the right person. It works with any backend, so LangGraph, CrewAI, Mastra, Google ADK, or a plain HTTP agent can connect through an existing endpoint. The same message can render as a Block Kit in Slack and as Adaptive Cards in Teams. In practice, the model and orchestration stay the same; it requires no migration or rewrite. It also handles human-in-the-loop approvals, persistence, and transcripts that carry state across platforms, so a thread started in Teams can continue in Slack. CopilotKit is open-source, and AG-UI is supported across every major agent framework, including LangGraph, CrewAI, Mastra, and Google ADK. Here's the repo: (don't forget to star it ⭐) The agent running in Slack no longer has to be a vendor's. It can be the one you already built. The video below shows this in action. Thanks to CopilotKit for working with me on this launch.

Another blow to Anthropic! They spent months building what's now fully open-source. Anthropic recently put Claude inside Slack, where you can tag it in a channel. It reads the thread, breaks the task into steps, and posts the result back. The problem is that it only runs Claude and only in the channels Anthropic supports. Running your own agent there is harder. The reasoning, tool calls, and state management are mostly handled by the framework. Connecting that agent to a messaging platform is not. Moreover, each platform has a different integration: - Slack renders messages with Block Kit - Teams uses Adaptive Cards - and each has its own SDK, auth flow, and delivery model. If an agent needs to run on three platforms, one must write three separate integrations against the same agent logic. That overhead explains why most custom agents never get deployed to Slack, and why the ones that do are usually a single vendor's hosted assistant. The alternative is to keep the agent in one place and add a per-platform adapter that translates its output into each platform's native format. The agent is written once, and each channel requires just another output target instead of a separate build. CopilotKit open-sourced this full implementation in the Channels SDK. Essentially, any agent that implements AG-UI can run in a messaging platform in a few lines of code, like Slack, Teams, Discord, WhatsApp, and many more. Because the agent runs inside the thread, it has that conversation's context, so it can summarize the discussion, open a ticket, or route to the right person. It works with any backend, so LangGraph, CrewAI, Mastra, Google ADK, or a plain HTTP agent can connect through an existing endpoint. The same message can render as a Block Kit in Slack and as Adaptive Cards in Teams. In practice, the model and orchestration stay the same; it requires no migration or rewrite. It also handles human-in-the-loop approvals, persistence, and transcripts that carry state across platforms, so a thread started in Teams can continue in Slack. CopilotKit is open-source, and AG-UI is supported across every major agent framework, including LangGraph, CrewAI, Mastra, and Google ADK. Here's the repo: (don't forget to star it ⭐) The agent running in Slack no longer has to be a vendor's. It can be the one you already built. The video below shows this in action. Thanks to CopilotKit for working with me on this launch.

Akshay 🚀

243,299 просмотров • 25 дней назад