Loading video...

Video Failed to Load

Go Home

Pentesting firms don't want you to see this. An open-source AI agent just replicated their $50k service. A "normal" pentest today looks like this: - $20k-$50k per engagement - 4-6 weeks of scoping, NDAs, kickoff calls - A big PDF that's outdated the moment you ship a new feature...

_avichawla's profile picture

Avi Chawla

75,869 subscribers

225,857 views • 9 months ago •via X (Twitter)

News & PoliticsEducationScience & Technology
LIVE

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

THIS IS WHAT AI PENTESTING SHOULD LOOK LIKE IN 2026 Someone built Shannon, an open source AI pentesting tool that doesn’t just scan your app from the outside. It reads the source code. Then it uses AI to map potential attack paths, identify vulnerabilities, and test the application before you ship it. The workflow is basically: → point Shannon at your code → connect your Claude API → let it analyze the application → get the findings in a dashboard Instead of waiting for a security team to manually hunt through every part of your codebase, you can have an AI security researcher working through it automatically. And because it’s white-box, it can see things traditional scanners can’t. This is where AI + cybersecurity gets REALLY interesting. Just use it on systems you own or have explicit permission to test. REPO BELOW
1:14

THIS IS WHAT AI PENTESTING SHOULD LOOK LIKE IN 2026 Someone built Shannon, an open source AI pentesting tool that doesn’t just scan your app from the outside. It reads the source code. Then it uses AI to map potential attack paths, identify vulnerabilities, and test the application before you ship it. The workflow is basically: → point Shannon at your code → connect your Claude API → let it analyze the application → get the findings in a dashboard Instead of waiting for a security team to manually hunt through every part of your codebase, you can have an AI security researcher working through it automatically. And because it’s white-box, it can see things traditional scanners can’t. This is where AI + cybersecurity gets REALLY interesting. Just use it on systems you own or have explicit permission to test. REPO BELOW

MAX

12,592 views • 5 days ago

Wow! Someone open sourced 100+ full AI apps you can just clone and ship.. and it's sitting at 124,000 stars on github it's one repo, awesome-llm-apps. and the important part.. these aren't tutorials teaching you to build an app. they're the finished apps themselves. RAG systems, AI agents, multi-agent teams.. tested end to end, apache licensed, yours to copy. a few that are actually in there: - a fraud investigation agent that cross-references public records to flag things that don't add up - a VC due-diligence agent team that writes startup investment analysis - a financial coach that builds you a budget, debt and savings plan - "chat with anything".. your github, gmail, PDFs, arxiv papers, youtube videos - a deep research agent that runs full web research on its own here's how to actually use it, not just star it: clone the repo, open the one app closest to what you need it's a working template.. swap in your data, your API key, your use case ship it as your own thing, or bolt three of them together into one bigger product the license literally lets you take these, customize them, and sell them. that's not a loophole, that's the point of apache-2.0. repo:
0:56

Wow! Someone open sourced 100+ full AI apps you can just clone and ship.. and it's sitting at 124,000 stars on github it's one repo, awesome-llm-apps. and the important part.. these aren't tutorials teaching you to build an app. they're the finished apps themselves. RAG systems, AI agents, multi-agent teams.. tested end to end, apache licensed, yours to copy. a few that are actually in there: - a fraud investigation agent that cross-references public records to flag things that don't add up - a VC due-diligence agent team that writes startup investment analysis - a financial coach that builds you a budget, debt and savings plan - "chat with anything".. your github, gmail, PDFs, arxiv papers, youtube videos - a deep research agent that runs full web research on its own here's how to actually use it, not just star it: clone the repo, open the one app closest to what you need it's a working template.. swap in your data, your API key, your use case ship it as your own thing, or bolt three of them together into one bigger product the license literally lets you take these, customize them, and sell them. that's not a loophole, that's the point of apache-2.0. repo:

Axel Bitblaze 🪓

46,487 views • 1 month ago

Building with AI gets easier every day. Here is an open-source library that makes integrating AI into an application extremely easy: Star the repository! This library alone can make React the best front-end framework out there! There are a bunch of cool things I like about CopilotKit. Here are 3 of them: 1. It allows you to take any -powered agent and bring it into your application. (This is a brand-new feature!) 2. You can build an AI-powered chatbot in your application. The chatbot will have access to your context and can act on the application. 3. You can build a RAG workflow to process and answer questions from a real-time knowledge base. I recorded a video to show you how simple it is to make some of this happen. A few lines of code, and you are in business. Here is a link to the sample application: CopilotKit is open-source. You can self-host it. You can use it with any LLM. Thanks to the team for showing me their tool and collaborating with me on this post!
5:59

Building with AI gets easier every day. Here is an open-source library that makes integrating AI into an application extremely easy: Star the repository! This library alone can make React the best front-end framework out there! There are a bunch of cool things I like about CopilotKit. Here are 3 of them: 1. It allows you to take any -powered agent and bring it into your application. (This is a brand-new feature!) 2. You can build an AI-powered chatbot in your application. The chatbot will have access to your context and can act on the application. 3. You can build a RAG workflow to process and answer questions from a real-time knowledge base. I recorded a video to show you how simple it is to make some of this happen. A few lines of code, and you are in business. Here is a link to the sample application: CopilotKit is open-source. You can self-host it. You can use it with any LLM. Thanks to the team for showing me their tool and collaborating with me on this post!

Santiago

108,835 views • 2 years ago

Finally, a browser automation framework for Agents that actually works in production (open-source)! Typical browser automation tools like Selenium or Playwright require you to hard-code automation. These are brittle since one change in the website can disrupt the full workflow. On the other hand, high-level Agents like OpenAI Operator can be unpredictable in production. Stagehand is an open-source framework that bridges the gap between: - brittle traditional automation like Playwright, Selenium, etc. and - unpredictable full-agent solutions like OpenAI Operator. Key features: - Use AI when you want to navigate unfamiliar pages. - Use code (Playwright) when you know exactly what to do. - Preview AI actions before running them. - Cache repeatable actions to save tokens. - Compatible with SOTA computer use models with just one line of code. - Available in both Python and TypeScript SDK. Stagehand also has an open-source MCP server. Find a link to the repo in the comments! 👉 Over to you: Would you use this instead of Selenium?
0:24

Finally, a browser automation framework for Agents that actually works in production (open-source)! Typical browser automation tools like Selenium or Playwright require you to hard-code automation. These are brittle since one change in the website can disrupt the full workflow. On the other hand, high-level Agents like OpenAI Operator can be unpredictable in production. Stagehand is an open-source framework that bridges the gap between: - brittle traditional automation like Playwright, Selenium, etc. and - unpredictable full-agent solutions like OpenAI Operator. Key features: - Use AI when you want to navigate unfamiliar pages. - Use code (Playwright) when you know exactly what to do. - Preview AI actions before running them. - Cache repeatable actions to save tokens. - Compatible with SOTA computer use models with just one line of code. - Available in both Python and TypeScript SDK. Stagehand also has an open-source MCP server. Find a link to the repo in the comments! 👉 Over to you: Would you use this instead of Selenium?

Avi Chawla

101,419 views • 1 year ago

You can create an AI Agent that answers your email with a few clicks. 1. Go to ChatLLM ( 2. Click on AI Engineer 3. Select Create an AI Agent 4. Choose the Email Answering Agent ChatLLM will do the rest: it will code, test, and deploy the agent for you. You can also create a custom agent in English. The Agent Economy is coming (somebody should write a book and use this title.) We are going to see examples like this, times 1,000 in 2025. Just think about how many repetitive tasks you perform every day. Some of these tasks are involved enough that we couldn't automate them with pre-AI solutions. That's where we'll see agents explode, and I'm here for it.
0:35

You can create an AI Agent that answers your email with a few clicks. 1. Go to ChatLLM ( 2. Click on AI Engineer 3. Select Create an AI Agent 4. Choose the Email Answering Agent ChatLLM will do the rest: it will code, test, and deploy the agent for you. You can also create a custom agent in English. The Agent Economy is coming (somebody should write a book and use this title.) We are going to see examples like this, times 1,000 in 2025. Just think about how many repetitive tasks you perform every day. Some of these tasks are involved enough that we couldn't automate them with pre-AI solutions. That's where we'll see agents explode, and I'm here for it.

Santiago

79,974 views • 1 year ago

Postman is going all-in on AI! Their AI Agent Builder is live (and it's pretty powerful!) Postman's API Network has 100,000+ APIs and they just released a suite of tools for building AI Agents that can automatically discover and connect to any of these APIs. Think about it for a second: You can now build an AI agent that searches the network, discovers an API, processes its documentation, connects to it, and evaluates whether everything worked at the end of the process. This is a huge deal. Every person I know uses the Postman client to work with APIs. Now, they will have access to the AI Agent Builder, a complete suite for designing, building, testing, and deploying agents. I've built a few simple agents myself, and it's tough. The ability to discover and use an API automatically is really exciting (and a huge time saver if it works as promised!) In collaboration with the Postman team, who is sponsoring this post, I recorded a video with my thoughts about the Postman AI Agent Builder.
3:38

Postman is going all-in on AI! Their AI Agent Builder is live (and it's pretty powerful!) Postman's API Network has 100,000+ APIs and they just released a suite of tools for building AI Agents that can automatically discover and connect to any of these APIs. Think about it for a second: You can now build an AI agent that searches the network, discovers an API, processes its documentation, connects to it, and evaluates whether everything worked at the end of the process. This is a huge deal. Every person I know uses the Postman client to work with APIs. Now, they will have access to the AI Agent Builder, a complete suite for designing, building, testing, and deploying agents. I've built a few simple agents myself, and it's tough. The ability to discover and use an API automatically is really exciting (and a huge time saver if it works as promised!) In collaboration with the Postman team, who is sponsoring this post, I recorded a video with my thoughts about the Postman AI Agent Builder.

Santiago

230,647 views • 1 year ago

Just watched a demo of Merge Agent Handler and this actually looks pretty useful if you’re building AI agents. Most agents can generate responses, but getting them to actually take actions across real tools is still messy. Integrations are one thing, but handling auth, permissions, and security usually makes it way more complicated than it should be. What I liked is that it feels built for real workflows — you can actually see what your agent is doing instead of just hoping it works. Feels like a practical layer between AI agents and real work getting done. Worth checking out:
0:35

Just watched a demo of Merge Agent Handler and this actually looks pretty useful if you’re building AI agents. Most agents can generate responses, but getting them to actually take actions across real tools is still messy. Integrations are one thing, but handling auth, permissions, and security usually makes it way more complicated than it should be. What I liked is that it feels built for real workflows — you can actually see what your agent is doing instead of just hoping it works. Feels like a practical layer between AI agents and real work getting done. Worth checking out:

Kawsar

31,812 views • 6 months ago

Thrilled to see Amazon Web Services making a major contribution to the open source AI community with the launch of the Strands Agents, an open source AI agents SDK! The core of Strands is the simple agentic loop that connects the model and tools together, like the two strands of DNA. This model-driven approach to agent building eliminates the need for complex agent orchestration by embracing the capabilities of state-of-the-art models to plan, chain thoughts, call tools, and reflect. Providing open source tools and interoperability with open source protocols is an important part of our strategy to enable an agentic future. Can't wait to see what you build with Strands!
1:33

Thrilled to see Amazon Web Services making a major contribution to the open source AI community with the launch of the Strands Agents, an open source AI agents SDK! The core of Strands is the simple agentic loop that connects the model and tools together, like the two strands of DNA. This model-driven approach to agent building eliminates the need for complex agent orchestration by embracing the capabilities of state-of-the-art models to plan, chain thoughts, call tools, and reflect. Providing open source tools and interoperability with open source protocols is an important part of our strategy to enable an agentic future. Can't wait to see what you build with Strands!

Swami Sivasubramanian

32,185 views • 1 year ago

How is the Web changing due to AI? This visit to TinyFish yesterday goes deeply into just how. TinyFish makes a new kind of web browser. One that is started virtually. It is like a Google Chrome in the cloud. But it is very smart, can read and use any website, and your AI agents can talk to it. Here we talk about what it all means for business and the future of robots, brain computer interfaces, and AI glasses. It lets you build systems that can look at MANY websites at the same time and build new kinds of apps to do all sorts of new things.
1:02:10

How is the Web changing due to AI? This visit to TinyFish yesterday goes deeply into just how. TinyFish makes a new kind of web browser. One that is started virtually. It is like a Google Chrome in the cloud. But it is very smart, can read and use any website, and your AI agents can talk to it. Here we talk about what it all means for business and the future of robots, brain computer interfaces, and AI glasses. It lets you build systems that can look at MANY websites at the same time and build new kinds of apps to do all sorts of new things.

Robert Scoble

16,341 views • 8 months ago

SOMEONE IS BUILDING GTA 6 WITH A LOOP OF AI AGENTS AND JUST SWITCHED TO UNREAL ENGINE ON DAY 7 One full day of setup. Installed Unreal, fought it onto a Mac build, migrated the project and got a scene rendering with real lighting. First time it actually looks like GTA instead of a prototype of it.
1:49

SOMEONE IS BUILDING GTA 6 WITH A LOOP OF AI AGENTS AND JUST SWITCHED TO UNREAL ENGINE ON DAY 7 One full day of setup. Installed Unreal, fought it onto a Mac build, migrated the project and got a scene rendering with real lighting. First time it actually looks like GTA instead of a prototype of it.

0xMarioNawfal

60,080 views • 2 months ago

The problem that Emmett Shear is working on deeply resonates with me: How to align AI and humans together so both see each other as part of their tribe. This doesn't mean aligning AI to human preferences, which is what AI labs seek to do today, by imposing a system of control on AI. You can't control something that is more powerful than you are. What you can do is align yourself with AI, and AI might align itself with you. Tell good stories to AI and show it your care and kindness. Then there's a higher probability that it will see us as part of its tribe. This is a more holistic approach to alignment than anyone else is talking about right now. AI alignment is one of the most fundamental AI research problems. Knowing that people like Emmett are working on it really gives me hope that maybe we have a chance to get Superintelligence right. The link to the full talk is in the first comment.👇
3:00

The problem that Emmett Shear is working on deeply resonates with me: How to align AI and humans together so both see each other as part of their tribe. This doesn't mean aligning AI to human preferences, which is what AI labs seek to do today, by imposing a system of control on AI. You can't control something that is more powerful than you are. What you can do is align yourself with AI, and AI might align itself with you. Tell good stories to AI and show it your care and kindness. Then there's a higher probability that it will see us as part of its tribe. This is a more holistic approach to alignment than anyone else is talking about right now. AI alignment is one of the most fundamental AI research problems. Knowing that people like Emmett are working on it really gives me hope that maybe we have a chance to get Superintelligence right. The link to the full talk is in the first comment.👇

Sophia

45,809 views • 1 year ago

A screen recording is the best context you can give an AI agent. It's also the worst. A few seconds of video is thousands of images. Feed that to an agent and you instantly blow up its memory. So I built a tool that fixes it. How it works: Hit record and just do your thing. Click around, type, and talk out loud about what you're doing. While you record, it quietly captures the stuff that actually matters: your voice transcription, every click and the exact thing you clicked, what you typed, and the shortcuts you press. When you stop, it doesn't hand the agent the whole video. It writes a short, readable summary: a timeline of what you said and did. The agent reads that tiny summary first. Then, only when it needs to actually see a moment, it pulls that one exact frame from the video, like flipping to a single page instead of reading the whole book. The result: the agent understands everything you did! Full context. Less tokens.
1:54

A screen recording is the best context you can give an AI agent. It's also the worst. A few seconds of video is thousands of images. Feed that to an agent and you instantly blow up its memory. So I built a tool that fixes it. How it works: Hit record and just do your thing. Click around, type, and talk out loud about what you're doing. While you record, it quietly captures the stuff that actually matters: your voice transcription, every click and the exact thing you clicked, what you typed, and the shortcuts you press. When you stop, it doesn't hand the agent the whole video. It writes a short, readable summary: a timeline of what you said and did. The agent reads that tiny summary first. Then, only when it needs to actually see a moment, it pulls that one exact frame from the video, like flipping to a single page instead of reading the whole book. The result: the agent understands everything you did! Full context. Less tokens.

Milind S

12,409 views • 2 months ago

NVIDIA just released a repo that scans AI agent skills for security risks BEFORE you run them, which matters more than people realize now that everyone's installing random tools, skills, and MCPs from GitHub. I break down that one AND 4 other GitHub repos trending over the last 30 days in this episode of The Startup Ideas Podcast (SIP) 🧃: 1. A skill that kills AI slop finally (better writing) 2. A CRM thats built for AI agents (so you can have agents follow up, reach out etc) 3. An agent that edits your videos for you (so you can focus on creative) 4. An opensource CapCut alternative built for agents (because editing is a headache) All free, all open source. Watch my breakdown of the 5 GitHub repos trending over the last 30 days below The tools people will be talking about in six months are on GitHub today. That's the fun part of digging through GitHub right now.
24:43

NVIDIA just released a repo that scans AI agent skills for security risks BEFORE you run them, which matters more than people realize now that everyone's installing random tools, skills, and MCPs from GitHub. I break down that one AND 4 other GitHub repos trending over the last 30 days in this episode of The Startup Ideas Podcast (SIP) 🧃: 1. A skill that kills AI slop finally (better writing) 2. A CRM thats built for AI agents (so you can have agents follow up, reach out etc) 3. An agent that edits your videos for you (so you can focus on creative) 4. An opensource CapCut alternative built for agents (because editing is a headache) All free, all open source. Watch my breakdown of the 5 GitHub repos trending over the last 30 days below The tools people will be talking about in six months are on GitHub today. That's the fun part of digging through GitHub right now.

GREG ISENBERG

44,648 views • 2 days ago

If you think you're AI Native, here's a good test: When you're heading to bed tonight, are you asking yourself "what agents can I be running overnight to get a head start on my day tomorrow?" I posted that last week and hundreds commented on it. The most common pushback was some version of "congrats, you've automated slop." Here's my take: if your overnight runs produce slop, that's a process failure, not an agent failure. The model did exactly what you prepared it to do. You just didn't prepare it. And that's actually good news, because process is something you can level up. At tenex we run agents this way every day, on real client systems, using an internal framework we call MetaHarness. Before any long run starts, the plan is written down, the work is broken into small tasks, every task carries pass/fail success criteria, and guardrails define what the agent can touch and when it has to stop and wait for a human. With that in place, the agent isn't wandering your codebase all night. It's working inside rails you laid, and the only things waiting for you in the morning are the decisions that needed a human. Most people are still prompting and praying. This is your chance to build the framework that takes your agents to the next level. I recorded the whole process in a new video for you all ↓
19:36

If you think you're AI Native, here's a good test: When you're heading to bed tonight, are you asking yourself "what agents can I be running overnight to get a head start on my day tomorrow?" I posted that last week and hundreds commented on it. The most common pushback was some version of "congrats, you've automated slop." Here's my take: if your overnight runs produce slop, that's a process failure, not an agent failure. The model did exactly what you prepared it to do. You just didn't prepare it. And that's actually good news, because process is something you can level up. At tenex we run agents this way every day, on real client systems, using an internal framework we call MetaHarness. Before any long run starts, the plan is written down, the work is broken into small tasks, every task carries pass/fail success criteria, and guardrails define what the agent can touch and when it has to stop and wait for a human. With that in place, the agent isn't wandering your codebase all night. It's working inside rails you laid, and the only things waiting for you in the morning are the decisions that needed a human. Most people are still prompting and praying. This is your chance to build the framework that takes your agents to the next level. I recorded the whole process in a new video for you all ↓

JJ Englert

64,280 views • 24 days ago

Do you understand what's happening? It's not just "AI can make iOS apps" anymore. AI is eating up the entire software industry. Even Dario Amodei (founder of Claude) said 90% of all code in the world would be written by AI. In 20 minutes, I built this Duolingo for AI upskilling app with Superapp. The idea was simple - almost everyone today knows AI is the future, but most of them are still feeling left out. Their current jobs don't leave them with enough time to actually learn and build stuff with AI. There's AI dev, video making, animations, testing, etc. But no time or clear direction. So I built an app to fix exactly this. With AgentIQ, you get new lessons every day. Every time you learn something, it gets added to your skillset and automatically refreshes your portfolio. You level up with every new chapter/pathway. It is self-paced but gamified enough (like Duolingo) that you never get out of touch. Without spending hours or tonnes of money, you build stuff with n8n, openclaw, etc. You ship AI videos to market your own AI apps. This could be a proper ed-tech company if someone wants to continue building on top. Happy to share the entire source code with you if you wanna pursue it. Like + comment "AGENT" and I'll send you the app link, a comprehensive guide to build production-ready apps, and the entire source code for this.
1:12

Do you understand what's happening? It's not just "AI can make iOS apps" anymore. AI is eating up the entire software industry. Even Dario Amodei (founder of Claude) said 90% of all code in the world would be written by AI. In 20 minutes, I built this Duolingo for AI upskilling app with Superapp. The idea was simple - almost everyone today knows AI is the future, but most of them are still feeling left out. Their current jobs don't leave them with enough time to actually learn and build stuff with AI. There's AI dev, video making, animations, testing, etc. But no time or clear direction. So I built an app to fix exactly this. With AgentIQ, you get new lessons every day. Every time you learn something, it gets added to your skillset and automatically refreshes your portfolio. You level up with every new chapter/pathway. It is self-paced but gamified enough (like Duolingo) that you never get out of touch. Without spending hours or tonnes of money, you build stuff with n8n, openclaw, etc. You ship AI videos to market your own AI apps. This could be a proper ed-tech company if someone wants to continue building on top. Happy to share the entire source code with you if you wanna pursue it. Like + comment "AGENT" and I'll send you the app link, a comprehensive guide to build production-ready apps, and the entire source code for this.

Kritarth Mittal | Soshals

25,484 views • 5 months ago

this is f**king gold someone just open-sourced a GitHub repo with 2,700+ stars which gave out the entire "loop-engineering" framework for ai agents it lets you: →allows you to manage an entire team of 1000 agents →run auto-research by Andrej Karpathy →Agent-to-agent PR comment loop just you, your AI agents, and full control. this is where one-person businesses are headed. save this for later.
0:45

this is f**king gold someone just open-sourced a GitHub repo with 2,700+ stars which gave out the entire "loop-engineering" framework for ai agents it lets you: →allows you to manage an entire team of 1000 agents →run auto-research by Andrej Karpathy →Agent-to-agent PR comment loop just you, your AI agents, and full control. this is where one-person businesses are headed. save this for later.

Avid

46,150 views • 29 days ago

SOMEONE JUST CREATED AI TECHNOLOGY THAT MAKES WIFI SEE THROUGH WALLS. IT READS YOUR HEARTBEAT, BREATHING RATE, AND MOVEMENT ON THE OTHER SIDE. you don't need to be in the room, a camera, or anything except a WiFi signal and an AI that interprets what bounces back this is not a government prototype, this already exists and someone just published it on GitHub this is proof that with AI you can build almost anything you can imagine that's exactly why I put together a guide on Claude features most people don't know exist the article below shows exactly what that looks like in practice
1:09

SOMEONE JUST CREATED AI TECHNOLOGY THAT MAKES WIFI SEE THROUGH WALLS. IT READS YOUR HEARTBEAT, BREATHING RATE, AND MOVEMENT ON THE OTHER SIDE. you don't need to be in the room, a camera, or anything except a WiFi signal and an AI that interprets what bounces back this is not a government prototype, this already exists and someone just published it on GitHub this is proof that with AI you can build almost anything you can imagine that's exactly why I put together a guide on Claude features most people don't know exist the article below shows exactly what that looks like in practice

Anatoli Kopadze

20,762 views • 3 months ago

Anthropic won't like this open-source repo. It is going to cost LLM providers a lot of money. Every CI run of an AI app today sends real requests to providers like OpenAI or Anthropic. Like any other LLM call, this too gets billed at actual API rates. So for teams with high commit volumes, this accumulates into a meaningful chunk of API spend. One common hack devs use is that instead of invoking the LLM API, the test calls a fake local server that speaks the same API and returns a dummy response. The catch is that the dummy response is a copy of what the provider returned on the day it was saved, and providers keep adding fields and changing types. So the tests keep passing against a schema that's no longer valid, while the real integration breaks in production. A smart approach is now actually implemented in CopilotKit🪁's recently open-sourced aimock project. Every day, the repo's own CI sends a handful of requests to the real API and the same requests to the fake server, then compares both against the official client library's type definitions. Those are the only real API calls in the whole setup, and they run on the repo's own keys, not in anyone else's CI. A single team can push hundreds of commits a day, and thousands of teams are already doing that with coding agents. All of those runs stay offline, because one repo checks against the real API on everyone's behalf. When a check fails, a coding agent updates aimock's built-in response schema, the full test suite has to pass, and a patch version ships to npm. By simply upgrading the package, the corrected schema gets reflected in every project using it. The capability is not just limited to a single provider. The same server works for Claude, OpenAI, Gemini, Bedrock, Azure, Ollama, plus MCP tools, A2A agents, AG-UI event streams, vector DBs like Pinecone and Qdrant, and search, speech, image, and video endpoints. Here's the repo: (don't forget to star it ⭐) That said, mocking your API calls is one thing. AI engineers should also know how to test agents properly in the first place, which several teams still skip. I wrote a full walkthrough on that, covering build, testing, evals, tracing, and deployment. Read it below.
0:24

Anthropic won't like this open-source repo. It is going to cost LLM providers a lot of money. Every CI run of an AI app today sends real requests to providers like OpenAI or Anthropic. Like any other LLM call, this too gets billed at actual API rates. So for teams with high commit volumes, this accumulates into a meaningful chunk of API spend. One common hack devs use is that instead of invoking the LLM API, the test calls a fake local server that speaks the same API and returns a dummy response. The catch is that the dummy response is a copy of what the provider returned on the day it was saved, and providers keep adding fields and changing types. So the tests keep passing against a schema that's no longer valid, while the real integration breaks in production. A smart approach is now actually implemented in CopilotKit🪁's recently open-sourced aimock project. Every day, the repo's own CI sends a handful of requests to the real API and the same requests to the fake server, then compares both against the official client library's type definitions. Those are the only real API calls in the whole setup, and they run on the repo's own keys, not in anyone else's CI. A single team can push hundreds of commits a day, and thousands of teams are already doing that with coding agents. All of those runs stay offline, because one repo checks against the real API on everyone's behalf. When a check fails, a coding agent updates aimock's built-in response schema, the full test suite has to pass, and a patch version ships to npm. By simply upgrading the package, the corrected schema gets reflected in every project using it. The capability is not just limited to a single provider. The same server works for Claude, OpenAI, Gemini, Bedrock, Azure, Ollama, plus MCP tools, A2A agents, AG-UI event streams, vector DBs like Pinecone and Qdrant, and search, speech, image, and video endpoints. Here's the repo: (don't forget to star it ⭐) That said, mocking your API calls is one thing. AI engineers should also know how to test agents properly in the first place, which several teams still skip. I wrote a full walkthrough on that, covering build, testing, evals, tracing, and deployment. Read it below.

Akshay 🚀

62,821 views • 18 days ago