正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

🚀 Introducing Cover-Agent 🧪 An open-source tool that includes a reimplementation of Meta's TestGen-LLM for automatically enhancing test suites. Manager: "We must improve old test suites for better code coverage. Can you handle it?" Me: "Sure, my favorite task... (Not!) 🤷‍♂️" Meta's team had the idea of using LLMs... show more

Itamar Friedman

6,187 subscribers

139,228 次观看 • 2 年前 •via X (Twitter)

科学技术教育

Anya Rossi• Live Now

Private livecam show

10 条评论

Itamar Friedman 的头像

Itamar Friedman2 年前

Original TestGen-LLM paper: Cover-Agent open-source that reimplements TestGen-LLM by Meta:

Itamar Friedman 的头像

Itamar Friedman2 年前

Some of the excitement about TestGen-LLM: Check out our blog for more info about TestGen-LLM and Cover-Agent:

Itamar Friedman 的头像

Itamar Friedman2 年前

Cover-Agent in action: ‣ Go example: ‣ Python example:

Qodo (formerly Codium) 的头像

Qodo (formerly Codium)2 年前

Here is a short how-to and demo video:

Dennis 的头像

Dennis2 年前

this is dope

Eyal Cohen 的头像

Eyal Cohen2 年前

Will try it out!

Gil Elbaz 的头像

Gil Elbaz2 年前

Looks great! Thnx @itamar_mar

Itamar Friedman 的头像

Itamar Friedman2 年前

Thank you! Counting on you to open Issues and maybe even PRs 😃

Hvipublik 的头像

Hvipublik2 年前

Awesome tool! Can ChatGPT swapped out for some locally running LLM (Mistral of something) for added IP safety?

Itamar Friedman 的头像

Itamar Friedman2 年前

Many are asking for it, we will enable it! You are, of course, welcome to contribute a solution that enables that. There are a few examples -- we will follow the solution implemented in PR-Agent

相关视频

The first open-source implementation of the paper that will change automatic test generation is now available! In February, Meta published a paper introducing a tool to automatically increase test coverage, guaranteeing improvements over an existing code base. This is a big deal, but Meta didn't release the code. Fortunately, we now have Cover-Agent, an open-source tool you can install that implements Meta's paper to generate unit tests automatically: I recorded a quick video showing Cover-Agent in action. There are two things I want to mention: 1. Automatically generating unit tests is not new, but doing it right is difficult. If you ask ChatGPT to do it, you'll get duplicate, non-working, and meaningless tests that don't improve your code. Meta's solution only generates unique tests that run and increase code coverage. 2. People who write tests before writing the code (TDD) will find this less helpful. That's okay. Not everyone does TDD, but we all need to improve test coverage. There are many good and bad applications of AI, but this is one I'm looking forward to make part of my life.

The first open-source implementation of the paper that will change automatic test generation is now available! In February, Meta published a paper introducing a tool to automatically increase test coverage, guaranteeing improvements over an existing code base. This is a big deal, but Meta didn't release the code. Fortunately, we now have Cover-Agent, an open-source tool you can install that implements Meta's paper to generate unit tests automatically: I recorded a quick video showing Cover-Agent in action. There are two things I want to mention: 1. Automatically generating unit tests is not new, but doing it right is difficult. If you ask ChatGPT to do it, you'll get duplicate, non-working, and meaningless tests that don't improve your code. Meta's solution only generates unique tests that run and increase code coverage. 2. People who write tests before writing the code (TDD) will find this less helpful. That's okay. Not everyone does TDD, but we all need to improve test coverage. There are many good and bad applications of AI, but this is one I'm looking forward to make part of my life.

Santiago

774,488 次观看 • 2 年前

i'm a little sick of chatgpt giving me obviously broken code i've found a "micro agent" approach to LLM code generation can work much better the LLM first generates a *test*, and then enters a loop where it generates and iterates on the code until the tests pass source below

i'm a little sick of chatgpt giving me obviously broken code i've found a "micro agent" approach to LLM code generation can work much better the LLM first generates a test, and then enters a loop where it generates and iterates on the code until the tests pass source below

Steve (Builder.io)

544,125 次观看 • 2 年前

Introducing Expect Test your agent code in a real browser Works with your agent. Fully open source

Introducing Expect Test your agent code in a real browser Works with your agent. Fully open source

Aiden Bai

73,293 次观看 • 2 个月前

Introducing Keak Agents! Free & open source tool that enables agents to interact with your website and tools. Because this is Keak, we also let you A/B test definitions and track how they impact agent usage of your website!

Introducing Keak Agents! Free & open source tool that enables agents to interact with your website and tools. Because this is Keak, we also let you A/B test definitions and track how they impact agent usage of your website!

Fab

300,826 次观看 • 3 个月前

we built the first sane way to debug your agent locally. you can see your traces. codex/claude code can too. this lets them write evals and test your agents automatically. best part: it's completely free and open source. install with 1 line. (github below)

we built the first sane way to debug your agent locally. you can see your traces. codex/claude code can too. this lets them write evals and test your agents automatically. best part: it's completely free and open source. install with 1 line. (github below)

ben hylak

193,021 次观看 • 1 个月前

CodiumAI PR-Agent is an open-source tool that assists developers in streamlining pull-request creation and review. It automatically analyzes the PR and can provide several types of feedback, including Auto-Description, PR Review, Q&A, Code Suggestion, and more.

CodiumAI PR-Agent is an open-source tool that assists developers in streamlining pull-request creation and review. It automatically analyzes the PR and can provide several types of feedback, including Auto-Description, PR Review, Q&A, Code Suggestion, and more.

Qodo

12,114,696 次观看 • 2 年前

New short course: Evaluating AI Agents! Evals are important for driving AI system improvements, and in this course you'll learn to systematically assess and improve an AI agent’s performance. This is built in partnership with Arize AI and taught by John Gilhuly, Head of Developer Relations, and , Director of Product. I've often found evals to be a critical tool in the agent development process - they can be the difference between picking the right thing to work on vs. wasting weeks of effort. Whether you’re building a shopping assistant, coding agent, or research assistant, having a structured evaluation process helps you refine its performance systematically, rather than relying on random trial and error. This course shows you how to structure your evals to assess the performance of each component of an agent and its end-to-end performance. For each component, you select the appropriate evaluators, test examples, and performance metrics. This helps you identify areas for improvement both during development and in production. (If you're familiar with error analysis in supervised learning, think of this as adapting those ideas to agentic workflows.) In this course, you'll build an AI agent, and add observability to visualize and debug its steps. You’ll learn about code-based evals, in which you write code explicitly to test a certain step, as well as LLM-as-a-Judge evals, in which you prompt an LLM to efficiently come up with ways to evaluate more open-ended outputs. In detail, you’ll: - Understand key differences between evaluating LLM-based systems and traditional software testing. - Add observability to an agent by collecting traces of the steps taken by the agent and visualizing them - Choose the appropriate evaluator - code-based, LLM-as-a-Judge, human-annotation based - for each component. - Compute a convergence score to evaluate if your agent can respond to a query in an efficient number of steps. - Run structured experiments to improve the agent’s performance by exploring changes to the prompt, LLM model, or the agent’s logic. - Understand how to deploy these evaluation techniques to monitor the agent’s performance in production. By the end of this course, you’ll know how to trace AI agents, systematically evaluate them, and improve their performance. Please sign up here:

New short course: Evaluating AI Agents! Evals are important for driving AI system improvements, and in this course you'll learn to systematically assess and improve an AI agent’s performance. This is built in partnership with Arize AI and taught by John Gilhuly, Head of Developer Relations, and , Director of Product. I've often found evals to be a critical tool in the agent development process - they can be the difference between picking the right thing to work on vs. wasting weeks of effort. Whether you’re building a shopping assistant, coding agent, or research assistant, having a structured evaluation process helps you refine its performance systematically, rather than relying on random trial and error. This course shows you how to structure your evals to assess the performance of each component of an agent and its end-to-end performance. For each component, you select the appropriate evaluators, test examples, and performance metrics. This helps you identify areas for improvement both during development and in production. (If you're familiar with error analysis in supervised learning, think of this as adapting those ideas to agentic workflows.) In this course, you'll build an AI agent, and add observability to visualize and debug its steps. You’ll learn about code-based evals, in which you write code explicitly to test a certain step, as well as LLM-as-a-Judge evals, in which you prompt an LLM to efficiently come up with ways to evaluate more open-ended outputs. In detail, you’ll: - Understand key differences between evaluating LLM-based systems and traditional software testing. - Add observability to an agent by collecting traces of the steps taken by the agent and visualizing them - Choose the appropriate evaluator - code-based, LLM-as-a-Judge, human-annotation based - for each component. - Compute a convergence score to evaluate if your agent can respond to a query in an efficient number of steps. - Run structured experiments to improve the agent’s performance by exploring changes to the prompt, LLM model, or the agent’s logic. - Understand how to deploy these evaluation techniques to monitor the agent’s performance in production. By the end of this course, you’ll know how to trace AI agents, systematically evaluate them, and improve their performance. Please sign up here:

Andrew Ng

126,355 次观看 • 1 年前

IBM dropped CUGA, open-source enterprise agent to automate boring tasks 🔥 > given workspace files, it writes and executes code to accomplish any task 🤯 > comes with a ton of tools built for enterprise tasks, supports MCPs > plug in your favorite LLM 👏 here's a small demo where it retrieves info from a file, calculates revenue by writing code, and drafts an e-mail 🤯 they release code, a blog and a demo 🙌🏻 you can run this locally

IBM dropped CUGA, open-source enterprise agent to automate boring tasks 🔥 > given workspace files, it writes and executes code to accomplish any task 🤯 > comes with a ton of tools built for enterprise tasks, supports MCPs > plug in your favorite LLM 👏 here's a small demo where it retrieves info from a file, calculates revenue by writing code, and drafts an e-mail 🤯 they release code, a blog and a demo 🙌🏻 you can run this locally

merve

32,866 次观看 • 6 个月前

The #1 problem with coding agents right now: Ask them to solve one problem, and they will make 10 other changes you didn't want. This happens to me every day. It happens to everyone I talk to as well. We have a solution for this now. The team Augment Code released a "Task List" feature for their coding assistant that solves this problem. Augment Code is partnering with me on this post. In case you haven't used them before: • Augment Code is a fully-fledged coding assistant • Their specialty are large projects • Fastest coding indexing I've seen • Has a free forever community edition Now, you can ask their coding agent to generate a Task List before doing anything. This will give you a plan you can review, edit, and augment if you need to. You can export this plan, load it on a different session, or even share it across projects. It makes a huge difference: The task list constrains the agent so you won't get any "unintended" changes anymore. It also puts you in control of everything the agent does. Check the video to see the agent working through a task list. You can also try this 100% free: (By the way, they also have support for remote agents. You can basically have those agents write your code while you are sleeping.)

The #1 problem with coding agents right now: Ask them to solve one problem, and they will make 10 other changes you didn't want. This happens to me every day. It happens to everyone I talk to as well. We have a solution for this now. The team Augment Code released a "Task List" feature for their coding assistant that solves this problem. Augment Code is partnering with me on this post. In case you haven't used them before: • Augment Code is a fully-fledged coding assistant • Their specialty are large projects • Fastest coding indexing I've seen • Has a free forever community edition Now, you can ask their coding agent to generate a Task List before doing anything. This will give you a plan you can review, edit, and augment if you need to. You can export this plan, load it on a different session, or even share it across projects. It makes a huge difference: The task list constrains the agent so you won't get any "unintended" changes anymore. It also puts you in control of everything the agent does. Check the video to see the agent working through a task list. You can also try this 100% free: (By the way, they also have support for remote agents. You can basically have those agents write your code while you are sleeping.)

Santiago

41,738 次观看 • 10 个月前

We can teach LLMs to write better robot code through natural language feedback. But can LLMs remember what they were taught and improve their teachability over time? Introducing our latest work, Learning to Learn Faster from Human Feedback with Language Model Predictive Control

We can teach LLMs to write better robot code through natural language feedback. But can LLMs remember what they were taught and improve their teachability over time? Introducing our latest work, Learning to Learn Faster from Human Feedback with Language Model Predictive Control

Jacky Liang

86,652 次观看 • 2 年前

Build AutoGen Agents without writing *any* code AutoGen studio is a low code / no code tool for prototyping agents in AutoGen. You can do the following: - Use the team builder to compose teams (drag and drop from a built in gallery of teams including a deep research agent team, web agents team etc). - Modify the team, agent, tools, models etc, validate and test behavior - Observe agent behavior - messages exchanged, token usage, LLM calls etc - Export and use python teams in AutoGen Studio and vice versa. For users who want a low-code experience and rapid prototyping of ideas, AutoGen Studio is good starting point. We just released version 0.4.1, take a moment to check it out (links to documentation, video walkthrough below)

Build AutoGen Agents without writing any code AutoGen studio is a low code / no code tool for prototyping agents in AutoGen. You can do the following: - Use the team builder to compose teams (drag and drop from a built in gallery of teams including a deep research agent team, web agents team etc). - Modify the team, agent, tools, models etc, validate and test behavior - Observe agent behavior - messages exchanged, token usage, LLM calls etc - Export and use python teams in AutoGen Studio and vice versa. For users who want a low-code experience and rapid prototyping of ideas, AutoGen Studio is good starting point. We just released version 0.4.1, take a moment to check it out (links to documentation, video walkthrough below)

AutoGen

14,225 次观看 • 1 年前

we built an open source version of operator its a web only agent with vision that you can easily run yourself it’s also very slow, likely to not work well but its a great reference (code + link below)

we built an open source version of operator its a web only agent with vision that you can easily run yourself it’s also very slow, likely to not work well but its a great reference (code + link below)

Sully

105,079 次观看 • 1 年前

🔎🤖LangSmith Insights Agent Really excited to launch our first in-product agent This agent lives inside LangSmith and combs through traces, giving you insights into: 🧑‍🤝‍🧑how users are using your agent ⁉️how your agent may be messing up 🛃{your custom insight here} The problem we saw was that people were launching agents... and didn't know how their users were actually using them! You put a chat box in front of people, and they may ask it anything - the surface area for agents is often super wide In addition - agents would fail silently. They could give a bad response - this wouldn't show up in error logs, but its good to know. If you know what look for, you can set up LLM as a judge evaluators. But what if you don't? (most people don't initially) The best way to figure this out - as Hamel Husain says - "look at your data". But LLMs are really good at looking at your data! So can they do it for you? This is exactly what insights agent attempts to do. It's live in LangSmith today. You can read more about it here:

🔎🤖LangSmith Insights Agent Really excited to launch our first in-product agent This agent lives inside LangSmith and combs through traces, giving you insights into: 🧑‍🤝‍🧑how users are using your agent ⁉️how your agent may be messing up 🛃{your custom insight here} The problem we saw was that people were launching agents... and didn't know how their users were actually using them! You put a chat box in front of people, and they may ask it anything - the surface area for agents is often super wide In addition - agents would fail silently. They could give a bad response - this wouldn't show up in error logs, but its good to know. If you know what look for, you can set up LLM as a judge evaluators. But what if you don't? (most people don't initially) The best way to figure this out - as Hamel Husain says - "look at your data". But LLMs are really good at looking at your data! So can they do it for you? This is exactly what insights agent attempts to do. It's live in LangSmith today. You can read more about it here:

Harrison Chase

98,520 次观看 • 8 个月前

"Orgs and cos building MCP servers are taking an LLM-first approach to what the API needs to expose to the agent(s)." - Nikunj Handa from OpenAI "For example, Stripe has a bunch of APIs that can be used to create a subscription/customer/product/price. For an LLM, it can just combine that into a single function." "Instead of returning this massive JSON object, they can return something very specific to the task being solved, so that the LLM can more easily understand what's happening." "It's an opportunity to rewrite your APIs to be very LLM-first. Why do 2 hours of work, when you can do it in 4 lines of code under a minute?"

"Orgs and cos building MCP servers are taking an LLM-first approach to what the API needs to expose to the agent(s)." - Nikunj Handa from OpenAI "For example, Stripe has a bunch of APIs that can be used to create a subscription/customer/product/price. For an LLM, it can just combine that into a single function." "Instead of returning this massive JSON object, they can return something very specific to the task being solved, so that the LLM can more easily understand what's happening." "It's an opportunity to rewrite your APIs to be very LLM-first. Why do 2 hours of work, when you can do it in 4 lines of code under a minute?"

TBPN

11,445 次观看 • 1 年前

Introducing /ui-test Give your agent a PR, and it'll test your feature in a real browser, generating an HTML report with UI fixes The planner agent generates adversarial test cases to break your app, assigns them to subagents, and evaluates the page using the Browserbase CLI We reviewed a merged PR on Cal.com and found a bunch of small UI nits:

Introducing /ui-test Give your agent a PR, and it'll test your feature in a real browser, generating an HTML report with UI fixes The planner agent generates adversarial test cases to break your app, assigns them to subagents, and evaluates the page using the Browserbase CLI We reviewed a merged PR on Cal.com and found a bunch of small UI nits:

Shrey Pandya

57,480 次观看 • 2 个月前

New Release: Claudex Mode For the first time in mankind claude code and codex will be on the same team On the Blackbox CLI you can send the same task for Claude Code to implement and Codex to verify and test. You can choose more ways where they can collaborate on the same task to get the highest level of performance on your tasks

New Release: Claudex Mode For the first time in mankind claude code and codex will be on the same team On the Blackbox CLI you can send the same task for Claude Code to implement and Codex to verify and test. You can choose more ways where they can collaborate on the same task to get the highest level of performance on your tasks

BLACKBOX AI

304,664 次观看 • 3 个月前

Studio MCP Server now supports full agentic loops, and Assistant can connect to external LLMs via Bring Your Own Key (BYOK). This allows creators to automatically edit, test, and refine their code with AI in plain language on Roblox using their preferred tools. The simple video shows a full loop with analyze, test, bug found, fix, test, pass.

Studio MCP Server now supports full agentic loops, and Assistant can connect to external LLMs via Bring Your Own Key (BYOK). This allows creators to automatically edit, test, and refine their code with AI in plain language on Roblox using their preferred tools. The simple video shows a full loop with analyze, test, bug found, fix, test, pass.

David Baszucki

241,980 次观看 • 4 个月前

BREAKING: Replit just launched mobile apps. Now you can build in Replit ⠕ and submit straight to the App Store for review. Describe your app → build it on Replit → test on your phone → publish to the App Store. I built a unicorn pet app for my niece: - Worked with Replit Agent to build it out - Scanned a QR code to download and test on my phone - One click to send to App Store Connect for review - A couple submissions later, it's live in the App Store! 🙌 I didn't write a single line of code! Just described what I wanted to Agent. Amazing! 😅 Go build something!! 👇

BREAKING: Replit just launched mobile apps. Now you can build in Replit ⠕ and submit straight to the App Store for review. Describe your app → build it on Replit → test on your phone → publish to the App Store. I built a unicorn pet app for my niece: - Worked with Replit Agent to build it out - Scanned a QR code to download and test on my phone - One click to send to App Store Connect for review - A couple submissions later, it's live in the App Store! 🙌 I didn't write a single line of code! Just described what I wanted to Agent. Amazing! 😅 Go build something!! 👇

Manny Bernabe

30,925 次观看 • 5 个月前

Better feedback = Better code. Artifacts are the agent’s proof of work. Review plans, screen recordings, and code changes. Then, leave feedback to keep the agent on track. Try it today:

Better feedback = Better code. Artifacts are the agent’s proof of work. Review plans, screen recordings, and code changes. Then, leave feedback to keep the agent on track. Try it today:

Google Antigravity

82,293 次观看 • 6 个月前