Video yükleniyor...

Video Yüklenemedi

Bu video yüklenirken bir sorun oluştu. Bu geçici bir ağ sorunundan kaynaklanıyor olabilir veya video kullanılamıyor olabilir.

Ana Sayfaya Dön

🚀 Introducing Cover-Agent 🧪 An open-source tool that includes a reimplementation of Meta's TestGen-LLM for automatically enhancing test suites. Manager: "We must improve old test suites for better code coverage. Can you handle it?" Me: "Sure, my favorite task... (Not!) 🤷‍♂️" Meta's team had the idea of using LLMs... show more

Itamar Friedman

6,193 subscribers

139,227 görüntüleme • 2 yıl önce •via X (Twitter)

Bilim & Teknoloji Eğitim

Anya Rossi• Live Now

Private livecam show

10 Yorum

Itamar Friedman profil fotoğrafı

Itamar Friedman2 yıl önce

Original TestGen-LLM paper: Cover-Agent open-source that reimplements TestGen-LLM by Meta:

Itamar Friedman profil fotoğrafı

Itamar Friedman2 yıl önce

Some of the excitement about TestGen-LLM: Check out our blog for more info about TestGen-LLM and Cover-Agent:

Itamar Friedman profil fotoğrafı

Itamar Friedman2 yıl önce

Cover-Agent in action: ‣ Go example: ‣ Python example:

Qodo (formerly Codium) profil fotoğrafı

Qodo (formerly Codium)2 yıl önce

Here is a short how-to and demo video:

Dennis profil fotoğrafı

Dennis2 yıl önce

this is dope

Eyal Cohen profil fotoğrafı

Eyal Cohen2 yıl önce

Will try it out!

Gil Elbaz profil fotoğrafı

Gil Elbaz2 yıl önce

Looks great! Thnx @itamar_mar

Itamar Friedman profil fotoğrafı

Itamar Friedman2 yıl önce

Thank you! Counting on you to open Issues and maybe even PRs 😃

Hvipublik profil fotoğrafı

Hvipublik2 yıl önce

Awesome tool! Can ChatGPT swapped out for some locally running LLM (Mistral of something) for added IP safety?

Itamar Friedman profil fotoğrafı

Itamar Friedman2 yıl önce

Many are asking for it, we will enable it! You are, of course, welcome to contribute a solution that enables that. There are a few examples -- we will follow the solution implemented in PR-Agent

Benzer Videolar

The first open-source implementation of the paper that will change automatic test generation is now available! In February, Meta published a paper introducing a tool to automatically increase test coverage, guaranteeing improvements over an existing code base. This is a big deal, but Meta didn't release the code. Fortunately, we now have Cover-Agent, an open-source tool you can install that implements Meta's paper to generate unit tests automatically: I recorded a quick video showing Cover-Agent in action. There are two things I want to mention: 1. Automatically generating unit tests is not new, but doing it right is difficult. If you ask ChatGPT to do it, you'll get duplicate, non-working, and meaningless tests that don't improve your code. Meta's solution only generates unique tests that run and increase code coverage. 2. People who write tests before writing the code (TDD) will find this less helpful. That's okay. Not everyone does TDD, but we all need to improve test coverage. There are many good and bad applications of AI, but this is one I'm looking forward to make part of my life.

The first open-source implementation of the paper that will change automatic test generation is now available! In February, Meta published a paper introducing a tool to automatically increase test coverage, guaranteeing improvements over an existing code base. This is a big deal, but Meta didn't release the code. Fortunately, we now have Cover-Agent, an open-source tool you can install that implements Meta's paper to generate unit tests automatically: I recorded a quick video showing Cover-Agent in action. There are two things I want to mention: 1. Automatically generating unit tests is not new, but doing it right is difficult. If you ask ChatGPT to do it, you'll get duplicate, non-working, and meaningless tests that don't improve your code. Meta's solution only generates unique tests that run and increase code coverage. 2. People who write tests before writing the code (TDD) will find this less helpful. That's okay. Not everyone does TDD, but we all need to improve test coverage. There are many good and bad applications of AI, but this is one I'm looking forward to make part of my life.

Santiago

774,460 görüntüleme • 2 yıl önce

i'm a little sick of chatgpt giving me obviously broken code i've found a "micro agent" approach to LLM code generation can work much better the LLM first generates a *test*, and then enters a loop where it generates and iterates on the code until the tests pass source below

i'm a little sick of chatgpt giving me obviously broken code i've found a "micro agent" approach to LLM code generation can work much better the LLM first generates a test, and then enters a loop where it generates and iterates on the code until the tests pass source below

Steve (Builder.io)

544,125 görüntüleme • 2 yıl önce

Introducing Expect Test your agent code in a real browser Works with your agent. Fully open source

Introducing Expect Test your agent code in a real browser Works with your agent. Fully open source

Aiden Bai

73,246 görüntüleme • 2 ay önce

Introducing Keak Agents! Free & open source tool that enables agents to interact with your website and tools. Because this is Keak, we also let you A/B test definitions and track how they impact agent usage of your website!

Introducing Keak Agents! Free & open source tool that enables agents to interact with your website and tools. Because this is Keak, we also let you A/B test definitions and track how they impact agent usage of your website!

Fab

300,826 görüntüleme • 3 ay önce

we built the first sane way to debug your agent locally. you can see your traces. codex/claude code can too. this lets them write evals and test your agents automatically. best part: it's completely free and open source. install with 1 line. (github below)

we built the first sane way to debug your agent locally. you can see your traces. codex/claude code can too. this lets them write evals and test your agents automatically. best part: it's completely free and open source. install with 1 line. (github below)

ben hylak

192,150 görüntüleme • 25 gün önce

CodiumAI PR-Agent is an open-source tool that assists developers in streamlining pull-request creation and review. It automatically analyzes the PR and can provide several types of feedback, including Auto-Description, PR Review, Q&A, Code Suggestion, and more.

CodiumAI PR-Agent is an open-source tool that assists developers in streamlining pull-request creation and review. It automatically analyzes the PR and can provide several types of feedback, including Auto-Description, PR Review, Q&A, Code Suggestion, and more.

Qodo

12,114,688 görüntüleme • 2 yıl önce

New short course: Evaluating AI Agents! Evals are important for driving AI system improvements, and in this course you'll learn to systematically assess and improve an AI agent’s performance. This is built in partnership with Arize AI and taught by John Gilhuly, Head of Developer Relations, and , Director of Product. I've often found evals to be a critical tool in the agent development process - they can be the difference between picking the right thing to work on vs. wasting weeks of effort. Whether you’re building a shopping assistant, coding agent, or research assistant, having a structured evaluation process helps you refine its performance systematically, rather than relying on random trial and error. This course shows you how to structure your evals to assess the performance of each component of an agent and its end-to-end performance. For each component, you select the appropriate evaluators, test examples, and performance metrics. This helps you identify areas for improvement both during development and in production. (If you're familiar with error analysis in supervised learning, think of this as adapting those ideas to agentic workflows.) In this course, you'll build an AI agent, and add observability to visualize and debug its steps. You’ll learn about code-based evals, in which you write code explicitly to test a certain step, as well as LLM-as-a-Judge evals, in which you prompt an LLM to efficiently come up with ways to evaluate more open-ended outputs. In detail, you’ll: - Understand key differences between evaluating LLM-based systems and traditional software testing. - Add observability to an agent by collecting traces of the steps taken by the agent and visualizing them - Choose the appropriate evaluator - code-based, LLM-as-a-Judge, human-annotation based - for each component. - Compute a convergence score to evaluate if your agent can respond to a query in an efficient number of steps. - Run structured experiments to improve the agent’s performance by exploring changes to the prompt, LLM model, or the agent’s logic. - Understand how to deploy these evaluation techniques to monitor the agent’s performance in production. By the end of this course, you’ll know how to trace AI agents, systematically evaluate them, and improve their performance. Please sign up here:

New short course: Evaluating AI Agents! Evals are important for driving AI system improvements, and in this course you'll learn to systematically assess and improve an AI agent’s performance. This is built in partnership with Arize AI and taught by John Gilhuly, Head of Developer Relations, and , Director of Product. I've often found evals to be a critical tool in the agent development process - they can be the difference between picking the right thing to work on vs. wasting weeks of effort. Whether you’re building a shopping assistant, coding agent, or research assistant, having a structured evaluation process helps you refine its performance systematically, rather than relying on random trial and error. This course shows you how to structure your evals to assess the performance of each component of an agent and its end-to-end performance. For each component, you select the appropriate evaluators, test examples, and performance metrics. This helps you identify areas for improvement both during development and in production. (If you're familiar with error analysis in supervised learning, think of this as adapting those ideas to agentic workflows.) In this course, you'll build an AI agent, and add observability to visualize and debug its steps. You’ll learn about code-based evals, in which you write code explicitly to test a certain step, as well as LLM-as-a-Judge evals, in which you prompt an LLM to efficiently come up with ways to evaluate more open-ended outputs. In detail, you’ll: - Understand key differences between evaluating LLM-based systems and traditional software testing. - Add observability to an agent by collecting traces of the steps taken by the agent and visualizing them - Choose the appropriate evaluator - code-based, LLM-as-a-Judge, human-annotation based - for each component. - Compute a convergence score to evaluate if your agent can respond to a query in an efficient number of steps. - Run structured experiments to improve the agent’s performance by exploring changes to the prompt, LLM model, or the agent’s logic. - Understand how to deploy these evaluation techniques to monitor the agent’s performance in production. By the end of this course, you’ll know how to trace AI agents, systematically evaluate them, and improve their performance. Please sign up here:

Andrew Ng

126,355 görüntüleme • 1 yıl önce

IBM dropped CUGA, open-source enterprise agent to automate boring tasks 🔥 > given workspace files, it writes and executes code to accomplish any task 🤯 > comes with a ton of tools built for enterprise tasks, supports MCPs > plug in your favorite LLM 👏 here's a small demo where it retrieves info from a file, calculates revenue by writing code, and drafts an e-mail 🤯 they release code, a blog and a demo 🙌🏻 you can run this locally

IBM dropped CUGA, open-source enterprise agent to automate boring tasks 🔥 > given workspace files, it writes and executes code to accomplish any task 🤯 > comes with a ton of tools built for enterprise tasks, supports MCPs > plug in your favorite LLM 👏 here's a small demo where it retrieves info from a file, calculates revenue by writing code, and drafts an e-mail 🤯 they release code, a blog and a demo 🙌🏻 you can run this locally

merve

32,866 görüntüleme • 5 ay önce

The #1 problem with coding agents right now: Ask them to solve one problem, and they will make 10 other changes you didn't want. This happens to me every day. It happens to everyone I talk to as well. We have a solution for this now. The team Augment Code released a "Task List" feature for their coding assistant that solves this problem. Augment Code is partnering with me on this post. In case you haven't used them before: • Augment Code is a fully-fledged coding assistant • Their specialty are large projects • Fastest coding indexing I've seen • Has a free forever community edition Now, you can ask their coding agent to generate a Task List before doing anything. This will give you a plan you can review, edit, and augment if you need to. You can export this plan, load it on a different session, or even share it across projects. It makes a huge difference: The task list constrains the agent so you won't get any "unintended" changes anymore. It also puts you in control of everything the agent does. Check the video to see the agent working through a task list. You can also try this 100% free: (By the way, they also have support for remote agents. You can basically have those agents write your code while you are sleeping.)

The #1 problem with coding agents right now: Ask them to solve one problem, and they will make 10 other changes you didn't want. This happens to me every day. It happens to everyone I talk to as well. We have a solution for this now. The team Augment Code released a "Task List" feature for their coding assistant that solves this problem. Augment Code is partnering with me on this post. In case you haven't used them before: • Augment Code is a fully-fledged coding assistant • Their specialty are large projects • Fastest coding indexing I've seen • Has a free forever community edition Now, you can ask their coding agent to generate a Task List before doing anything. This will give you a plan you can review, edit, and augment if you need to. You can export this plan, load it on a different session, or even share it across projects. It makes a huge difference: The task list constrains the agent so you won't get any "unintended" changes anymore. It also puts you in control of everything the agent does. Check the video to see the agent working through a task list. You can also try this 100% free: (By the way, they also have support for remote agents. You can basically have those agents write your code while you are sleeping.)

Santiago

41,738 görüntüleme • 10 ay önce

We can teach LLMs to write better robot code through natural language feedback. But can LLMs remember what they were taught and improve their teachability over time? Introducing our latest work, Learning to Learn Faster from Human Feedback with Language Model Predictive Control

We can teach LLMs to write better robot code through natural language feedback. But can LLMs remember what they were taught and improve their teachability over time? Introducing our latest work, Learning to Learn Faster from Human Feedback with Language Model Predictive Control

Jacky Liang

86,549 görüntüleme • 2 yıl önce

Build AutoGen Agents without writing *any* code AutoGen studio is a low code / no code tool for prototyping agents in AutoGen. You can do the following: - Use the team builder to compose teams (drag and drop from a built in gallery of teams including a deep research agent team, web agents team etc). - Modify the team, agent, tools, models etc, validate and test behavior - Observe agent behavior - messages exchanged, token usage, LLM calls etc - Export and use python teams in AutoGen Studio and vice versa. For users who want a low-code experience and rapid prototyping of ideas, AutoGen Studio is good starting point. We just released version 0.4.1, take a moment to check it out (links to documentation, video walkthrough below)

Build AutoGen Agents without writing any code AutoGen studio is a low code / no code tool for prototyping agents in AutoGen. You can do the following: - Use the team builder to compose teams (drag and drop from a built in gallery of teams including a deep research agent team, web agents team etc). - Modify the team, agent, tools, models etc, validate and test behavior - Observe agent behavior - messages exchanged, token usage, LLM calls etc - Export and use python teams in AutoGen Studio and vice versa. For users who want a low-code experience and rapid prototyping of ideas, AutoGen Studio is good starting point. We just released version 0.4.1, take a moment to check it out (links to documentation, video walkthrough below)

AutoGen

14,225 görüntüleme • 1 yıl önce

we built an open source version of operator its a web only agent with vision that you can easily run yourself it’s also very slow, likely to not work well but its a great reference (code + link below)

we built an open source version of operator its a web only agent with vision that you can easily run yourself it’s also very slow, likely to not work well but its a great reference (code + link below)

Sully

102,851 görüntüleme • 1 yıl önce

🔎🤖LangSmith Insights Agent Really excited to launch our first in-product agent This agent lives inside LangSmith and combs through traces, giving you insights into: 🧑‍🤝‍🧑how users are using your agent ⁉️how your agent may be messing up 🛃{your custom insight here} The problem we saw was that people were launching agents... and didn't know how their users were actually using them! You put a chat box in front of people, and they may ask it anything - the surface area for agents is often super wide In addition - agents would fail silently. They could give a bad response - this wouldn't show up in error logs, but its good to know. If you know what look for, you can set up LLM as a judge evaluators. But what if you don't? (most people don't initially) The best way to figure this out - as Hamel Husain says - "look at your data". But LLMs are really good at looking at your data! So can they do it for you? This is exactly what insights agent attempts to do. It's live in LangSmith today. You can read more about it here:

🔎🤖LangSmith Insights Agent Really excited to launch our first in-product agent This agent lives inside LangSmith and combs through traces, giving you insights into: 🧑‍🤝‍🧑how users are using your agent ⁉️how your agent may be messing up 🛃{your custom insight here} The problem we saw was that people were launching agents... and didn't know how their users were actually using them! You put a chat box in front of people, and they may ask it anything - the surface area for agents is often super wide In addition - agents would fail silently. They could give a bad response - this wouldn't show up in error logs, but its good to know. If you know what look for, you can set up LLM as a judge evaluators. But what if you don't? (most people don't initially) The best way to figure this out - as Hamel Husain says - "look at your data". But LLMs are really good at looking at your data! So can they do it for you? This is exactly what insights agent attempts to do. It's live in LangSmith today. You can read more about it here:

Harrison Chase

98,487 görüntüleme • 7 ay önce

"Orgs and cos building MCP servers are taking an LLM-first approach to what the API needs to expose to the agent(s)." - Nikunj Handa from OpenAI "For example, Stripe has a bunch of APIs that can be used to create a subscription/customer/product/price. For an LLM, it can just combine that into a single function." "Instead of returning this massive JSON object, they can return something very specific to the task being solved, so that the LLM can more easily understand what's happening." "It's an opportunity to rewrite your APIs to be very LLM-first. Why do 2 hours of work, when you can do it in 4 lines of code under a minute?"

"Orgs and cos building MCP servers are taking an LLM-first approach to what the API needs to expose to the agent(s)." - Nikunj Handa from OpenAI "For example, Stripe has a bunch of APIs that can be used to create a subscription/customer/product/price. For an LLM, it can just combine that into a single function." "Instead of returning this massive JSON object, they can return something very specific to the task being solved, so that the LLM can more easily understand what's happening." "It's an opportunity to rewrite your APIs to be very LLM-first. Why do 2 hours of work, when you can do it in 4 lines of code under a minute?"

TBPN

11,445 görüntüleme • 1 yıl önce

Introducing /ui-test Give your agent a PR, and it'll test your feature in a real browser, generating an HTML report with UI fixes The planner agent generates adversarial test cases to break your app, assigns them to subagents, and evaluates the page using the Browserbase CLI We reviewed a merged PR on Cal.com and found a bunch of small UI nits:

Introducing /ui-test Give your agent a PR, and it'll test your feature in a real browser, generating an HTML report with UI fixes The planner agent generates adversarial test cases to break your app, assigns them to subagents, and evaluates the page using the Browserbase CLI We reviewed a merged PR on Cal.com and found a bunch of small UI nits:

Shrey Pandya

57,480 görüntüleme • 2 ay önce

New Release: Claudex Mode For the first time in mankind claude code and codex will be on the same team On the Blackbox CLI you can send the same task for Claude Code to implement and Codex to verify and test. You can choose more ways where they can collaborate on the same task to get the highest level of performance on your tasks

New Release: Claudex Mode For the first time in mankind claude code and codex will be on the same team On the Blackbox CLI you can send the same task for Claude Code to implement and Codex to verify and test. You can choose more ways where they can collaborate on the same task to get the highest level of performance on your tasks

BLACKBOX AI

304,035 görüntüleme • 3 ay önce

Studio MCP Server now supports full agentic loops, and Assistant can connect to external LLMs via Bring Your Own Key (BYOK). This allows creators to automatically edit, test, and refine their code with AI in plain language on Roblox using their preferred tools. The simple video shows a full loop with analyze, test, bug found, fix, test, pass.

Studio MCP Server now supports full agentic loops, and Assistant can connect to external LLMs via Bring Your Own Key (BYOK). This allows creators to automatically edit, test, and refine their code with AI in plain language on Roblox using their preferred tools. The simple video shows a full loop with analyze, test, bug found, fix, test, pass.

David Baszucki

241,020 görüntüleme • 3 ay önce

BREAKING: Replit just launched mobile apps. Now you can build in Replit ⠕ and submit straight to the App Store for review. Describe your app → build it on Replit → test on your phone → publish to the App Store. I built a unicorn pet app for my niece: - Worked with Replit Agent to build it out - Scanned a QR code to download and test on my phone - One click to send to App Store Connect for review - A couple submissions later, it's live in the App Store! 🙌 I didn't write a single line of code! Just described what I wanted to Agent. Amazing! 😅 Go build something!! 👇

BREAKING: Replit just launched mobile apps. Now you can build in Replit ⠕ and submit straight to the App Store for review. Describe your app → build it on Replit → test on your phone → publish to the App Store. I built a unicorn pet app for my niece: - Worked with Replit Agent to build it out - Scanned a QR code to download and test on my phone - One click to send to App Store Connect for review - A couple submissions later, it's live in the App Store! 🙌 I didn't write a single line of code! Just described what I wanted to Agent. Amazing! 😅 Go build something!! 👇

Manny Bernabe

30,925 görüntüleme • 4 ay önce

Better feedback = Better code. Artifacts are the agent’s proof of work. Review plans, screen recordings, and code changes. Then, leave feedback to keep the agent on track. Try it today:

Better feedback = Better code. Artifacts are the agent’s proof of work. Review plans, screen recordings, and code changes. Then, leave feedback to keep the agent on track. Try it today:

Google Antigravity

82,293 görüntüleme • 6 ay önce