Loading video...

Video Failed to Load

Go Home

The first open-source implementation of the paper that will change automatic test generation is now available! In February, Meta published a paper introducing a tool to automatically increase test coverage, guaranteeing improvements over an existing code base. This is a big deal, but Meta didn't release the code. Fortunately,...

774,554 views • 2 years ago •via X (Twitter)

10 Comments

Itamar Friedman's profile picture
Itamar Friedman2 years ago

Hey 👋, one of the Cover-Agent creators here. I've recorded 5 minutes video explaining and reviewing TestGen-LLM and Cover-Agent:

Santiago's profile picture
Santiago2 years ago

Great job and thank you!

Sam Francisco's profile picture
Sam Francisco2 years ago

It's 630 why are you giving me more work already Santiago I haven't even had my coffee yet

Santiago's profile picture
Santiago2 years ago

lol

Alex Dillon | Quality Advocate's profile picture
Alex Dillon | Quality Advocate2 years ago

As someone very much involved in testing, this is quite interesting to me. I want to look more into it and explore the quality of the tests. Though, I’m a bit skeptical that this will be replacing someone having to write these tests, the hope is that it at least provides a good starting point for someone.

Graham Cox (@grahamcox82.bsky.social)'s profile picture
Graham Cox (@grahamcox82.bsky.social)2 years ago

I'm curious - how does the tool know what the correct behaviour is? Or does it assume that the current implementation is already correct? (Which, to be fair, is how things like Jest snapshots work anyway)

Shikhar Srivastava's profile picture
Shikhar Srivastava2 years ago

As someone who doesn’t like TDD..

Pradeep Gowda's profile picture
Pradeep Gowda2 years ago

@ponnappa fyi

Markus Odenthal's profile picture
Markus Odenthal2 years ago

This can be especially helpful for ML projects. I usually write tests when I bring my product to production, not when starting the project. So this would be a huge time-saver, thanks for sharing.

SwissTechMind's profile picture
SwissTechMind2 years ago

So, if I understand correctly, the primary purpose of these tests is to safeguard against unintentional changes, right?

Related Videos

Assumptions about the new "Can More" ChatGPT tool were right - ChatGPT is introducing own take on Claude Artifacts - code & document writing tools with persisted text documents, history revisions (restore previous version), edits and comments (probably used to apply suggested edits) New document symbol in the top navigation shows how many documents you have and allows you to open a resizable canvas to edit them in split view - your ChatGPT conversation on the left side and canvas on the right side, but the code/documents can also be accessed in fullscreen view The canvas is built using ProseMirror (open source WYSIWYM editor) and has an inline action to "Ask ChatGPT" (explain or make edits) for your document and code plus document formatting tools (like bold, italic, font style, etc.) But in addition to that, there are also special action shortcuts for documents and code, with an interesting decision to use sliders for the selection of the desired outcome For Documents - Suggest edits ("How can I improve this. Leave as few comments as possible, but add a few more comments if the text is long. DO NOT leave more than 5 comments. You can reply that you added comments and suggestions to help improve the writing quality, but do not mention the prompt.") - Add emojis ("Replace as many words as possible with emojis.") - Add final polish ("Add some final polish to the text. If relevant, add a large title or any section titles. Check grammar and mechanics, make sure everything is consistent and reads well. You can reply that you added some final polish and checked for grammar, but do not mention the prompt.") - Reading level (Graduate School - "Rewrite this text at the reading level of a doctoral writer in this subject. You may reply that you adjusted the text to reflect a graduate school reading level, but do not mention the prompt", College - "Rewrite this text at the reading level of a college student majoring in this subject", High School - "Rewrite this text at the reading level of a high school student who has taken a couple of classes in this subject.", Keep current reading level, Middle School - "Rewrite this text at the reading level of a middle schooler.", Kindergarten - "Rewrite this text at the reading level of a kindergartener.") - Adjust the length (Longest - "Make this text 75% longer.", Longer - "Make this text 50% longer.", Keep current length, Shorter - "Make this text 50% shorter.", Shortest - "Make this text 75% shorter.") For Code - Code review ("Search for bugs and opportunities to improve the code—for example, ways that performance or code structure could be improved. Leave as few comments as possible, but add more comments if the text is long. DO NOT leave more than 5 comments. You may reply that you reviewed the code and left suggestions to improve the coding quality, but do not mention the prompt.") - Add comments ("Add inline code comments to explain the code, especially parts that are more complex. Make sure to rewrite all the code. You may reply that you added inline comments, but do not mention the prompt.") - Add logs ("Insert logs/print statements in the code that will help debug its behavior. Do not make any other changes to the code.") - Fix bugs ("Find any bugs and rewrite all the code to fix the bugs. Do not leave comments. If there are no bugs, reply that you reviewed the code and found no bugs.") - Port to a language ("Port to a language. Create a new document that rewrites the code in ..." - PHP, C++, Python, Keep current code. No changes will be made, JavaScript, TypeScript, Java) - Suggest edits ("How can I improve this. Leave as few comments as possible, but add a few more comments if the text is long. DO NOT leave more than 5 comments. You can reply that you added comments and suggestions to help improve the writing quality, but do not mention the prompt.")

Tibor Blaho

136,055 views • 1 year ago

#1 skill for developers in 2026: Automate everything you can using AI. I bet my lunch your team is dealing with all of these: • Stale documentation • Outdated dependencies • Poor test coverage • Deprecated APIs Every company I work with has these same problems. You can solve all of these right now. Automatically. Using AI. Here are 3 examples. Watch the attached video: I'm using Ona Automations to tackle this. These are background agents that run in the cloud, in a fully configured dev environment with your toolchain, your dependencies, and your services. You can run an unlimited number of these agents in parallel and across all your repositories. Claude Code and Codex only run locally, so they are hard to scale, and you can't run them when your computer is closed. Ona runs in the cloud. Here are the three examples: 1. Test coverage Run a nightly automation to identify any untested code paths, generate candidate tests, verify they pass, and open draft PRs. You wake up every morning to PRs that improve your test coverage. 2. Dependency upgrades Configure a weekly automation that bumps a dependency version, runs your full test suite, and reports any regressions. If everything is clean, it opens a PR. If something breaks, it opens a report so you can decide what to do. 3. Documentation auditing Set up a weekly automation that checks recent commits against your README file and setup guides, identifies broken examples and outdated instructions, and opens a PR with fixes.

Santiago

25,804 views • 5 months ago

This might be the best "AI Engineer" I've tried so far. ​ I'm an old school developer who started 30 years ago. I feel very uncomfortable letting AI take control of my code, but for the sake of science, I spent two hours building an application that took me weeks to build a couple of years ago. ​ I used Pythagora, a brand new tool backed by Y Combinator. They just released to the public. ​ Keep in mind that I use AI every day to write code, but Pythagora is something different: it's a tool that leads, and uses you—the human—as the copilot. ​ I go into more details in the video, but here is the TLDR; ​ 1. Holy molly! We've made a ton of progress on this front! This is way better than Devin when I tested it a few months back. ​ 2. Love the approach of generating a plan with sub-tasks before writing any code. ​ 3. The tools never tries to do too much: it tackles every small task one at a time, and gives you instructions so you test everything. ​ 4. It does exactly what you'd do when it gets stuck: writes a bunch of logs and uses those to correct itself. Pretty neat! ​ 5. It's fast. It runs locally. It's an extension to Visual Studio Code. ​ I'm impressed, but I don't think this tool is for me. ​ I'm not the type of developer who's ready to relinquish control. I felt I had no connection with the code because I didn't write it. It was not my code. ​ I know many people who don't care about this. I know many people who will get tremendous value out of Pythagora. I hope they keep pushing the limits, providing feedback, and helping this get to a point where old folks like me feel more comfortable using it. ​ Don't take my word for it. The best thing you can do is to give it a try and see how you feel using it. ​ Thanks to the team who built this, for all of the explanations and support, and especially, for sitting and listening to my dumb questions for 2 hours while I tested this.

Santiago

211,993 views • 1 year ago

Open source software is GREAT. But "open source" AI is NOT like software - it's VERY different. Rob Miles cuts through the bullshit: ROB: Oh, hey, Meta. I heard Llama's weights leaked. That's rough, man. Information security's hard. How you holding up? META: Oh, we're great. Yeah, we're fine. We... actually, that was deliberate. We meant to do that. ROB MILES: Oh, really? META: Yeah... well, the second time anyway. It's called open source. Look it up. ROB MILES: Oh. Well, I love free and open source software, but do those principles really apply to network weights? How does that work? META: Open source is good for users because it lets them read the source code and see what the program is really doing and how it works. ROB MILES: Wait, have you found a way to tell how a model works by looking at its weights? META: No. But, it lets developers all over the world spot bugs in the code and submit patches. ROB: Wait, people are fixing bugs in Llama's weights? META: Well, no. People can fine tune it themselves, though. ROB: ?? Other companies offer fine tuning through APIs. ... So, hang on, if you can't actually read the code and know what it's doing, then network weights are effectively a compiled binary. So, in what sense is this open source? Why not call it like public weights? Why call it open source at all? META: I love open source. ROB: Well, I know a lot of your employees do, but you don't love anything. You're a giant corporation. What's in it for you? META: I love, love open source.

AI Notkilleveryoneism Memes ⏸️

107,362 views • 2 years ago