Loading video...
Video Failed to Load
Introducing PaperQA2, the first AI agent that conducts entire scientific literature reviews on its own. PaperQA2 is also the first agent to beat PhD and Postdoc-level biology researchers on multiple literature research tasks, as measured both by accuracy on objective benchmarks and assessments by human experts. We are publishing... show more
555,733 views • 1 year ago •via X (Twitter)
10 Comments

PaperQA2 finds and summarizes relevant literature, refines its search parameters based on what it finds, and provides cited, factually grounded answers that are more accurate on average than answers provided by PhD and postdoc-level biologists. When applied to answer highly specific questions, like this one, it obtains SOTA performance on LitQA2, part of LAB-Bench focused on information retrieval. 2/

PaperQA2 can also do broad-based literature reviews. WikiCrow, which is an agent based on PaperQA2, writes Wikipedia-style articles that are significantly more accurate on average than actual human-written articles on Wikipedia, as judged by PhD and postdoc-level biologists. We are using WikiCrow to write updated summaries of all 20,000 genes in the human genome. They are still being written, but in the meantime see a preview: 3/

We spent a lot of effort on making our open source version be excellent. We put together a new system of building metadata of arbitrary PDFs, full-text search, and more. See it here: 4/

Also, see our preprint for details, here: 5/

And, of course, this was all made possible through the wonderful generosity of Eric and Wendy Schmidt, and all of our other funders, including Open Philanthropy for supporting our work on LitQA2, the NSF National AI Resource program, and others! 6/

This is great work @SGRodriques! If I understand this correctly, the open source version requires downloaded pdfs in the folder structure. Just imagine what you could achieve if there was easy programmatic access to all publications and precomputed embeddings. Can someone just buy out the journal rights? :)

"Exceeding human performance" which translates into "still has errors". Using lossy techniques like AI are fine for art, music, etc. For subject matter that must be absolute, lossy techniques will not work. Compress your perfect data in a lossy compression format, uncompress it, and you have AI equivalent results.

Can you load it up with the 1000 most cited studies in psychology then put on an open url so we can ask it questions

Awesome work Sam and team!!

@Plinz Massive respect to you and team! This is incredibly important and thank you for open sourcing 🙏
