Can we teach LLMs to write long articles from scratch, grounded in trustworthy sources? Do Wikipedia editors think this can assist them? 📣Announcing STORM, a system that writes Wikipedia-like articles based on Internet search. I now use STORM in my daily research!🧵

Generating long articles with citations is hard to do & hard to evaluate! We break this problem down into two steps: 1️⃣Pre-writing, in which the system collects references and generates an outline. 2️⃣Writing, in which the system generates the final article with citations.

“Pre-writing” requires researching a topic from scratch. That makes it hard even for expert humans. And directly prompting the LM to generate questions doesn’t work well! The questions lack depth and have limited breadth. STORM is designed to teach LMs to *ask good questions*.

STORM improves question asking by automatically discovering perspectives for researching the topic and adding the perspective in the prompt. It also simulates information-seeking conversations to encourage follow-up questions which are usually more in-depth.

We build FreshWiki to mitigate data leakage into LM training data for evaluation. To measure quality, we introduce heading soft recall and heading entity recall. Outline eval makes it easier to prototype methods for pre-writing. STORM outperforms well-designed RAG baselines!

In the final writing stage, STORM generates text with citations and writes the full article section by section. Articles produced by STORM are favored by both automatic metrics *and* experienced Wikipedia editors!

Such expository writing should always be grounded. We assess citation quality and ask Wikipedia editors to rate verifiability. We find the major challenge stems from red herring rather than widely discussed factual hallucination. This calls for research beyond fact-checking!

We also ask Wikipedia editors for the perceived usefulness of STORM. It’s exciting that all participants agree that STORM is helpful for their pre-writing stage. Also, I use STORM myself to learn concepts in-depth in my research 😎(check out our demo video if you haven’t).

It’s worth mentioning that STORM is a carefully designed pipeline for knowledge curation rather than a single prompt or model. We build STORM using DSPy which provides very neat modularization - this allows us to keep extending our work without getting lost in many prompt files.

We are working on making the demo public to let more people try out STORM. Stay tuned! Read our Arxiv paper to learn more: Thanks @_Yucheng_Jiang , Theo, Peter, @lateinteraction , and @MonicaSLam for the amazing collaboration!!

Congratulations Yijia!! Amazing!! 🤯🎉 - I love the task decomposition into pre-writing and writing! 🧱 - The detail into question asking is 🔥, I love that the sort of "Hello world" first example of DSPy is multi-hop QA -- super powerful and underrated in the RAG world. Perspective-Guided and Conversational Question Asking is absolutely next level, fascinating! 🧠 - So cool to see the comparison of STORM with direct generation, standard RAG, and outline-driven RAG on both GPT-3.5 and GPT-4!! The ablation on Perspective-Guided / Conversational is incredible -- hmm, I would have thought that Perspective-Guided would be more powerful, maybe chat fine-tune slightly biases Conversational QA to outperform Perspective-Guided? - Wow!! Love the human annotation and deeper dive into fact checking. Looks like we're coming a long way from FEVER benchmarking haha. The general statement of perceived usefulness is also super fascinating! Also cool to see citation quality judged by Mistral 7B-Instruct, love seeing these multi-model DSPy systems!! 🤯🤯 Congratulations again! Excited to dive in further, amazing work! 🔥🎉