New package + paper drop 📄 - Introducing KGGen – a simple library to transform unstructured text into knowledge graphs. Text is abundant, but good knowledge graphs are scarce. Feed it raw text, and KGGen generates a structured network of entities and relationships. (1/7)

We tried several existing text-to-knowledge graph generation tools and found they didn’t perform very well. So we built KGGen and a new benchmark, Measure of Information in Nodes & Edges (MINE). Paper: Install: `pip install kg-gen` (2/7)

KGGen uses an LLM-driven, multi-stage pipeline to improve graph sparsity issues: 1. Extract entities & relations 2. Aggregate info across multiple docs 3. Cluster entities and relations based on semantic similarity (e.g. "Supplier A LLC" and "Supplier-A" are 1 node) (3/7)

We compare KGGen to OpenIE and GraphRAG on MINE, a benchmark to evaluate how well a text-to-KG extractor captures information. We use 100 documents with ~15 key facts each, and check if those facts appear in the generated KGs. On the MINE benchmark, KGGen captured ~66% of the key facts on average, while GraphRAG managed ~48% and OpenIE only ~30%​. (there is still a ways to go!) (4/7)

Why is this important? Well, knowledge graphs may be used: - To assist with RAG (Retrieval-Augmented Generation) - For multi-hop question answering - As data for model training and testing - To visualize the relationships between concepts in your source text In particular, graph RAG is an increasingly useful way to retrieve contextually relevant information. It is however limited by the completeness of the knowledge graph. If you look at human-curated datasets like WikiGraphs, you may see that they take a long time to craft and are usually are missing relations. (5/7)

You can see in the video that we cluster similar nodes and edges respectively, which helps with curating a denser, richer graph. e.g. “strength”, “strong”, “strengths” are treated as one node. This compression is contextual! You may pass in a context string to `kg-gen` to describe your clustering criteria (6/7)

with @JoshuaK92829 @KyssenYu @proudmpala @sanmikoyejo and others! Github: tl;dr, KGGen turns text dumps into useful knowledge! The result is richer, cleaner KGs for powering tasks like question answering, knowledge base completion, or retrieval-augmented generation. (7/7)

@KyssenYu @proudmpala @sanmikoyejo and special thanks to Lisa Yu for making the video!

Thoughts on KG normalization across many entities, beyond what can fit in a context window?

I think a hierarchical approach could work! So it is localized step by step

so cute omg! but I want your voice

this is super cool, congrats on shipping this, excited to try it out!

hey, did you guys happen to compile the dspy program by any chance?

@threadreaderapp compile if you don't mind

What about comparison with Graphiti from Zep AI?

Super cool use of LLM-as-an-engine. Is this enabled by growing context windows, that weren't possible before?

that helps for sure