1/ Introducing Lynx - the leading hallucination detection model 🚀👀 - Beats GPT-4o on hallucination tasks - Open source, open weights, open data - Excels in real-world domains like medicine and finance We are excited to launch Lynx with Day 1 integration partners: NVIDIA, MongoDB, and Nomic 🔥

2/ Download the HuggingFace model here: You can use quantized Lynx-8B locally, deploy Lynx-70B with GPUs, or reach out to Patronus AI for easy API access 😀

3/ While LLM-as-judge models like GPT-4 have become popular for detecting hallucinations in RAG systems, they are often unreliable and inconsistent. See this example where GPT-4o and Claude-3-Sonnet both failed to identify the hallucinated answer to plant identification ❌ whereas Lynx correctly reasoned that the correct answer is “genus”, based on the document provided ✅

4/ What makes Lynx unique? - Lynx-70B is the largest and most powerful hallucination detection model to date ⚡ - Lynx not only produces a score but can also reason about it, like a human grader, making AI outputs more explainable and interpretable 🔍 - Lynx is especially strong at catching hard-to-detect and ambiguous hallucinations. This is because of novel approaches we used in training the model, including semantic perturbations and Chain-of-Thought reasoning! Read our research paper here:

5/ We are also open sourcing our new hallucination benchmark HaluBench. HaluBench is a large-scale 15k sample dataset that contains challenging hallucination tasks and supports diverse real world domains like finance and medicine. 🏦👩‍⚕️ Access HaluBench here:

6/ We are excited to support a number of integrations in the Lynx ecosystem! Check out this blog by @nomic_ai to see how we used Nomic Atlas and the power of embeddings to significantly improve the quality of HaluBench:

@nomic_ai 7/ Want to use Lynx with @nvidia's Nemo Guardrails? Use the dev branch in the Nemo Guardrails repo here:

@nomic_ai 8/ At Patronus AI, our mission is to make high quality LLM evaluation accessible to everyone. We are releasing Lynx, HaluBench, and our evaluation code for public access. Reach out to us to access Lynx via our API! Join our Discord for updates 🚀

@nomic_ai 9/ Check out the full paper on ArXiv:

@nvidia Really nice work from @PatronusAI. This has a potential to become a very useful tool. Also thanks for making it open source. 👏🙏

@nvidia