We are excited to announce Lilac (Lilac, joining Databricks!), a new Open Source product to analyze, structure and clean data with AI. Our mission is to make unstructured data visible, quantifiable, and malleable. 🌸 Announcement blog:

🔥 Check out our quick start guide:

🔗 Check out a live demo of the OpenOrca dataset! You can duplicate this HuggingFace space, make it private, and upload your own data – with no Python installation!

🔨 Lilac lets you: - Visually explore datasets - Enrich unstructured data with structured metadata using "Signals", e.g. near-dup and personal information detection. This allows us to compute statistics, find problematic slices, and eventually measure changes over time.

- Create and refine "Concepts" which are embedding-based AI models that can be built via the UI and can be used to find and score text from datasets that match a semantic concept you have in your mind. - Download the results of the enrichments for downstream applications.

🪄 Concepts are a powerful way of exploring and analyzing data. With a few clicks, make your own text classifier to instantly find text in a large dataset. Concepts can be built and refined through the UI or Python! Try the source code detection concept:

🔒 We are focused on making the product fast and useable *on-device*. That’s why many of the concepts in our demos are using on-device Open Source embeddings, like GTE. If you prefer to use a more powerful embedding, like OpenAI, PaLM, or Cohere, you can use your own API key.

⚡ We believe the best way to improve the quality and malleability of datasets is a hybrid of a Python API and a visual UI, built Open Source. We also believe a community is the best way to grow useful signals and concepts. Check out our website:

🚀 This is just the beginning of a long journey to make AI datasets more visible and improve their quality. Find us on GitHub if you have issues or suggestions!

⏰ Countdown alert! 🚀 Get free access to our new AI Writing feature before Dec 1st! Don't miss your chance to explore its power. Try it now—no cost, no catch! 🔗

Yes!! So proud of y'all for getting this released - and jazzed to see Lilac out in the world, making life a bit brighter for the data-inclined. ❤️

@lilac_ai Thanks Paige :) Your support means a lot!