My new app is out !! ✨The Common Crawl Pipeline Creator ✨ Create your pipeline easily: ✔Run Text Extraction✂️ ✔Define Language Filters🌐 ✔Customize text quality💯 ✔See Live Results👀 ✔Get Python code 🐍 Based on famous LLM research like Gopher, C4 or FineWeb

Keep an eye on the data you are filtering out, you don't want to discard interesting quality & diverse data ! You can also check how much data is filtered out at each step

The python code is based on the `datatrove` library that was used to build the FineWeb dataset, which has a text quality so good it makes training LLMs faster !

Try the app by yourself !

@qlhoest awesome app!

@qlhoest this is so cool!

@qlhoest Amazing job! What is the aim of the code line "partial(increment_num_warc_samples, num_warc_samples_per_doc=2000 / 1687)" in the app code? I mean, why 2000 / 1687 ?

I load the data from a cache from an intermediate step that has 300+ examples filtered out