what if visualizing your data was actually a video game? flying through 30 million jina-v5-nano embeddings of 12 billion tokens of multilingual FineWeb + starcoder + pile + RedPajama looking at your data is supposed to be like eating your veggies, what if we made tasty veggies?

each voxel is a bin that holds umap data points, so we can preview whats in it with a heatmap (to show density), summary statistics (data set composition) or experimenting with an identicon (icon generated from content hash). i have more ideas but the weekend is over

Heatmap view is awesome.

You should make this into a @huggingface Space (even cooler if you can use Jobs so someone can set this up for any hub dataset!)

Veggies attract fruit flies - I see a pattern here :)

so cool

so cool

Severance would like a call

This is very cool, I’ve thought of doing something like this before. Are there some practical applications you are using this with?

This is amazing ❤️

I'd go looking for the single-point voxels you filtered out. Those samples might be more fun to explore than the dense neighborhoods.

mindcraft

30 million embeddings is the scale where 'explore vs. report' stops being a UX question and starts being an epistemology one

How many voxels?

3 million. but i filtered the view to only voxels with >5 points and its 500k (1.5m have just 1 point in them)