Sortex: our data quality platform built to search and filter the world's largest egocentric dataset. Across 50+ environments and 20k+ tasks, it uses an in-house model and evaluation harness to score each clip's usability, selecting for specific training objectives by: → Narrating tasks and steps → Checking hand and... action visibility → Comparing against the rest of the corpus We also tested Sortex on popular open datasets and found they differ widely in quality and coverage.show more

From raw recording to selection, Sortex: 1. Narrates each clip step by step, using sampled frames for coarse classification and video where timing matters 2. Structures the annotations and measurements in a defined taxonomy 3. Scores each clip, compares it with the rest of the corpus, and picks footage for the objective Outputs are stored, so we can change a filter without decoding and labeling every recording again.

Ego data is scaling quickly, but hours alone say little about action visibility or repetition. Each training objective calls for different footage, including: • Precise hand-object interactions • Multi-step task sequences • The same action across different environments Unlike broad category labels, Sortex narrates every step in each clip, then compares it with the rest of the corpus to select for each objective.

A model that filters too aggressively can mistake gloves, unusual tools, or partial occlusion for unusable video. We measure what our checks wrongly reject as well as what they catch, and keep the raw measurements so thresholds can change per dataset. Sortex clusters similar clips and applies limits by site, task, and environment, so we can see what each new recording adds.

We also ran Sortex on random clips from three open egocentric datasets. About one in six clips failed screening in EPIC-KITCHENS-100 and RekaDaily-10k, mostly for darkness, shake, blur, and faces. In Gen-HumanEgo, 42% of the clips that passed were near-duplicates of another clip in the sample.

Sortex was built by Addea Gupta (@addeagupta) and David Guo (@dgmonsoon). Read more below.

Very cool!

Looks great

👍

Testing the open datasets is the most useful line in this post. Everyone treats the free ones as a fair baseline. If their qualities varies this much, a lot of results built on top of them are partly measuring the data.

Can we contribute?

what gets a clip cut most, hands out of frame or unfinished tasks?