HOT TAKES: GPT-6 Astra killed task-level annotation. I created a highly accurate dense task annotation system from scratch, using Astra. BEST PART IS: I packaged it as a reusable LLM skill ! Link further down

i don't like: "tend the setting omelet" "work the omelet in the pan" wtf do those even mean it also missed the seasoning step

🤦‍♂️🤦‍♂️

🦾

Backstory: - Robot data annotation requires, beyond hand poses and so on, tasks, subtasks, and action details. - This type of annotation requires much more human hours than basic segmentation tasks, and is also more subjective. - I noticed Astra was pretty good at video understanding, so I figured there might be something worth building.

Long story short, the system understands: - Tasks - Subtasks - Actions - Overlap - Hand decomposition - Observed outcome and limits - Visible evidence

Check my live demo on 6 samples from open source datasets: It also comes with a detailed action ledger.

Now the main question is: now that egocentric data generation seems to get closer to full automation, is data annotation solved as well?

Do you want the packaged skill ? 📟 DM me, I would love to see what you build with it !

A pretty insightful source by @_varunnair:

what's the $$ per video hour my guy

Good question. I used my OpenAI subscription, but at current API rate, it would amount to about $0.22 per video minute.

This isn't new; any standard off-the-shelf VLM can ace this. Astra, in fact, is pretty heavy-duty for this specific task; the hot take is that it's killing the VLAs/WAMs themselves that these annotations are being used to train

please show me your demo then, I would love to see

Seems pretty interesting!

It sure is!

packaging it as a reusable skill is the part I care about. everyone shows the one-off demo, nobody shows the thing you can rerun tomorrow. what's the skill actually wrapping, a prompt chain or a callable tool?

a skill is a md file with instructions

Amazing work, Is the skill released?

Yes it is, DM me and I will send oyu the link

Is there any benchmark for the accuracy of the annotations Astra creates?

unfortunately no; but if can get my hands on a human annotated dataset, I could test and benchmark Astra over it

@andrewheejay You can try HD-EPIC? we have dense human annotations at multiple levels of activity: high level cooking recipe -> prep/step -> detailed running action annotations

@andrewheejay Feel free to scrape my webpage and compare to your results. Curious to see how it goes ;)