today we're launching automatic captioning at Instance. send us your robot data, and every episode is segmented into captioned subtasks, graded success/fail plus 1-5 on quality and speed. if this could be useful for you, reach out and we'll caption an episode for free!

does it work for longform videos?

yes!

Incredible stuff

Dino+vlm pipeline?

Awesome!

Super coool! We should chat!

this is so cool!

Amazing, qq: how captioning helps robotic companies? During their pre-training or fine tuning model ? Whats the difference in outcome for them with captioning and without it ?

yes helps both pretraining and finetuning! the captions help train a VLA to do low-level subtasks, and the grades help filter out bad data

Understood, thanks for the explanation. Are you using open source VLM with fine tuning to achieve this or current LLM (like claude/Gemini etc) works for you ? Because I feel nowadays these LLM are so much reliable for visual understanding.

Will customers really give us their data ?

some will!

This gives me motivation :)

this looks amazing! How the failed vs succeeded subtasks can help with training or finetuning VLAs?

pi0.7 for example now accepts metadata tags (quality, speed) so you can train on suboptimal data instead of wasting it. and the failed subtasks can be filtered out of training data

@nourzahzah Let’s go this is awesome!

super cool

Is this done using LLMs?

@nolan_gg1 VLMs!

okay, so since all robotics companies are already dealing with VLAs, why would they need any 3rd party company to do captioning? can't they do it themselves? I mean it would be pretty easy for them to use VLMs and do the same, right?

so cool

We’re doing similar work + more layers, high time we have a standard benchmark for automated labelling/curation to help us compare and keep improving on the quality.

amazing! I did something similar. Is public your code?

How well does it handle ambiguous transitions between subtasks? That's usually where auto-labeling falls apart.

how does it use ? containing with subtask didn't very help on language generalization