Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

today we're launching automatic captioning at Instance. send us your robot data, and every episode is segmented into captioned subtasks, graded success/fail plus 1-5 on quality and speed. if this could be useful for you, reach out and we'll caption an episode for free!

16,139 Aufrufe • vor 3 Monaten •via X (Twitter)

25 Kommentare

Profilbild von vishnu manoj
vishnu manojvor 3 Monaten

does it work for longform videos?

Profilbild von Lucy Cai
Lucy Caivor 3 Monaten

yes!

Profilbild von ^HI^
^HI^vor 3 Monaten

Incredible stuff

Profilbild von daniel wu
daniel wuvor 3 Monaten

Dino+vlm pipeline?

Profilbild von shrish
shrishvor 3 Monaten

Awesome!

Profilbild von Kayoum
Kayoumvor 3 Monaten

Super coool! We should chat!

Profilbild von Yousef
Yousefvor 3 Monaten

this is so cool!

Profilbild von Sujeet Soni
Sujeet Sonivor 3 Monaten

Amazing, qq: how captioning helps robotic companies? During their pre-training or fine tuning model ? Whats the difference in outcome for them with captioning and without it ?

Profilbild von Lucy Cai
Lucy Caivor 3 Monaten

yes helps both pretraining and finetuning! the captions help train a VLA to do low-level subtasks, and the grades help filter out bad data

Profilbild von Sujeet Soni
Sujeet Sonivor 3 Monaten

Understood, thanks for the explanation. Are you using open source VLM with fine tuning to achieve this or current LLM (like claude/Gemini etc) works for you ? Because I feel nowadays these LLM are so much reliable for visual understanding.

Profilbild von Prabhu
Prabhuvor 3 Monaten

Will customers really give us their data ?

Profilbild von Lucy Cai
Lucy Caivor 3 Monaten

some will!

Profilbild von Prabhu
Prabhuvor 3 Monaten

This gives me motivation :)

Profilbild von 🤖Sergio
🤖Sergiovor 3 Monaten

this looks amazing! How the failed vs succeeded subtasks can help with training or finetuning VLAs?

Profilbild von Lucy Cai
Lucy Caivor 3 Monaten

pi0.7 for example now accepts metadata tags (quality, speed) so you can train on suboptimal data instead of wasting it. and the failed subtasks can be filtered out of training data

Profilbild von Shaamil
Shaamilvor 3 Monaten

@nourzahzah Let’s go this is awesome!

Profilbild von Nour Zahzah
Nour Zahzahvor 3 Monaten

super cool

Profilbild von Shuddhatma Jain
Shuddhatma Jainvor 3 Monaten

Is this done using LLMs?

Profilbild von Lucy Cai
Lucy Caivor 3 Monaten

@nolan_gg1 VLMs!

Profilbild von Shuddhatma Jain
Shuddhatma Jainvor 3 Monaten

okay, so since all robotics companies are already dealing with VLAs, why would they need any 3rd party company to do captioning? can't they do it themselves? I mean it would be pretty easy for them to use VLMs and do the same, right?

Profilbild von Frank Lee
Frank Leevor 3 Monaten

so cool

Profilbild von Senthilnathan K
Senthilnathan Kvor 3 Monaten

We’re doing similar work + more layers, high time we have a standard benchmark for automated labelling/curation to help us compare and keep improving on the quality.

Profilbild von Johnny Núñez
Johnny Núñezvor 2 Monaten

amazing! I did something similar. Is public your code?

Profilbild von Riley Quantum
Riley Quantumvor 3 Monaten

How well does it handle ambiguous transitions between subtasks? That's usually where auto-labeling falls apart.

Profilbild von AI路子野
AI路子野vor 3 Monaten

how does it use ? containing with subtask didn't very help on language generalization

Ähnliche Videos