Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

today we're launching automatic captioning at Instance. send us your robot data, and every episode is segmented into captioned subtasks, graded success/fail plus 1-5 on quality and speed. if this could be useful for you, reach out and we'll caption an episode for free!

16,139 görüntüleme • 3 ay önce •via X (Twitter)

25 Yorum

vishnu manoj profil fotoğrafı
vishnu manoj3 ay önce

does it work for longform videos?

Lucy Cai profil fotoğrafı
Lucy Cai3 ay önce

yes!

^HI^ profil fotoğrafı
^HI^3 ay önce

Incredible stuff

daniel wu profil fotoğrafı
daniel wu3 ay önce

Dino+vlm pipeline?

shrish profil fotoğrafı
shrish3 ay önce

Awesome!

Kayoum profil fotoğrafı
Kayoum3 ay önce

Super coool! We should chat!

Yousef profil fotoğrafı
Yousef3 ay önce

this is so cool!

Sujeet Soni profil fotoğrafı
Sujeet Soni3 ay önce

Amazing, qq: how captioning helps robotic companies? During their pre-training or fine tuning model ? Whats the difference in outcome for them with captioning and without it ?

Lucy Cai profil fotoğrafı
Lucy Cai3 ay önce

yes helps both pretraining and finetuning! the captions help train a VLA to do low-level subtasks, and the grades help filter out bad data

Sujeet Soni profil fotoğrafı
Sujeet Soni3 ay önce

Understood, thanks for the explanation. Are you using open source VLM with fine tuning to achieve this or current LLM (like claude/Gemini etc) works for you ? Because I feel nowadays these LLM are so much reliable for visual understanding.

Prabhu profil fotoğrafı
Prabhu3 ay önce

Will customers really give us their data ?

Lucy Cai profil fotoğrafı
Lucy Cai3 ay önce

some will!

Prabhu profil fotoğrafı
Prabhu3 ay önce

This gives me motivation :)

🤖Sergio profil fotoğrafı
🤖Sergio3 ay önce

this looks amazing! How the failed vs succeeded subtasks can help with training or finetuning VLAs?

Lucy Cai profil fotoğrafı
Lucy Cai3 ay önce

pi0.7 for example now accepts metadata tags (quality, speed) so you can train on suboptimal data instead of wasting it. and the failed subtasks can be filtered out of training data

Shaamil profil fotoğrafı
Shaamil3 ay önce

@nourzahzah Let’s go this is awesome!

Nour Zahzah profil fotoğrafı
Nour Zahzah3 ay önce

super cool

Shuddhatma Jain profil fotoğrafı
Shuddhatma Jain3 ay önce

Is this done using LLMs?

Lucy Cai profil fotoğrafı
Lucy Cai3 ay önce

@nolan_gg1 VLMs!

Shuddhatma Jain profil fotoğrafı
Shuddhatma Jain3 ay önce

okay, so since all robotics companies are already dealing with VLAs, why would they need any 3rd party company to do captioning? can't they do it themselves? I mean it would be pretty easy for them to use VLMs and do the same, right?

Frank Lee profil fotoğrafı
Frank Lee3 ay önce

so cool

Senthilnathan K profil fotoğrafı
Senthilnathan K3 ay önce

We’re doing similar work + more layers, high time we have a standard benchmark for automated labelling/curation to help us compare and keep improving on the quality.

Johnny Núñez profil fotoğrafı
Johnny Núñez2 ay önce

amazing! I did something similar. Is public your code?

Riley Quantum profil fotoğrafı
Riley Quantum3 ay önce

How well does it handle ambiguous transitions between subtasks? That's usually where auto-labeling falls apart.

AI路子野 profil fotoğrafı
AI路子野3 ay önce

how does it use ? containing with subtask didn't very help on language generalization

Benzer Videolar