GPT-4V picks up an object with my robot hand. 1 hour to setup 0 tuning. Using laptop camera. Output is wrist angles and grip type. #Robotics

So freaking cool! Is this done through in-context learning from tokenized manual-demo examples in the prompt, like this one?

@TeslaBotJournal That’s a cool way to do it. I just told it to plan a sequence of actions to pick up the yellow toy. No demos or pre-training or previous context. It took a few tries and I was surprised it worked.

Super cool! Nice work.

This is really cool! Have you considered using LLaVA or CoGVLM to see how well they perform?

Thanks. I’ll try LLaVA!

BASED BASED BASED BASED. Imagine feeding it an image that contains a previous pose + parameters and an image with current pose + parameters and let it compare so it knows best how to move the fingers

@RoboDepot cool! would love to have a look at the code/prompt

Can you share more details of how this works? What prompts did you use, etc.?

This is indeed awesome, but be aware that the compute/memory requirements for running an LLM far exceed what you’d want in an autonomous robot. Also, you can see this runs really slowly. So you’d want to create a highly optimized robot LLM if you use this approach in production.

Agreed! Its was painfully slow :D