Loading video...

Video Failed to Load

Go Home

Gemini + π0 = actually useful robots! (Similar to what Physical Intelligence did with "Hi Robot") I can now verbally tell the robot that I'm building a red Lego wall or wooden tower, and it will infer the next steps by itself and pass me the necessary pieces, tools,...

25,485 views • 1 year ago •via X (Twitter)

10 Comments

Shreyas Gite's profile picture
Shreyas Gite1 year ago

And sometimes when pi0 wakes-up it does a wild-drunken behaviours:D But super exciting, thanks @RemiCadene and @LeRobotHF community! We are at stage we can actually build useful robots for constrained environments and specific tasks that genuinely generate economic value.

Shreyas Gite's profile picture
Shreyas Gite1 year ago

Here is the video with Legos. Also one of the wooden bars is too far to reach, so the arm moves to the next target. I think with the next generation of VLA models trained on top of video generation models, we should automatically see a step-change improvement in accuracy.

Saketh Saketh's profile picture
Saketh Saketh1 year ago

@physical_int Try GROOT Mimic

Tran Nguyen Le's profile picture
Tran Nguyen Le1 year ago

@physical_int Hi @shreyasgite, great work! I will soon get my hand on the SO-ARM101, and I would like to apply pi0 for the arm as well. Any guidance how to get on with it? Thanks :D

Shreyas Gite's profile picture
Shreyas Gite1 year ago

@physical_int For training you can use lerobot as it is; for inference I changed few things in control_utils.py file under robot devices;) Will make a quick tutorial this week or next✌️

H4sh3's profile picture
H4sh31 year ago

@physical_int Did you finetune or something?

Saba Khalilnaji's profile picture
Saba Khalilnaji1 year ago

@physical_int Totally sick!! What are you using the grasping points for if you’re using pi0 and not conditioning with it? Also which gemni model?

Shreyas Gite's profile picture
Shreyas Gite1 year ago

@physical_int Gemini 2.0 flash. Not using grasping points in this one, but used it for other training set as grasping the objects correctly is still a bit tricky.

Krishna's profile picture
Krishna1 year ago

@physical_int Maybe a simple Readme about how all the pieces fit together, please? Will get us noobs acclimated super-fast. Thank you for your valuable contributions.

Shreyas Gite's profile picture
Shreyas Gite1 year ago

@physical_int Here you go;)

Related Videos

Learning from Human Demonstrations: Show the Robot How to Act! The pipeline is very similar to older experiments using Gemini & pi0 with LeRobot. Pi-zero runs locally, while Gemini Flash generates the affordances and the high-level task. (More details are in the thread.) The new component is learning from demonstrations via Gemini 2.5 Pro. I capture a video while demoing & take one of the last frames. Gemini 2.5 Pro then extracts the instructions & passes them to Gemini Flash to process the scene. The fun part is that there's no fancy insight that came from me; other than the days spent figuring out the right prompts. It's the bitter lesson hitting you in the face -> Enhanced Gemini capabilities make this possible. For example, Gemini Flash cannot do Russian doll stacking, but Gemini 2.5 Pro can do it consistently. The current limitation is low-level manipulation: - As you can see, I'm aligning the objects so they are easy to grasp using the same technique from the training data. I couldn't get Gemini Flash to consistently output an accurate grasping angle, and Gemini 1.5 Pro was too expensive and slow for real-time deployment. - Getting a symmetrical gripper should also help a lot. Adding rubber to the tips would probably also help prevent objects from slipping. Collecting & curating the data was the most time consuming & labor intensive part. Next, to improve low-level manipulation and make the system more real-time, I'm shifting to focus more on sims & synthetic data. This aligns better with my core competence. I'm open to tips and suggestions.

Shreyas Gite

22,555 views • 1 year ago

More moves to AI-first living. Got the Genspark browser that just came out. One thing I notice is that I like having separate browsers working on different tasks. I'm not a big user of MCPs, since I'm not really a developer. Are you? They have a store of a bunch of MCPs available to use. Most of my AI work is research related, or answering emails and scheduling things, I'm focusing more of my effort there. Genspark has been shipping a ton of stuff the past few weeks, from ability to make slides to a new AI secretary that can schedule things, or help you answer your emails. First I looked at, "does it make sense to move over to this new way of working?" The problem with that is you gotta learn a new browser. Well, it looks like Google Chrome, except it has a prompt window. Click the icons underneath it and you will see you can add X, Notion, Google stuff like calendar and gmail, or a variety of others. But the real power is just talking to it. I asked it what kinds of tasks it can do, and it answered: "Just tell me what you want to accomplish! For example: "Research the latest developments in spatial computing and create a presentation" "Analyze my Gmail for any important emails from this week" "Find information about AI robotics companies and create a spreadsheet" "Generate a video about emerging tech trends" "Help me plan a trip to CES 2026" And so further into AI-first living I go. Have you tried it yet? What do you suggest I do next to use more AI to run my life and build my business? I have another browser tab building a video for me. What a beautiful time to be alive. Download it at:

Robert Scoble

84,349 views • 1 year ago

Zack Snyder on his dyslexia: "It was a challenge for me when I was, you know, young in school, and all I wanted to do was make movies because that was the thing that I got great pleasure from and reward from. I love books, and I'm an avid reader, but I just have a hard time because of the way that I perceive. "I've had a great sort of - one side of me anyways - was really satisfied by art and drawing and sculpture and sort of visual expression. And I think that that started to, you know, was the thing that kind of made me feel un-frustrated. And also the way the system was designed, sort of not to support me when I was in high school at that time. "It was very difficult, you know, there was a lot of, you know, just, difficulty. My English teacher in high school was worried about what my career would be, and I'm like. He would be happy to know that I'm in the Writers Guild of America now. "But, I think that that all those things are, they're all... you can transcend all those things with perseverance and with interest and with with help. And I think that that's an important part of it. "And I just think I've had to adapt, and sort of... I have my own style of the way I write, I write all, you know, but I'm pretty prolific. And I love- I listen to tons of audio books on tape, unabridged hours and hours and hours. That's all I do when I'm driving in the car or wherever I'm doing. And it's helped me a lot. "And yeah, I mean, I just hope that anyone who is- feels trapped or frustrated by the world in general. You know, they need to just, I think that we all have like a magic spark, and you need to just find the thing that makes you, you know, inspires you and, and gets you excited and pursue it as hard as you can find your passion in the world. That's a, that's a great motivator."

Zack Snyder Film

11,128 views • 6 months ago