Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Learning from Human Demonstrations: Show the Robot How to Act! The pipeline is very similar to older experiments using Gemini & pi0 with LeRobot. Pi-zero runs locally, while Gemini Flash generates the affordances and the high-level task. (More details are in the thread.) The new component is learning from... demonstrations via Gemini 2.5 Pro. I capture a video while demoing & take one of the last frames. Gemini 2.5 Pro then extracts the instructions & passes them to Gemini Flash to process the scene. The fun part is that there's no fancy insight that came from me; other than the days spent figuring out the right prompts. It's the bitter lesson hitting you in the face -> Enhanced Gemini capabilities make this possible. For example, Gemini Flash cannot do Russian doll stacking, but Gemini 2.5 Pro can do it consistently. The current limitation is low-level manipulation: - As you can see, I'm aligning the objects so they are easy to grasp using the same technique from the training data. I couldn't get Gemini Flash to consistently output an accurate grasping angle, and Gemini 1.5 Pro was too expensive and slow for real-time deployment. - Getting a symmetrical gripper should also help a lot. Adding rubber to the tips would probably also help prevent objects from slipping. Collecting & curating the data was the most time consuming & labor intensive part. Next, to improve low-level manipulation and make the system more real-time, I'm shifting to focus more on sims & synthetic data. This aligns better with my core competence. I'm open to tips and suggestions.show more

Shreyas Gite

2,223 subscribers

22,555 views • 1 year ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

5 Comments

Shreyas Gite1 year ago

Same pipeline as this one. The code is based on older LeRobot. We update soon to make compatible with the latest refactor.

Shreyas Gite1 year ago

Also have deep dive here:

atharva1 year ago

@LeRobotHF so. cool

00ri1 year ago

@LeRobotHF Nice work! How do the high level task outputs from gemini look?

Benedetti1 year ago

@LeRobotHF I will try to create my own version, it looks nice for a college project ❕

Related Videos

The pace at which Gemini 2.5 Pro is improving in code is wild! I generally leaned towards using Claude for code, but now I rely more on Gemini 2.5 Pro. It's just that good! Watch me use Gemini-2.5-Pro-05-06 to go from a YT video to an interactive slide deck app in minutes!

The pace at which Gemini 2.5 Pro is improving in code is wild! I generally leaned towards using Claude for code, but now I rely more on Gemini 2.5 Pro. It's just that good! Watch me use Gemini-2.5-Pro-05-06 to go from a YT video to an interactive slide deck app in minutes!

elvis

81,882 views • 1 year ago

Listen up 🔊 We’ve made some updates to our Gemini Audio models and capabilities: — Gemini’s live speech-to-speech translation capability is rolling out in a beta experience to the Google Translate app, bringing you real-time audio translation that captures the nuance of human speech — Gemini 2.5 Flash and 2.5 Pro Text-to-Speech preview models bring improved adherence to style prompts, precision pacing with context-aware speed adjustments, and character voice consistency for multi-speaker scenarios — Gemini 2.5 Flash Native Audio is now updated, with improvements to handle complex workflows, navigate user instructions, and hold natural conversations

Listen up 🔊 We’ve made some updates to our Gemini Audio models and capabilities: — Gemini’s live speech-to-speech translation capability is rolling out in a beta experience to the Google Translate app, bringing you real-time audio translation that captures the nuance of human speech — Gemini 2.5 Flash and 2.5 Pro Text-to-Speech preview models bring improved adherence to style prompts, precision pacing with context-aware speed adjustments, and character voice consistency for multi-speaker scenarios — Gemini 2.5 Flash Native Audio is now updated, with improvements to handle complex workflows, navigate user instructions, and hold natural conversations

Google AI

1,093,874 views • 8 months ago

Let’s talk about Gemini 3 Pro. There’s really no other way to say it. The benchmarks are incredible. The jump was dramatic – much more than most people expected. Anew agentic development platform named Antigravity was also launched today. Essentially, it’s a full VS Code fork, which will disappoint some – but with Gemini 3 Pro built directly into it. Now, it will take a few days for the developer community to determine whether it outperforms Cursor and some of the other IDEs. But the advantages that Antigravity has, is that it’s coupled with Gemini 3 Pro (SOTA) Gemini 2.5 Computer Use and Nano Banana for rich media generation. Excited to dive deeper into the Gemini 3 Pro launch. A LOT to unpack today.

Let’s talk about Gemini 3 Pro. There’s really no other way to say it. The benchmarks are incredible. The jump was dramatic – much more than most people expected. Anew agentic development platform named Antigravity was also launched today. Essentially, it’s a full VS Code fork, which will disappoint some – but with Gemini 3 Pro built directly into it. Now, it will take a few days for the developer community to determine whether it outperforms Cursor and some of the other IDEs. But the advantages that Antigravity has, is that it’s coupled with Gemini 3 Pro (SOTA) Gemini 2.5 Computer Use and Nano Banana for rich media generation. Excited to dive deeper into the Gemini 3 Pro launch. A LOT to unpack today.

Roberto Nickson

60,506 views • 9 months ago

Also starting today, Gemini Advanced users can create with Canvas on Gemini 2.5 Pro. This model’s advanced coding abilities make it easy for anyone to collaborate, create, and share imaginative prototypes (like this example of a 3D dino world, made in under 5 minutes with just a few prompts). Here’s how to get started: 1) Go to 2) Choose the “2.5 Pro (experimental)” model 3) Select “Canvas” and start prompting We’re working hard to make Gemini 2.5 Pro available for more people to try soon–watch this space for updates.

Also starting today, Gemini Advanced users can create with Canvas on Gemini 2.5 Pro. This model’s advanced coding abilities make it easy for anyone to collaborate, create, and share imaginative prototypes (like this example of a 3D dino world, made in under 5 minutes with just a few prompts). Here’s how to get started: 1) Go to 2) Choose the “2.5 Pro (experimental)” model 3) Select “Canvas” and start prompting We’re working hard to make Gemini 2.5 Pro available for more people to try soon–watch this space for updates.

Google Gemini

302,724 views • 1 year ago

Love this example! Congrats to Google DeepMind! I successfully replicated the Boggle game on SWE Arena. TLDR: Gemini 2.0 Pro won! In the case of Claude 3.5 Sonnet vs Gemini 2.0 Pro, both are given the image and prompt to generate the algo to get the results. The image and prompt are exactly the same. Try it out on SWE Arena: BTW can we get a bigger streaming rate limit for our API, so that ppl can better compare the latest Gemini models with other frontier models? Jeff Dean Logan Kilpatrick 👀

Love this example! Congrats to Google DeepMind! I successfully replicated the Boggle game on SWE Arena. TLDR: Gemini 2.0 Pro won! In the case of Claude 3.5 Sonnet vs Gemini 2.0 Pro, both are given the image and prompt to generate the algo to get the results. The image and prompt are exactly the same. Try it out on SWE Arena: BTW can we get a bigger streaming rate limit for our API, so that ppl can better compare the latest Gemini models with other frontier models? Jeff Dean Logan Kilpatrick 👀

Terry Yue Zhuo

48,220 views • 1 year ago

I was so impressed with the Astra demo at Google I/O yesterday that I decided to build my own version using Gemini 1.5 Pro Flash. It's so fast and really good. ⚡️ It was even able to detect the gate! Content is streamed directly from my camera. Voice via

I was so impressed with the Astra demo at Google I/O yesterday that I decided to build my own version using Gemini 1.5 Pro Flash. It's so fast and really good. ⚡️ It was even able to detect the gate! Content is streamed directly from my camera. Voice via

Pietro Schirano

107,742 views • 2 years ago

For developers, Gemini 3 Pro offers unparalleled results across every major AI benchmark compared to Gemini 2.5 Pro. Here three ways Gemini 3 Pro can improve your workflow today: 1. Gemini 3 Pro fits right into existing production agent and coding workflows. It is also integrated into development environment experiences across the ecosystem of developer products like Google AI Studio, Gemini CLI, and Android Studio. 2. We introduced Google Google Antigravity, a new agentic development platform that reimagines how the model and IDE work together. It enables developers to manage agents to work at a higher, task-oriented level collaborating with agents while they build. 3. We’ve built Google AI Studio to be your fastest path from a prompt to an AI-native app. Build mode lets you add AI capabilities faster than ever, automatically wiring up the right models and APIs, while features like annotations enable fast and intuitive iteration. Whether you’re building a one-shot prompt game or an interactive landing page from unstructured voice notes, you can bring any idea to life with Gemini 3.

For developers, Gemini 3 Pro offers unparalleled results across every major AI benchmark compared to Gemini 2.5 Pro. Here three ways Gemini 3 Pro can improve your workflow today: 1. Gemini 3 Pro fits right into existing production agent and coding workflows. It is also integrated into development environment experiences across the ecosystem of developer products like Google AI Studio, Gemini CLI, and Android Studio. 2. We introduced Google Google Antigravity, a new agentic development platform that reimagines how the model and IDE work together. It enables developers to manage agents to work at a higher, task-oriented level collaborating with agents while they build. 3. We’ve built Google AI Studio to be your fastest path from a prompt to an AI-native app. Build mode lets you add AI capabilities faster than ever, automatically wiring up the right models and APIs, while features like annotations enable fast and intuitive iteration. Whether you’re building a one-shot prompt game or an interactive landing page from unstructured voice notes, you can bring any idea to life with Gemini 3.

Google AI

136,648 views • 9 months ago

Gemini 3 Pro is the best model in the world for multimodal understanding. One of its most exciting capabilities is document understanding and reasoning. This means you can convert information in any format and into the medium that works best for you. Gemini 3 also has leading multilingual capabilities, enabling it to process, reason and even capture cultural relevance across a variety of languages. For example, here Gemini 3 is translating handwritten recipes in Korean and English to build a digital family cookbook in different languages.

Gemini 3 Pro is the best model in the world for multimodal understanding. One of its most exciting capabilities is document understanding and reasoning. This means you can convert information in any format and into the medium that works best for you. Gemini 3 also has leading multilingual capabilities, enabling it to process, reason and even capture cultural relevance across a variety of languages. For example, here Gemini 3 is translating handwritten recipes in Korean and English to build a digital family cookbook in different languages.

Google AI

36,696 views • 9 months ago

I love Hermes Agent and Gemini Live is also one of my for voice conversation, so over the weekend I built a small vibe-coding project that brings the two together called IRIS. The idea is simple: Gemini Live becomes the conversational interface, while Hermes Agent handles execution, automation, and long-running tasks behind the scenes. I can talk to Gemini naturally, and whenever something needs real work, Gemini formulates the request and hands it off to Hermes. The best part is that the conversation never stops. Gemini keeps talking with me naturally while Hermes works in the background. When Hermes finishes, Gemini proactively tells me the results are ready, switches context to the Hermes output, helps me review it, and then we return to our normal conversation. That manager/worker flow feels really good. Because I wanted the app to be as hands-free as possible, I also added a camera-based gesture layer using MediaPipe Gesture Recognizer. I can point at Hermes task cards, open details, scroll through results with my hand, and close them using gestures. Small weekend project, but I had a lot of fun building it. Also, the anti-flicker macOS app I vibe-coded yesterday ended up being incredibly useful for this project. It fixed my webcam flickering issue, so I can finally record videos properly. 🙂

I love Hermes Agent and Gemini Live is also one of my for voice conversation, so over the weekend I built a small vibe-coding project that brings the two together called IRIS. The idea is simple: Gemini Live becomes the conversational interface, while Hermes Agent handles execution, automation, and long-running tasks behind the scenes. I can talk to Gemini naturally, and whenever something needs real work, Gemini formulates the request and hands it off to Hermes. The best part is that the conversation never stops. Gemini keeps talking with me naturally while Hermes works in the background. When Hermes finishes, Gemini proactively tells me the results are ready, switches context to the Hermes output, helps me review it, and then we return to our normal conversation. That manager/worker flow feels really good. Because I wanted the app to be as hands-free as possible, I also added a camera-based gesture layer using MediaPipe Gesture Recognizer. I can point at Hermes task cards, open details, scroll through results with my hand, and close them using gestures. Small weekend project, but I had a lot of fun building it. Also, the anti-flicker macOS app I vibe-coded yesterday ended up being incredibly useful for this project. It fixed my webcam flickering issue, so I can finally record videos properly. 🙂

AshutoshShrivastava

149,739 views • 2 months ago

Got an output from the strongest pre-release Gemini 3.5 Pro checkpoint 👀 I compared it to Claude Fable 5 and GPT-5.5 on the same exact prompt Yeah... This model is mostly dead on arrival (tried it on other prompts too ofc). It will likely be the "poor man's model" next to 3.5 flash. 1st video: Gemini 3.5 Pro (pre-release) 2nd video: GPT-5.5 3rd video: Claude Fable 5

Got an output from the strongest pre-release Gemini 3.5 Pro checkpoint 👀 I compared it to Claude Fable 5 and GPT-5.5 on the same exact prompt Yeah... This model is mostly dead on arrival (tried it on other prompts too ofc). It will likely be the "poor man's model" next to 3.5 flash. 1st video: Gemini 3.5 Pro (pre-release) 2nd video: GPT-5.5 3rd video: Claude Fable 5

Lentils

50,395 views • 2 months ago

Video Game Bench introduce a research preview of VideoGameBench, a benchmark which challenges vision-language models to complete, in real-time, a suite of 20 different popular video games from both hand-held consoles and PC GPT-4o, Claude Sonnet 3.7, Gemini 2.5 Pro, and Gemini 2.0 Flash playing Doom II (default difficulty) on VideoGameBench-Lite with the same input prompt! Models achieve varying levels of success but none are able to pass even the first level.

Video Game Bench introduce a research preview of VideoGameBench, a benchmark which challenges vision-language models to complete, in real-time, a suite of 20 different popular video games from both hand-held consoles and PC GPT-4o, Claude Sonnet 3.7, Gemini 2.5 Pro, and Gemini 2.0 Flash playing Doom II (default difficulty) on VideoGameBench-Lite with the same input prompt! Models achieve varying levels of success but none are able to pass even the first level.

AK

154,221 views • 1 year ago

Really happy to see the interest around our “Hands-on with Gemini” video. In our developer blog yesterday, we broke down how Gemini was used to create it. We gave Gemini sequences of different modalities — image and text in this case — and had it respond by predicting what might come next. Devs can try similar things when access to Pro opens on 12/13 🚀. The knitting demo used Ultra⚡ All the user prompts and outputs in the video are real, shortened for brevity. The video illustrates what the multimodal user experiences built with Gemini could look like. We made it to inspire developers. When you’re building an app, you can get similar results (there’s always some variability with LLMs) by prompting Gemini with an instruction that allows the user to "configure" the behavior of the model, like inputting “you are an expert in science …” before a user can engage in the same kind of back and forth dialogue. Here’s a clip of what this looks like in AI Studio with Gemini Pro. We’ve come a long way since Flamingo 🦩 & PALI, looking forward to seeing what people build with it!

Really happy to see the interest around our “Hands-on with Gemini” video. In our developer blog yesterday, we broke down how Gemini was used to create it. We gave Gemini sequences of different modalities — image and text in this case — and had it respond by predicting what might come next. Devs can try similar things when access to Pro opens on 12/13 🚀. The knitting demo used Ultra⚡ All the user prompts and outputs in the video are real, shortened for brevity. The video illustrates what the multimodal user experiences built with Gemini could look like. We made it to inspire developers. When you’re building an app, you can get similar results (there’s always some variability with LLMs) by prompting Gemini with an instruction that allows the user to "configure" the behavior of the model, like inputting “you are an expert in science …” before a user can engage in the same kind of back and forth dialogue. Here’s a clip of what this looks like in AI Studio with Gemini Pro. We’ve come a long way since Flamingo 🦩 & PALI, looking forward to seeing what people build with it!

Oriol Vinyals

180,991 views • 2 years ago

With this new release, Gemini applies advanced reasoning to make sense of complex, changing information in real-time. 3.1 Pro can utilize advanced reasoning to bridge the gap between complex APIs and user-friendly design. For example, Gemini built a live aerospace dashboard, taking live telemetry from a public API and designing a front end to track the Space Station's orbit.

With this new release, Gemini applies advanced reasoning to make sense of complex, changing information in real-time. 3.1 Pro can utilize advanced reasoning to bridge the gap between complex APIs and user-friendly design. For example, Gemini built a live aerospace dashboard, taking live telemetry from a public API and designing a front end to track the Space Station's orbit.

Google Gemini

106,360 views • 6 months ago

You can now try Gemini 3 Pro in Figma Make! What stands out to me from early testing is how useful Gemini 3 is for exploring visual directions + ideas that designers can then shape and push further with their craft. My thoughts (and some 🤯 demos) in the video below...

You can now try Gemini 3 Pro in Figma Make! What stands out to me from early testing is how useful Gemini 3 is for exploring visual directions + ideas that designers can then shape and push further with their craft. My thoughts (and some 🤯 demos) in the video below...

Dylan Field

116,294 views • 9 months ago

Sergey has a new habit. He talks to Gemini Live while driving, discussing things like data center power and cost. It’s classic Google dogfooding — obsessively testing your own product. It reminds me of Bill Gates removing his car radio so he could think about Microsoft nonstop. Every founder should have this level of obsession. Sergey also says the Gemini model he’s using in the car is way better than what’s available now. My guess is that Gemini 3 Flash is coming soon!

Sergey has a new habit. He talks to Gemini Live while driving, discussing things like data center power and cost. It’s classic Google dogfooding — obsessively testing your own product. It reminds me of Bill Gates removing his car radio so he could think about Microsoft nonstop. Every founder should have this level of obsession. Sergey also says the Gemini model he’s using in the car is way better than what’s available now. My guess is that Gemini 3 Flash is coming soon!

Yuchen Jin

652,835 views • 8 months ago

Gemini 2.5 Flash Image (Nano Banana) and the collage method + generating new camera angles and then feeding that to Kling 2.1 start and end frames + Veo 3 is becoming my favorite combo. I just created this for fun to test the workflow. I've included some additional insight in this thread if you want to see how it was brought together 👍

Gemini 2.5 Flash Image (Nano Banana) and the collage method + generating new camera angles and then feeding that to Kling 2.1 start and end frames + Veo 3 is becoming my favorite combo. I just created this for fun to test the workflow. I've included some additional insight in this thread if you want to see how it was brought together 👍

Travis Davids

140,428 views • 1 year ago

Playing Pokemon with LLMs became a benchmark, but what about Super Mario? Turns out Google DeepMind Gemini 2.0 Flash can play Super Mario in real-time due to its fast latency, multimodal input and long context! 🎮 Gemini 2.0 Flash receives screenshots of the game and then generates Python code (PyAutoGUI commands) for each screenshot to control the game for a short period (the next 1 or 2 seconds). Add a while loop to it and you have it play in real-time. 🤯

Playing Pokemon with LLMs became a benchmark, but what about Super Mario? Turns out Google DeepMind Gemini 2.0 Flash can play Super Mario in real-time due to its fast latency, multimodal input and long context! 🎮 Gemini 2.0 Flash receives screenshots of the game and then generates Python code (PyAutoGUI commands) for each screenshot to control the game for a short period (the next 1 or 2 seconds). Add a while loop to it and you have it play in real-time. 🤯

Philipp Schmid

153,086 views • 1 year ago

We put the new Gemini 3.6 Flash from Google DeepMind to work in Box Agent on a pharmaceutical manufacturing deviation. It reconciled a draft investigation using an obsolete SOP with minute-by-minute historian data, equipment audit logs, maintenance records, an executed batch record, and a prior CAPA. Applying the current SOP, Gemini uncovered a second excursion and calculated 147 cumulative minutes against a 120-minute limit. It traced the likely control failure to an unrestored equipment setpoint, then created a cited investigation review for the Quality Unit. Customers can now choose Gemini 3.6 Flash from in Box AI Studio to build custom agents grounded in their enterprise content.

We put the new Gemini 3.6 Flash from Google DeepMind to work in Box Agent on a pharmaceutical manufacturing deviation. It reconciled a draft investigation using an obsolete SOP with minute-by-minute historian data, equipment audit logs, maintenance records, an executed batch record, and a prior CAPA. Applying the current SOP, Gemini uncovered a second excursion and calculated 147 cumulative minutes against a 120-minute limit. It traced the likely control failure to an unrestored equipment setpoint, then created a cited investigation review for the Quality Unit. Customers can now choose Gemini 3.6 Flash from in Box AI Studio to build custom agents grounded in their enterprise content.

Box

348,850 views • 1 month ago