Skip navigation
Combining real-time interactivity, task understanding, and full-body action prediction on a humanoid…
0:00 / 1:07

Combining real-time interactivity, task understanding, and full-body action prediction on a humanoid…

Annie Chen

@_anniechen_1.4K followers

Visit on X
244 likesDownloadOriginal post

41,907 views

Combining real-time interactivity, task understanding, and full-body action prediction on a humanoid is so, so hard. Here's an example where we bring all of these together in Gemini Robotics 2 🤖🧠

12 Comments

Comments from the original post on X (Twitter)

  • Annie Chen's profile picture

    @_anniechen_

    Under the hood, this requires: - Action model that coordinates whole-body movement and fine motor skills simultaneously, while following instructions and recovering on-the-fly - Embodied reasoning (ER) agent with SOTA video understanding, long-horizon reasoning about user intent, and interruption handling

    8 likes

  • Annie Chen's profile picture

    @_anniechen_

    This was a huge team effort @GoogleDeepMind! hard to fathom the amount of work that goes into making full systems like this, but it's incredibly rewarding when things come together, and it's so fun to see how responsive the models are :)

    5 likes

  • Jerry Hong's profile picture

    @jhoer100

    this brings me back to when you interrupted my toy cleanup session in toyon and i started crashing out. huge moment.

    2 likes

  • Harsh Tripathi's profile picture

    @htripathi06

    @DvijKalaria

    1 like

  • Cortexist's profile picture

    @cortexist

    The response feels too slow. If you begin streaming tokens as soon as you start generating, and flush the first speakable clause as early as possible, the user experience would improve significantly—even though it has no real impact on the underlying task.

    1 like

  • Alfred Cueva's profile picture

    @alfie14x

    Apollo never hesitated, great!

    1 like

  • T.'s profile picture

    @TimoS163822

    i want Apollo 🥰

  • Black Eyed Hat Man's profile picture

    @BlackEyedHatMan

    You sure do look nervous... suppose, it's normal with beta versions 👉 keep us updated! 🍀

  • Extruder Pro's profile picture

    @xpguru3dprint

    All three on a humanoid in real-time? No joke.

  • Zain Shah's profile picture

    @zainshah_x

    @StanfordAILab Yeah! But the passion to make it happen is the thing that move people forward

  • Dmitriy Sandzhiev's profile picture

    @dsandzhiev

    The interactivity part is what stands out. Most demos show a robot executing, this one shows it adjusting to a person mid-task. Curious how much of that came from data with humans actually in the loop, versus scripted rollouts.

  • Mr. House's profile picture

    @USArmyPhoenix

    I think having a human test subject do the same task, and while they are performing the task, ask them to describe why, and how, they are making the decisions, and the movements, would help go a long way for training AI/Robotics. The more detailed the human subject gets and explains their through process helps train the AI/Robot to do the same thing. AI/Robotics need to know more than how to do something, they need to know why as well. There are many ways to perform a task, first train them on the different approaches with extractable expirations, then focus optimization through efficiency and effectiveness. The why is a very crucial part of learning.