Combining real-time interactivity, task understanding, and full-body action prediction on a humanoid…
@_anniechen_1.4K followers
41,907 views
12 Comments
Comments from the original post on X (Twitter)

@_anniechen_
Under the hood, this requires: - Action model that coordinates whole-body movement and fine motor skills simultaneously, while following instructions and recovering on-the-fly - Embodied reasoning (ER) agent with SOTA video understanding, long-horizon reasoning about user intent, and interruption handling
8 likes

@_anniechen_
This was a huge team effort @GoogleDeepMind! hard to fathom the amount of work that goes into making full systems like this, but it's incredibly rewarding when things come together, and it's so fun to see how responsive the models are :)
5 likes

@jhoer100
this brings me back to when you interrupted my toy cleanup session in toyon and i started crashing out. huge moment.
2 likes

@htripathi06
@DvijKalaria
1 like

@cortexist
The response feels too slow. If you begin streaming tokens as soon as you start generating, and flush the first speakable clause as early as possible, the user experience would improve significantly—even though it has no real impact on the underlying task.
1 like

@alfie14x
Apollo never hesitated, great!
1 like

@TimoS163822
i want Apollo 🥰

@BlackEyedHatMan
You sure do look nervous... suppose, it's normal with beta versions 👉 keep us updated! 🍀

@xpguru3dprint
All three on a humanoid in real-time? No joke.

@zainshah_x
@StanfordAILab Yeah! But the passion to make it happen is the thing that move people forward

@dsandzhiev
The interactivity part is what stands out. Most demos show a robot executing, this one shows it adjusting to a person mid-task. Curious how much of that came from data with humans actually in the loop, versus scripted rollouts.

@USArmyPhoenix
I think having a human test subject do the same task, and while they are performing the task, ask them to describe why, and how, they are making the decisions, and the movements, would help go a long way for training AI/Robotics. The more detailed the human subject gets and explains their through process helps train the AI/Robot to do the same thing. AI/Robotics need to know more than how to do something, they need to know why as well. There are many ways to perform a task, first train them on the different approaches with extractable expirations, then focus optimization through efficiency and effectiveness. The why is a very crucial part of learning.