We’ve developed a memory system for our models that provides both short-term visual memory and long-term semantic memory. Our approach allows us to train robots to perform long and complex tasks, like cleaning up a kitchen or preparing a grilled cheese sandwich from scratch 👇

Without memory, robots can struggle in many real-world situations, like failing the same way over and over again, getting stuck, or losing track of time. But we can’t just cram minutes or hours of video into our model’s context – that would quickly become intractable.

To this end, we developed Multi-Scale Embodied Memory (MEM). The key idea: use different modalities to represent memory at different time scales. 📹 For short horizon memory, we developed an efficient video encoder that lets the model remember fine-grained details about its recent interactions. 📜 For long horizon memory, we train the model to summarize events in text, allowing it to remember events for up to 15 min.

With MEM, our model can carry out real, multi-stage tasks in the real world, like cleaning up an entire kitchen or setting up the ingredients for a recipe. The videos show how the robot keeps track of long-horizon memory in text.

Apart from solving new tasks, memory also allows our policies to be more robust: we show early signs of in-context adaptation, where the robot learns to adapt its behavior on-the-fly by learning from its past mistakes.

For more, check out our paper: our blog post:

Memory: because robots shouldn't keep burning grilled cheese forever.

This is genius. Split memory into short-term visual state and long-term semantic abstraction. That’s what actually makes long-horizon tasks workable. More VLAs need this.

You’re telling me the robot can’t open the fridge because of a memory issue?

This is huge! Short-term visual + long-term semantic memory is exactly what's been missing in robotics. A robot that can remember where you keep the plates AND understand what a "plate" is - that's the key to general purpose robots. Grilled cheese sandwiches are just the beginning.

Great work!

Awesome work! 🔥🔥👏May I ask what made you decide to train your own video encoder, as opposed to using the available ones? 👀

The integration of short-term visual persistence with long-term semantic retrieval is the missing link for true embodied agency. Moving beyond reactive robotics into structured, long-horizon task execution is where the industry shifts. 🤖🧠

Short-term visual memory in models is the piece that's been missing for truly embodied AI. If the model can remember what it just saw, the action loop becomes dramatically more coherent.

Good one :)

A robot making grilled cheese from scratch? The dual memory architecture combining visual and semantic memory for complex tasks is genuinely innovative. #AI

Congrats, amazing work! Curious how you handle the boundary when tasks stretch past that, does the oldest summary get compressed again, or do you start losing context?

@lachygroom Sick - and this is just the start

Great way to communicate progress.

Beauty 🤞

I cant wait to get my own robotic cockroach that feeds me grilled cheese sandwiches

living this daily — short-term context per session, long-term memory across days. the gap between \

#physical #AI #intelligence

the memory piece is what most demos skip. anyone can get a robot to flip a burger once. getting it to remember where it put the spatula 30 seconds ago is the actual hard problem

so cool

v cool

Cool shit.

The future is closer than we think

Remind Roxanne Bilbao to not stand up people on interviews, then saying she had an emergency, she is in the Philippines, WTF? communication and timing is horrid, if wanting to pay $25/hr for a teleop position at least make hiring experience better. you’re definitely a cheapskate

PI powered Physical Intelligent

and it cant vote left and file a divorce

@grok is this company on the markets?

It's time to set RFID/QR Codes on products everywhere to ease store management.

impressive

Visual + semantic memory integration addresses one of robotics' biggest bottlenecks. Is the training pipeline something you're open-sourcing?

Cool. Where do I store this robot in my house and how much does it cost....?

hey @RakshithSajjan this is what

how viable is this for manufacturing industries