What should a robot remember? Can it learn to remember better than we tell it to? Excited to share MEMI: Reverse-Engineering What Robots Should Remember 📈 97.0% end-to-end, 99.4% w/ low-level controller on RoboMME, nearly solving it for the first time 🧵👇

Why reverse-engineer memory? Demos record what a robot did, never what it remembered. Just like the rewards behind demos in IRL, memory is inherently hidden We thus cast memory learning as an inverse problem: infer memory from decisions

But this inverse problem is underdetermined: many memories explain the same actions So we pin it down with three memory principles: ✅ Sufficient: reproduces the decisions 🔁 Predictable: from the past only, no peeking ahead ✂️ Necessary: no detail the policy ignores

MEMI uses a reasoning model in two roles -- annotator & policy -- self-improving in 3 steps ♻️ 1. Annotator infers memory from demonstrations 2. Policy uses memory to act 3. Three principles guide feedback → refine both Fully offline, without prescribed memory labels

On RoboMME's 16 memory-dependent tasks: 🥇 Astra-Plan: 92.4 → 99.4%, nearly solving it 🥈 Astra-Act: 88.3 → 97.0%, new end-to-end SOTA Gains hold across all 4 reasoning models. Even cost-efficient Claude Haiku climbs 73.9 → 90.8%, reaching human-level performance

Really enjoyed this sprint w/ @sooyoung_ryu @YinpeiD @du_yilun @chelseabfinn Code, prompts, and logs are all released! 🚀 🌐 Blog: 💻 Code: More coming soon, stay tuned!

Memory, not manipulation, is the real robotics bottleneck. At 97.0% end-to-end and 99.4% with a low-level controller, MEMI shows the next leap comes from what a robot retains, not just what it can move.

A robot that learns what to remember is a robot that stops needing you to babysit it — that's the real leap

Moving an object off camera is a good test of when to forget.

Great point! In fact, we have something related coming out very soon. Stay tuned!

A task-family split would be informative: optimize the memory prompts on one set of tasks, freeze them, then evaluate on unfamiliar tasks. That would help separate reusable memory strategies from task-specific hints.

StopCube 的例子很直观：只记次数和位置不够，还得保留能估计速度的历史。🧠 好奇换任务后，“必要性”约束会不会删掉后来有用的信息？这类迁移测试很值得看。

Great point! "Necessity" is like sparsity in neural networks. It's particularly valuable as the horizon grows for efficiency, but enforcing it too strictly might limit memory expressivity. We haven’t explored this trade-off in depth yet, but could be interesting to study!

谢谢澄清！把它看成稀疏性约束很有帮助。🧠 我会特别关注“先压缩记忆、再切换任务”的测试：固定记忆预算，比较不同约束强度下的迁移成功率，或许能看清效率与表达力的边界。

does it pick different stuff than people would by hand