🦍 Introducing Memorilla: memory as a modality for LLMs Memorilla turns everything a frozen LLM needs to remember into just 16 memory embeddings ⚡ It uses up to 4,212× fewer tokens and is 27× faster than full context inference while more accurate 🎯 Pushes the Pareto frontier over RAG... and agentic memory in personalization, long-document and knowledge QA A thread 🧵:show more

A frozen encoder embeds each document once into a persistent store. Conditioned on the question, 16 memory slots read the store and produce embeddings fed directly into the LLM, like image embeddings. The LLM stays frozen. Only the memory module is trained.

We tested it on personalization, long-document and knowledge QA against full context, RAG, agentic memory and compression methods. Memorilla sits on the Pareto frontier of all 8 benchmarks and leads on 5.

Memorilla also works as an agent's memory. In TextWorld, every turn becomes a new memory, and RL on the game reward trains only the memory module. Compared to in-context learning alone, the same frozen agent solves 24.2% of games instead of 18.6%.

"What are some suitable foods for me to boost my energy during the day?" RAG suggests oats and brown rice. Memorilla remembers the user's iron-deficiency anemia from an old chat, even though that chat has little semantic connection to the question. On PersonaMem-v2 it gets 44.6% vs 30.9% for the next best method.

It's also better at connecting facts. On FactKG, RAG-50 drops from 79% to 52% as a claim combines more facts. Memorilla stays between 80% and 89%.

Memorilla and RAG get different questions right. Their union covers 94.3% of FactKG, vs 84.3% for Memorilla and 78.4% for RAG-50 alone.

Latency stays flat as memory grows. With 512 documents on one H100, Memorilla answers in 0.74 s and full context takes 19.9 s.

It of course has limitations Memorilla can lose fine-grained information, so RAG still wins "who" and "which" questions on NarrativeQA. The memory module is also tied to the LLM it was trained with, and moving to another LLM or a new domain takes some fine-tuning.

Humans have both episodic and semantic memory. If RAG is episodic, Memorilla is semantic. Just like us, LLMs need both. Memorilla isn't perfect, but we see it as a 0→1 change in how LLMs use memory, and we look forward to seeing where the community takes these ideas.

Big thank you the entire amazing team! @Radii2323 @kvignesh1420 @ShayanPardis @nikitamounier @ShirleyYXWu @jure @Stanford Website: Paper: Code: HuggingFace:

@jure @Stanford @kvignesh1420 @Radii2323 @ShirleyYXWu @nikitamounier @ShayanPardis nice to see this out !

@jure @Stanford @kvignesh1420 @Radii2323 @ShirleyYXWu @nikitamounier @ShayanPardis thank you for the great support ben!

@jure @Stanford @kvignesh1420 @Radii2323 @ShirleyYXWu @nikitamounier @ShayanPardis Very sick, ultra cool, congrats!

@jure @Stanford @kvignesh1420 @Radii2323 @ShirleyYXWu @nikitamounier @ShayanPardis Thank you!

@jure @Stanford @kvignesh1420 @Radii2323 @ShirleyYXWu @nikitamounier @ShayanPardis Adib you legend

@jure @Stanford @kvignesh1420 @Radii2323 @ShirleyYXWu @nikitamounier @ShayanPardis 🔥

@jure @Stanford @kvignesh1420 @Radii2323 @ShirleyYXWu @nikitamounier @ShayanPardis sick!

@jure @Stanford @kvignesh1420 @Radii2323 @ShirleyYXWu @nikitamounier @ShayanPardis 🔥

@jure @Stanford @kvignesh1420 @Radii2323 @ShirleyYXWu @nikitamounier @ShayanPardis Cool! In what ways would you adapt Memorilla for biomedical reasoning? [like a bio-specific encoder]

@jure @Stanford @kvignesh1420 @Radii2323 @ShirleyYXWu @nikitamounier @ShayanPardis Haha that is BioReason territory now

@jure @Stanford @kvignesh1420 @Radii2323 @ShirleyYXWu @nikitamounier @ShayanPardis awesome work, congrats!

@jure @Stanford @kvignesh1420 @Radii2323 @ShirleyYXWu @nikitamounier @ShayanPardis thank you!

@jure @Stanford @kvignesh1420 @Radii2323 @ShirleyYXWu @nikitamounier @ShayanPardis Hype!!

@jure @Stanford @kvignesh1420 @Radii2323 @ShirleyYXWu @nikitamounier @ShayanPardis 😀

@jure @Stanford @kvignesh1420 @Radii2323 @ShirleyYXWu @nikitamounier @ShayanPardis holf f***

@jure @Stanford @kvignesh1420 @Radii2323 @ShirleyYXWu @nikitamounier @ShayanPardis indeed

@jure @Stanford @kvignesh1420 @Radii2323 @ShirleyYXWu @nikitamounier @ShayanPardis This is really cool

@jure @Stanford @kvignesh1420 @Radii2323 @ShirleyYXWu @nikitamounier @ShayanPardis Thank you 😀

@jure @Stanford @kvignesh1420 @Radii2323 @ShirleyYXWu @nikitamounier @ShayanPardis 16 slots is a clean compression. what happens when two memories contradict and the newer one is the true one? that update path is what breaks agent memory in production, and it's rarely measured.

@jure @Stanford @kvignesh1420 @Radii2323 @ShirleyYXWu @nikitamounier @ShayanPardis this is so cool!

@jure @Stanford @kvignesh1420 @Radii2323 @ShirleyYXWu @nikitamounier @ShayanPardis 😀

@jure @Stanford @kvignesh1420 @Radii2323 @ShirleyYXWu @nikitamounier @ShayanPardis 16 embeddings for everything is wild compression. What happens with exact recall, like a specific number or ID from 50 turns ago? Feels like soft memory is great for gist but loses precise facts. Do you pair it with retrieval for those?

@jure @Stanford @kvignesh1420 @Radii2323 @ShirleyYXWu @nikitamounier @ShayanPardis This is pretty cool

@jure @Stanford @kvignesh1420 @Radii2323 @ShirleyYXWu @nikitamounier @ShayanPardis very cool work!

@jure @Stanford @kvignesh1420 @Radii2323 @ShirleyYXWu @nikitamounier @ShayanPardis yoo this is amazing, congrats!!

@jure @Stanford @kvignesh1420 @Radii2323 @ShirleyYXWu @nikitamounier @ShayanPardis thank you Hamza!

@jure @Stanford @kvignesh1420 @Radii2323 @ShirleyYXWu @nikitamounier @ShayanPardis this is amazing !!

@jure @Stanford @kvignesh1420 @Radii2323 @ShirleyYXWu @nikitamounier @ShayanPardis thank you mersad!

@jure @Stanford @kvignesh1420 @Radii2323 @ShirleyYXWu @nikitamounier @ShayanPardis Compressing down to 16 soft tokens usually shifts the compute burden to an encoder step. Does updating the memory require reprocessing the whole context from scratch?

@jure @Stanford @kvignesh1420 @Radii2323 @ShirleyYXWu @nikitamounier @ShayanPardis v cool work Adib!

@jure @Stanford @kvignesh1420 @Radii2323 @ShirleyYXWu @nikitamounier @ShayanPardis thank you!

@jure @Stanford @kvignesh1420 @Radii2323 @ShirleyYXWu @nikitamounier @ShayanPardis super cool work!

@jure @Stanford @kvignesh1420 @Radii2323 @ShirleyYXWu @nikitamounier @ShayanPardis thank you!

@jure @Stanford @kvignesh1420 @Radii2323 @ShirleyYXWu @nikitamounier @ShayanPardis 27x faster is the number i'd care about most. latency kills more products than accuracy does

@jure @Stanford @kvignesh1420 @Radii2323 @ShirleyYXWu @nikitamounier @ShayanPardis Saw the website a few days ago, looks great. Nice to see you working also on non-bio stuff!

@jure @Stanford @kvignesh1420 @Radii2323 @ShirleyYXWu @nikitamounier @ShayanPardis side quest!

@jure @Stanford @kvignesh1420 @Radii2323 @ShirleyYXWu @nikitamounier @ShayanPardis RETRO but much more deployment friendly Great work!

@jure @Stanford @kvignesh1420 @Radii2323 @ShirleyYXWu @nikitamounier @ShayanPardis 16 memory embeddings replacing a huge context window is a clean reduction - curious if lookup stability holds across different base checkpoints

@jure @Stanford @kvignesh1420 @Radii2323 @ShirleyYXWu @nikitamounier @ShayanPardis 16 embeddings replacing full context is wild if it holds on long messy inputs. curious how it degrades when the memory has to carry conflicting facts.

@jure @Stanford @kvignesh1420 @Radii2323 @ShirleyYXWu @nikitamounier @ShayanPardis

@jure @Stanford @kvignesh1420 @Radii2323 @ShirleyYXWu @nikitamounier @ShayanPardis is this speedup using precomputed document embeddings or the actual online speed up ❓