Interesting approach to long context/continual learning here from Baseten at Cursor Compact long trajectories by compressing a prefix of the KV cache using an MLP/autoencoder. You can train this "compactor" MLP by learning to reconstruct activations that the original KV cache would produce on subsequent tokens. This maximally reconstructs... information from long context that's useful for subsequent outputs. If you run an agent over extremely long context and run this compaction recursively, the activations of this compressed KV cache become like trained weights. Similar to task-specific LoRA or "cartridges". Would not be surprised if OpenAI is running a similar algorithm for their blackbox compaction. Clear benefits here if you can avoid busting the cache, as e.g. compacting via writing to a text file would. Seeing many emerging approaches to fixed size latent states for LLMs that seem promising. If building a task-specific KV cache compression ends up being more sample efficient than running backprop, getting this to "work" feels like one of the 2-3 remaining breakthroughs on the path to true AGIshow more

@cursor_ai

@cursor_ai Go @mudithj! 🚀

@baseten @cursor_ai Long context yeah, but not continual learning.

@baseten @cursor_ai what, to you, is the difference between these

I might be using “continual learning” more narrowly. To me long context is still basically memory during an episode, even if it’s compressed KV cache. Continual learning would mean the system actually changes from experience in a durable way, so future runs are different after the context is gone. This feels like a useful substrate for that, but not quite the thing itself.

@baseten @cursor_ai Both approaches attempt to use information gleaned from past tokens to refine future tokens. This is the essence of learning. They are only distinct insofar as they are different implementation methods with associated resource constraints and performance characteristics.

@baseten @cursor_ai Compressing the KV cache into a trained latent state is the right direction. The open question is whether that compactor stays stable when the task distribution shifts or whether you just trained a very expensive task-specific cache.

@baseten @cursor_ai That is the interesting target: preserve useful KV state without rebuilding the prompt prefix. On constrained GPUs, cache layout is already the practical limiter long before model weights are

@baseten @cursor_ai Training is studying before the exam. Using past tokens is checking the last few lines while writing the next answer. Similar idea, very different bill.

@baseten @cursor_ai wdym it if gets sample efficient? Cartridge training is already more sample efficient than LoRA or RL, just has to be compressed enough to allow for a "normal" context window.

@baseten @cursor_ai 这个思路实操门槛不低。压缩器训练要模型内部 KV cache，API 用户拿不到。如果 1 个月内有人把它跑在 llama.cpp 上开源，agent 长上下文成本能降一个数量级。