👀Humans compare images by looking back and forth. Many open-weight VLMs encode each image independently, and defer comparison to the LM. We introduce SVE: Stateful Visual Encoders for Vision-Language Models, where the visual encoder itself becomes change-aware. 🌐Project: 📰Paper: 💻Code: 1/n

VLMs are increasingly used for interactive tasks that require comparative visual reasoning, yet their upstream visual encoders have largely remained stateless. Each image is encoded independently for dozens of layers before the LLM ever sees the visual features. We hypothesize that this creates a bottleneck for change-aware visual representations, so we make the visual encoder stateful. 2/n

We tested several ways to make visual encoders stateful: * extend past-image features into the current image’s K/V * use AdaLN-Zero-style feature modulation * add cross-attention layers over past features The strongest design adds a cross-attention layer with its own FFN. But optimization matters. For stability, we stop gradients to past features, initialize added attention/FFN layers from nearby pretrained layers, and zero-initialize the final matrix in each added block. This lets SVE start close to the original stateless model, then learn to use prior visual context. 3/n

Across input resolutions, model sizes, and VLM backbones, making the visual encoder stateful consistently improves over the stateless baseline. The gains are especially clear when the baseline is weaker, and they persist across Qwen3-VL, Qwen3.5, GLM, InternVL, and Gemma backbones. 4/n

SVE also improves real-world visual comparison. In longitudinal radiology, image editing, and remote sensing change detection, the stateless baseline often misses, flips, or hallucinates visual differences. With stateful visual encoding, the model better tracks what actually changed across the image pair. 5/n

This project was done during my time at @VoioInc. Huge thanks to @yujunwei04, @YalaTweets, @_dmchan, @profjoeyg, @trevordarrell, @berkeley_ai, and @BerkeleySky for making this project possible, and to everyone who gave feedback on the idea, experiments, writing, and figures. We hope SVE is a useful step toward VLMs that reason over visual change more directly. More details here: 📷 n/n

Very nice work! Our recent work might be of interest to you as well as your formulation of cross-attention state integration is similar to what we discovered as the best way to integrate temporal state across video frames:

Yes, I think this is very elegant and relevant! Thanks for sharing your work!! As we scale up to dozens of images (what we haven't done), keeping a fixed state than retrieving from all past images should def be the more efficient/scalable approach

Great work!

Very cool