🤔 Why do we still rely on the final layer of an LLM, when different layers encode different information? 🤔 In our new work, “Improving LLM Final Representations with Inter-Layer Geometry” (ICLR 2026 Workshop on Geometry-grounded Representation Learning and Generative Modeling) we show that actually, LLMs do not have... one “best” layer. We introduce the Cayley-Encoder: an efficient and effective geometric encoder that learns one strong representation from all layer representations of the LLM, without biasing the representation toward any specific layer. While adding at most 0.1% learned parameters to the LLM, the Cayley-Encoder achieves large empirical gains over LoRA fine-tuning, final-layer representations, expensive attention-based aggregation, and methods that optimize specific layers for the task.show more

The Cayley-Encoder builds on two previous works work: (1) “Graph Neural Networks Use Graphs When They Shouldn’t,” where we showed that GNNs tend to overfit graph structures and that regular graphs are robust to this overfitting; and (2) “Cayley-Graph Propagation,” with @PetarV_93 and @sonjj74 , where we showed how to leverage Cayley Graphs to break computational bottlenecks in GNNs. Together, these make Cayley Graphs a natural candidate for sparse, efficient, and robust communication between layer representations. Congrats to Tom Ulanovski and Eyal Blyachman for leading this great work!

The core idea of the Cayley-Encoder is to let all layers communicate through an efficient sparse symmetric structure that is robust to structural overfitting. We do so by introducing a Cayley graph of the Special Linear Group over the LLM’s layer representations and learning over it with a small Graph Neural Network. This enables efficient information flow across the full depth of the model, without overfitting to arbitrary layer-interaction patterns or biasing the representation toward any single layer.

Improving LLM Final Representations with Inter-Layer Geometry:

super solid work. 🤝 just gave you a follow :)

Really cool! This seems to be very similar to the idea of jumping knowledge networks for GNNs (but done in a clever way).

What about the computational overhead of training and maintaining this multi-layer representation?

Your welcome.

Metacognition is the solution for consciousness of any kind I like how your works complement each other