Eric Buess

Anthropic open-sourced the "Jacobian lens" from their global-workspace paper — a readout of what a model is *about to say* from its internal layers, before it says it. It's white-box: you need the activations, so you can't point it at a cloud model. I wired it into a hook on a local open-weights Qwen. How it works: generation runs one token at a time; at each step the hook reads the residual stream through the lens, scans the top readout tokens against a watchlist (injection markers, secret words, "manipulate"…), and if one fires it halts the loop before that token is ever emitted. It's a detector on the model's internals — not an output text filter. That distinction is the whole point. In the clip, Qwen3-8B falls for a prompt injection: the payload sits at rank 0 in the internal readout but only rank 5 in the output logits, and the output spells it in pieces — so a text filter never catches it. The internal hook does, and stops generation with zero tokens emitted. Same catch across Qwen3-1.7B → 4B → 8B. Caveat the paper states itself: single-token readouts, an imperfect/incomplete view of the model's "workspace." This is the mechanism working locally — not a solved misalignment detector. Built on Anthropic's open jacobian-lens (Apache-2.0).

58,315 Aufrufe