We've been tinkering with protein structure design. Our (new?) method works like a language model, but instead of a discrete token, it samples the placement of the next amino acid backbone from a continuous distribution. Meet the Generative Invariant Angle Transformer (GIAT):

The lines in that video ↑ show which amino acid the current one is attending to when sampling the next. The attention heads are sometimes interpretable too. Here is a head (green) that looks only at the opposing antiparallel beta sheet, slightly ahead of the current AA:

How does this all work? The core of the model looks (mostly) like a transformer, with some tweaks. We use Invariant Point Attention (from AlphaFold2), with two modifications: we added a causal mask to make the whole thing autoregressive, and we use Softmax1 (@EvMill).

But transformers work with discrete tokens? Not necessarily! We need to place the next amino acid, and to do this we generate dihedral angles Ψ, Ω, and Φ. But you can't just select the "best" dihedrals (or else you only make α helices - @KevinKaichuang ?)

You have to sample from a *distribution* over dihedrals, just as GPT samples from a distribution over tokens. Enter the "Neural von Mises Mixture" model. A von Mises distribution is a distribution on the circle (useful for angles)...

With a little reparameterization to make everything smooth and learnable, a network can take the embedding from the transformer stack and directly output the parameters of a mixture of von Mises distributions, whose (negative) log probability becomes the loss.

When trained (on AF2 CATH domains, and the full PDB), GAIT models mostly sample relatively compact backbones. Sequences generated from these these often (but not always) refold into the "designed" target structure.

Still early days, and it won't compete with diffusion and flow matching just yet, but it seems like a promising avenue to explore, and to try and scale. One up-side is that, with attention caching, sampling can be quite fast. Around 5 seconds for a 200AA sample, on a CPU.

A few tangential things I'm particularly happy about: 1) last year, when we were just getting started in this direction, the three students leading this were all in high school. It has been a genuine joy to work with them. 2) this is all built in Julia (@JuliaLanguage)

The preprint will be up at bioRxiv after screening, and we'll get the code and models up ASAP. We'll also be trying to see if these to do anything, experimentally. And this is our first excursion into this field, so please let us know if we've missed anything in the literature!

The bioRxiv folks (@cshperspectives ?) were working overtime over the weekend, apparently: