1/ Can molecular AI move past hard-coded Graph Neural Networks and embrace scalable Transformers that discover molecular structure on their own? We demonstrate that you can train a 1B parameter Transformer model without any graph priors or physical inductive biases. And surprisingly, not only can it maintain competitive performance... under equal compute on the Open Molecules 2025 dataset… it’s faster than a 6M parameter equivariant GNN, and exhibits scaling laws that don’t saturate. We use this as a starting point to investigate emergent internal representations, and find that it adaptively discovers molecular structure! Check out the interactive demo on our website: And our paper: In collaboration with Toby Kreiman, Yutong Bai, Fadi, Elizabeth, and Eric Qu. Here’s a video showing how the Transformer learns distance-aware attention patterns (purple gradient) that adapt to atomic environments 👇show more

Super nice!!! We've been working on a scalable approach too; our transformer is both scalable *and* equivariant 😉, but we haven't deployed at this scale yet. We'll share the paper soon 😀. But wow, cool paper, what an amazing demo of scale at work! I love it ❤️

Excited to see it 😃! The beauty of this approach (fully unmodified Transformer) is we can fairly easily scale to 1B+ params, and it's clear that the improvement trend is going to predictably continue to hold. So far, with constraints, it seems one reaches saturation much sooner

Nice, yeah that's insane! I'm looking forward to learn more about equivariance at scale

2/ A Transformer can adaptively learn attention patterns based on atomic environment, whereas GNNs often use a fixed graph construction algorithm (e.g., k-nearest neighbors or fixed radius cutoff)

3/ The model predictably scales with training resources, with no signs of saturating!

4/ The model learns attention scores that decay inversely with interatomic distance in early layers, and in later layers learns attention that remains roughly constant with distance. This suggests that the Transformer naturally first learns local feature extraction before performing global aggregation in later layers. The more isolated an atom is, the farther out attention adaptively expands.

5/ We really do mean an unmodified Transformer: no explicit calculation of pairwise distances, no graph-based features, no rotational equivariance, etc. Leveraging modern software and hardware, a 1B parameter Transformer trains and runs inference faster than a 6M parameter equivariant GNN.

6/ Our results demonstrate that many favorable properties of GNNs can emerge adaptively and more flexibly in Transformers, challenging the necessity of hard-coded graph inductive biases and pointing toward standardized, scalable architectures for molecular modeling. This has been a hot topic in the community, and we hope that this adds more to the discussion!

Absolutely huge work!!! I have a feeling I am seeing into the future. 🔮

Super interesting! Faster and smaller models will be the future

Thanks for sending, and apologies that we forgot to cite (will add). This work also goes in the category that we discuss in our paper of GNNs that incorporate an attention-based mechanism: it's still operating on a predefined graph, and a new graph is still being constructed for every input using a radius cutoff (and then doing message passing). In this case study, we're using a completely unmodified Transformer with no graph-based features