I trained a tiny transformer to add numbers mod 113 and saved its embeddings every 20 steps. When it Grokked, it had put the numbers on circles and was adding them with the angle addition formula from trigonometry.

Hey @tetsuoai Im sure you just saw the recent news about @UsePaid On Crypto Twitter We just made you a coin to claim x fees 79FCJqG2DrYzH2G3eCQedRoWRbjri7MSgMkyGeZ7cLrm

That’s interesting 🔥

Very cool

Wooow

best circles

Hackers always w1n. PS. I run the entire department. Imagine the following I'd have if I gave the public my play book. "The action is the juice"

40,000 steps does help with weight decay

Here we grokke 🌌

the voice in this is really cool. should be doing radio! 🔥

That's so cool! The model really spontaneously learned the underlying mathematical pattern.

Check ur X money

Yo @tetsuoai why you unfollowed me lmao

are you from the us? do you have X money?

so much fucking cognitive dissonance

wild how it lands on trig without being told

Hmm wonder where that idea came from 😂

Broo do a new x

Embeddings are cool 😎

Wow🔥🔥

wild how geometry just shows up in embeddings after grokking

The circle geometry showing up in the embeddings is such a neat way for the model to represent modular addition.