Xander Chin's banner

Xander Chin

@XanderChin • 3,619 subscribers

inference @nvidia | eng @westernu @schulichleaders | building and learning for fun

Shorts

hand-controlled boids

hand-controlled boids

741,616 次观看

as an intro to mechanistic interpretability, i decided to look into the formation of induction heads, which are circuits that allow LLMs to perform in-context learning by searching for previous occurrences of a sequence to predict the next token to form these circuits i trained attention-only transformers to repeat varying sequence lengths of random tokens. by randomizing the sequence length, i prevented the models from relying on rote memorization, forcing them to instead develop a generalizable circuit during this dive, i recorded some really cool findings and saw some interesting visual patterns emerging:

as an intro to mechanistic interpretability, i decided to look into the formation of induction heads, which are circuits that allow LLMs to perform in-context learning by searching for previous occurrences of a sequence to predict the next token to form these circuits i trained attention-only transformers to repeat varying sequence lengths of random tokens. by randomizing the sequence length, i prevented the models from relying on rote memorization, forcing them to instead develop a generalizable circuit during this dive, i recorded some really cool findings and saw some interesting visual patterns emerging:

26,486 次观看

Videos

Anya Rossi

sweetdream.ai

SweetDream.ai•Sponsored•Livecam

Watch Anya Live

Anya is streaming live right now! Join her private show and enjoy exclusive content.

Exclusive private shows

1.2k viewers online

Private Show

Join now for exclusive access

Free preview available • Premium content

wrote a piece on how we do tiling and gradient descent on TinyTPU included some animations of the weight and bias gradient descent

wrote a piece on how we do tiling and gradient descent on TinyTPU included some animations of the weight and bias gradient descent

33,195 次观看 • 1 年前

没有更多内容可加载