I spent my summer building TinyTPU : An open source ML inference and training chip.
@suryasure053.6K followers
357,205 views
48 Comments
Comments from the original post on X (Twitter)

@suryasure05
I worked on this with @evanliin, @XanderChin, and @kennykgguo — incredibly smart people! check out our article to see how our chip works and how we went about developing it: you can also find the code here and play with it yourself:
101 likes

@suryasure05
our first step was to decide the scale of this project. we decided to target the simplest possible neural network — the XOR problem. however, we still wanted to make this scalable so a core design philosophy for us was to ensure all of our mechanisms could scale to larger networks.
51 likes

@suryasure05
before we got to designing any hardware, we started off with properly understanding the math behind MLPs. we worked out the math by hand for inference and training of our network.
68 likes

@suryasure05
then we build the heart of any TPU: the systolic array! each processing element (PE) in the systolic array performs a multiply-accumulate in one clock cycle. when connected in a grid, multiple output matrix elements compute simultaneously. this allows us to very efficiently perform matrix multiplication, the most compute-heavy operation in neural networks!
34 likes

@suryasure05
the next operation is adding the bias, for which we made a bias module. since this is an element wise operation, we placed 2 bias modules right under the systolic array (one for each column). we structured the activation module similarly — we chose the leaky ReLU and placed one module under each column of the systolic array.
30 likes

@suryasure05
however, we had a big problem with our systolic array — computation at the end of every layer to load in the weights of the new layer (this was negligible with a 2x2 systolic array, but it would very inefficient if we scaled it) to solve this, we introduced double buffering to the systolic array. Each PE in the systolic array would have two buffers — an active and inactive buffer. while the outputs of the previous layer are being computed, we can load in the weights of the next layer in the inactive buffers. once the previous layer is finished, we can instantly start computation on the next layer. this nearly DOUBLES the speed of our TPU when we scaled to a larger systolic array size.
26 likes

@suryasure05
now we can move on to training! this was easily the most challenging part since we couldn’t find any resource where someone has done this before. a key insight we had while doing the math for training was that the long chain in the computational graph of the backdrop was identical to the forward pass computation graph. this meant we could calculate the long chain first, cache the gradients and then compute the individual weight and bias gradients!
27 likes

@suryasure05
the only problem was...we didn't have any on-chip to store the gradients (LOL) so we decided to make a unified buffer. as we developed this, we realized the unified buffer could replace our accumulators as well, making our design more elegant!
23 likes

@suryasure05
the VPU was the next big change we made. all those modules UNDERNEATH the systolic array (bias, activation, loss, derivatives) process column vectors element-wise. we unified them into one scalable unit with pathway bits to enable/skip operations, which is a lot more elegant than interfacing with N number of individual modules (N scales with systolic array size).
21 likes

@suryasure05
and finally, here's our 94-bit VLIW instruction set:
23 likes

@suryasure05
this was our final TPU architecture:
33 likes

@suryasure05
having no hardware knowledge or experience at all until just 6 months ago, this was a very ambitious project to work on. I had no idea how difficult this would be or if I could even complete it without the "prerequisites". but throughout the last 4 months, I developed a style of thinking and solving problems that I can carry forward with me. it can be encompassed by these two philosophies: - always try the dumb ideas first - DRAW EVERYTHING OUT to TRULY understand a concept you can read more about our background and thought process at and with that here's our final waveform that shows the outputs of inference and training:
70 likes

@alvarolloretlop
Bro’s bout to receive a job offer from an AI lab
15 likes

@evanliin
@karpathy
11 likes

@XanderChin
greatest thing ive ever helped create
8 likes

@threadreaderapp
Your thread is going viral! #TopUnroll 🙏🏼@ain3sh for 🥇unroll
7 likes

@krupaad
@karpathy ‼️
6 likes

@noah_vandal
wow this is very nice. were you at all tempted to use verilog and fpgas?
5 likes

@suryasure05
yes we did use verilog to build this if you look at we did everything in simulation so far, but plan to deploy on an fpga very soon
15 likes

@noah_vandal
wow that is cool























