Skip navigation
I spent my summer building TinyTPU : An open source ML inference and training chip.
0:00 / 0:27

I spent my summer building TinyTPU : An open source ML inference and training chip.

surya

@suryasure053.6K followers

Visit on X
3,433 likesDownloadOriginal post

357,205 views

I spent my summer building TinyTPU : An open source ML inference and training chip. it can do end to end inference + training ENTIRELY on chip. here's how I did it👇:

48 Comments

Comments from the original post on X (Twitter)

  • surya's profile picture

    @suryasure05

    I worked on this with @evanliin, @XanderChin, and @kennykgguo — incredibly smart people! check out our article to see how our chip works and how we went about developing it: you can also find the code here and play with it yourself:

    101 likes

  • surya's profile picture

    @suryasure05

    our first step was to decide the scale of this project. we decided to target the simplest possible neural network — the XOR problem. however, we still wanted to make this scalable so a core design philosophy for us was to ensure all of our mechanisms could scale to larger networks.

    51 likes

  • surya's profile picture

    @suryasure05

    before we got to designing any hardware, we started off with properly understanding the math behind MLPs. we worked out the math by hand for inference and training of our network.

    68 likes

  • surya's profile picture

    @suryasure05

    then we build the heart of any TPU: the systolic array! each processing element (PE) in the systolic array performs a multiply-accumulate in one clock cycle. when connected in a grid, multiple output matrix elements compute simultaneously. this allows us to very efficiently perform matrix multiplication, the most compute-heavy operation in neural networks!

    34 likes

  • surya's profile picture

    @suryasure05

    the next operation is adding the bias, for which we made a bias module. since this is an element wise operation, we placed 2 bias modules right under the systolic array (one for each column). we structured the activation module similarly — we chose the leaky ReLU and placed one module under each column of the systolic array.

    30 likes

  • surya's profile picture

    @suryasure05

    however, we had a big problem with our systolic array — computation at the end of every layer to load in the weights of the new layer (this was negligible with a 2x2 systolic array, but it would very inefficient if we scaled it) to solve this, we introduced double buffering to the systolic array. Each PE in the systolic array would have two buffers — an active and inactive buffer. while the outputs of the previous layer are being computed, we can load in the weights of the next layer in the inactive buffers. once the previous layer is finished, we can instantly start computation on the next layer. this nearly DOUBLES the speed of our TPU when we scaled to a larger systolic array size.

    26 likes

  • surya's profile picture

    @suryasure05

    now we can move on to training! this was easily the most challenging part since we couldn’t find any resource where someone has done this before. a key insight we had while doing the math for training was that the long chain in the computational graph of the backdrop was identical to the forward pass computation graph. this meant we could calculate the long chain first, cache the gradients and then compute the individual weight and bias gradients!

    27 likes

  • surya's profile picture

    @suryasure05

    the only problem was...we didn't have any on-chip to store the gradients (LOL) so we decided to make a unified buffer. as we developed this, we realized the unified buffer could replace our accumulators as well, making our design more elegant!

    23 likes

  • surya's profile picture

    @suryasure05

    the VPU was the next big change we made. all those modules UNDERNEATH the systolic array (bias, activation, loss, derivatives) process column vectors element-wise. we unified them into one scalable unit with pathway bits to enable/skip operations, which is a lot more elegant than interfacing with N number of individual modules (N scales with systolic array size).

    21 likes

  • surya's profile picture

    @suryasure05

    and finally, here's our 94-bit VLIW instruction set:

    23 likes

  • surya's profile picture

    @suryasure05

    this was our final TPU architecture:

    33 likes

  • surya's profile picture

    @suryasure05

    having no hardware knowledge or experience at all until just 6 months ago, this was a very ambitious project to work on. I had no idea how difficult this would be or if I could even complete it without the "prerequisites". but throughout the last 4 months, I developed a style of thinking and solving problems that I can carry forward with me. it can be encompassed by these two philosophies: - always try the dumb ideas first - DRAW EVERYTHING OUT to TRULY understand a concept you can read more about our background and thought process at and with that here's our final waveform that shows the outputs of inference and training:

    70 likes

  • Lloret's profile picture

    @alvarolloretlop

    Bro’s bout to receive a job offer from an AI lab

    15 likes

  • evan's profile picture

    @evanliin

    @karpathy

    11 likes

  • Xander Chin's profile picture

    @XanderChin

    greatest thing ive ever helped create

    8 likes

  • Thread Reader App's profile picture

    @threadreaderapp

    Your thread is going viral! #TopUnroll 🙏🏼@ain3sh for 🥇unroll

    7 likes

  • krupa's profile picture

    @krupaad

    @karpathy ‼️

    6 likes

  • Noah Vandal's profile picture

    @noah_vandal

    wow this is very nice. were you at all tempted to use verilog and fpgas?

    5 likes

  • surya's profile picture

    @suryasure05

    yes we did use verilog to build this if you look at we did everything in simulation so far, but plan to deploy on an fpga very soon

    15 likes

  • Noah Vandal's profile picture

    @noah_vandal

    wow that is cool