Skip navigation
➡️ Machine Learning from Scratch, part 42: why GPU and not CPU.
0:00 / 2:43

➡️ Machine Learning from Scratch, part 42: why GPU and not CPU.

41,000 views #machinelearning #ai #gpu

➡️ Machine Learning from Scratch, part 42: why GPU and not CPU. A neural network spends almost all of its time multiplying numbers and adding them up. In every layer each neuron receives the whole input vector, multiplies each number by its own weight and adds them up, so with the weights of all neurons as the rows of one table, the layer becomes one matrix times the input vector. In training a whole mini batch goes in at once, every layer becomes a matrix times a matrix, and the backward pass for the gradients needs about twice as many multiplications again (about 4N against 2N per token in the scaling law paper by Kaplan et al., 2020). The code (gpu_cpu.py, 16 lines of numpy, seed 42) counts the multiplications of one training step for a batch of 32 examples through a small network with three layers (4, 64, 64, 1): 423,936, a third forward and two thirds backward. Then it does the same multiplications once one at a time in a Python loop (0.072 seconds) and once in bulk as matrix products (77 millionths of a second), about 930 times faster. Honest notes: our test ran on one computer without a graphics card, so it shows the idea of doing many independent multiplications at once, and the gap also contains the overhead of the Python loop. A processor has a few fast, clever cores for many different tasks one after another; a graphics card has thousands of simpler cores that run the same instruction on many numbers at the same time. #machinelearning #ai #gpu #neuralnetworks #deeplearning

Comments

Comments from the original post on TikTok

No comments yet