➡️ Machine Learning from Scratch, part 36: the learning rate.
MachineLearningToGo
Visit on TikTok29K followers
12,100 views #machinelearning #ai #coding
➡️ Machine Learning from Scratch, part 36: the learning rate. When a model trains, a loss function measures its error. The gradient tells the model which way is downhill, and the learning rate decides how big each step in that direction is. Think of a ball rolling into a valley, where the lowest point is the smallest error. Too small, and every step is tiny. Too big, and the model jumps over the bottom, lands higher on the other side, and the error can grow with every step. With a good learning rate the steps are big at first and get smaller near the bottom, because the slope there is flatter. Our own run, nothing downloaded. lr.py (16 lines of numpy) trains a tiny network with exactly two weights: the input is multiplied by weight 1, goes through an S curve, and is multiplied by weight 2. Because there are only two weights, the whole error landscape can be drawn exactly, and every ball on screen sits on it at every step. Same start, 20 steps of gradient descent, three learning rates ten times apart. The error starts at 4.0. With 0.1 it is 2.65 after 20 steps, with 1 it is 0.017, and with 10 it grows every single step, to about 4 times 10 to the 34 (numpy even warns about an overflow). Honest notes: we picked the start and the three learning rates in a design study on other random seeds before the real run, so the three cases show up clearly. Which value counts as too small or too big depends on the model and the data, so in practice people try a few values, often ten times apart, and many training runs lower the learning rate over time. Real networks have millions of weights, so their landscape cannot be drawn like this. #machinelearning #ai #coding #gradientdescent #neuralnetwork
Comments
Comments from the original post on TikTok
No comments yet
