Skip navigation
➡️ Machine Learning from Scratch, part 45: the Adam optimizer.
0:00 / 1:37

➡️ Machine Learning from Scratch, part 45: the Adam optimizer.

30,100 views #machinelearning #ai #coding

➡️ Machine Learning from Scratch, part 45: the Adam optimizer. Plain gradient descent moves every weight by one learning rate times its gradient, so a learning rate that is small enough for a steep weight barely moves a flat one. Adam keeps two running averages for each weight, one of its recent gradients and one of its recent squared gradients, and divides the first by the square root of the second. That way every weight takes a step of about the same size, however steep its slope, and the first average also works like momentum. The code (adam.py, 16 lines of numpy, seed 45) fits two weights to 200 points where one input is about ten times larger than the other. With a learning rate right at its limit (0.0099, the limit is 0.00998), plain gradient descent needs 269 steps until both weights are within 0.01 of the best answer. Adam with a learning rate of 0.05 needs 59. Honest notes: both methods run close to their best learning rate in our sweep, so the comparison is fair. Adam was introduced by Diederik Kingma and Jimmy Ba in 2014. #machinelearning #ai #coding #gradientdescent #deeplearning

Comments

Comments from the original post on TikTok

No comments yet