➡️ Machine Learning from Scratch part 19: backpropagation and gradient descent connected Our data is…
MachineLearningToGo
Visit on TikTok29K followers
21,300 views #machinelearning #backpropagation #gradientdescent
➡️ Machine Learning from Scratch part 19: backpropagation and gradient descent connected Our data is simulated. The program makes up 20 points along an S-shaped curve, 3 * sig(2x) plus noise (seed 19). The network has one input, one hidden neuron and one output: exactly two weights, so we can draw its whole loss landscape. How they connect: The loss is the average of the squared errors, 2.38 at the start. A derivative is a slope: how much the loss changes when a weight moves a tiny bit. Taken for one weight while the other holds still, it is a partial derivative, and the list of all partial derivatives is the gradient. Backpropagation computes that gradient. It starts at the loss and works backward through the network, multiplying the slopes of each link (the chain rule) and reusing the slope at the output for both weights. Gradient descent then uses exactly that gradient: new weight = weight - learning rate * slope, with a learning rate of 0.5. The check: nudging the first weight by one millionth gives a slope of -1.2214294, backpropagation gives -1.2214297. The first seven digits agree. The rest is the nudge's own error, since one millionth is not zero. After 100 steps the loss falls from 2.38 to 0.028, and the weights end at 2.15 and 2.89, close to the 2 and 3 our program used to make the points. In PyTorch you see the same split: loss.backward() runs backpropagation, optimizer.step() takes the gradient step. #machinelearning #backpropagation #gradientdescent #deeplearning #python
Comments
Comments from the original post on TikTok
No comments yet
