#crossentropy
➡️ Machine Learning from Scratch, part 38: cross entropy. Softmax (part 37) turns the raw scores of a network into one probability per class. Cross entropy is the loss function that scores how wrong those probabilities are, with one number. It looks only at the probability of the right class and takes minus its natural log. Right with 99 % costs 0.01. An unsure answer that gives the right class 40 % costs 0.92. If the network is 99 % sure of a wrong class, the right class gets 1 % and the cost jumps to 4.61. Training averages this cost over all examples and pushes the average down. Together with softmax, the push on every raw score is simply the predicted probability minus the truth, so the right class is pushed up, the wrong classes are pushed down, and a confident mistake gets the strongest push. With squared error through softmax, that push on a confident miss can be much weaker, which is one reason classifiers usually learn with cross entropy. Our own run, nothing downloaded. cross_entropy.py (16 lines of numpy) prints the three costs above, makes 300 points in three classes and trains the small network from part 37 (2 inputs, 4 hidden neurons, 3 output neurons) with cross entropy. The average loss falls from 1.01 to 0.13 in 1,000 steps. After training it gets 283 of the 300 points right, and the 17 it still gets wrong make up 73 % of the remaining loss. Honest notes: we train and measure on the same 300 points, so this is the training loss. On these easy points, the same network trained with squared error ends up about as accurate (284 of 300). The difference shows in the push on single confident mistakes: for our most confident miss at the start, 0.98 with cross entropy against 0.07 with squared error. The 3D surfaces show both losses over two raw scores of that real point, with the same 12 steps on each. #machinelearning #ai #crossentropy #neuralnetwork #deeplearning
MachineLearningToGo11,400 görüntüleme • 1 ay önce
Daha fazla içerik yok.
