#softmax
➡️ Machine Learning from Scratch, part 37: softmax. Every neuron first adds up its inputs times their weights, then an activation function reshapes the result. Without activation functions, any number of layers would act like one single linear layer. Softmax is the special activation at the very end of a network that sorts things into classes. The last layer has one output neuron per class, and each one gives a raw score that can be any number, even negative. Softmax looks at all of them at once: it raises e to the power of each score, then divides by the sum. Now every output lies between 0 and 1, all outputs add up to exactly 1, and the class with the highest one is the prediction. Unlike an S curve on each neuron, softmax couples the whole layer: if one score rises, the other probabilities fall. Our own run, nothing downloaded. softmax.py (16 lines of numpy) makes 300 points in three classes and trains a small network with 2 inputs, 4 hidden neurons and 3 output neurons on 200 of them. On the 100 new points it gets 90 right. For the point (-1, 0) it says 95% class 1, 1% class 2 and 4% class 3. Honest notes: these percentages are the model's own confidence. On the test points our model is 94.9% sure on average but right only 90% of the time, so it is a bit too confident, which is common for neural networks. Our learning rate sits close to the edge of stability, so a tiny change in the start can move the last digit (we show our exact run). The 3D hills are the real probabilities of our trained network over the input plane. #machinelearning #coding #softmax #neuralnetwork #deeplearning
MachineLearningToGo16,500 views • 1 month ago
No more content to load
