Skip navigation
➡️ Machine Learning from Scratch, part 33: word embeddings.
0:00 / 1:44

➡️ Machine Learning from Scratch, part 33: word embeddings.

14,200 views #machinelearning #ai #embeddings

➡️ Machine Learning from Scratch, part 33: word embeddings. A language model turns every word, or token, into an embedding: a long list of numbers, often 768 and in big models several thousand (=basically a vector). Each list is a point in a space with that many dimensions. The model learns these points during training, and words that show up in similar sentences end up close together. To write the next word, the model turns the text so far into one new vector in the same space, compares it with every word vector (the score counts how similar the directions are and how long the word vector is), and softmax turns the scores into probabilities. Usually the best fit wins, sometimes a slightly less likely word. The point in space marks the concept's exact location, while the direction represents its core meaning and relationships. Our own run, nothing downloaded. corpus.py writes a tiny text of 144 sentences like "cat and dog eat meat" with 32 words in four groups (pets, farm animals, kids, adults). emb.py (16 lines of numpy) gives every word exactly 3 numbers, averages the words so far, turns that average with a learned 3 by 3 matrix, scores all words with the same table and is trained from scratch with hand-written backpropagation. 12 word pairs were never trained on: on their 36 sentences it gets 87 of 144 next words right. 96 is the best any model can do here, because which word comes after "cat and", or which verb follows, is pure chance in our text. After "cat and dog eat" it puts 75 percent on "meat", although it never saw "cat and dog" together. Honest notes: real models use hundreds to thousands of dimensions; 3D is a simplification so we can draw it. Our text is made up and very small. In our space cat, dog, kitten and puppy land almost on the same spot. The king, man, woman, queen picture is from classic word vectors, not from our model. In our run "cake" points in the most similar direction, but "meat" wins because its vector is longer. Embeddings copy the patterns and biases of their training text, and one vector per word mixes up meanings like bank. #machinelearning #ai #embeddings #coding #python

Comments

Comments from the original post on TikTok

No comments yet