➡️ Part 12 of learning ML code from scratch: Does your AI really think, or is it just doing math?
MachineLearningToGo
Visit on TikTok29K followers
16,000 views #machinelearning #ai #coding
➡️ Part 12 of learning ML code from scratch: Does your AI really think, or is it just doing math? We open the code and find the exact place where its thinking happens. Two questions guide the video: where in the code is the thinking, and is it anything like yours? Step 1, the loop. One line writes the text. It picks the next word from its chances, and that word goes right back in to pick the one after it. Step 2, the thinking. Everything the model does before a word happens in one small function, chances(). It takes 29 scores, one for each possible next word, and turns them into chances. After "the", cat gets 37.2 %, bird 31.8 % and dog 30.6 %. That is plain arithmetic: e to the power of 7.13 is 1,252, and divided by the sum of all 29 values, 3,363, it gives cat its 37.2 %. Then comes a random pick. Step 3, big models. They do the same kind of arithmetic, just far more of it. GPT-3 sends every word through 96 layers with 175 billion numbers, and still, every word gets exactly one pass. Step 4, the limit. One pass has a fixed number of steps, and some questions need more. 17 × 24 is hard in your head, and on paper it is easy: 17 × 20 = 340, 17 × 4 = 68, together 408. Step 5, reasoning. The model writes its own paper: notes in plain words, before the answer. Each note is one more pass, and every new word reads all the notes again. When your app shows "Thinking", this is what it is doing. Our tiny model only reads the last word, so notes would be wasted on it. Reasoning models learned to write useful notes in training, because notes that led to a right answer were rewarded. Step 6, like you? In some ways, yes. You also break hard problems into steps and write them down, and artificial neurons were modeled on brain cells, which add up signals and fire. While the model writes, though, nothing inside it changes, only the text grows. Its notes also do not always show what really happened inside. And whether it truly understands anything, nobody can measure yet. The short answers. Where is the thinking? It is just math, done once for every word, plus the notes it writes to itself. Is it like yours? It works step by step, just like you, but whether it truly understands is still an open question. #machinelearning #ai #coding #reasoning #chatgpt
Comments
Comments from the original post on TikTok
No comments yet
