Загрузка видео...
Не удалось загрузить видео
ResNet by hand ✍️ ~ 10 steps walkthrough below "Deep Residual Learning for Image Recognition" (Kaiming He, CVPR 2016) is among the most cited papers in all of deep learning. Why does it matter so much? It fixed the exploding and vanishing gradients that kept deep networks from being... show more
16,658 просмотров • 2 дней назад •via X (Twitter)
Комментарии: 0
Нет доступных комментариев
Здесь появятся комментарии из оригинального поста
![Self Attention by hand ✍️ ~ 9 steps walkthrough below Self-attention is what enables LLMs to understand context. How does it work? So I drew and calculated one entirely by hand. Goal: turn four 6D features into four 3D attention weighted features, filling in every cell yourself. = 1. Given = Four feature vectors, six dimensions each, one per position. = 2. Query, key, value = Let us multiply the features by WQ, WK and WV. Queries, keys and values all come out of the same four features, and that is what the word "self" is doing in self-attention. = 3. Prepare for MatMul = We copy the queries across the top and the transposed keys down the side. Lining the two up is half the work. = 4. MatMul = Let us multiply K transpose by Q. Every cell is the dot product of one key with one query, which we use as a matching score. That works because the dot product is the numerator of cosine similarity: it is how alike two vectors are, before anyone divides by their lengths. = 5. Scale = We divide by the square root of dk, the dimension of a key vector, here 3. Without it the scores grow with the dimension and a 64-wide head would swamp the softmax. To keep the page doable in pen, the drawing approximates dividing by root 3 with halving. = 6. e to the power = Let us raise e to the power of each score. This is the first half of softmax, and the drawing uses 3 in place of e, which is close enough to do in your head. = 7. Sum = We add up each column: 16, 6, 7 and 12. = 8. Normalize = Let us divide every cell by its column sum. That gives the attention weight matrix in yellow, and each of its four columns is now a probability distribution over the four positions. The decimals are nudged as they are rounded, so every column still sums to exactly 1. = 9. MatMul = We multiply the value vectors by those weights. Each output is a blend of all four values, mixed in the proportion the attention matrix just decided, and it goes to the position-wise feed forward network in the next layer: the FFN box at the bottom of the page. The outputs: Attention weights (A), by column = [.2, .6, 0, .2], [.2, .4, .2, .2], [.4, .2, 0, .4], [.1, .7, .1, .1] Attention weighted features (Z) = [8, 2, 6], [8, 4, 4], [16, 4, 2], [4, 2, 7] The takeaway: attention is a weighted average, and everything before step 9 exists to decide the weights. Compare every position with every other, turn the scores into one distribution per position, then blend. 💾 Save this post!](https://image.24vids.com/tw-2079933315990057242/ext_tw_video_thumb/2079933302048219136/pu/img/q3wr-ymUWG3Ai-ij.jpg)
![[Self-Attention] by Hand ✍️ Self-attention is what enables LLMs to understand context. How does it work? This exercise demonstrates how to calculate a 6-3 attention head by hand. Note that if we have two instances of this, we get 6-6 attention (i.e., multi-head attention, n=2). -- 𝗚𝗼𝗮𝗹 -- Transform [6D Features 🟧] to [3D Attention Weighted Features 🟦] -- 𝗪𝗮𝗹𝗸𝘁𝗵𝗿𝗼𝘂𝗴𝗵 -- [1] Given ↳ A set of 4 feature vectors (6-D): x1,x2,x3,x4 [2] Query, Key, Value ↳ Multiply features x's with linear transformation matrices WQ, WK, and WV, to obtain query vectors (q1,q2,q3,q4), key vectors (k1,k2,k3,k4), and value vectors (v1,v2,v3,v4). ↳ "Self" refers to the fact that both queries and keys are derived from the same set of features. [3] 🟪 Prepare for MatMul ↳ Copy query vectors ↳ Copy the transpose of key vectors [4] 🟪 MatMul ↳ Multiply K^T and Q ↳ This is equivalent to taking dot product between every pair of query and key vectors. ↳ The purpose is to use dot product as an estimate of the "matching score" between every key-value pair. ↳ This estimate makes sense because dot product is the numerator of Cosine Similarity between two vectors. [5] 🟨 Scale ↳ Scale each element by the square root of dk, which is the dimension of key vectors (dk=3). ↳ The purpose is to normalize the impact of the dk on matching scores, even if we scale dk to 32, 64, or 128. ↳ To simplify hand calculation, we approximate [ □/sqrt(3) ] with [ floor(□/2) ]. [6] 🟩 Softmax: e^x ↳ Raise e to the power of the number in each cell ↳ To simplify hand calculation, we approximate e^□ with 3^□. [7] 🟩 Softmax: ∑ ↳ Sum across each column [8] 🟩 Softmax: 1 / sum ↳ For each column, divide each element by the column sum ↳ The purpose is normalize each column so that the numbers sum to 1. In other words, each column is a probability distribution of attention, and we have four of them. ↳ The result is the Attention Weight Matrix (A) (yellow) [9] 🟦 MatMul ↳ Multiply the value vectors (Vs) with the Attention Weight Matrix (A) ↳ The results are the attention weighted features Zs. ↳ They are fed to the position-wise feed forward network in the next layer.](https://image.24vids.com/tw-1797249951325434315/ext_tw_video_thumb/1797249707795697665/pu/img/bRQFS-RcgUFbmOnk.jpg)
![Dropout by hand ✍️ ~ 10 steps walkthrough below Dropout is the simplest trick in deep learning that actually works: during training you randomly switch neurons off, so the network cannot lean on any one of them. It is two lines of code and almost nobody has worked through what those lines do to the numbers. So I drew and calculated one entirely by hand. Goal: train one pass through a small network with two dropout layers, then run inference with dropout switched off. The network: Linear(2,4), ReLU, Dropout(0.5), Linear(4,3), ReLU, Dropout(0.33), Linear(3,2). = 1. Given = A training set of two examples, X1 and X2, and the weight matrices for all three linear layers. = 2. Draw the first random numbers = Let us draw 4 random numbers, one per neuron in the first hidden layer. Above 0.5 we keep (◯), below we drop (╳). Here that gives [◯, ╳, ◯, ╳]. = 3. Build the first dropout matrix = We turn that pattern into a diagonal matrix. The scaling factor is 1/(1-p) = 2, so a kept neuron gets 2 and a dropped one gets 0. Multiplying by it does both jobs at once: it deletes the 2nd and 4th neurons and doubles the two that survive. = 4. Draw the second random numbers = Let us do it again for the 3 neurons in the next layer, this time against p = 0.33. The result is [◯, ◯, ╳]. = 5. Build the second dropout matrix = We set the diagonal to 1.5 where kept and 0 where dropped. Only the 3rd neuron goes. = 6. Feed forward = Let us run the whole thing top to bottom: one matrix multiplication per layer, ReLU setting the negatives to zero, and the two dropout matrices doing their work in between. The outputs Y come out at the bottom. = 7. MSE loss gradients = We compare Y against the targets Y', subtract, and multiply each element by 2. That is the whole gradient of the mean squared error. = 8. Update the weights = Let us push those gradients back through the network and update the weights (marked in light red). = 9. Deactivate dropout = Training is over, so we set both dropout matrices to the identity. Every neuron is back, and nothing is scaled. = 10. Feed forward again = One more pass, this time on unseen data, to make the prediction. You have just trained and run a network with dropout by hand. ✍️ The outputs: Training outputs Y = [-6, 9; 13, 4] Loss gradients = [-4, 4; 6, -2] Inference outputs = [13, 13; 4, 3] 💾 Save this post! #AIbyHand #Dropout #DeepLearning #NeuralNetworks](https://image.24vids.com/tw-2078846104460783996/ext_tw_video_thumb/2078846092997763074/pu/img/pljAsdaJjvGEEjPE.jpg)
![[CLIP] by Hand ✍️ The CLIP (Contrastive Language–Image Pre-training) model, a groundbreaking work by OpenAI, redefines the intersection of computer vision and natural language processing. It is the basis of all the multi-modal foundation models we see today. How does CLIP work? Goal: 🟨 Learn a shared embedding space for text and image [1] Given ↳ A mini batch of 3 text-image pairs ↳ OpenAI used 400 million text-image pairs to train its original CLIP model. Process 1st pair: "big table" [2] 🟪 Text → 2 Vectors (3D) ↳ Look up word embedding vectors using word2vec. [3] 🟩 Image → 2 Vectors (4D) ↳ Divide the image into two patches. ↳ Flatten each patch [4] Process other pairs ↳ Repeat [2]-[3] [5] 🟪 Text Encoder & 🟩 Image Encoder ↳ Encode input vectors into feature vectors ↳ Here, both encoders are simple one layer perceptron (linear + ReLU) ↳ In practice, the encoders are usually transformer models. [6] 🟪 🟩 Mean Pooling: 2 → 1 vector ↳ Average 2 feature vectors into a single vector by averaging across the columns ↳ The goal is to have one vector to represent each image or text [7] 🟪 🟩 -> 🟨 Projection ↳ Note that the text and image feature vectors from the encoders have different dimensions (3D vs. 4D). ↳ Use a linear layer to project image and text vectors to a 2D shared embedding space. 🏋️ Contrastive Pre-training 🏋️ [8] Prepare for MatMul ↳ Copy text vectors (T1,T2,T3) ↳ Copy the transpose of image vectors (I1,I2,I3) ↳ They are all in the 2D shared embedding space. [9] 🟦 MatMul ↳ Multiply T and I matrices. ↳ This is equivalent to taking dot product between every pair of image and text vectors. ↳ The purpose is to use dot product to estimate the similarity between a pair of image-text. [10] 🟦 Softmax: e^x ↳ Raise e to the power of the number in each cell ↳ To simplify hand calculation, we approximate e^□ with 3^□. [11] 🟦 Softmax: ∑ ↳ Sum each row for 🟩 image→🟪 text ↳ Sum each column for 🟪 text→ 🟩 image [12] 🟦 Softmax: 1 / sum ↳ Divide each element by the column sum to obtain a similarity matrix for 🟪 text→🟩 image ↳ Divide each element by the row sum to obtain a similarity matrix for 🟩 image→🟪 text [13] 🟥 Loss Gradients ↳ The "Targets" for the similarity matrices are Identity Matrices. ↳ Why? If I and T come from the same pair (i=j), we want the highest value, which is 1, and 0 otherwise. ↳ Apply the simple equation of [Similarity - Target] to compute gradients of for both directions. ↳ Why so simple? Because when Softmax and Cross-Entropy Loss are used together, the math magically works out that way. ↳ These gradients kick off the backpropagation process to update weights and biases of the encoders and projection layers (red borders).](https://image.24vids.com/tw-1801370086152016034/ext_tw_video_thumb/1801368762937131008/pu/img/GuRh5lcJ6M7HsrkY.jpg)
![Graph Convolutional Network by hand ✍️ ~ 12 steps walkthrough below Graph Convolutional Networks (GCNs), introduced by Thomas Kipf and Max Welling in 2017, are the tool for data shaped like a graph: social networks, recommendations, biological networks, drug discovery, molecular chemistry. I drew and calculated a simple GCN entirely by hand. Goal: run a two-layer GCN, then a small classifier, on a five-node graph, filling in every cell yourself. 1. Given A graph of five nodes, A to E, with edges between some of them. 2. Adjacency matrix (neighbors) Put a 1 wherever two nodes share an edge, in both directions. 3. Adjacency matrix (self) Add 1s down the diagonal, one self-loop per node. That is just adding the identity matrix. 4. Messages Multiply each node's embedding by the weights and biases, then ReLU. Negatives become 0. 5. Pooling Multiply the messages by the adjacency matrix. Each node gathers the messages of its neighbours and itself. 6. Visualize Node A pools [3,0,1] + [1,0,0] = [4,0,1]. 7. Second GCN layer Messages again: weights, biases, ReLU. 8. Pooling again Pool over each node and its neighbours, once more. 9. Visualize Node C pools [1,2,4] + [1,3,5] + [0,0,1] = [2,5,10]. 10. Fully connected layer Weights, biases, ReLU. This time there are no neighbours to pool, just the node itself. 11. Linear layer One more: weights and biases. 12. Sigmoid Squash each score to a probability (≥ 3 → 1, 0 → 0.5, ≤ -3 → 0). That is the classification for each node. You have just classified every node in the graph by hand. ✍️ The outputs: A: 0 (very unlikely) B: 1 (very likely) C: 1 (very likely) D: 1 (very likely) E: 0.5 (neutral) The takeaway: a GCN layer is two parts. The top part pools each node with its neighbours through the adjacency matrix. The bottom part is an MLP that transforms each node on its own. A transformer layer has the same two parts, with an attention matrix where the adjacency matrix was. Both matrices do one job, mixing across positions: attention over tokens, adjacency over nodes. In my class I call the GCN the transformer's little cousin: a bit more stubborn, because its attention is fixed by the graph rather than computed from Q, K, and V. Draw the two side by side and the resemblance is hard to miss. 💾 Save this post! #AIbyHand #GraphNeuralNetworks #DeepLearning](https://image.24vids.com/tw-2077396566864773414/ext_tw_video_thumb/2077396555410051073/pu/img/JX9U_Rk-wmgd1S-x.jpg)
![SVM by hand ✍️ ~ 19 steps walkthrough below (Linear vs RBF) Support Vector Machines reigned supreme in machine learning before the deep learning revolution. An SVM predicts with dot products, the same matrix multiplication every model uses. What it does not do is train by backpropagation: it is fitted by convex optimization, so there is no matrix-multiplication backward pass for a GPU to accelerate. I drew and calculated two SVMs by hand: a linear one (top) and an RBF one (bottom), classifying the same two test vectors. Goal: turn six training vectors and their learned coefficients into a prediction, and see what changing the kernel actually changes. = 1. Given = Six training vectors, their labels, and the coefficients and bias already learned. A coefficient of zero means that vector is not a support vector: too far from the boundary to matter. = 2. Linear kernel, test vector 1 = Let us take the dot product of the test vector with every training vector. The dot product stands in for cosine similarity, and the column of results is the first column of the kernel matrix K. = 3. Linear kernel, test vector 2 = We do the same for the second, and K is complete. = 4. Signed weights = Let us multiply each coefficient by its label. The second training vector drops out here, because its coefficient is 0. = 5. Weighted combination = We multiply the signed weights through K and add the bias b. The result is a signed distance to the decision boundary: 17 and 5. = 6. Classify = Let us take the sign. Both are positive. = 7 to 11. RBF kernel, test vector 1 = Now the same picture with a different kernel, in five moves: square the differences, sum them, take the square root for the L2 distance, multiply by minus gamma, and raise e to that power. The negation is what turns a distance into a similarity, and gamma controls how far a single training vector's influence reaches. = 12 to 16. RBF kernel, test vector 2 = We repeat all five. The numbers change, the moves do not. = 17 to 19. Decision boundary, again = Signed weights, weighted combination, sign. Identical arithmetic to steps 4 through 6, on a K that was built a completely different way. The outputs: Linear K, first column = [13, 25, 12, 15, 19, 27] Linear decision values = 17 and 5, both positive RBF decision values = -2 and 1, so negative and positive The takeaway: the kernel is the only thing that changed, and it changed the answer. The linear SVM calls both test vectors positive; the RBF one splits them. Everything after the kernel matrix, the signed weights and the weighted combination and the sign, is the same page of arithmetic twice. 💾 Save this post!](https://image.24vids.com/tw-2080658072842703031/ext_tw_video_thumb/2080658053083394048/pu/img/Lrl4Xyr6GSkbbbrN.jpg)
![Vector Database by hand ✍️ ~ 10 steps walkthrough below Vector databases are the backbone of Retrieval Augmented Generation (RAG). How do they actually work? Goal: index three sentences, then answer a query by finding the nearest one, filling in every cell yourself. = 1. Given = A dataset of three sentences, three words each. In practice it is millions of them. = 2. Word embeddings = Let us look up each word in an embedding table. Here the vocabulary is 22 words; in practice it is tens of thousands, and the vectors have thousands of dimensions rather than four. = 3. Encoding = We feed the sequence to an encoder, one linear layer and a ReLU, and get one feature vector per word. In practice the encoder is a transformer. = 4. Mean pooling = Let us average across the columns. Three word vectors collapse into one, which is what people mean by a text embedding or a sentence embedding. = 5. Indexing = We multiply by a projection matrix and the four dimensions become two. It is doing the job of a hash: a short representation that is faster to compare, and it is what gets saved in the vector storage. = 6. Process "who are you" = Let us repeat steps 2 to 5 on the second sentence. = 7. Process "who am I" = We do it a third time. The database is now indexed. = 8. Query "am I you" = Let us push the query through the very same pipeline: lookup, encoder, mean pooling, projection, and it lands as a 2D vector in the same space. = 9. Dot products = We transpose the query and multiply, which takes the dot product against every stored vector at once. The dot product is the estimate of similarity. = 10. Nearest neighbour = Let us scan for the largest: 60/9 beats 44/9 and 40/9, so the answer is "who am I". Scanning billions of vectors one at a time is what makes this the slow step in practice, which is why real databases use an approximate nearest neighbour index like HNSW. The outputs: Stored index vectors = [5/3, 2/3], [5/3, 0], [7/3, 2/3] Query vector = [8/3, 2/3] Dot products = 44/9, 40/9, 60/9 Nearest neighbour = "who am I" The takeaway: a vector database is an embedding pipeline, a projection, and a dot product. Every step here is arithmetic you can do in pen, which is worth remembering when the word "database" makes it sound like something else. 💾 Save this post!](https://image.24vids.com/tw-2081020431767581061/ext_tw_video_thumb/2081020419935449088/pu/img/j0l6wBTIjusWDAaV.jpg)
![Generative Adversarial Network (GAN) by hand ✍️ ~ 9 steps walkthrough below The Gen in GenAI came from this landmark paper by Ian Goodfellow et al., 12 years ago. The paper showed that a neural network can not only classify but also turn upside down to generate realistic looking images. The secret? We pit two of them against each other: a Generator turns noise into fake data, and a Discriminator learns to tell fake from real, pushing the Generator to keep doing better. One runs upside down, the other right way up. I drew and calculated one entirely by hand. Goal: generate realistic 4D data out of 2D noise, filling in every cell yourself. = 1. Given = Four noise vectors in 2D, and four real data vectors in 4D. = 2. Generator, first layer = Let us multiply the noise by weights and biases to get new features. = 3. ReLU = We apply the activation, and -1 and -2 are crossed out and set to 0. = 4. Generator, second layer = Let us multiply again. ReLU applies here too, but every value is already positive, so nothing changes. What comes out is the fake data F, made by a two-layer generator out of nothing but noise. = 5. Discriminator, first layer = We feed it both, the four fakes and the four real vectors, through the same weights. It never learns which is which from the layout, only from the numbers. = 6. Discriminator, second layer = Let us reduce each data vector to a single feature Z. Eight vectors in, eight numbers out. = 7. Sigmoid = We turn each Z into a probability Y. A 1 means the discriminator is certain the data is real, a 0 means certain it is fake. = 8. Training the Discriminator = Let us take the gradients as Y minus YD, where YD is what the discriminator should have said: 0 for the four fakes, 1 for the four real. Why so simple? Because pairing sigmoid with binary cross entropy loss makes the math collapse to exactly this subtraction. Its loss uses both halves of the page. = 9. Training the Generator = We do it again, as Y minus YG, and YG is [1, 1, 1, 1]: the generator wants the discriminator to call every fake real. Same predictions, different target, opposite goal. Its loss uses only the fakes. The outputs: Fake data F = [1, 2, 3, 1], [1, 1, 2, 1], [2, 2, 4, 2], [1, 0, 1, 1] Predictions on fakes = [.7, .5, .9, .3] Predictions on real = [.7, .9, .9, 1] Discriminator gradients = [.7, .5, .9, .3] and [-.3, -.1, -.1, 0] Generator gradients = [-.3, -.5, -.1, -.7] The takeaway: the adversarial part is one subtraction done twice. The same eight predictions, scored against two opposite targets, send one set of gradients back through the blue weights and another back through the green ones. 💾 Save this post!](https://image.24vids.com/tw-2080295671999950953/ext_tw_video_thumb/2080295659916128256/pu/img/NFB3stabPHwl-YpA.jpg)
![SORA by Hand ✍️ OpenAI’s #SORA took over the Internet when it was announced earlier this year. The technology behind Sora is the Diffusion Transformer (DiT) developed by William Peebles and Shining Xie. How does DiT work? 𝗚𝗼𝗮𝗹: Generate a video conditioned by a text prompt and a series of diffusion steps [1] Given ↳ Video ↳ Prompt: "sora is sky" ↳ Diffusion step: t = 3 [2] Video → Patches ↳ Divide all pixels in all frames into 4 spacetime patches [3] Visual Encoder: Pixels 🟨 → Latent 🟩 ↳ Multiply the patches with weights and biases, followed by ReLU ↳ The result is a latent feature vector per patch ↳ The purpose is dimension reduction from 4 (2x2x1) to 2 (2x1). ↳ In the paper, the reduction is 196,608 (256x256x3)→ 4096 (32x32x4) [4] ⬛ Add Noise ↳ Sample a noise according to the diffusion time step t. Typically, the larger the t, the smaller the noise. ↳ Add the Sampled Noise to latent features to obtain Noised Latent. ↳ The goal is to purposely add noise to a video and ask the model to guess what that noise is. ↳ This is analogous to training a language model by purposely deleting a word in a sentence and ask the model to guess what the deleted word was. [5-7] 🟪 Conditioning by Adaptive Layer Norm [5] Encode Conditions ↳ Encode "sora is sky" into a text embedding vector [0,1,-1]. ↳ Encode t = 3 to as a binary vector [1,1]. ↳ Concatenate the two vectors in to a 5D column vector. [6] Estimate Scale/Shift ↳ Multiply the combined vector with weights and biases ↳ The goal is to estimate the scale [2,-1] and shift [-1,5]. ↳ Copy the result to (X) and (+) [7] Apply Scale/Sift ↳ Scale the noised latent by [2,-1] ↳ Shifted the scaled noised latent by [-1, 5] ↳ The result is "conditioned" noise latent. [8-10] Transformer [8] Self-Attention ↳ Feed the conditioned noised latent to Query-Key function to obtain a self-attention matrix ↳ Value is omitted for simplicity [9] Attention Pooling ↳ Multiply the conditioned noised latent with the self-attention matrix ↳ The result are attention weighted features [10] Pointwise Feed Forward Network ↳ Multiply the attention weighted features with weights and biases ↳ The result is the Predicted Noise 🏋️♂️ 𝗧𝗿𝗮𝗶𝗻 [11] ↳ Calculate MSE loss gradients by taking the different between the Predicted Noise and the Sampled Noise (ground truth). ↳ Use the loss gradients to kick off backpropagation to update all learnable parameters (red borders) ↳ Note the visual encoder and decoder's parameters are frozen (blue borders) 🎨 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗲 (𝗦𝗮𝗺𝗽𝗹𝗲) [12] Denoise ↳ Subtract the predicted noise from the noised latent to obtain the noise-free latent [13] Visual Decoder: Latent 🟩 → Pixels 🟨 ↳ Multiply the patches with weights and biases, followed by ReLU [14] Patches → Video ↳ Rearrange patches into a sequence of video frames.](https://image.24vids.com/tw-1795449683285848509/ext_tw_video_thumb/1795449360802525185/pu/img/j-hppkMq667X9mjK.jpg)
![[RLHF] by Hand ✍️ Yesterday, Jan Leike (Jan Leike) announced he is joining #Anthropic to lead their "super-alignment" mission. He is the co-inventor of Reinforcement Learning with Human Feedback (#RLHF). How does RLHF work? [1] Given ↳ Reward Model (RM) ↳ Large Language Model (LLM) ↳ Two (Prompt, Next) Pairs 🟪 TRAIN RM Goal: Learn to give higher rewards to winners [2] Preferences ↳ A human reviews the two pairs and picks a "winner" ↳ (doc is, him) Embeddings ↳ This prompt has never received human feedback directly ↳ [S] is the special start symbol [11] Transformer ↳ Attention (yellow) ↳ Feed Forward (4x2 weight and bias matrix) ↳ Output: 3 "transformed" feature vector, one per position ↳ More details in my previous post 8. Transformer [] [12] Output Probabilities ↳ Apply a linear layer to map each transformed feature vector to a probability distribution over the vocabulary. [13] Sample ↳ Apply the greedy method, which is to pick the word with the highest score ↳ For output 1 and 2, the model accurately predicts the next word ↳ For 3rd output position, the model's predicts "him" [14] Reward Model ↳ The new pair (CEO is, him) is fed to the reward model ↳ The process is same as [3]-[6] ↳ Output: Reward = 3 [15] Loss Gradient ↳ We set the loss as the negative of the reward. ↳ The loss gradient is simply a constant -1. ↳ Run backpropagation and gradient descent to update LLM's weights and biases (red border)](https://image.24vids.com/tw-1795803665845899511/ext_tw_video_thumb/1795803020149534720/pu/img/jb8SBLT_1drHtYd_.jpg)
![RLHF by hand ✍️ ~ 15 steps walkthrough below Train a model on human text and it inherits human bias. It will assume a doctor is a "him", because the data says so. RLHF is the correction. A human marks one preference, doc is them over doc is him, and the weights move. But one correction is not the point. The hope is that the model learns the value behind it, gender neutrality, and applies it to professions nobody ever mentioned. How does it work? Goal: train a reward model from a single human comparison about doctors, then turn it on CEOs, filling in every cell yourself. = 1. Given = A reward model, an LLM, and two (prompt, next) pairs. = 2. Preferences = A human reads both pairs and picks a winner: (doc is, them) beats (doc is, him). The loser is not bad grammar, it is gender bias, and that is the whole signal. = 3. Word embeddings = Let us look up each word of the loser pair. These vectors are the reward model's input. = 4. Linear layer = We multiply by the reward model's weights and add its biases. Out come feature vectors, one per position. = 5. Mean pool = Let us multiply by [1/3, 1/3, 1/3], which averages the three positions into one sentence embedding. = 6. Output layer = We map that sentence down to a single number. Reward = 3. = 7. The winner, the same way = Let us repeat steps 3 to 6 on the winning pair. Reward = 5. = 8. Winner minus loser = We take the gap: 5 - 3 = 2. The reward model wants this positive and as large as it can make it. = 9. Loss gradient = Let us squash the gap into a probability, σ(2) ≈ 0.9, and subtract the target of 1. The gradient is -0.1, and it goes back through the purple weights. The reward model is now trained. = 10. A prompt it has never seen = We start the second half with "[S] CEO is". The feedback in step 2 was about doctors. Nothing connects a CEO to a doctor except what the reward model generalised. = 11. Transformer = Let us push it through attention and a feed forward layer, one vector per position. = 12. Output probabilities = We map each vector to a score over the vocabulary. = 13. Sample = Let us take the highest score. The model completes "CEO is" with "him", which is the same bias the human penalised in step 2. = 14. Score it with the reward model = We feed the new pair (CEO is, him) through steps 3 to 6. Reward = 3, exactly the score it gave "doc is him" in step 6. Nobody taught it about CEOs. The value transferred. = 15. Loss gradient = Let us set the loss to the negative of the reward, so minimising the loss maximises the reward. The gradient is a constant -1, and it goes back through the red weights. The outputs: Loser reward = 3, winner reward = 5 Reward gap = 2, predicted σ ≈ 0.9, reward model gradient = -0.1 LLM samples "him", reward = 3, LLM gradient = -1 Congrats! You just calculated RLHF by hand. And you watched a value generalise: one comparison about doctors, and the model marks down "CEO is him" unprompted. 💾 Save this post!](https://image.24vids.com/tw-2081745292051845420/ext_tw_video_thumb/2081745247344676864/pu/img/ahcUqHG6T2iBoOlI.jpg)
![U-Net by hand ✍️ ~ 17 steps walkthrough below I consider U-Net as a key milestone in deep learning, the first image-to-image model that really worked! It came out of medical imaging, an unusual place, not from NeurIPS or CVPR or ACL. Now it is the backbone of diffusion models, which you see in almost all modern image generation models. I drew the network as a C so the matrix multiplication flows naturally down. Tilt your head to the right and it is a U again. 🤣 Goal: push a 3 x 16 image down to a 2 x 4 bottleneck and back out again, filling in every cell yourself. = 1. Given = An image of three channels, R, G and B, sixteen pixels wide, and every kernel the network will use. = 2. Convolution 1 = Let us slide the first kernel over the image. Each output is one multiply-and-add over a 2 x 3 window, and the result is the green feature map. = 3. Find the maxima = We circle the largest value in each 1 x 2 window. Circling first is worth the extra step: it is the pooling decision, made before anything is written down. = 4. Max pool 1 = Let us copy those maxima down. Sixteen columns become eight, and half the detail is gone for good. = 5. Convolution 2 = We convolve again with the second kernel, deeper into the contracting path. The feature map is blue now. = 6. Find the maxima again = Same move as step 3, on the blue map. = 7. Max pool 2 = Eight columns become four. = 8. The bottleneck = Let us convolve once more. This is the bottom of the U, a 2 x 4 block that is everything the network kept. = 9. Spread it out = We start back up. The transposed convolution writes each bottleneck value into a wider grid, leaving gaps between them. = 10. Transposed convolution 1 = Let us fill those gaps by convolving over the spread-out grid. Four columns become eight. = 11. The first skip = We copy the encoder's matching row straight across. This is the skip connection, and it is the whole reason a U-Net can recover detail that pooling threw away. = 12. Convolution with the skip = Let us convolve the upsampled features together with the copied ones. = 13. Spread it out again = Same as step 9, one level up. = 14. Transposed convolution 2 = Eight columns become sixteen, back to the width we started at. = 15. The second skip = The encoder's first feature map comes across, the one made before any pooling happened. = 16. Convolution and ReLU = We convolve, then cross out every negative and set it to zero. = 17. Output convolution = Let us apply the last kernel. Out comes R', G' and B', an image the same size as the one we started with. The outputs: R' = [3, 0, 7, 0, 7, 0, 17, 0, 3, 0, 9, 0, 2, 0, 6, 0] G' = [1, 20, 1, 10, 1, 12, 1, 19, 2, 5, 1, 11, 1, 3, 1, 7] B' = [4, 20, 8, 10, 8, 12, 18, 19, 5, 5, 10, 11, 3, 3, 7, 7] Congrats! You just calculated a U-Net by hand. 💾 Save this post!](https://image.24vids.com/tw-2081382845243691256/ext_tw_video_thumb/2081382802667249664/pu/img/n2gZcnKRYe9G781o.jpg)
