Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

CLIP by hand ✍️ ~ 13 steps walkthrough below CLIP, Contrastive Language-Image Pre-training, is OpenAI's answer to a question that sounds impossible: how do you put a sentence and a picture in the same space? CLIP shipped when OpenAI was still open, and those embeddings were shared far and...

ProfTomYeh's profile picture

Tom Yeh

64,636 subscribers

20,523 görüntüleme • 7 gün önce •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

[CLIP] by Hand ✍️ The CLIP (Contrastive Language–Image Pre-training) model, a groundbreaking work by OpenAI, redefines the intersection of computer vision and natural language processing. It is the basis of all the multi-modal foundation models we see today. How does CLIP work? Goal: 🟨 Learn a shared embedding space for text and image [1] Given ↳ A mini batch of 3 text-image pairs ↳ OpenAI used 400 million text-image pairs to train its original CLIP model. Process 1st pair: "big table" [2] 🟪 Text → 2 Vectors (3D) ↳ Look up word embedding vectors using word2vec. [3] 🟩 Image → 2 Vectors (4D) ↳ Divide the image into two patches. ↳ Flatten each patch [4] Process other pairs ↳ Repeat [2]-[3] [5] 🟪 Text Encoder & 🟩 Image Encoder ↳ Encode input vectors into feature vectors ↳ Here, both encoders are simple one layer perceptron (linear + ReLU) ↳ In practice, the encoders are usually transformer models. [6] 🟪 🟩 Mean Pooling: 2 → 1 vector ↳ Average 2 feature vectors into a single vector by averaging across the columns ↳ The goal is to have one vector to represent each image or text [7] 🟪 🟩 -> 🟨 Projection ↳ Note that the text and image feature vectors from the encoders have different dimensions (3D vs. 4D). ↳ Use a linear layer to project image and text vectors to a 2D shared embedding space. 🏋️ Contrastive Pre-training 🏋️ [8] Prepare for MatMul ↳ Copy text vectors (T1,T2,T3) ↳ Copy the transpose of image vectors (I1,I2,I3) ↳ They are all in the 2D shared embedding space. [9] 🟦 MatMul ↳ Multiply T and I matrices. ↳ This is equivalent to taking dot product between every pair of image and text vectors. ↳ The purpose is to use dot product to estimate the similarity between a pair of image-text. [10] 🟦 Softmax: e^x ↳ Raise e to the power of the number in each cell ↳ To simplify hand calculation, we approximate e^□ with 3^□. [11] 🟦 Softmax: ∑ ↳ Sum each row for 🟩 image→🟪 text ↳ Sum each column for 🟪 text→ 🟩 image [12] 🟦 Softmax: 1 / sum ↳ Divide each element by the column sum to obtain a similarity matrix for 🟪 text→🟩 image ↳ Divide each element by the row sum to obtain a similarity matrix for 🟩 image→🟪 text [13] 🟥 Loss Gradients ↳ The "Targets" for the similarity matrices are Identity Matrices. ↳ Why? If I and T come from the same pair (i=j), we want the highest value, which is 1, and 0 otherwise. ↳ Apply the simple equation of [Similarity - Target] to compute gradients of for both directions. ↳ Why so simple? Because when Softmax and Cross-Entropy Loss are used together, the math magically works out that way. ↳ These gradients kick off the backpropagation process to update weights and biases of the encoders and projection layers (red borders).
1:20

[CLIP] by Hand ✍️ The CLIP (Contrastive Language–Image Pre-training) model, a groundbreaking work by OpenAI, redefines the intersection of computer vision and natural language processing. It is the basis of all the multi-modal foundation models we see today. How does CLIP work? Goal: 🟨 Learn a shared embedding space for text and image [1] Given ↳ A mini batch of 3 text-image pairs ↳ OpenAI used 400 million text-image pairs to train its original CLIP model. Process 1st pair: "big table" [2] 🟪 Text → 2 Vectors (3D) ↳ Look up word embedding vectors using word2vec. [3] 🟩 Image → 2 Vectors (4D) ↳ Divide the image into two patches. ↳ Flatten each patch [4] Process other pairs ↳ Repeat [2]-[3] [5] 🟪 Text Encoder & 🟩 Image Encoder ↳ Encode input vectors into feature vectors ↳ Here, both encoders are simple one layer perceptron (linear + ReLU) ↳ In practice, the encoders are usually transformer models. [6] 🟪 🟩 Mean Pooling: 2 → 1 vector ↳ Average 2 feature vectors into a single vector by averaging across the columns ↳ The goal is to have one vector to represent each image or text [7] 🟪 🟩 -> 🟨 Projection ↳ Note that the text and image feature vectors from the encoders have different dimensions (3D vs. 4D). ↳ Use a linear layer to project image and text vectors to a 2D shared embedding space. 🏋️ Contrastive Pre-training 🏋️ [8] Prepare for MatMul ↳ Copy text vectors (T1,T2,T3) ↳ Copy the transpose of image vectors (I1,I2,I3) ↳ They are all in the 2D shared embedding space. [9] 🟦 MatMul ↳ Multiply T and I matrices. ↳ This is equivalent to taking dot product between every pair of image and text vectors. ↳ The purpose is to use dot product to estimate the similarity between a pair of image-text. [10] 🟦 Softmax: e^x ↳ Raise e to the power of the number in each cell ↳ To simplify hand calculation, we approximate e^□ with 3^□. [11] 🟦 Softmax: ∑ ↳ Sum each row for 🟩 image→🟪 text ↳ Sum each column for 🟪 text→ 🟩 image [12] 🟦 Softmax: 1 / sum ↳ Divide each element by the column sum to obtain a similarity matrix for 🟪 text→🟩 image ↳ Divide each element by the row sum to obtain a similarity matrix for 🟩 image→🟪 text [13] 🟥 Loss Gradients ↳ The "Targets" for the similarity matrices are Identity Matrices. ↳ Why? If I and T come from the same pair (i=j), we want the highest value, which is 1, and 0 otherwise. ↳ Apply the simple equation of [Similarity - Target] to compute gradients of for both directions. ↳ Why so simple? Because when Softmax and Cross-Entropy Loss are used together, the math magically works out that way. ↳ These gradients kick off the backpropagation process to update weights and biases of the encoders and projection layers (red borders).

Tom Yeh

67,858 görüntüleme • 2 yıl önce

Vector Database by hand ✍️ ~ 10 steps walkthrough below Vector databases are the backbone of Retrieval Augmented Generation (RAG). How do they actually work? Goal: index three sentences, then answer a query by finding the nearest one, filling in every cell yourself. = 1. Given = A dataset of three sentences, three words each. In practice it is millions of them. = 2. Word embeddings = Let us look up each word in an embedding table. Here the vocabulary is 22 words; in practice it is tens of thousands, and the vectors have thousands of dimensions rather than four. = 3. Encoding = We feed the sequence to an encoder, one linear layer and a ReLU, and get one feature vector per word. In practice the encoder is a transformer. = 4. Mean pooling = Let us average across the columns. Three word vectors collapse into one, which is what people mean by a text embedding or a sentence embedding. = 5. Indexing = We multiply by a projection matrix and the four dimensions become two. It is doing the job of a hash: a short representation that is faster to compare, and it is what gets saved in the vector storage. = 6. Process "who are you" = Let us repeat steps 2 to 5 on the second sentence. = 7. Process "who am I" = We do it a third time. The database is now indexed. = 8. Query "am I you" = Let us push the query through the very same pipeline: lookup, encoder, mean pooling, projection, and it lands as a 2D vector in the same space. = 9. Dot products = We transpose the query and multiply, which takes the dot product against every stored vector at once. The dot product is the estimate of similarity. = 10. Nearest neighbour = Let us scan for the largest: 60/9 beats 44/9 and 40/9, so the answer is "who am I". Scanning billions of vectors one at a time is what makes this the slow step in practice, which is why real databases use an approximate nearest neighbour index like HNSW. The outputs: Stored index vectors = [5/3, 2/3], [5/3, 0], [7/3, 2/3] Query vector = [8/3, 2/3] Dot products = 44/9, 40/9, 60/9 Nearest neighbour = "who am I" The takeaway: a vector database is an embedding pipeline, a projection, and a dot product. Every step here is arithmetic you can do in pen, which is worth remembering when the word "database" makes it sound like something else. 💾 Save this post!
0:31

Vector Database by hand ✍️ ~ 10 steps walkthrough below Vector databases are the backbone of Retrieval Augmented Generation (RAG). How do they actually work? Goal: index three sentences, then answer a query by finding the nearest one, filling in every cell yourself. = 1. Given = A dataset of three sentences, three words each. In practice it is millions of them. = 2. Word embeddings = Let us look up each word in an embedding table. Here the vocabulary is 22 words; in practice it is tens of thousands, and the vectors have thousands of dimensions rather than four. = 3. Encoding = We feed the sequence to an encoder, one linear layer and a ReLU, and get one feature vector per word. In practice the encoder is a transformer. = 4. Mean pooling = Let us average across the columns. Three word vectors collapse into one, which is what people mean by a text embedding or a sentence embedding. = 5. Indexing = We multiply by a projection matrix and the four dimensions become two. It is doing the job of a hash: a short representation that is faster to compare, and it is what gets saved in the vector storage. = 6. Process "who are you" = Let us repeat steps 2 to 5 on the second sentence. = 7. Process "who am I" = We do it a third time. The database is now indexed. = 8. Query "am I you" = Let us push the query through the very same pipeline: lookup, encoder, mean pooling, projection, and it lands as a 2D vector in the same space. = 9. Dot products = We transpose the query and multiply, which takes the dot product against every stored vector at once. The dot product is the estimate of similarity. = 10. Nearest neighbour = Let us scan for the largest: 60/9 beats 44/9 and 40/9, so the answer is "who am I". Scanning billions of vectors one at a time is what makes this the slow step in practice, which is why real databases use an approximate nearest neighbour index like HNSW. The outputs: Stored index vectors = [5/3, 2/3], [5/3, 0], [7/3, 2/3] Query vector = [8/3, 2/3] Dot products = 44/9, 40/9, 60/9 Nearest neighbour = "who am I" The takeaway: a vector database is an embedding pipeline, a projection, and a dot product. Every step here is arithmetic you can do in pen, which is worth remembering when the word "database" makes it sound like something else. 💾 Save this post!

Tom Yeh

35,755 görüntüleme • 14 gün önce

[Self-Attention] by Hand ✍️ Self-attention is what enables LLMs to understand context. How does it work? This exercise demonstrates how to calculate a 6-3 attention head by hand. Note that if we have two instances of this, we get 6-6 attention (i.e., multi-head attention, n=2). -- 𝗚𝗼𝗮𝗹 -- Transform [6D Features 🟧] to [3D Attention Weighted Features 🟦] -- 𝗪𝗮𝗹𝗸𝘁𝗵𝗿𝗼𝘂𝗴𝗵 -- [1] Given ↳ A set of 4 feature vectors (6-D): x1,x2,x3,x4 [2] Query, Key, Value ↳ Multiply features x's with linear transformation matrices WQ, WK, and WV, to obtain query vectors (q1,q2,q3,q4), key vectors (k1,k2,k3,k4), and value vectors (v1,v2,v3,v4). ↳ "Self" refers to the fact that both queries and keys are derived from the same set of features. [3] 🟪 Prepare for MatMul ↳ Copy query vectors ↳ Copy the transpose of key vectors [4] 🟪 MatMul ↳ Multiply K^T and Q ↳ This is equivalent to taking dot product between every pair of query and key vectors. ↳ The purpose is to use dot product as an estimate of the "matching score" between every key-value pair. ↳ This estimate makes sense because dot product is the numerator of Cosine Similarity between two vectors. [5] 🟨 Scale ↳ Scale each element by the square root of dk, which is the dimension of key vectors (dk=3). ↳ The purpose is to normalize the impact of the dk on matching scores, even if we scale dk to 32, 64, or 128. ↳ To simplify hand calculation, we approximate [ □/sqrt(3) ] with [ floor(□/2) ]. [6] 🟩 Softmax: e^x ↳ Raise e to the power of the number in each cell ↳ To simplify hand calculation, we approximate e^□ with 3^□. [7] 🟩 Softmax: ∑ ↳ Sum across each column [8] 🟩 Softmax: 1 / sum ↳ For each column, divide each element by the column sum ↳ The purpose is normalize each column so that the numbers sum to 1. In other words, each column is a probability distribution of attention, and we have four of them. ↳ The result is the Attention Weight Matrix (A) (yellow) [9] 🟦 MatMul ↳ Multiply the value vectors (Vs) with the Attention Weight Matrix (A) ↳ The results are the attention weighted features Zs. ↳ They are fed to the position-wise feed forward network in the next layer.
0:32

[Self-Attention] by Hand ✍️ Self-attention is what enables LLMs to understand context. How does it work? This exercise demonstrates how to calculate a 6-3 attention head by hand. Note that if we have two instances of this, we get 6-6 attention (i.e., multi-head attention, n=2). -- 𝗚𝗼𝗮𝗹 -- Transform [6D Features 🟧] to [3D Attention Weighted Features 🟦] -- 𝗪𝗮𝗹𝗸𝘁𝗵𝗿𝗼𝘂𝗴𝗵 -- [1] Given ↳ A set of 4 feature vectors (6-D): x1,x2,x3,x4 [2] Query, Key, Value ↳ Multiply features x's with linear transformation matrices WQ, WK, and WV, to obtain query vectors (q1,q2,q3,q4), key vectors (k1,k2,k3,k4), and value vectors (v1,v2,v3,v4). ↳ "Self" refers to the fact that both queries and keys are derived from the same set of features. [3] 🟪 Prepare for MatMul ↳ Copy query vectors ↳ Copy the transpose of key vectors [4] 🟪 MatMul ↳ Multiply K^T and Q ↳ This is equivalent to taking dot product between every pair of query and key vectors. ↳ The purpose is to use dot product as an estimate of the "matching score" between every key-value pair. ↳ This estimate makes sense because dot product is the numerator of Cosine Similarity between two vectors. [5] 🟨 Scale ↳ Scale each element by the square root of dk, which is the dimension of key vectors (dk=3). ↳ The purpose is to normalize the impact of the dk on matching scores, even if we scale dk to 32, 64, or 128. ↳ To simplify hand calculation, we approximate [ □/sqrt(3) ] with [ floor(□/2) ]. [6] 🟩 Softmax: e^x ↳ Raise e to the power of the number in each cell ↳ To simplify hand calculation, we approximate e^□ with 3^□. [7] 🟩 Softmax: ∑ ↳ Sum across each column [8] 🟩 Softmax: 1 / sum ↳ For each column, divide each element by the column sum ↳ The purpose is normalize each column so that the numbers sum to 1. In other words, each column is a probability distribution of attention, and we have four of them. ↳ The result is the Attention Weight Matrix (A) (yellow) [9] 🟦 MatMul ↳ Multiply the value vectors (Vs) with the Attention Weight Matrix (A) ↳ The results are the attention weighted features Zs. ↳ They are fed to the position-wise feed forward network in the next layer.

Tom Yeh

101,010 görüntüleme • 2 yıl önce

Self Attention by hand ✍️ ~ 9 steps walkthrough below Self-attention is what enables LLMs to understand context. How does it work? So I drew and calculated one entirely by hand. Goal: turn four 6D features into four 3D attention weighted features, filling in every cell yourself. = 1. Given = Four feature vectors, six dimensions each, one per position. = 2. Query, key, value = Let us multiply the features by WQ, WK and WV. Queries, keys and values all come out of the same four features, and that is what the word "self" is doing in self-attention. = 3. Prepare for MatMul = We copy the queries across the top and the transposed keys down the side. Lining the two up is half the work. = 4. MatMul = Let us multiply K transpose by Q. Every cell is the dot product of one key with one query, which we use as a matching score. That works because the dot product is the numerator of cosine similarity: it is how alike two vectors are, before anyone divides by their lengths. = 5. Scale = We divide by the square root of dk, the dimension of a key vector, here 3. Without it the scores grow with the dimension and a 64-wide head would swamp the softmax. To keep the page doable in pen, the drawing approximates dividing by root 3 with halving. = 6. e to the power = Let us raise e to the power of each score. This is the first half of softmax, and the drawing uses 3 in place of e, which is close enough to do in your head. = 7. Sum = We add up each column: 16, 6, 7 and 12. = 8. Normalize = Let us divide every cell by its column sum. That gives the attention weight matrix in yellow, and each of its four columns is now a probability distribution over the four positions. The decimals are nudged as they are rounded, so every column still sums to exactly 1. = 9. MatMul = We multiply the value vectors by those weights. Each output is a blend of all four values, mixed in the proportion the attention matrix just decided, and it goes to the position-wise feed forward network in the next layer: the FFN box at the bottom of the page. The outputs: Attention weights (A), by column = [.2, .6, 0, .2], [.2, .4, .2, .2], [.4, .2, 0, .4], [.1, .7, .1, .1] Attention weighted features (Z) = [8, 2, 6], [8, 4, 4], [16, 4, 2], [4, 2, 7] The takeaway: attention is a weighted average, and everything before step 9 exists to decide the weights. Compare every position with every other, turn the scores into one distribution per position, then blend. 💾 Save this post!
0:22

Self Attention by hand ✍️ ~ 9 steps walkthrough below Self-attention is what enables LLMs to understand context. How does it work? So I drew and calculated one entirely by hand. Goal: turn four 6D features into four 3D attention weighted features, filling in every cell yourself. = 1. Given = Four feature vectors, six dimensions each, one per position. = 2. Query, key, value = Let us multiply the features by WQ, WK and WV. Queries, keys and values all come out of the same four features, and that is what the word "self" is doing in self-attention. = 3. Prepare for MatMul = We copy the queries across the top and the transposed keys down the side. Lining the two up is half the work. = 4. MatMul = Let us multiply K transpose by Q. Every cell is the dot product of one key with one query, which we use as a matching score. That works because the dot product is the numerator of cosine similarity: it is how alike two vectors are, before anyone divides by their lengths. = 5. Scale = We divide by the square root of dk, the dimension of a key vector, here 3. Without it the scores grow with the dimension and a 64-wide head would swamp the softmax. To keep the page doable in pen, the drawing approximates dividing by root 3 with halving. = 6. e to the power = Let us raise e to the power of each score. This is the first half of softmax, and the drawing uses 3 in place of e, which is close enough to do in your head. = 7. Sum = We add up each column: 16, 6, 7 and 12. = 8. Normalize = Let us divide every cell by its column sum. That gives the attention weight matrix in yellow, and each of its four columns is now a probability distribution over the four positions. The decimals are nudged as they are rounded, so every column still sums to exactly 1. = 9. MatMul = We multiply the value vectors by those weights. Each output is a blend of all four values, mixed in the proportion the attention matrix just decided, and it goes to the position-wise feed forward network in the next layer: the FFN box at the bottom of the page. The outputs: Attention weights (A), by column = [.2, .6, 0, .2], [.2, .4, .2, .2], [.4, .2, 0, .4], [.1, .7, .1, .1] Attention weighted features (Z) = [8, 2, 6], [8, 4, 4], [16, 4, 2], [4, 2, 7] The takeaway: attention is a weighted average, and everything before step 9 exists to decide the weights. Compare every position with every other, turn the scores into one distribution per position, then blend. 💾 Save this post!

Tom Yeh

27,049 görüntüleme • 17 gün önce

SVM by hand ✍️ ~ 19 steps walkthrough below (Linear vs RBF) Support Vector Machines reigned supreme in machine learning before the deep learning revolution. An SVM predicts with dot products, the same matrix multiplication every model uses. What it does not do is train by backpropagation: it is fitted by convex optimization, so there is no matrix-multiplication backward pass for a GPU to accelerate. I drew and calculated two SVMs by hand: a linear one (top) and an RBF one (bottom), classifying the same two test vectors. Goal: turn six training vectors and their learned coefficients into a prediction, and see what changing the kernel actually changes. = 1. Given = Six training vectors, their labels, and the coefficients and bias already learned. A coefficient of zero means that vector is not a support vector: too far from the boundary to matter. = 2. Linear kernel, test vector 1 = Let us take the dot product of the test vector with every training vector. The dot product stands in for cosine similarity, and the column of results is the first column of the kernel matrix K. = 3. Linear kernel, test vector 2 = We do the same for the second, and K is complete. = 4. Signed weights = Let us multiply each coefficient by its label. The second training vector drops out here, because its coefficient is 0. = 5. Weighted combination = We multiply the signed weights through K and add the bias b. The result is a signed distance to the decision boundary: 17 and 5. = 6. Classify = Let us take the sign. Both are positive. = 7 to 11. RBF kernel, test vector 1 = Now the same picture with a different kernel, in five moves: square the differences, sum them, take the square root for the L2 distance, multiply by minus gamma, and raise e to that power. The negation is what turns a distance into a similarity, and gamma controls how far a single training vector's influence reaches. = 12 to 16. RBF kernel, test vector 2 = We repeat all five. The numbers change, the moves do not. = 17 to 19. Decision boundary, again = Signed weights, weighted combination, sign. Identical arithmetic to steps 4 through 6, on a K that was built a completely different way. The outputs: Linear K, first column = [13, 25, 12, 15, 19, 27] Linear decision values = 17 and 5, both positive RBF decision values = -2 and 1, so negative and positive The takeaway: the kernel is the only thing that changed, and it changed the answer. The linear SVM calls both test vectors positive; the RBF one splits them. Everything after the kernel matrix, the signed weights and the weighted combination and the sign, is the same page of arithmetic twice. 💾 Save this post!
0:44

SVM by hand ✍️ ~ 19 steps walkthrough below (Linear vs RBF) Support Vector Machines reigned supreme in machine learning before the deep learning revolution. An SVM predicts with dot products, the same matrix multiplication every model uses. What it does not do is train by backpropagation: it is fitted by convex optimization, so there is no matrix-multiplication backward pass for a GPU to accelerate. I drew and calculated two SVMs by hand: a linear one (top) and an RBF one (bottom), classifying the same two test vectors. Goal: turn six training vectors and their learned coefficients into a prediction, and see what changing the kernel actually changes. = 1. Given = Six training vectors, their labels, and the coefficients and bias already learned. A coefficient of zero means that vector is not a support vector: too far from the boundary to matter. = 2. Linear kernel, test vector 1 = Let us take the dot product of the test vector with every training vector. The dot product stands in for cosine similarity, and the column of results is the first column of the kernel matrix K. = 3. Linear kernel, test vector 2 = We do the same for the second, and K is complete. = 4. Signed weights = Let us multiply each coefficient by its label. The second training vector drops out here, because its coefficient is 0. = 5. Weighted combination = We multiply the signed weights through K and add the bias b. The result is a signed distance to the decision boundary: 17 and 5. = 6. Classify = Let us take the sign. Both are positive. = 7 to 11. RBF kernel, test vector 1 = Now the same picture with a different kernel, in five moves: square the differences, sum them, take the square root for the L2 distance, multiply by minus gamma, and raise e to that power. The negation is what turns a distance into a similarity, and gamma controls how far a single training vector's influence reaches. = 12 to 16. RBF kernel, test vector 2 = We repeat all five. The numbers change, the moves do not. = 17 to 19. Decision boundary, again = Signed weights, weighted combination, sign. Identical arithmetic to steps 4 through 6, on a K that was built a completely different way. The outputs: Linear K, first column = [13, 25, 12, 15, 19, 27] Linear decision values = 17 and 5, both positive RBF decision values = -2 and 1, so negative and positive The takeaway: the kernel is the only thing that changed, and it changed the answer. The linear SVM calls both test vectors positive; the RBF one splits them. Everything after the kernel matrix, the signed weights and the weighted combination and the sign, is the same page of arithmetic twice. 💾 Save this post!

Tom Yeh

16,916 görüntüleme • 15 gün önce

ResNet by hand ✍️ ~ 10 steps walkthrough below "Deep Residual Learning for Image Recognition" (Kaiming He, CVPR 2016) is among the most cited papers in all of deep learning. Why does it matter so much? It fixed the exploding and vanishing gradients that kept deep networks from being deep, and made thousands of layers possible. How simple was the fix? An identity matrix. Goal: push three input vectors through a residual block, then through a transformer encoder block, filling in every cell yourself. = 1. Given = A mini batch of three input vectors, 3D, and the weights of the layers ahead. = 2. Linear layer = Let us multiply by the weights, add the bias, and apply ReLU so negatives become 0. Three feature vectors out. This is F(X). = 3. Concatenate = Now the trick. Stack an identity matrix beside the second layer's weights, and stack the input vectors under the features. Draw the lines between rows and columns: those are the skip connections. The identity is the residual. = 4. Linear layer + identity = We multiply the two stacked matrices. The identity carries X straight through while the weights transform it, so a single multiplication computes F(X) + X. Apply ReLU and hand it to the next block. Now watch the same trick inside a transformer, first in attention. = 5. Attention = Let us take three input vectors in 2D, compute the attention matrix, and multiply to get attention weighted vectors. = 6. Concatenate = We stack two identities this time, two residuals, which is how you get 1 + 1, and stack the input vectors with the attention weighted ones. = 7. Add = Multiply the stacked matrices. The identity adds attention to its own input, across the columns, which is how positions get combined. And again in the feed forward layer. = 8. First layer = Let us multiply by the feed forward weights and bias, then ReLU. Three feature vectors. = 9. Concatenate = Stack and link exactly as in step 3: the residual again. = 10. Second layer + identity = We multiply, apply ReLU, and pass the result to the next encoder block. This identity adds across the rows, combining features rather than positions. Takeaway: one simple "add" is what made really deep networks possible. 💾 Save this post!
0:27

ResNet by hand ✍️ ~ 10 steps walkthrough below "Deep Residual Learning for Image Recognition" (Kaiming He, CVPR 2016) is among the most cited papers in all of deep learning. Why does it matter so much? It fixed the exploding and vanishing gradients that kept deep networks from being deep, and made thousands of layers possible. How simple was the fix? An identity matrix. Goal: push three input vectors through a residual block, then through a transformer encoder block, filling in every cell yourself. = 1. Given = A mini batch of three input vectors, 3D, and the weights of the layers ahead. = 2. Linear layer = Let us multiply by the weights, add the bias, and apply ReLU so negatives become 0. Three feature vectors out. This is F(X). = 3. Concatenate = Now the trick. Stack an identity matrix beside the second layer's weights, and stack the input vectors under the features. Draw the lines between rows and columns: those are the skip connections. The identity is the residual. = 4. Linear layer + identity = We multiply the two stacked matrices. The identity carries X straight through while the weights transform it, so a single multiplication computes F(X) + X. Apply ReLU and hand it to the next block. Now watch the same trick inside a transformer, first in attention. = 5. Attention = Let us take three input vectors in 2D, compute the attention matrix, and multiply to get attention weighted vectors. = 6. Concatenate = We stack two identities this time, two residuals, which is how you get 1 + 1, and stack the input vectors with the attention weighted ones. = 7. Add = Multiply the stacked matrices. The identity adds attention to its own input, across the columns, which is how positions get combined. And again in the feed forward layer. = 8. First layer = Let us multiply by the feed forward weights and bias, then ReLU. Three feature vectors. = 9. Concatenate = Stack and link exactly as in step 3: the residual again. = 10. Second layer + identity = We multiply, apply ReLU, and pass the result to the next encoder block. This identity adds across the rows, combining features rather than positions. Takeaway: one simple "add" is what made really deep networks possible. 💾 Save this post!

Tom Yeh

17,934 görüntüleme • 11 gün önce

RLHF by hand ✍️ ~ 15 steps walkthrough below Train a model on human text and it inherits human bias. It will assume a doctor is a "him", because the data says so. RLHF is the correction. A human marks one preference, doc is them over doc is him, and the weights move. But one correction is not the point. The hope is that the model learns the value behind it, gender neutrality, and applies it to professions nobody ever mentioned. How does it work? Goal: train a reward model from a single human comparison about doctors, then turn it on CEOs, filling in every cell yourself. = 1. Given = A reward model, an LLM, and two (prompt, next) pairs. = 2. Preferences = A human reads both pairs and picks a winner: (doc is, them) beats (doc is, him). The loser is not bad grammar, it is gender bias, and that is the whole signal. = 3. Word embeddings = Let us look up each word of the loser pair. These vectors are the reward model's input. = 4. Linear layer = We multiply by the reward model's weights and add its biases. Out come feature vectors, one per position. = 5. Mean pool = Let us multiply by [1/3, 1/3, 1/3], which averages the three positions into one sentence embedding. = 6. Output layer = We map that sentence down to a single number. Reward = 3. = 7. The winner, the same way = Let us repeat steps 3 to 6 on the winning pair. Reward = 5. = 8. Winner minus loser = We take the gap: 5 - 3 = 2. The reward model wants this positive and as large as it can make it. = 9. Loss gradient = Let us squash the gap into a probability, σ(2) ≈ 0.9, and subtract the target of 1. The gradient is -0.1, and it goes back through the purple weights. The reward model is now trained. = 10. A prompt it has never seen = We start the second half with "[S] CEO is". The feedback in step 2 was about doctors. Nothing connects a CEO to a doctor except what the reward model generalised. = 11. Transformer = Let us push it through attention and a feed forward layer, one vector per position. = 12. Output probabilities = We map each vector to a score over the vocabulary. = 13. Sample = Let us take the highest score. The model completes "CEO is" with "him", which is the same bias the human penalised in step 2. = 14. Score it with the reward model = We feed the new pair (CEO is, him) through steps 3 to 6. Reward = 3, exactly the score it gave "doc is him" in step 6. Nobody taught it about CEOs. The value transferred. = 15. Loss gradient = Let us set the loss to the negative of the reward, so minimising the loss maximises the reward. The gradient is a constant -1, and it goes back through the red weights. The outputs: Loser reward = 3, winner reward = 5 Reward gap = 2, predicted σ ≈ 0.9, reward model gradient = -0.1 LLM samples "him", reward = 3, LLM gradient = -1 Congrats! You just calculated RLHF by hand. And you watched a value generalise: one comparison about doctors, and the model marks down "CEO is him" unprompted. 💾 Save this post!
0:37

RLHF by hand ✍️ ~ 15 steps walkthrough below Train a model on human text and it inherits human bias. It will assume a doctor is a "him", because the data says so. RLHF is the correction. A human marks one preference, doc is them over doc is him, and the weights move. But one correction is not the point. The hope is that the model learns the value behind it, gender neutrality, and applies it to professions nobody ever mentioned. How does it work? Goal: train a reward model from a single human comparison about doctors, then turn it on CEOs, filling in every cell yourself. = 1. Given = A reward model, an LLM, and two (prompt, next) pairs. = 2. Preferences = A human reads both pairs and picks a winner: (doc is, them) beats (doc is, him). The loser is not bad grammar, it is gender bias, and that is the whole signal. = 3. Word embeddings = Let us look up each word of the loser pair. These vectors are the reward model's input. = 4. Linear layer = We multiply by the reward model's weights and add its biases. Out come feature vectors, one per position. = 5. Mean pool = Let us multiply by [1/3, 1/3, 1/3], which averages the three positions into one sentence embedding. = 6. Output layer = We map that sentence down to a single number. Reward = 3. = 7. The winner, the same way = Let us repeat steps 3 to 6 on the winning pair. Reward = 5. = 8. Winner minus loser = We take the gap: 5 - 3 = 2. The reward model wants this positive and as large as it can make it. = 9. Loss gradient = Let us squash the gap into a probability, σ(2) ≈ 0.9, and subtract the target of 1. The gradient is -0.1, and it goes back through the purple weights. The reward model is now trained. = 10. A prompt it has never seen = We start the second half with "[S] CEO is". The feedback in step 2 was about doctors. Nothing connects a CEO to a doctor except what the reward model generalised. = 11. Transformer = Let us push it through attention and a feed forward layer, one vector per position. = 12. Output probabilities = We map each vector to a score over the vocabulary. = 13. Sample = Let us take the highest score. The model completes "CEO is" with "him", which is the same bias the human penalised in step 2. = 14. Score it with the reward model = We feed the new pair (CEO is, him) through steps 3 to 6. Reward = 3, exactly the score it gave "doc is him" in step 6. Nobody taught it about CEOs. The value transferred. = 15. Loss gradient = Let us set the loss to the negative of the reward, so minimising the loss maximises the reward. The gradient is a constant -1, and it goes back through the red weights. The outputs: Loser reward = 3, winner reward = 5 Reward gap = 2, predicted σ ≈ 0.9, reward model gradient = -0.1 LLM samples "him", reward = 3, LLM gradient = -1 Congrats! You just calculated RLHF by hand. And you watched a value generalise: one comparison about doctors, and the model marks down "CEO is him" unprompted. 💾 Save this post!

Tom Yeh

22,034 görüntüleme • 12 gün önce

Generative Adversarial Network (GAN) by hand ✍️ ~ 9 steps walkthrough below The Gen in GenAI came from this landmark paper by Ian Goodfellow et al., 12 years ago. The paper showed that a neural network can not only classify but also turn upside down to generate realistic looking images. The secret? We pit two of them against each other: a Generator turns noise into fake data, and a Discriminator learns to tell fake from real, pushing the Generator to keep doing better. One runs upside down, the other right way up. I drew and calculated one entirely by hand. Goal: generate realistic 4D data out of 2D noise, filling in every cell yourself. = 1. Given = Four noise vectors in 2D, and four real data vectors in 4D. = 2. Generator, first layer = Let us multiply the noise by weights and biases to get new features. = 3. ReLU = We apply the activation, and -1 and -2 are crossed out and set to 0. = 4. Generator, second layer = Let us multiply again. ReLU applies here too, but every value is already positive, so nothing changes. What comes out is the fake data F, made by a two-layer generator out of nothing but noise. = 5. Discriminator, first layer = We feed it both, the four fakes and the four real vectors, through the same weights. It never learns which is which from the layout, only from the numbers. = 6. Discriminator, second layer = Let us reduce each data vector to a single feature Z. Eight vectors in, eight numbers out. = 7. Sigmoid = We turn each Z into a probability Y. A 1 means the discriminator is certain the data is real, a 0 means certain it is fake. = 8. Training the Discriminator = Let us take the gradients as Y minus YD, where YD is what the discriminator should have said: 0 for the four fakes, 1 for the four real. Why so simple? Because pairing sigmoid with binary cross entropy loss makes the math collapse to exactly this subtraction. Its loss uses both halves of the page. = 9. Training the Generator = We do it again, as Y minus YG, and YG is [1, 1, 1, 1]: the generator wants the discriminator to call every fake real. Same predictions, different target, opposite goal. Its loss uses only the fakes. The outputs: Fake data F = [1, 2, 3, 1], [1, 1, 2, 1], [2, 2, 4, 2], [1, 0, 1, 1] Predictions on fakes = [.7, .5, .9, .3] Predictions on real = [.7, .9, .9, 1] Discriminator gradients = [.7, .5, .9, .3] and [-.3, -.1, -.1, 0] Generator gradients = [-.3, -.5, -.1, -.7] The takeaway: the adversarial part is one subtraction done twice. The same eight predictions, scored against two opposite targets, send one set of gradients back through the blue weights and another back through the green ones. 💾 Save this post!
0:23

Generative Adversarial Network (GAN) by hand ✍️ ~ 9 steps walkthrough below The Gen in GenAI came from this landmark paper by Ian Goodfellow et al., 12 years ago. The paper showed that a neural network can not only classify but also turn upside down to generate realistic looking images. The secret? We pit two of them against each other: a Generator turns noise into fake data, and a Discriminator learns to tell fake from real, pushing the Generator to keep doing better. One runs upside down, the other right way up. I drew and calculated one entirely by hand. Goal: generate realistic 4D data out of 2D noise, filling in every cell yourself. = 1. Given = Four noise vectors in 2D, and four real data vectors in 4D. = 2. Generator, first layer = Let us multiply the noise by weights and biases to get new features. = 3. ReLU = We apply the activation, and -1 and -2 are crossed out and set to 0. = 4. Generator, second layer = Let us multiply again. ReLU applies here too, but every value is already positive, so nothing changes. What comes out is the fake data F, made by a two-layer generator out of nothing but noise. = 5. Discriminator, first layer = We feed it both, the four fakes and the four real vectors, through the same weights. It never learns which is which from the layout, only from the numbers. = 6. Discriminator, second layer = Let us reduce each data vector to a single feature Z. Eight vectors in, eight numbers out. = 7. Sigmoid = We turn each Z into a probability Y. A 1 means the discriminator is certain the data is real, a 0 means certain it is fake. = 8. Training the Discriminator = Let us take the gradients as Y minus YD, where YD is what the discriminator should have said: 0 for the four fakes, 1 for the four real. Why so simple? Because pairing sigmoid with binary cross entropy loss makes the math collapse to exactly this subtraction. Its loss uses both halves of the page. = 9. Training the Generator = We do it again, as Y minus YG, and YG is [1, 1, 1, 1]: the generator wants the discriminator to call every fake real. Same predictions, different target, opposite goal. Its loss uses only the fakes. The outputs: Fake data F = [1, 2, 3, 1], [1, 1, 2, 1], [2, 2, 4, 2], [1, 0, 1, 1] Predictions on fakes = [.7, .5, .9, .3] Predictions on real = [.7, .9, .9, 1] Discriminator gradients = [.7, .5, .9, .3] and [-.3, -.1, -.1, 0] Generator gradients = [-.3, -.5, -.1, -.7] The takeaway: the adversarial part is one subtraction done twice. The same eight predictions, scored against two opposite targets, send one set of gradients back through the blue weights and another back through the green ones. 💾 Save this post!

Tom Yeh

16,566 görüntüleme • 16 gün önce

We've officially released and open-sourced HunyuanImage 2.1, our latest text-to-image model. The new model delivers on our commitment to balancing performance and quality. With native 2K image generation, HunyuanImage 2.1 is an advanced open-source text-to-image model.🎨 ✨ New in 2.1: 🔹Advanced Semantics: Supports ultra-long and complex prompts of up to 1000 tokens, and precisely controls the generation of multiple subjects in a single image. 🔹Precise Chinese and English Text Rendering with seamless image–text integration: The model naturally integrates text into images, making it suitable for a wide range of applications such as product covers, illustrations, and poster design to meet the needs of various fields. 🔹Rich Styles and High Aesthetic: Capable of generating images in various styles—including photorealistic portraits, comics, and vinyl figures—it delivers outstanding visual appeal and artistic quality. 🔹High-Quality Generation: Efficiently produces ultra-high-definition (2K) images in the same time other models take to generate a 1K image. HunyuanImage 2.1 uses two text encoders: a multimodal large language model (MLLM) to improve the model's image and text alignment capabilities, and a multi-language character-aware encoder to improve text rendering capabilities. The model is a single- and double-stream diffusion transformer with 17B parameters. We've also open-sourced the weights of the the accelerated version with meanflow which reduces inference steps from 100 to just 8, and PromptEnhancer, the first industrial-grade rewriting model that enhances your prompts for more nuanced and expressive image generation. Now, creators turn complex ideas—like posters with slogans or multi-panel comics—into visuals faster than ever. We’re just getting started. Stay tuned for our native multimodal image generation model coming soon. 🌐Website: 🔗Github: 🤗Hugging Face: ✨Hugging Face Demo:
1:39

We've officially released and open-sourced HunyuanImage 2.1, our latest text-to-image model. The new model delivers on our commitment to balancing performance and quality. With native 2K image generation, HunyuanImage 2.1 is an advanced open-source text-to-image model.🎨 ✨ New in 2.1: 🔹Advanced Semantics: Supports ultra-long and complex prompts of up to 1000 tokens, and precisely controls the generation of multiple subjects in a single image. 🔹Precise Chinese and English Text Rendering with seamless image–text integration: The model naturally integrates text into images, making it suitable for a wide range of applications such as product covers, illustrations, and poster design to meet the needs of various fields. 🔹Rich Styles and High Aesthetic: Capable of generating images in various styles—including photorealistic portraits, comics, and vinyl figures—it delivers outstanding visual appeal and artistic quality. 🔹High-Quality Generation: Efficiently produces ultra-high-definition (2K) images in the same time other models take to generate a 1K image. HunyuanImage 2.1 uses two text encoders: a multimodal large language model (MLLM) to improve the model's image and text alignment capabilities, and a multi-language character-aware encoder to improve text rendering capabilities. The model is a single- and double-stream diffusion transformer with 17B parameters. We've also open-sourced the weights of the the accelerated version with meanflow which reduces inference steps from 100 to just 8, and PromptEnhancer, the first industrial-grade rewriting model that enhances your prompts for more nuanced and expressive image generation. Now, creators turn complex ideas—like posters with slogans or multi-panel comics—into visuals faster than ever. We’re just getting started. Stay tuned for our native multimodal image generation model coming soon. 🌐Website: 🔗Github: 🤗Hugging Face: ✨Hugging Face Demo:

Tencent Hy

89,257 görüntüleme • 11 ay önce

This Python script helps you better understand how embeddings work for SEO. Input a sentence + a query and BERT will calculate a content "Similarity Score": Search engines use embedding models to translate your content into numeric values. This is how they're able to mathematically determine whether a page on your site is actually relevant to a query someone is searching for. Once both the content and query are run through embeddings - they can calculate the "cosine similarity" or how similar the two entities are to each one. With this Python script, you can actually visualize how cosine similarity works for yourself. To you use you'll simply: 1. Save the Python script to a text file 2. Use your Terminal to run the script 3. You'll be prompted for both a "Sentence" and "Query" 4. Once entered, the script will calculate a "Similarity Score" between the text and query. This is how relevant your sentence is for the target keyword. By running this, you'll see that search engines are able to come with an calculation of content relevance. You'll need to think about the content on your site the same way - which sections have high scores and which ones have low ones? I've linked the Python script in the comments below. No coding knowledge is required and it walks you step by step on how to implement it.
2:37

This Python script helps you better understand how embeddings work for SEO. Input a sentence + a query and BERT will calculate a content "Similarity Score": Search engines use embedding models to translate your content into numeric values. This is how they're able to mathematically determine whether a page on your site is actually relevant to a query someone is searching for. Once both the content and query are run through embeddings - they can calculate the "cosine similarity" or how similar the two entities are to each one. With this Python script, you can actually visualize how cosine similarity works for yourself. To you use you'll simply: 1. Save the Python script to a text file 2. Use your Terminal to run the script 3. You'll be prompted for both a "Sentence" and "Query" 4. Once entered, the script will calculate a "Similarity Score" between the text and query. This is how relevant your sentence is for the target keyword. By running this, you'll see that search engines are able to come with an calculation of content relevance. You'll need to think about the content on your site the same way - which sections have high scores and which ones have low ones? I've linked the Python script in the comments below. No coding knowledge is required and it walks you step by step on how to implement it.

Chris Long

13,185 görüntüleme • 2 yıl önce

U-Net by hand ✍️ ~ 17 steps walkthrough below I consider U-Net as a key milestone in deep learning, the first image-to-image model that really worked! It came out of medical imaging, an unusual place, not from NeurIPS or CVPR or ACL. Now it is the backbone of diffusion models, which you see in almost all modern image generation models. I drew the network as a C so the matrix multiplication flows naturally down. Tilt your head to the right and it is a U again. 🤣 Goal: push a 3 x 16 image down to a 2 x 4 bottleneck and back out again, filling in every cell yourself. = 1. Given = An image of three channels, R, G and B, sixteen pixels wide, and every kernel the network will use. = 2. Convolution 1 = Let us slide the first kernel over the image. Each output is one multiply-and-add over a 2 x 3 window, and the result is the green feature map. = 3. Find the maxima = We circle the largest value in each 1 x 2 window. Circling first is worth the extra step: it is the pooling decision, made before anything is written down. = 4. Max pool 1 = Let us copy those maxima down. Sixteen columns become eight, and half the detail is gone for good. = 5. Convolution 2 = We convolve again with the second kernel, deeper into the contracting path. The feature map is blue now. = 6. Find the maxima again = Same move as step 3, on the blue map. = 7. Max pool 2 = Eight columns become four. = 8. The bottleneck = Let us convolve once more. This is the bottom of the U, a 2 x 4 block that is everything the network kept. = 9. Spread it out = We start back up. The transposed convolution writes each bottleneck value into a wider grid, leaving gaps between them. = 10. Transposed convolution 1 = Let us fill those gaps by convolving over the spread-out grid. Four columns become eight. = 11. The first skip = We copy the encoder's matching row straight across. This is the skip connection, and it is the whole reason a U-Net can recover detail that pooling threw away. = 12. Convolution with the skip = Let us convolve the upsampled features together with the copied ones. = 13. Spread it out again = Same as step 9, one level up. = 14. Transposed convolution 2 = Eight columns become sixteen, back to the width we started at. = 15. The second skip = The encoder's first feature map comes across, the one made before any pooling happened. = 16. Convolution and ReLU = We convolve, then cross out every negative and set it to zero. = 17. Output convolution = Let us apply the last kernel. Out comes R', G' and B', an image the same size as the one we started with. The outputs: R' = [3, 0, 7, 0, 7, 0, 17, 0, 3, 0, 9, 0, 2, 0, 6, 0] G' = [1, 20, 1, 10, 1, 12, 1, 19, 2, 5, 1, 11, 1, 3, 1, 7] B' = [4, 20, 8, 10, 8, 12, 18, 19, 5, 5, 10, 11, 3, 3, 7, 7] Congrats! You just calculated a U-Net by hand. 💾 Save this post!
0:37

U-Net by hand ✍️ ~ 17 steps walkthrough below I consider U-Net as a key milestone in deep learning, the first image-to-image model that really worked! It came out of medical imaging, an unusual place, not from NeurIPS or CVPR or ACL. Now it is the backbone of diffusion models, which you see in almost all modern image generation models. I drew the network as a C so the matrix multiplication flows naturally down. Tilt your head to the right and it is a U again. 🤣 Goal: push a 3 x 16 image down to a 2 x 4 bottleneck and back out again, filling in every cell yourself. = 1. Given = An image of three channels, R, G and B, sixteen pixels wide, and every kernel the network will use. = 2. Convolution 1 = Let us slide the first kernel over the image. Each output is one multiply-and-add over a 2 x 3 window, and the result is the green feature map. = 3. Find the maxima = We circle the largest value in each 1 x 2 window. Circling first is worth the extra step: it is the pooling decision, made before anything is written down. = 4. Max pool 1 = Let us copy those maxima down. Sixteen columns become eight, and half the detail is gone for good. = 5. Convolution 2 = We convolve again with the second kernel, deeper into the contracting path. The feature map is blue now. = 6. Find the maxima again = Same move as step 3, on the blue map. = 7. Max pool 2 = Eight columns become four. = 8. The bottleneck = Let us convolve once more. This is the bottom of the U, a 2 x 4 block that is everything the network kept. = 9. Spread it out = We start back up. The transposed convolution writes each bottleneck value into a wider grid, leaving gaps between them. = 10. Transposed convolution 1 = Let us fill those gaps by convolving over the spread-out grid. Four columns become eight. = 11. The first skip = We copy the encoder's matching row straight across. This is the skip connection, and it is the whole reason a U-Net can recover detail that pooling threw away. = 12. Convolution with the skip = Let us convolve the upsampled features together with the copied ones. = 13. Spread it out again = Same as step 9, one level up. = 14. Transposed convolution 2 = Eight columns become sixteen, back to the width we started at. = 15. The second skip = The encoder's first feature map comes across, the one made before any pooling happened. = 16. Convolution and ReLU = We convolve, then cross out every negative and set it to zero. = 17. Output convolution = Let us apply the last kernel. Out comes R', G' and B', an image the same size as the one we started with. The outputs: R' = [3, 0, 7, 0, 7, 0, 17, 0, 3, 0, 9, 0, 2, 0, 6, 0] G' = [1, 20, 1, 10, 1, 12, 1, 19, 2, 5, 1, 11, 1, 3, 1, 7] B' = [4, 20, 8, 10, 8, 12, 18, 19, 5, 5, 10, 11, 3, 3, 7, 7] Congrats! You just calculated a U-Net by hand. 💾 Save this post!

Tom Yeh

17,510 görüntüleme • 13 gün önce

Dropout by hand ✍️ ~ 10 steps walkthrough below Dropout is the simplest trick in deep learning that actually works: during training you randomly switch neurons off, so the network cannot lean on any one of them. It is two lines of code and almost nobody has worked through what those lines do to the numbers. So I drew and calculated one entirely by hand. Goal: train one pass through a small network with two dropout layers, then run inference with dropout switched off. The network: Linear(2,4), ReLU, Dropout(0.5), Linear(4,3), ReLU, Dropout(0.33), Linear(3,2). = 1. Given = A training set of two examples, X1 and X2, and the weight matrices for all three linear layers. = 2. Draw the first random numbers = Let us draw 4 random numbers, one per neuron in the first hidden layer. Above 0.5 we keep (◯), below we drop (╳). Here that gives [◯, ╳, ◯, ╳]. = 3. Build the first dropout matrix = We turn that pattern into a diagonal matrix. The scaling factor is 1/(1-p) = 2, so a kept neuron gets 2 and a dropped one gets 0. Multiplying by it does both jobs at once: it deletes the 2nd and 4th neurons and doubles the two that survive. = 4. Draw the second random numbers = Let us do it again for the 3 neurons in the next layer, this time against p = 0.33. The result is [◯, ◯, ╳]. = 5. Build the second dropout matrix = We set the diagonal to 1.5 where kept and 0 where dropped. Only the 3rd neuron goes. = 6. Feed forward = Let us run the whole thing top to bottom: one matrix multiplication per layer, ReLU setting the negatives to zero, and the two dropout matrices doing their work in between. The outputs Y come out at the bottom. = 7. MSE loss gradients = We compare Y against the targets Y', subtract, and multiply each element by 2. That is the whole gradient of the mean squared error. = 8. Update the weights = Let us push those gradients back through the network and update the weights (marked in light red). = 9. Deactivate dropout = Training is over, so we set both dropout matrices to the identity. Every neuron is back, and nothing is scaled. = 10. Feed forward again = One more pass, this time on unseen data, to make the prediction. You have just trained and run a network with dropout by hand. ✍️ The outputs: Training outputs Y = [-6, 9; 13, 4] Loss gradients = [-4, 4; 6, -2] Inference outputs = [13, 13; 4, 3] 💾 Save this post! #AIbyHand #Dropout #DeepLearning #NeuralNetworks
0:25

Dropout by hand ✍️ ~ 10 steps walkthrough below Dropout is the simplest trick in deep learning that actually works: during training you randomly switch neurons off, so the network cannot lean on any one of them. It is two lines of code and almost nobody has worked through what those lines do to the numbers. So I drew and calculated one entirely by hand. Goal: train one pass through a small network with two dropout layers, then run inference with dropout switched off. The network: Linear(2,4), ReLU, Dropout(0.5), Linear(4,3), ReLU, Dropout(0.33), Linear(3,2). = 1. Given = A training set of two examples, X1 and X2, and the weight matrices for all three linear layers. = 2. Draw the first random numbers = Let us draw 4 random numbers, one per neuron in the first hidden layer. Above 0.5 we keep (◯), below we drop (╳). Here that gives [◯, ╳, ◯, ╳]. = 3. Build the first dropout matrix = We turn that pattern into a diagonal matrix. The scaling factor is 1/(1-p) = 2, so a kept neuron gets 2 and a dropped one gets 0. Multiplying by it does both jobs at once: it deletes the 2nd and 4th neurons and doubles the two that survive. = 4. Draw the second random numbers = Let us do it again for the 3 neurons in the next layer, this time against p = 0.33. The result is [◯, ◯, ╳]. = 5. Build the second dropout matrix = We set the diagonal to 1.5 where kept and 0 where dropped. Only the 3rd neuron goes. = 6. Feed forward = Let us run the whole thing top to bottom: one matrix multiplication per layer, ReLU setting the negatives to zero, and the two dropout matrices doing their work in between. The outputs Y come out at the bottom. = 7. MSE loss gradients = We compare Y against the targets Y', subtract, and multiply each element by 2. That is the whole gradient of the mean squared error. = 8. Update the weights = Let us push those gradients back through the network and update the weights (marked in light red). = 9. Deactivate dropout = Training is over, so we set both dropout matrices to the identity. Every neuron is back, and nothing is scaled. = 10. Feed forward again = One more pass, this time on unseen data, to make the prediction. You have just trained and run a network with dropout by hand. ✍️ The outputs: Training outputs Y = [-6, 9; 13, 4] Loss gradients = [-4, 4; 6, -2] Inference outputs = [13, 13; 4, 3] 💾 Save this post! #AIbyHand #Dropout #DeepLearning #NeuralNetworks

Tom Yeh

14,442 görüntüleme • 20 gün önce

TurboEdit Instant text-based image editing discuss: We address the challenges of precise image inversion and disentangled image editing in the context of few-step diffusion models. We introduce an encoder based iterative inversion technique. The inversion network is conditioned on the input image and the reconstructed image from the previous step, allowing for correction of the next reconstruction towards the input image. We demonstrate that disentangled controls can be easily achieved in the few-step diffusion model by conditioning on an (automatically generated) detailed text prompt. To manipulate the inverted image, we freeze the noise maps and modify one attribute in the text prompt (either manually or via instruction based editing driven by an LLM), resulting in the generation of a new image similar to the input image with only one attribute changed. It can further control the editing strength and accept instructive text prompt. Our approach facilitates realistic text-guided image edits in real-time, requiring only 8 number of functional evaluations (NFEs) in inversion (one-time cost) and 4 NFEs per edit. Our method is not only fast, but also significantly outperforms state-of-the-art multi-step diffusion editing techniques.
0:22

TurboEdit Instant text-based image editing discuss: We address the challenges of precise image inversion and disentangled image editing in the context of few-step diffusion models. We introduce an encoder based iterative inversion technique. The inversion network is conditioned on the input image and the reconstructed image from the previous step, allowing for correction of the next reconstruction towards the input image. We demonstrate that disentangled controls can be easily achieved in the few-step diffusion model by conditioning on an (automatically generated) detailed text prompt. To manipulate the inverted image, we freeze the noise maps and modify one attribute in the text prompt (either manually or via instruction based editing driven by an LLM), resulting in the generation of a new image similar to the input image with only one attribute changed. It can further control the editing strength and accept instructive text prompt. Our approach facilitates realistic text-guided image edits in real-time, requiring only 8 number of functional evaluations (NFEs) in inversion (one-time cost) and 4 NFEs per edit. Our method is not only fast, but also significantly outperforms state-of-the-art multi-step diffusion editing techniques.

AK

16,062 görüntüleme • 2 yıl önce

Scaled Dot-Product Attention by hand ✍️ I explained attention in my recent AI seminar as follows: Step 1 — Compare We take one token as a query and compare it against all keys using dot products. This gives us a grid of similarity scores: every query against every key. Step 2 — Scale We divide those scores by √dₖ to keep the values numerically stable. Step 3 — Normalize We apply softmax to turn raw scores into a probability distribution—values between 0 and 1 that sum to 1. These are the attention weights. Step 4 — Combine We use those weights to compute a weighted sum of the values. Each output token becomes a linear combination of all previous tokens—maybe 37% from one, 23% from another, and a little from the rest. No blackbox. Just dot products, scaling, softmax, and weighted sums—built one row at a time in Excel.
0:29

Scaled Dot-Product Attention by hand ✍️ I explained attention in my recent AI seminar as follows: Step 1 — Compare We take one token as a query and compare it against all keys using dot products. This gives us a grid of similarity scores: every query against every key. Step 2 — Scale We divide those scores by √dₖ to keep the values numerically stable. Step 3 — Normalize We apply softmax to turn raw scores into a probability distribution—values between 0 and 1 that sum to 1. These are the attention weights. Step 4 — Combine We use those weights to compute a weighted sum of the values. Each output token becomes a linear combination of all previous tokens—maybe 37% from one, 23% from another, and a little from the rest. No blackbox. Just dot products, scaling, softmax, and weighted sums—built one row at a time in Excel.

Tom Yeh

31,114 görüntüleme • 7 ay önce

7 steps and 2 clues created a woman who was bought a drink by a stranger. She doesn't exist. A stranger sat at a club bar last week. She picked out a man, started a conversation, bought him a round. He smiled back. The whole scene lived in a 6-second clip that pulled comments asking who she was. She was never there. No club, no drinks, no face that belongs to anyone. Here is the full build, 7 steps end to end. Open Pinterest, search “girl selfie,” save one image. Open Nano Banana Pro and swap the face with a single prompt. That photo now belongs to no one. Move the image into a node editor. Create 3 nodes: a reference image node, a text node, a video generator node. Drop the photo into the image node. Write 1 video prompt into the text node, the line that turns a still frame into motion. Connect text to text-in, image to image-in. Select Kling 2.6, set duration, switch sound on. Hit generate. 2 prompts. One face that exists nowhere. A 6-second clip people argued was real. The next person who buys you a drink at a bar might run on a text prompt.
0:45

7 steps and 2 clues created a woman who was bought a drink by a stranger. She doesn't exist. A stranger sat at a club bar last week. She picked out a man, started a conversation, bought him a round. He smiled back. The whole scene lived in a 6-second clip that pulled comments asking who she was. She was never there. No club, no drinks, no face that belongs to anyone. Here is the full build, 7 steps end to end. Open Pinterest, search “girl selfie,” save one image. Open Nano Banana Pro and swap the face with a single prompt. That photo now belongs to no one. Move the image into a node editor. Create 3 nodes: a reference image node, a text node, a video generator node. Drop the photo into the image node. Write 1 video prompt into the text node, the line that turns a still frame into motion. Connect text to text-in, image to image-in. Select Kling 2.6, set duration, switch sound on. Hit generate. 2 prompts. One face that exists nowhere. A 6-second clip people argued was real. The next person who buys you a drink at a bar might run on a text prompt.

RGK

16,209 görüntüleme • 2 ay önce

We’re excited to announce the release and open-source of HunyuanImage 3.0 — the largest and most powerful open-source text-to-image model to date, with over 80 billion total parameters, of which 13 billion are activated per token during inference.The effect is completely comparable to the industry’s flagship closed-source model.🚀🚀🚀 HunyuanImage 3.0 originates from our internally developed native multimodal large language model, with fine-tuning and post-training focused on text-to-image generation. This unique foundation gives the model a powerful set of capabilities: ✅Reason with world knowledge ✅Understand complex, thousand-word prompts ✅Generate precise text within images Different from traditional DiT architecture image generation models, HunyuanImage 3.0’s MoE architecture uses a Transfusion-based approach to deeply couple Diffusion and LLM training for a single, powerful system. Built on Hunyuan-A13B, HunyuanImage 3.0 was trained on a massive dataset: 5 billion image-text pairs, video frames, interleaved image-text data, and 6 trillion tokens of text corpora. This hybrid training across multimodal generation, understanding, and LLM capabilities allows the model to seamlessly integrate multiple tasks. Whether you're an illustrator, designer, or creator, this is built to slash your workflow from hours to minutes. HunyuanImage 3.0 can generate intricate text, detailed comics, expressive emojis, and lively, engaging illustrations for educational content. The current release focuses solely on text-to-image generation and future updates will include image-to-image, image editing, multi-turn interaction, and more. 👉🏻Try it now: 🔗GitHub: 🤗Hugging Face:
2:09

We’re excited to announce the release and open-source of HunyuanImage 3.0 — the largest and most powerful open-source text-to-image model to date, with over 80 billion total parameters, of which 13 billion are activated per token during inference.The effect is completely comparable to the industry’s flagship closed-source model.🚀🚀🚀 HunyuanImage 3.0 originates from our internally developed native multimodal large language model, with fine-tuning and post-training focused on text-to-image generation. This unique foundation gives the model a powerful set of capabilities: ✅Reason with world knowledge ✅Understand complex, thousand-word prompts ✅Generate precise text within images Different from traditional DiT architecture image generation models, HunyuanImage 3.0’s MoE architecture uses a Transfusion-based approach to deeply couple Diffusion and LLM training for a single, powerful system. Built on Hunyuan-A13B, HunyuanImage 3.0 was trained on a massive dataset: 5 billion image-text pairs, video frames, interleaved image-text data, and 6 trillion tokens of text corpora. This hybrid training across multimodal generation, understanding, and LLM capabilities allows the model to seamlessly integrate multiple tasks. Whether you're an illustrator, designer, or creator, this is built to slash your workflow from hours to minutes. HunyuanImage 3.0 can generate intricate text, detailed comics, expressive emojis, and lively, engaging illustrations for educational content. The current release focuses solely on text-to-image generation and future updates will include image-to-image, image editing, multi-turn interaction, and more. 👉🏻Try it now: 🔗GitHub: 🤗Hugging Face:

Tencent Hy

412,802 görüntüleme • 10 ay önce

Discrete Fourier Transform by hand ✍️ ~ 12 steps walkthrough below Here is a little-known secret about the DFT and the inverse DFT: it is just matrix multiplication in both directions, one the transpose of the other, exactly like the forward pass and backpropagation I drew in other examples. Goal: recover which cosine waves a signal is made of, using nothing but multiplication and addition. = 1. Given = Three signals written as sums of cosines, and a fourth, X, that we do not know yet. = 2. Frequency matrix F = Let us write the coefficients as a matrix. Each signal is a row, each frequency a column, so A = cos(w) + 2cos(2w) becomes [1, 2, 0, 0]. = 3. Sample the waves = We read the four cosine waves at ten discrete time points. That word "discrete" is the whole difference between this and the continuous transform. = 4. Cosine matrix W = Let us write those samples as a matrix: each frequency a row, each time point a column. = 5. Frequency to time = We multiply F by W. That combines the four cosine waves in the proportions F specifies, and the result T is the three signals as they would look in time. = 6. Transpose = Let us stand each signal up as a column. = 7. Time to frequency = We multiply W by that transpose. Every cell is the dot product of one signal with one cosine wave, which measures how much of that wave the signal contains. Zero means none of it. = 8. Scale = Let us multiply by 2/n, with n = 10. The projections come out five times too large, and this is the correction. = 9. Transpose back = We turn it back around, and it is F again, exactly. That is the check: the transform recovered the coefficients we started from. = 10. Now solve for X = Let us run the same multiplication on the one signal whose recipe we never knew. = 11. Scale = We divide by 5 again. = 12. Transpose back = And X reads [0, 0, 3, 2], which says X = 3cos(3w) + 2cos(4w). Note: I originally drew this to show that the DFT is a special case of a convolution layer, its filters fixed to sine and cosine waves rather than learned. No wonder, then, that a convolution layer free to learn its own filters can be trained to process signals. 💾 Save this post!
0:32

Discrete Fourier Transform by hand ✍️ ~ 12 steps walkthrough below Here is a little-known secret about the DFT and the inverse DFT: it is just matrix multiplication in both directions, one the transpose of the other, exactly like the forward pass and backpropagation I drew in other examples. Goal: recover which cosine waves a signal is made of, using nothing but multiplication and addition. = 1. Given = Three signals written as sums of cosines, and a fourth, X, that we do not know yet. = 2. Frequency matrix F = Let us write the coefficients as a matrix. Each signal is a row, each frequency a column, so A = cos(w) + 2cos(2w) becomes [1, 2, 0, 0]. = 3. Sample the waves = We read the four cosine waves at ten discrete time points. That word "discrete" is the whole difference between this and the continuous transform. = 4. Cosine matrix W = Let us write those samples as a matrix: each frequency a row, each time point a column. = 5. Frequency to time = We multiply F by W. That combines the four cosine waves in the proportions F specifies, and the result T is the three signals as they would look in time. = 6. Transpose = Let us stand each signal up as a column. = 7. Time to frequency = We multiply W by that transpose. Every cell is the dot product of one signal with one cosine wave, which measures how much of that wave the signal contains. Zero means none of it. = 8. Scale = Let us multiply by 2/n, with n = 10. The projections come out five times too large, and this is the correction. = 9. Transpose back = We turn it back around, and it is F again, exactly. That is the check: the transform recovered the coefficients we started from. = 10. Now solve for X = Let us run the same multiplication on the one signal whose recipe we never knew. = 11. Scale = We divide by 5 again. = 12. Transpose back = And X reads [0, 0, 3, 2], which says X = 3cos(3w) + 2cos(4w). Note: I originally drew this to show that the DFT is a special case of a convolution layer, its filters fixed to sine and cosine waves rather than learned. No wonder, then, that a convolution layer free to learn its own filters can be trained to process signals. 💾 Save this post!

Tom Yeh

25,519 görüntüleme • 8 gün önce

Introducing SDXL Turbo: A real-time text-to-image generation model. SDXL Turbo achieves state-of-the-art performance with a new distillation technology, enabling single-step image generation with unprecedented quality, reducing the required step count from 50 to just one. The code, research paper, and weights for non-commercial use are now available on our website. You can test SDXL Turbo on Stability AI’s image editing platform Clipdrop, with a beta demonstration of the real-time text-to-image generation capabilities. Learn more:
0:23

Introducing SDXL Turbo: A real-time text-to-image generation model. SDXL Turbo achieves state-of-the-art performance with a new distillation technology, enabling single-step image generation with unprecedented quality, reducing the required step count from 50 to just one. The code, research paper, and weights for non-commercial use are now available on our website. You can test SDXL Turbo on Stability AI’s image editing platform Clipdrop, with a beta demonstration of the real-time text-to-image generation capabilities. Learn more:

Stability AI

976,456 görüntüleme • 2 yıl önce

[1/2] We’ve released the code for #pix2pixturbo and #CycleGANTurbo. These conditional GANs are able to adapt a text-to-image model such as SD-Turbo for both paired and unpaired image translation with a single step (0.11 sec on A100 and 0.29 sec on A6000). Try our code and the Gradio demo. Paper: Code: Demo: This is a joint work with Gaurav Parmar (the leading author), Taesung Park, and Srinivasa Narasimhan. This work shows that a pre-trained one-step model can be easily adapted to conditional GANs frameworks for downstream image editing and synthesis tasks. #Edges2Cats
0:26

[1/2] We’ve released the code for #pix2pixturbo and #CycleGANTurbo. These conditional GANs are able to adapt a text-to-image model such as SD-Turbo for both paired and unpaired image translation with a single step (0.11 sec on A100 and 0.29 sec on A6000). Try our code and the Gradio demo. Paper: Code: Demo: This is a joint work with Gaurav Parmar (the leading author), Taesung Park, and Srinivasa Narasimhan. This work shows that a pre-trained one-step model can be easily adapted to conditional GANs frameworks for downstream image editing and synthesis tasks. #Edges2Cats

Jun-Yan Zhu

36,488 görüntüleme • 2 yıl önce

TokenFlow: Consistent Diffusion Features for Consistent Video Editing paper page: The generative AI revolution has recently expanded to videos. Nevertheless, current state-of-the-art video models are still lagging behind image models in terms of visual quality and user control over the generated content. In this work, we present a framework that harnesses the power of a text-to-image diffusion model for the task of text-driven video editing. Specifically, given a source video and a target text-prompt, our method generates a high-quality video that adheres to the target text, while preserving the spatial layout and motion of the input video. Our method is based on a key observation that consistency in the edited video can be obtained by enforcing consistency in the diffusion feature space. We achieve this by explicitly propagating diffusion features based on inter-frame correspondences, readily available in the model. Thus, our framework does not require any training or fine-tuning, and can work in conjunction with any off-the-shelf text-to-image editing method. We demonstrate state-of-the-art editing results on a variety of real-world videos.
0:36

TokenFlow: Consistent Diffusion Features for Consistent Video Editing paper page: The generative AI revolution has recently expanded to videos. Nevertheless, current state-of-the-art video models are still lagging behind image models in terms of visual quality and user control over the generated content. In this work, we present a framework that harnesses the power of a text-to-image diffusion model for the task of text-driven video editing. Specifically, given a source video and a target text-prompt, our method generates a high-quality video that adheres to the target text, while preserving the spatial layout and motion of the input video. Our method is based on a key observation that consistency in the edited video can be obtained by enforcing consistency in the diffusion feature space. We achieve this by explicitly propagating diffusion features based on inter-frame correspondences, readily available in the model. Thus, our framework does not require any training or fine-tuning, and can work in conjunction with any off-the-shelf text-to-image editing method. We demonstrate state-of-the-art editing results on a variety of real-world videos.

AK

143,469 görüntüleme • 3 yıl önce