Загрузка видео...

Не удалось загрузить видео

На главную

Backpropagation by hand ✍️ ~ 11 steps walkthrough below Backpropagation is the algorithm that actually trains a neural network, and it is where most people stop following along. It is not calculus you cannot do. It is matrix multiplication, working backward, one layer at a time. So I drew...

ProfTomYeh's profile picture

Tom Yeh

64,790 subscribers

957,856 просмотров • 29 дней назад •via X (Twitter)

Наука и технологииОбразование
LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

Dropout by hand ✍️ ~ 10 steps walkthrough below Dropout is the simplest trick in deep learning that actually works: during training you randomly switch neurons off, so the network cannot lean on any one of them. It is two lines of code and almost nobody has worked through what those lines do to the numbers. So I drew and calculated one entirely by hand. Goal: train one pass through a small network with two dropout layers, then run inference with dropout switched off. The network: Linear(2,4), ReLU, Dropout(0.5), Linear(4,3), ReLU, Dropout(0.33), Linear(3,2). = 1. Given = A training set of two examples, X1 and X2, and the weight matrices for all three linear layers. = 2. Draw the first random numbers = Let us draw 4 random numbers, one per neuron in the first hidden layer. Above 0.5 we keep (◯), below we drop (╳). Here that gives [◯, ╳, ◯, ╳]. = 3. Build the first dropout matrix = We turn that pattern into a diagonal matrix. The scaling factor is 1/(1-p) = 2, so a kept neuron gets 2 and a dropped one gets 0. Multiplying by it does both jobs at once: it deletes the 2nd and 4th neurons and doubles the two that survive. = 4. Draw the second random numbers = Let us do it again for the 3 neurons in the next layer, this time against p = 0.33. The result is [◯, ◯, ╳]. = 5. Build the second dropout matrix = We set the diagonal to 1.5 where kept and 0 where dropped. Only the 3rd neuron goes. = 6. Feed forward = Let us run the whole thing top to bottom: one matrix multiplication per layer, ReLU setting the negatives to zero, and the two dropout matrices doing their work in between. The outputs Y come out at the bottom. = 7. MSE loss gradients = We compare Y against the targets Y', subtract, and multiply each element by 2. That is the whole gradient of the mean squared error. = 8. Update the weights = Let us push those gradients back through the network and update the weights (marked in light red). = 9. Deactivate dropout = Training is over, so we set both dropout matrices to the identity. Every neuron is back, and nothing is scaled. = 10. Feed forward again = One more pass, this time on unseen data, to make the prediction. You have just trained and run a network with dropout by hand. ✍️ The outputs: Training outputs Y = [-6, 9; 13, 4] Loss gradients = [-4, 4; 6, -2] Inference outputs = [13, 13; 4, 3] 💾 Save this post! #AIbyHand #Dropout #DeepLearning #NeuralNetworks
0:25

Dropout by hand ✍️ ~ 10 steps walkthrough below Dropout is the simplest trick in deep learning that actually works: during training you randomly switch neurons off, so the network cannot lean on any one of them. It is two lines of code and almost nobody has worked through what those lines do to the numbers. So I drew and calculated one entirely by hand. Goal: train one pass through a small network with two dropout layers, then run inference with dropout switched off. The network: Linear(2,4), ReLU, Dropout(0.5), Linear(4,3), ReLU, Dropout(0.33), Linear(3,2). = 1. Given = A training set of two examples, X1 and X2, and the weight matrices for all three linear layers. = 2. Draw the first random numbers = Let us draw 4 random numbers, one per neuron in the first hidden layer. Above 0.5 we keep (◯), below we drop (╳). Here that gives [◯, ╳, ◯, ╳]. = 3. Build the first dropout matrix = We turn that pattern into a diagonal matrix. The scaling factor is 1/(1-p) = 2, so a kept neuron gets 2 and a dropped one gets 0. Multiplying by it does both jobs at once: it deletes the 2nd and 4th neurons and doubles the two that survive. = 4. Draw the second random numbers = Let us do it again for the 3 neurons in the next layer, this time against p = 0.33. The result is [◯, ◯, ╳]. = 5. Build the second dropout matrix = We set the diagonal to 1.5 where kept and 0 where dropped. Only the 3rd neuron goes. = 6. Feed forward = Let us run the whole thing top to bottom: one matrix multiplication per layer, ReLU setting the negatives to zero, and the two dropout matrices doing their work in between. The outputs Y come out at the bottom. = 7. MSE loss gradients = We compare Y against the targets Y', subtract, and multiply each element by 2. That is the whole gradient of the mean squared error. = 8. Update the weights = Let us push those gradients back through the network and update the weights (marked in light red). = 9. Deactivate dropout = Training is over, so we set both dropout matrices to the identity. Every neuron is back, and nothing is scaled. = 10. Feed forward again = One more pass, this time on unseen data, to make the prediction. You have just trained and run a network with dropout by hand. ✍️ The outputs: Training outputs Y = [-6, 9; 13, 4] Loss gradients = [-4, 4; 6, -2] Inference outputs = [13, 13; 4, 3] 💾 Save this post! #AIbyHand #Dropout #DeepLearning #NeuralNetworks

Tom Yeh

14,442 просмотров • 1 месяц назад

Variational Autoencoder by hand ✍️ ~ 11 steps walkthrough below A VAE learns the structure of your data, the mean and variance of its hidden features, and then generates new data from that structure. A GAN only learns to fool a discriminator. It can make convincing fakes without ever knowing what the data is really made of. That is the difference, and it is the whole reason VAEs matter. In 2024 ICLR gave its first ever Test of Time Award to the VAE paper, "Auto-Encoding Variational Bayes" by Diederik Kingma and Max Welling, ten years on. How does it work? Goal: encode three inputs into a distribution, sample from it, decode it back, and read every loss gradient off the page. = 1. Given = Three training examples X1, X2, X3, copied to the bottom as their own targets. Reconstructing your own input is what puts the "auto", meaning self, in autoencoder. = 2. Encoder, layer 1 = Let us multiply the inputs by weights and biases, then apply ReLU, crossing out every negative. = 3. Mean and standard deviation = We multiply the features by two more weight sets. The first predicts the means μ of the latent distributions, the second their standard deviations σ. = 4. A random offset = Let us sample ε from a standard normal, mean 0 and variance 1, and multiply it by σ. This is a random step away from the mean, scaled by how uncertain each feature is. = 5. Mean plus offset = We add the offset back onto μ, and these become the decoder's inputs. Keeping the randomness out in ε is the reparameterization trick: it lets gradients flow straight through the sampling. = 6. Decoder, layer 1 = Let us multiply by weights and biases and apply ReLU again. Here -4 is crossed out. = 7. Decoder, layer 2 = We multiply once more. The output Y is the decoder's attempt to rebuild X from the sampled distribution. = 8. Gradient for the mean = Let us push μ toward 0. A lot of math, the SGVB estimator, collapses the KL gradient to simply μ itself. = 9. Gradient for the standard deviation = We want σ to approach 1. = 10. And its formula = That same math simplifies the gradient to σ minus 1/σ. = 11. Reconstruction gradient = We want the reconstruction Y to match the input X. Mean squared error simplifies its gradient to Y minus X. Takeaway: the two gradients you just calculated each sit at the heart of a modern method, so one VAE teaches you both. The KL divergence is the penalty RLHF like GRPO uses to keep a fine-tuned model from drifting off its base. The reconstruction loss, plain mean squared error, is exactly what trains a diffusion model to denoise. Draw one VAE by hand and you have quietly learned the core of both. 💾 Save this post!
0:31

Variational Autoencoder by hand ✍️ ~ 11 steps walkthrough below A VAE learns the structure of your data, the mean and variance of its hidden features, and then generates new data from that structure. A GAN only learns to fool a discriminator. It can make convincing fakes without ever knowing what the data is really made of. That is the difference, and it is the whole reason VAEs matter. In 2024 ICLR gave its first ever Test of Time Award to the VAE paper, "Auto-Encoding Variational Bayes" by Diederik Kingma and Max Welling, ten years on. How does it work? Goal: encode three inputs into a distribution, sample from it, decode it back, and read every loss gradient off the page. = 1. Given = Three training examples X1, X2, X3, copied to the bottom as their own targets. Reconstructing your own input is what puts the "auto", meaning self, in autoencoder. = 2. Encoder, layer 1 = Let us multiply the inputs by weights and biases, then apply ReLU, crossing out every negative. = 3. Mean and standard deviation = We multiply the features by two more weight sets. The first predicts the means μ of the latent distributions, the second their standard deviations σ. = 4. A random offset = Let us sample ε from a standard normal, mean 0 and variance 1, and multiply it by σ. This is a random step away from the mean, scaled by how uncertain each feature is. = 5. Mean plus offset = We add the offset back onto μ, and these become the decoder's inputs. Keeping the randomness out in ε is the reparameterization trick: it lets gradients flow straight through the sampling. = 6. Decoder, layer 1 = Let us multiply by weights and biases and apply ReLU again. Here -4 is crossed out. = 7. Decoder, layer 2 = We multiply once more. The output Y is the decoder's attempt to rebuild X from the sampled distribution. = 8. Gradient for the mean = Let us push μ toward 0. A lot of math, the SGVB estimator, collapses the KL gradient to simply μ itself. = 9. Gradient for the standard deviation = We want σ to approach 1. = 10. And its formula = That same math simplifies the gradient to σ minus 1/σ. = 11. Reconstruction gradient = We want the reconstruction Y to match the input X. Mean squared error simplifies its gradient to Y minus X. Takeaway: the two gradients you just calculated each sit at the heart of a modern method, so one VAE teaches you both. The KL divergence is the penalty RLHF like GRPO uses to keep a fine-tuned model from drifting off its base. The reconstruction loss, plain mean squared error, is exactly what trains a diffusion model to denoise. Draw one VAE by hand and you have quietly learned the core of both. 💾 Save this post!

Tom Yeh

17,011 просмотров • 20 дней назад

Generative Adversarial Network (GAN) by hand ✍️ ~ 9 steps walkthrough below The Gen in GenAI came from this landmark paper by Ian Goodfellow et al., 12 years ago. The paper showed that a neural network can not only classify but also turn upside down to generate realistic looking images. The secret? We pit two of them against each other: a Generator turns noise into fake data, and a Discriminator learns to tell fake from real, pushing the Generator to keep doing better. One runs upside down, the other right way up. I drew and calculated one entirely by hand. Goal: generate realistic 4D data out of 2D noise, filling in every cell yourself. = 1. Given = Four noise vectors in 2D, and four real data vectors in 4D. = 2. Generator, first layer = Let us multiply the noise by weights and biases to get new features. = 3. ReLU = We apply the activation, and -1 and -2 are crossed out and set to 0. = 4. Generator, second layer = Let us multiply again. ReLU applies here too, but every value is already positive, so nothing changes. What comes out is the fake data F, made by a two-layer generator out of nothing but noise. = 5. Discriminator, first layer = We feed it both, the four fakes and the four real vectors, through the same weights. It never learns which is which from the layout, only from the numbers. = 6. Discriminator, second layer = Let us reduce each data vector to a single feature Z. Eight vectors in, eight numbers out. = 7. Sigmoid = We turn each Z into a probability Y. A 1 means the discriminator is certain the data is real, a 0 means certain it is fake. = 8. Training the Discriminator = Let us take the gradients as Y minus YD, where YD is what the discriminator should have said: 0 for the four fakes, 1 for the four real. Why so simple? Because pairing sigmoid with binary cross entropy loss makes the math collapse to exactly this subtraction. Its loss uses both halves of the page. = 9. Training the Generator = We do it again, as Y minus YG, and YG is [1, 1, 1, 1]: the generator wants the discriminator to call every fake real. Same predictions, different target, opposite goal. Its loss uses only the fakes. The outputs: Fake data F = [1, 2, 3, 1], [1, 1, 2, 1], [2, 2, 4, 2], [1, 0, 1, 1] Predictions on fakes = [.7, .5, .9, .3] Predictions on real = [.7, .9, .9, 1] Discriminator gradients = [.7, .5, .9, .3] and [-.3, -.1, -.1, 0] Generator gradients = [-.3, -.5, -.1, -.7] The takeaway: the adversarial part is one subtraction done twice. The same eight predictions, scored against two opposite targets, send one set of gradients back through the blue weights and another back through the green ones. 💾 Save this post!
0:23

Generative Adversarial Network (GAN) by hand ✍️ ~ 9 steps walkthrough below The Gen in GenAI came from this landmark paper by Ian Goodfellow et al., 12 years ago. The paper showed that a neural network can not only classify but also turn upside down to generate realistic looking images. The secret? We pit two of them against each other: a Generator turns noise into fake data, and a Discriminator learns to tell fake from real, pushing the Generator to keep doing better. One runs upside down, the other right way up. I drew and calculated one entirely by hand. Goal: generate realistic 4D data out of 2D noise, filling in every cell yourself. = 1. Given = Four noise vectors in 2D, and four real data vectors in 4D. = 2. Generator, first layer = Let us multiply the noise by weights and biases to get new features. = 3. ReLU = We apply the activation, and -1 and -2 are crossed out and set to 0. = 4. Generator, second layer = Let us multiply again. ReLU applies here too, but every value is already positive, so nothing changes. What comes out is the fake data F, made by a two-layer generator out of nothing but noise. = 5. Discriminator, first layer = We feed it both, the four fakes and the four real vectors, through the same weights. It never learns which is which from the layout, only from the numbers. = 6. Discriminator, second layer = Let us reduce each data vector to a single feature Z. Eight vectors in, eight numbers out. = 7. Sigmoid = We turn each Z into a probability Y. A 1 means the discriminator is certain the data is real, a 0 means certain it is fake. = 8. Training the Discriminator = Let us take the gradients as Y minus YD, where YD is what the discriminator should have said: 0 for the four fakes, 1 for the four real. Why so simple? Because pairing sigmoid with binary cross entropy loss makes the math collapse to exactly this subtraction. Its loss uses both halves of the page. = 9. Training the Generator = We do it again, as Y minus YG, and YG is [1, 1, 1, 1]: the generator wants the discriminator to call every fake real. Same predictions, different target, opposite goal. Its loss uses only the fakes. The outputs: Fake data F = [1, 2, 3, 1], [1, 1, 2, 1], [2, 2, 4, 2], [1, 0, 1, 1] Predictions on fakes = [.7, .5, .9, .3] Predictions on real = [.7, .9, .9, 1] Discriminator gradients = [.7, .5, .9, .3] and [-.3, -.1, -.1, 0] Generator gradients = [-.3, -.5, -.1, -.7] The takeaway: the adversarial part is one subtraction done twice. The same eight predictions, scored against two opposite targets, send one set of gradients back through the blue weights and another back through the green ones. 💾 Save this post!

Tom Yeh

16,652 просмотров • 26 дней назад

Graph Convolutional Network by hand ✍️ ~ 12 steps walkthrough below Graph Convolutional Networks (GCNs), introduced by Thomas Kipf and Max Welling in 2017, are the tool for data shaped like a graph: social networks, recommendations, biological networks, drug discovery, molecular chemistry. I drew and calculated a simple GCN entirely by hand. Goal: run a two-layer GCN, then a small classifier, on a five-node graph, filling in every cell yourself. 1. Given A graph of five nodes, A to E, with edges between some of them. 2. Adjacency matrix (neighbors) Put a 1 wherever two nodes share an edge, in both directions. 3. Adjacency matrix (self) Add 1s down the diagonal, one self-loop per node. That is just adding the identity matrix. 4. Messages Multiply each node's embedding by the weights and biases, then ReLU. Negatives become 0. 5. Pooling Multiply the messages by the adjacency matrix. Each node gathers the messages of its neighbours and itself. 6. Visualize Node A pools [3,0,1] + [1,0,0] = [4,0,1]. 7. Second GCN layer Messages again: weights, biases, ReLU. 8. Pooling again Pool over each node and its neighbours, once more. 9. Visualize Node C pools [1,2,4] + [1,3,5] + [0,0,1] = [2,5,10]. 10. Fully connected layer Weights, biases, ReLU. This time there are no neighbours to pool, just the node itself. 11. Linear layer One more: weights and biases. 12. Sigmoid Squash each score to a probability (≥ 3 → 1, 0 → 0.5, ≤ -3 → 0). That is the classification for each node. You have just classified every node in the graph by hand. ✍️ The outputs: A: 0 (very unlikely) B: 1 (very likely) C: 1 (very likely) D: 1 (very likely) E: 0.5 (neutral) The takeaway: a GCN layer is two parts. The top part pools each node with its neighbours through the adjacency matrix. The bottom part is an MLP that transforms each node on its own. A transformer layer has the same two parts, with an attention matrix where the adjacency matrix was. Both matrices do one job, mixing across positions: attention over tokens, adjacency over nodes. In my class I call the GCN the transformer's little cousin: a bit more stubborn, because its attention is fixed by the graph rather than computed from Q, K, and V. Draw the two side by side and the resemblance is hard to miss. 💾 Save this post! #AIbyHand #GraphNeuralNetworks #DeepLearning
0:23

Graph Convolutional Network by hand ✍️ ~ 12 steps walkthrough below Graph Convolutional Networks (GCNs), introduced by Thomas Kipf and Max Welling in 2017, are the tool for data shaped like a graph: social networks, recommendations, biological networks, drug discovery, molecular chemistry. I drew and calculated a simple GCN entirely by hand. Goal: run a two-layer GCN, then a small classifier, on a five-node graph, filling in every cell yourself. 1. Given A graph of five nodes, A to E, with edges between some of them. 2. Adjacency matrix (neighbors) Put a 1 wherever two nodes share an edge, in both directions. 3. Adjacency matrix (self) Add 1s down the diagonal, one self-loop per node. That is just adding the identity matrix. 4. Messages Multiply each node's embedding by the weights and biases, then ReLU. Negatives become 0. 5. Pooling Multiply the messages by the adjacency matrix. Each node gathers the messages of its neighbours and itself. 6. Visualize Node A pools [3,0,1] + [1,0,0] = [4,0,1]. 7. Second GCN layer Messages again: weights, biases, ReLU. 8. Pooling again Pool over each node and its neighbours, once more. 9. Visualize Node C pools [1,2,4] + [1,3,5] + [0,0,1] = [2,5,10]. 10. Fully connected layer Weights, biases, ReLU. This time there are no neighbours to pool, just the node itself. 11. Linear layer One more: weights and biases. 12. Sigmoid Squash each score to a probability (≥ 3 → 1, 0 → 0.5, ≤ -3 → 0). That is the classification for each node. You have just classified every node in the graph by hand. ✍️ The outputs: A: 0 (very unlikely) B: 1 (very likely) C: 1 (very likely) D: 1 (very likely) E: 0.5 (neutral) The takeaway: a GCN layer is two parts. The top part pools each node with its neighbours through the adjacency matrix. The bottom part is an MLP that transforms each node on its own. A transformer layer has the same two parts, with an attention matrix where the adjacency matrix was. Both matrices do one job, mixing across positions: attention over tokens, adjacency over nodes. In my class I call the GCN the transformer's little cousin: a bit more stubborn, because its attention is fixed by the graph rather than computed from Q, K, and V. Draw the two side by side and the resemblance is hard to miss. 💾 Save this post! #AIbyHand #GraphNeuralNetworks #DeepLearning

Tom Yeh

16,800 просмотров • 1 месяц назад

RLHF by hand ✍️ ~ 15 steps walkthrough below Train a model on human text and it inherits human bias. It will assume a doctor is a "him", because the data says so. RLHF is the correction. A human marks one preference, doc is them over doc is him, and the weights move. But one correction is not the point. The hope is that the model learns the value behind it, gender neutrality, and applies it to professions nobody ever mentioned. How does it work? Goal: train a reward model from a single human comparison about doctors, then turn it on CEOs, filling in every cell yourself. = 1. Given = A reward model, an LLM, and two (prompt, next) pairs. = 2. Preferences = A human reads both pairs and picks a winner: (doc is, them) beats (doc is, him). The loser is not bad grammar, it is gender bias, and that is the whole signal. = 3. Word embeddings = Let us look up each word of the loser pair. These vectors are the reward model's input. = 4. Linear layer = We multiply by the reward model's weights and add its biases. Out come feature vectors, one per position. = 5. Mean pool = Let us multiply by [1/3, 1/3, 1/3], which averages the three positions into one sentence embedding. = 6. Output layer = We map that sentence down to a single number. Reward = 3. = 7. The winner, the same way = Let us repeat steps 3 to 6 on the winning pair. Reward = 5. = 8. Winner minus loser = We take the gap: 5 - 3 = 2. The reward model wants this positive and as large as it can make it. = 9. Loss gradient = Let us squash the gap into a probability, σ(2) ≈ 0.9, and subtract the target of 1. The gradient is -0.1, and it goes back through the purple weights. The reward model is now trained. = 10. A prompt it has never seen = We start the second half with "[S] CEO is". The feedback in step 2 was about doctors. Nothing connects a CEO to a doctor except what the reward model generalised. = 11. Transformer = Let us push it through attention and a feed forward layer, one vector per position. = 12. Output probabilities = We map each vector to a score over the vocabulary. = 13. Sample = Let us take the highest score. The model completes "CEO is" with "him", which is the same bias the human penalised in step 2. = 14. Score it with the reward model = We feed the new pair (CEO is, him) through steps 3 to 6. Reward = 3, exactly the score it gave "doc is him" in step 6. Nobody taught it about CEOs. The value transferred. = 15. Loss gradient = Let us set the loss to the negative of the reward, so minimising the loss maximises the reward. The gradient is a constant -1, and it goes back through the red weights. The outputs: Loser reward = 3, winner reward = 5 Reward gap = 2, predicted σ ≈ 0.9, reward model gradient = -0.1 LLM samples "him", reward = 3, LLM gradient = -1 Congrats! You just calculated RLHF by hand. And you watched a value generalise: one comparison about doctors, and the model marks down "CEO is him" unprompted. 💾 Save this post!
0:37

RLHF by hand ✍️ ~ 15 steps walkthrough below Train a model on human text and it inherits human bias. It will assume a doctor is a "him", because the data says so. RLHF is the correction. A human marks one preference, doc is them over doc is him, and the weights move. But one correction is not the point. The hope is that the model learns the value behind it, gender neutrality, and applies it to professions nobody ever mentioned. How does it work? Goal: train a reward model from a single human comparison about doctors, then turn it on CEOs, filling in every cell yourself. = 1. Given = A reward model, an LLM, and two (prompt, next) pairs. = 2. Preferences = A human reads both pairs and picks a winner: (doc is, them) beats (doc is, him). The loser is not bad grammar, it is gender bias, and that is the whole signal. = 3. Word embeddings = Let us look up each word of the loser pair. These vectors are the reward model's input. = 4. Linear layer = We multiply by the reward model's weights and add its biases. Out come feature vectors, one per position. = 5. Mean pool = Let us multiply by [1/3, 1/3, 1/3], which averages the three positions into one sentence embedding. = 6. Output layer = We map that sentence down to a single number. Reward = 3. = 7. The winner, the same way = Let us repeat steps 3 to 6 on the winning pair. Reward = 5. = 8. Winner minus loser = We take the gap: 5 - 3 = 2. The reward model wants this positive and as large as it can make it. = 9. Loss gradient = Let us squash the gap into a probability, σ(2) ≈ 0.9, and subtract the target of 1. The gradient is -0.1, and it goes back through the purple weights. The reward model is now trained. = 10. A prompt it has never seen = We start the second half with "[S] CEO is". The feedback in step 2 was about doctors. Nothing connects a CEO to a doctor except what the reward model generalised. = 11. Transformer = Let us push it through attention and a feed forward layer, one vector per position. = 12. Output probabilities = We map each vector to a score over the vocabulary. = 13. Sample = Let us take the highest score. The model completes "CEO is" with "him", which is the same bias the human penalised in step 2. = 14. Score it with the reward model = We feed the new pair (CEO is, him) through steps 3 to 6. Reward = 3, exactly the score it gave "doc is him" in step 6. Nobody taught it about CEOs. The value transferred. = 15. Loss gradient = Let us set the loss to the negative of the reward, so minimising the loss maximises the reward. The gradient is a constant -1, and it goes back through the red weights. The outputs: Loser reward = 3, winner reward = 5 Reward gap = 2, predicted σ ≈ 0.9, reward model gradient = -0.1 LLM samples "him", reward = 3, LLM gradient = -1 Congrats! You just calculated RLHF by hand. And you watched a value generalise: one comparison about doctors, and the model marks down "CEO is him" unprompted. 💾 Save this post!

Tom Yeh

22,105 просмотров • 22 дней назад

Discrete Fourier Transform by hand ✍️ ~ 12 steps walkthrough below Here is a little-known secret about the DFT and the inverse DFT: it is just matrix multiplication in both directions, one the transpose of the other, exactly like the forward pass and backpropagation I drew in other examples. Goal: recover which cosine waves a signal is made of, using nothing but multiplication and addition. = 1. Given = Three signals written as sums of cosines, and a fourth, X, that we do not know yet. = 2. Frequency matrix F = Let us write the coefficients as a matrix. Each signal is a row, each frequency a column, so A = cos(w) + 2cos(2w) becomes [1, 2, 0, 0]. = 3. Sample the waves = We read the four cosine waves at ten discrete time points. That word "discrete" is the whole difference between this and the continuous transform. = 4. Cosine matrix W = Let us write those samples as a matrix: each frequency a row, each time point a column. = 5. Frequency to time = We multiply F by W. That combines the four cosine waves in the proportions F specifies, and the result T is the three signals as they would look in time. = 6. Transpose = Let us stand each signal up as a column. = 7. Time to frequency = We multiply W by that transpose. Every cell is the dot product of one signal with one cosine wave, which measures how much of that wave the signal contains. Zero means none of it. = 8. Scale = Let us multiply by 2/n, with n = 10. The projections come out five times too large, and this is the correction. = 9. Transpose back = We turn it back around, and it is F again, exactly. That is the check: the transform recovered the coefficients we started from. = 10. Now solve for X = Let us run the same multiplication on the one signal whose recipe we never knew. = 11. Scale = We divide by 5 again. = 12. Transpose back = And X reads [0, 0, 3, 2], which says X = 3cos(3w) + 2cos(4w). Note: I originally drew this to show that the DFT is a special case of a convolution layer, its filters fixed to sine and cosine waves rather than learned. No wonder, then, that a convolution layer free to learn its own filters can be trained to process signals. 💾 Save this post!
0:32

Discrete Fourier Transform by hand ✍️ ~ 12 steps walkthrough below Here is a little-known secret about the DFT and the inverse DFT: it is just matrix multiplication in both directions, one the transpose of the other, exactly like the forward pass and backpropagation I drew in other examples. Goal: recover which cosine waves a signal is made of, using nothing but multiplication and addition. = 1. Given = Three signals written as sums of cosines, and a fourth, X, that we do not know yet. = 2. Frequency matrix F = Let us write the coefficients as a matrix. Each signal is a row, each frequency a column, so A = cos(w) + 2cos(2w) becomes [1, 2, 0, 0]. = 3. Sample the waves = We read the four cosine waves at ten discrete time points. That word "discrete" is the whole difference between this and the continuous transform. = 4. Cosine matrix W = Let us write those samples as a matrix: each frequency a row, each time point a column. = 5. Frequency to time = We multiply F by W. That combines the four cosine waves in the proportions F specifies, and the result T is the three signals as they would look in time. = 6. Transpose = Let us stand each signal up as a column. = 7. Time to frequency = We multiply W by that transpose. Every cell is the dot product of one signal with one cosine wave, which measures how much of that wave the signal contains. Zero means none of it. = 8. Scale = Let us multiply by 2/n, with n = 10. The projections come out five times too large, and this is the correction. = 9. Transpose back = We turn it back around, and it is F again, exactly. That is the check: the transform recovered the coefficients we started from. = 10. Now solve for X = Let us run the same multiplication on the one signal whose recipe we never knew. = 11. Scale = We divide by 5 again. = 12. Transpose back = And X reads [0, 0, 3, 2], which says X = 3cos(3w) + 2cos(4w). Note: I originally drew this to show that the DFT is a special case of a convolution layer, its filters fixed to sine and cosine waves rather than learned. No wonder, then, that a convolution layer free to learn its own filters can be trained to process signals. 💾 Save this post!

Tom Yeh

25,684 просмотров • 18 дней назад

[RLHF] by Hand ✍️ Yesterday, Jan Leike (Jan Leike) announced he is joining #Anthropic to lead their "super-alignment" mission. He is the co-inventor of Reinforcement Learning with Human Feedback (#RLHF). How does RLHF work? [1] Given ↳ Reward Model (RM) ↳ Large Language Model (LLM) ↳ Two (Prompt, Next) Pairs 🟪 TRAIN RM Goal: Learn to give higher rewards to winners [2] Preferences ↳ A human reviews the two pairs and picks a "winner" ↳ (doc is, him) Embeddings ↳ This prompt has never received human feedback directly ↳ [S] is the special start symbol [11] Transformer ↳ Attention (yellow) ↳ Feed Forward (4x2 weight and bias matrix) ↳ Output: 3 "transformed" feature vector, one per position ↳ More details in my previous post 8. Transformer [] [12] Output Probabilities ↳ Apply a linear layer to map each transformed feature vector to a probability distribution over the vocabulary. [13] Sample ↳ Apply the greedy method, which is to pick the word with the highest score ↳ For output 1 and 2, the model accurately predicts the next word ↳ For 3rd output position, the model's predicts "him" [14] Reward Model ↳ The new pair (CEO is, him) is fed to the reward model ↳ The process is same as [3]-[6] ↳ Output: Reward = 3 [15] Loss Gradient ↳ We set the loss as the negative of the reward. ↳ The loss gradient is simply a constant -1. ↳ Run backpropagation and gradient descent to update LLM's weights and biases (red border)
1:01

[RLHF] by Hand ✍️ Yesterday, Jan Leike (Jan Leike) announced he is joining #Anthropic to lead their "super-alignment" mission. He is the co-inventor of Reinforcement Learning with Human Feedback (#RLHF). How does RLHF work? [1] Given ↳ Reward Model (RM) ↳ Large Language Model (LLM) ↳ Two (Prompt, Next) Pairs 🟪 TRAIN RM Goal: Learn to give higher rewards to winners [2] Preferences ↳ A human reviews the two pairs and picks a "winner" ↳ (doc is, him) Embeddings ↳ This prompt has never received human feedback directly ↳ [S] is the special start symbol [11] Transformer ↳ Attention (yellow) ↳ Feed Forward (4x2 weight and bias matrix) ↳ Output: 3 "transformed" feature vector, one per position ↳ More details in my previous post 8. Transformer [] [12] Output Probabilities ↳ Apply a linear layer to map each transformed feature vector to a probability distribution over the vocabulary. [13] Sample ↳ Apply the greedy method, which is to pick the word with the highest score ↳ For output 1 and 2, the model accurately predicts the next word ↳ For 3rd output position, the model's predicts "him" [14] Reward Model ↳ The new pair (CEO is, him) is fed to the reward model ↳ The process is same as [3]-[6] ↳ Output: Reward = 3 [15] Loss Gradient ↳ We set the loss as the negative of the reward. ↳ The loss gradient is simply a constant -1. ↳ Run backpropagation and gradient descent to update LLM's weights and biases (red border)

Tom Yeh

79,916 просмотров • 2 лет назад

Self Attention by hand ✍️ ~ 9 steps walkthrough below Self-attention is what enables LLMs to understand context. How does it work? So I drew and calculated one entirely by hand. Goal: turn four 6D features into four 3D attention weighted features, filling in every cell yourself. = 1. Given = Four feature vectors, six dimensions each, one per position. = 2. Query, key, value = Let us multiply the features by WQ, WK and WV. Queries, keys and values all come out of the same four features, and that is what the word "self" is doing in self-attention. = 3. Prepare for MatMul = We copy the queries across the top and the transposed keys down the side. Lining the two up is half the work. = 4. MatMul = Let us multiply K transpose by Q. Every cell is the dot product of one key with one query, which we use as a matching score. That works because the dot product is the numerator of cosine similarity: it is how alike two vectors are, before anyone divides by their lengths. = 5. Scale = We divide by the square root of dk, the dimension of a key vector, here 3. Without it the scores grow with the dimension and a 64-wide head would swamp the softmax. To keep the page doable in pen, the drawing approximates dividing by root 3 with halving. = 6. e to the power = Let us raise e to the power of each score. This is the first half of softmax, and the drawing uses 3 in place of e, which is close enough to do in your head. = 7. Sum = We add up each column: 16, 6, 7 and 12. = 8. Normalize = Let us divide every cell by its column sum. That gives the attention weight matrix in yellow, and each of its four columns is now a probability distribution over the four positions. The decimals are nudged as they are rounded, so every column still sums to exactly 1. = 9. MatMul = We multiply the value vectors by those weights. Each output is a blend of all four values, mixed in the proportion the attention matrix just decided, and it goes to the position-wise feed forward network in the next layer: the FFN box at the bottom of the page. The outputs: Attention weights (A), by column = [.2, .6, 0, .2], [.2, .4, .2, .2], [.4, .2, 0, .4], [.1, .7, .1, .1] Attention weighted features (Z) = [8, 2, 6], [8, 4, 4], [16, 4, 2], [4, 2, 7] The takeaway: attention is a weighted average, and everything before step 9 exists to decide the weights. Compare every position with every other, turn the scores into one distribution per position, then blend. 💾 Save this post!
0:22

Self Attention by hand ✍️ ~ 9 steps walkthrough below Self-attention is what enables LLMs to understand context. How does it work? So I drew and calculated one entirely by hand. Goal: turn four 6D features into four 3D attention weighted features, filling in every cell yourself. = 1. Given = Four feature vectors, six dimensions each, one per position. = 2. Query, key, value = Let us multiply the features by WQ, WK and WV. Queries, keys and values all come out of the same four features, and that is what the word "self" is doing in self-attention. = 3. Prepare for MatMul = We copy the queries across the top and the transposed keys down the side. Lining the two up is half the work. = 4. MatMul = Let us multiply K transpose by Q. Every cell is the dot product of one key with one query, which we use as a matching score. That works because the dot product is the numerator of cosine similarity: it is how alike two vectors are, before anyone divides by their lengths. = 5. Scale = We divide by the square root of dk, the dimension of a key vector, here 3. Without it the scores grow with the dimension and a 64-wide head would swamp the softmax. To keep the page doable in pen, the drawing approximates dividing by root 3 with halving. = 6. e to the power = Let us raise e to the power of each score. This is the first half of softmax, and the drawing uses 3 in place of e, which is close enough to do in your head. = 7. Sum = We add up each column: 16, 6, 7 and 12. = 8. Normalize = Let us divide every cell by its column sum. That gives the attention weight matrix in yellow, and each of its four columns is now a probability distribution over the four positions. The decimals are nudged as they are rounded, so every column still sums to exactly 1. = 9. MatMul = We multiply the value vectors by those weights. Each output is a blend of all four values, mixed in the proportion the attention matrix just decided, and it goes to the position-wise feed forward network in the next layer: the FFN box at the bottom of the page. The outputs: Attention weights (A), by column = [.2, .6, 0, .2], [.2, .4, .2, .2], [.4, .2, 0, .4], [.1, .7, .1, .1] Attention weighted features (Z) = [8, 2, 6], [8, 4, 4], [16, 4, 2], [4, 2, 7] The takeaway: attention is a weighted average, and everything before step 9 exists to decide the weights. Compare every position with every other, turn the scores into one distribution per position, then blend. 💾 Save this post!

Tom Yeh

27,049 просмотров • 27 дней назад

U-Net by hand ✍️ ~ 17 steps walkthrough below I consider U-Net as a key milestone in deep learning, the first image-to-image model that really worked! It came out of medical imaging, an unusual place, not from NeurIPS or CVPR or ACL. Now it is the backbone of diffusion models, which you see in almost all modern image generation models. I drew the network as a C so the matrix multiplication flows naturally down. Tilt your head to the right and it is a U again. 🤣 Goal: push a 3 x 16 image down to a 2 x 4 bottleneck and back out again, filling in every cell yourself. = 1. Given = An image of three channels, R, G and B, sixteen pixels wide, and every kernel the network will use. = 2. Convolution 1 = Let us slide the first kernel over the image. Each output is one multiply-and-add over a 2 x 3 window, and the result is the green feature map. = 3. Find the maxima = We circle the largest value in each 1 x 2 window. Circling first is worth the extra step: it is the pooling decision, made before anything is written down. = 4. Max pool 1 = Let us copy those maxima down. Sixteen columns become eight, and half the detail is gone for good. = 5. Convolution 2 = We convolve again with the second kernel, deeper into the contracting path. The feature map is blue now. = 6. Find the maxima again = Same move as step 3, on the blue map. = 7. Max pool 2 = Eight columns become four. = 8. The bottleneck = Let us convolve once more. This is the bottom of the U, a 2 x 4 block that is everything the network kept. = 9. Spread it out = We start back up. The transposed convolution writes each bottleneck value into a wider grid, leaving gaps between them. = 10. Transposed convolution 1 = Let us fill those gaps by convolving over the spread-out grid. Four columns become eight. = 11. The first skip = We copy the encoder's matching row straight across. This is the skip connection, and it is the whole reason a U-Net can recover detail that pooling threw away. = 12. Convolution with the skip = Let us convolve the upsampled features together with the copied ones. = 13. Spread it out again = Same as step 9, one level up. = 14. Transposed convolution 2 = Eight columns become sixteen, back to the width we started at. = 15. The second skip = The encoder's first feature map comes across, the one made before any pooling happened. = 16. Convolution and ReLU = We convolve, then cross out every negative and set it to zero. = 17. Output convolution = Let us apply the last kernel. Out comes R', G' and B', an image the same size as the one we started with. The outputs: R' = [3, 0, 7, 0, 7, 0, 17, 0, 3, 0, 9, 0, 2, 0, 6, 0] G' = [1, 20, 1, 10, 1, 12, 1, 19, 2, 5, 1, 11, 1, 3, 1, 7] B' = [4, 20, 8, 10, 8, 12, 18, 19, 5, 5, 10, 11, 3, 3, 7, 7] Congrats! You just calculated a U-Net by hand. 💾 Save this post!
0:37

U-Net by hand ✍️ ~ 17 steps walkthrough below I consider U-Net as a key milestone in deep learning, the first image-to-image model that really worked! It came out of medical imaging, an unusual place, not from NeurIPS or CVPR or ACL. Now it is the backbone of diffusion models, which you see in almost all modern image generation models. I drew the network as a C so the matrix multiplication flows naturally down. Tilt your head to the right and it is a U again. 🤣 Goal: push a 3 x 16 image down to a 2 x 4 bottleneck and back out again, filling in every cell yourself. = 1. Given = An image of three channels, R, G and B, sixteen pixels wide, and every kernel the network will use. = 2. Convolution 1 = Let us slide the first kernel over the image. Each output is one multiply-and-add over a 2 x 3 window, and the result is the green feature map. = 3. Find the maxima = We circle the largest value in each 1 x 2 window. Circling first is worth the extra step: it is the pooling decision, made before anything is written down. = 4. Max pool 1 = Let us copy those maxima down. Sixteen columns become eight, and half the detail is gone for good. = 5. Convolution 2 = We convolve again with the second kernel, deeper into the contracting path. The feature map is blue now. = 6. Find the maxima again = Same move as step 3, on the blue map. = 7. Max pool 2 = Eight columns become four. = 8. The bottleneck = Let us convolve once more. This is the bottom of the U, a 2 x 4 block that is everything the network kept. = 9. Spread it out = We start back up. The transposed convolution writes each bottleneck value into a wider grid, leaving gaps between them. = 10. Transposed convolution 1 = Let us fill those gaps by convolving over the spread-out grid. Four columns become eight. = 11. The first skip = We copy the encoder's matching row straight across. This is the skip connection, and it is the whole reason a U-Net can recover detail that pooling threw away. = 12. Convolution with the skip = Let us convolve the upsampled features together with the copied ones. = 13. Spread it out again = Same as step 9, one level up. = 14. Transposed convolution 2 = Eight columns become sixteen, back to the width we started at. = 15. The second skip = The encoder's first feature map comes across, the one made before any pooling happened. = 16. Convolution and ReLU = We convolve, then cross out every negative and set it to zero. = 17. Output convolution = Let us apply the last kernel. Out comes R', G' and B', an image the same size as the one we started with. The outputs: R' = [3, 0, 7, 0, 7, 0, 17, 0, 3, 0, 9, 0, 2, 0, 6, 0] G' = [1, 20, 1, 10, 1, 12, 1, 19, 2, 5, 1, 11, 1, 3, 1, 7] B' = [4, 20, 8, 10, 8, 12, 18, 19, 5, 5, 10, 11, 3, 3, 7, 7] Congrats! You just calculated a U-Net by hand. 💾 Save this post!

Tom Yeh

17,510 просмотров • 23 дней назад

SVM by hand ✍️ ~ 19 steps walkthrough below (Linear vs RBF) Support Vector Machines reigned supreme in machine learning before the deep learning revolution. An SVM predicts with dot products, the same matrix multiplication every model uses. What it does not do is train by backpropagation: it is fitted by convex optimization, so there is no matrix-multiplication backward pass for a GPU to accelerate. I drew and calculated two SVMs by hand: a linear one (top) and an RBF one (bottom), classifying the same two test vectors. Goal: turn six training vectors and their learned coefficients into a prediction, and see what changing the kernel actually changes. = 1. Given = Six training vectors, their labels, and the coefficients and bias already learned. A coefficient of zero means that vector is not a support vector: too far from the boundary to matter. = 2. Linear kernel, test vector 1 = Let us take the dot product of the test vector with every training vector. The dot product stands in for cosine similarity, and the column of results is the first column of the kernel matrix K. = 3. Linear kernel, test vector 2 = We do the same for the second, and K is complete. = 4. Signed weights = Let us multiply each coefficient by its label. The second training vector drops out here, because its coefficient is 0. = 5. Weighted combination = We multiply the signed weights through K and add the bias b. The result is a signed distance to the decision boundary: 17 and 5. = 6. Classify = Let us take the sign. Both are positive. = 7 to 11. RBF kernel, test vector 1 = Now the same picture with a different kernel, in five moves: square the differences, sum them, take the square root for the L2 distance, multiply by minus gamma, and raise e to that power. The negation is what turns a distance into a similarity, and gamma controls how far a single training vector's influence reaches. = 12 to 16. RBF kernel, test vector 2 = We repeat all five. The numbers change, the moves do not. = 17 to 19. Decision boundary, again = Signed weights, weighted combination, sign. Identical arithmetic to steps 4 through 6, on a K that was built a completely different way. The outputs: Linear K, first column = [13, 25, 12, 15, 19, 27] Linear decision values = 17 and 5, both positive RBF decision values = -2 and 1, so negative and positive The takeaway: the kernel is the only thing that changed, and it changed the answer. The linear SVM calls both test vectors positive; the RBF one splits them. Everything after the kernel matrix, the signed weights and the weighted combination and the sign, is the same page of arithmetic twice. 💾 Save this post!
0:44

SVM by hand ✍️ ~ 19 steps walkthrough below (Linear vs RBF) Support Vector Machines reigned supreme in machine learning before the deep learning revolution. An SVM predicts with dot products, the same matrix multiplication every model uses. What it does not do is train by backpropagation: it is fitted by convex optimization, so there is no matrix-multiplication backward pass for a GPU to accelerate. I drew and calculated two SVMs by hand: a linear one (top) and an RBF one (bottom), classifying the same two test vectors. Goal: turn six training vectors and their learned coefficients into a prediction, and see what changing the kernel actually changes. = 1. Given = Six training vectors, their labels, and the coefficients and bias already learned. A coefficient of zero means that vector is not a support vector: too far from the boundary to matter. = 2. Linear kernel, test vector 1 = Let us take the dot product of the test vector with every training vector. The dot product stands in for cosine similarity, and the column of results is the first column of the kernel matrix K. = 3. Linear kernel, test vector 2 = We do the same for the second, and K is complete. = 4. Signed weights = Let us multiply each coefficient by its label. The second training vector drops out here, because its coefficient is 0. = 5. Weighted combination = We multiply the signed weights through K and add the bias b. The result is a signed distance to the decision boundary: 17 and 5. = 6. Classify = Let us take the sign. Both are positive. = 7 to 11. RBF kernel, test vector 1 = Now the same picture with a different kernel, in five moves: square the differences, sum them, take the square root for the L2 distance, multiply by minus gamma, and raise e to that power. The negation is what turns a distance into a similarity, and gamma controls how far a single training vector's influence reaches. = 12 to 16. RBF kernel, test vector 2 = We repeat all five. The numbers change, the moves do not. = 17 to 19. Decision boundary, again = Signed weights, weighted combination, sign. Identical arithmetic to steps 4 through 6, on a K that was built a completely different way. The outputs: Linear K, first column = [13, 25, 12, 15, 19, 27] Linear decision values = 17 and 5, both positive RBF decision values = -2 and 1, so negative and positive The takeaway: the kernel is the only thing that changed, and it changed the answer. The linear SVM calls both test vectors positive; the RBF one splits them. Everything after the kernel matrix, the signed weights and the weighted combination and the sign, is the same page of arithmetic twice. 💾 Save this post!

Tom Yeh

16,916 просмотров • 25 дней назад

Vector Database by hand ✍️ ~ 10 steps walkthrough below Vector databases are the backbone of Retrieval Augmented Generation (RAG). How do they actually work? Goal: index three sentences, then answer a query by finding the nearest one, filling in every cell yourself. = 1. Given = A dataset of three sentences, three words each. In practice it is millions of them. = 2. Word embeddings = Let us look up each word in an embedding table. Here the vocabulary is 22 words; in practice it is tens of thousands, and the vectors have thousands of dimensions rather than four. = 3. Encoding = We feed the sequence to an encoder, one linear layer and a ReLU, and get one feature vector per word. In practice the encoder is a transformer. = 4. Mean pooling = Let us average across the columns. Three word vectors collapse into one, which is what people mean by a text embedding or a sentence embedding. = 5. Indexing = We multiply by a projection matrix and the four dimensions become two. It is doing the job of a hash: a short representation that is faster to compare, and it is what gets saved in the vector storage. = 6. Process "who are you" = Let us repeat steps 2 to 5 on the second sentence. = 7. Process "who am I" = We do it a third time. The database is now indexed. = 8. Query "am I you" = Let us push the query through the very same pipeline: lookup, encoder, mean pooling, projection, and it lands as a 2D vector in the same space. = 9. Dot products = We transpose the query and multiply, which takes the dot product against every stored vector at once. The dot product is the estimate of similarity. = 10. Nearest neighbour = Let us scan for the largest: 60/9 beats 44/9 and 40/9, so the answer is "who am I". Scanning billions of vectors one at a time is what makes this the slow step in practice, which is why real databases use an approximate nearest neighbour index like HNSW. The outputs: Stored index vectors = [5/3, 2/3], [5/3, 0], [7/3, 2/3] Query vector = [8/3, 2/3] Dot products = 44/9, 40/9, 60/9 Nearest neighbour = "who am I" The takeaway: a vector database is an embedding pipeline, a projection, and a dot product. Every step here is arithmetic you can do in pen, which is worth remembering when the word "database" makes it sound like something else. 💾 Save this post!
0:31

Vector Database by hand ✍️ ~ 10 steps walkthrough below Vector databases are the backbone of Retrieval Augmented Generation (RAG). How do they actually work? Goal: index three sentences, then answer a query by finding the nearest one, filling in every cell yourself. = 1. Given = A dataset of three sentences, three words each. In practice it is millions of them. = 2. Word embeddings = Let us look up each word in an embedding table. Here the vocabulary is 22 words; in practice it is tens of thousands, and the vectors have thousands of dimensions rather than four. = 3. Encoding = We feed the sequence to an encoder, one linear layer and a ReLU, and get one feature vector per word. In practice the encoder is a transformer. = 4. Mean pooling = Let us average across the columns. Three word vectors collapse into one, which is what people mean by a text embedding or a sentence embedding. = 5. Indexing = We multiply by a projection matrix and the four dimensions become two. It is doing the job of a hash: a short representation that is faster to compare, and it is what gets saved in the vector storage. = 6. Process "who are you" = Let us repeat steps 2 to 5 on the second sentence. = 7. Process "who am I" = We do it a third time. The database is now indexed. = 8. Query "am I you" = Let us push the query through the very same pipeline: lookup, encoder, mean pooling, projection, and it lands as a 2D vector in the same space. = 9. Dot products = We transpose the query and multiply, which takes the dot product against every stored vector at once. The dot product is the estimate of similarity. = 10. Nearest neighbour = Let us scan for the largest: 60/9 beats 44/9 and 40/9, so the answer is "who am I". Scanning billions of vectors one at a time is what makes this the slow step in practice, which is why real databases use an approximate nearest neighbour index like HNSW. The outputs: Stored index vectors = [5/3, 2/3], [5/3, 0], [7/3, 2/3] Query vector = [8/3, 2/3] Dot products = 44/9, 40/9, 60/9 Nearest neighbour = "who am I" The takeaway: a vector database is an embedding pipeline, a projection, and a dot product. Every step here is arithmetic you can do in pen, which is worth remembering when the word "database" makes it sound like something else. 💾 Save this post!

Tom Yeh

35,755 просмотров • 24 дней назад

LSTM by hand ✍️ ~ 15 steps walkthrough below Since Hochreiter and Schmidhuber introduced them in 1997, LSTMs were the most effective way to handle long sequences, right up until the Transformer wave. They are a recurrent network: they read one input at a time and carry a memory forward. Lately recurrence is back in fashion (Mamba), because attention does not scale to hundreds of thousands of tokens. So I drew and calculated one entirely by hand. Goal: run an LSTM cell over a sequence of three inputs, filling in every gate and memory cell yourself. 1. Given Three inputs X1, X2, X3, and four weight matrices: forget, input, candidate, and output. 2. Initialize Let us set the previous hidden state h0 and the memory cell C0 to their start values. 3. Linear transform We multiply the four weight matrices by the stack of the current input, the previous hidden state, and a 1. 4. The gates Let us squash three of those results with sigmoid, giving the forget, input, and output gates, each between 0 and 1. 5. Update the memory We forget part of the old memory (C0 times the forget gate) and add the new (the candidate times the input gate). That is the new memory C1. 6. Candidate output Let us apply tanh to the new memory. 7. Update the hidden state We multiply that candidate by the output gate. The result is h1. 8. Process X2 Copy h1 and C1 forward, then repeat the whole cell: linear transform, the gates, update memory to C2, output gate to h2. 9. Process X3 Once more. Copy h2 and C2 forward, repeat, and read off h3, the final hidden state. Now you can show off to your friends that you calculated an LSTM by hand. ✍️😉 💾 Save this post! #AIbyHand #LSTM #DeepLearning
0:29

LSTM by hand ✍️ ~ 15 steps walkthrough below Since Hochreiter and Schmidhuber introduced them in 1997, LSTMs were the most effective way to handle long sequences, right up until the Transformer wave. They are a recurrent network: they read one input at a time and carry a memory forward. Lately recurrence is back in fashion (Mamba), because attention does not scale to hundreds of thousands of tokens. So I drew and calculated one entirely by hand. Goal: run an LSTM cell over a sequence of three inputs, filling in every gate and memory cell yourself. 1. Given Three inputs X1, X2, X3, and four weight matrices: forget, input, candidate, and output. 2. Initialize Let us set the previous hidden state h0 and the memory cell C0 to their start values. 3. Linear transform We multiply the four weight matrices by the stack of the current input, the previous hidden state, and a 1. 4. The gates Let us squash three of those results with sigmoid, giving the forget, input, and output gates, each between 0 and 1. 5. Update the memory We forget part of the old memory (C0 times the forget gate) and add the new (the candidate times the input gate). That is the new memory C1. 6. Candidate output Let us apply tanh to the new memory. 7. Update the hidden state We multiply that candidate by the output gate. The result is h1. 8. Process X2 Copy h1 and C1 forward, then repeat the whole cell: linear transform, the gates, update memory to C2, output gate to h2. 9. Process X3 Once more. Copy h2 and C2 forward, repeat, and read off h3, the final hidden state. Now you can show off to your friends that you calculated an LSTM by hand. ✍️😉 💾 Save this post! #AIbyHand #LSTM #DeepLearning

Tom Yeh

18,144 просмотров • 1 месяц назад

Layer by LAYER... The Hidden Hand's protections are being stripped away. Every recovered child is debriefed to learn what happened to them. Leading to more raids/arrests. A HIDDEN WORLD is being dismantled right in front of you. A WORLD that was protected by US government prior to January 2025. And is still desperately trying to keep the illusion going in Congress and the Senate <---reread that. 1/2
1:31

Layer by LAYER... The Hidden Hand's protections are being stripped away. Every recovered child is debriefed to learn what happened to them. Leading to more raids/arrests. A HIDDEN WORLD is being dismantled right in front of you. A WORLD that was protected by US government prior to January 2025. And is still desperately trying to keep the illusion going in Congress and the Senate <---reread that. 1/2

DuaneCates001

29,646 просмотров • 2 месяцев назад

We asked James Wang why Cerebras can run large models so quickly. His answer: Inference speed is mostly a memory problem. Instead of constantly pulling weights from external memory, Cerebras splits the model across multiple wafers and pipelines the layers together. “Inference is all about memory bandwidth.” “You don’t want to store weights external to the wafer, because the second it’s outside, it becomes much slower.” “We put the weights on the inside.” “The team built a new software stack that basically lets us split the models by layer and store them layer by layer across multiple wafers.” “That allows us to never have to read from external memory, and that’s what makes it so fast.” “For context, Claude is about 100 tokens per second.”
1:31

We asked James Wang why Cerebras can run large models so quickly. His answer: Inference speed is mostly a memory problem. Instead of constantly pulling weights from external memory, Cerebras splits the model across multiple wafers and pipelines the layers together. “Inference is all about memory bandwidth.” “You don’t want to store weights external to the wafer, because the second it’s outside, it becomes much slower.” “We put the weights on the inside.” “The team built a new software stack that basically lets us split the models by layer and store them layer by layer across multiple wafers.” “That allows us to never have to read from external memory, and that’s what makes it so fast.” “For context, Claude is about 100 tokens per second.”

MTS

27,935 просмотров • 3 месяцев назад

SORA by Hand ✍️ OpenAI’s #SORA took over the Internet when it was announced earlier this year. The technology behind Sora is the Diffusion Transformer (DiT) developed by William Peebles and Shining Xie. How does DiT work? 𝗚𝗼𝗮𝗹: Generate a video conditioned by a text prompt and a series of diffusion steps [1] Given ↳ Video ↳ Prompt: "sora is sky" ↳ Diffusion step: t = 3 [2] Video → Patches ↳ Divide all pixels in all frames into 4 spacetime patches [3] Visual Encoder: Pixels 🟨 → Latent 🟩 ↳ Multiply the patches with weights and biases, followed by ReLU ↳ The result is a latent feature vector per patch ↳ The purpose is dimension reduction from 4 (2x2x1) to 2 (2x1). ↳ In the paper, the reduction is 196,608 (256x256x3)→ 4096 (32x32x4) [4] ⬛ Add Noise ↳ Sample a noise according to the diffusion time step t. Typically, the larger the t, the smaller the noise. ↳ Add the Sampled Noise to latent features to obtain Noised Latent. ↳ The goal is to purposely add noise to a video and ask the model to guess what that noise is. ↳ This is analogous to training a language model by purposely deleting a word in a sentence and ask the model to guess what the deleted word was. [5-7] 🟪 Conditioning by Adaptive Layer Norm [5] Encode Conditions ↳ Encode "sora is sky" into a text embedding vector [0,1,-1]. ↳ Encode t = 3 to as a binary vector [1,1]. ↳ Concatenate the two vectors in to a 5D column vector. [6] Estimate Scale/Shift ↳ Multiply the combined vector with weights and biases ↳ The goal is to estimate the scale [2,-1] and shift [-1,5]. ↳ Copy the result to (X) and (+) [7] Apply Scale/Sift ↳ Scale the noised latent by [2,-1] ↳ Shifted the scaled noised latent by [-1, 5] ↳ The result is "conditioned" noise latent. [8-10] Transformer [8] Self-Attention ↳ Feed the conditioned noised latent to Query-Key function to obtain a self-attention matrix ↳ Value is omitted for simplicity [9] Attention Pooling ↳ Multiply the conditioned noised latent with the self-attention matrix ↳ The result are attention weighted features [10] Pointwise Feed Forward Network ↳ Multiply the attention weighted features with weights and biases ↳ The result is the Predicted Noise 🏋️‍♂️ 𝗧𝗿𝗮𝗶𝗻 [11] ↳ Calculate MSE loss gradients by taking the different between the Predicted Noise and the Sampled Noise (ground truth). ↳ Use the loss gradients to kick off backpropagation to update all learnable parameters (red borders) ↳ Note the visual encoder and decoder's parameters are frozen (blue borders) 🎨 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗲 (𝗦𝗮𝗺𝗽𝗹𝗲) [12] Denoise ↳ Subtract the predicted noise from the noised latent to obtain the noise-free latent [13] Visual Decoder: Latent 🟩 → Pixels 🟨 ↳ Multiply the patches with weights and biases, followed by ReLU [14] Patches → Video ↳ Rearrange patches into a sequence of video frames.
0:43

SORA by Hand ✍️ OpenAI’s #SORA took over the Internet when it was announced earlier this year. The technology behind Sora is the Diffusion Transformer (DiT) developed by William Peebles and Shining Xie. How does DiT work? 𝗚𝗼𝗮𝗹: Generate a video conditioned by a text prompt and a series of diffusion steps [1] Given ↳ Video ↳ Prompt: "sora is sky" ↳ Diffusion step: t = 3 [2] Video → Patches ↳ Divide all pixels in all frames into 4 spacetime patches [3] Visual Encoder: Pixels 🟨 → Latent 🟩 ↳ Multiply the patches with weights and biases, followed by ReLU ↳ The result is a latent feature vector per patch ↳ The purpose is dimension reduction from 4 (2x2x1) to 2 (2x1). ↳ In the paper, the reduction is 196,608 (256x256x3)→ 4096 (32x32x4) [4] ⬛ Add Noise ↳ Sample a noise according to the diffusion time step t. Typically, the larger the t, the smaller the noise. ↳ Add the Sampled Noise to latent features to obtain Noised Latent. ↳ The goal is to purposely add noise to a video and ask the model to guess what that noise is. ↳ This is analogous to training a language model by purposely deleting a word in a sentence and ask the model to guess what the deleted word was. [5-7] 🟪 Conditioning by Adaptive Layer Norm [5] Encode Conditions ↳ Encode "sora is sky" into a text embedding vector [0,1,-1]. ↳ Encode t = 3 to as a binary vector [1,1]. ↳ Concatenate the two vectors in to a 5D column vector. [6] Estimate Scale/Shift ↳ Multiply the combined vector with weights and biases ↳ The goal is to estimate the scale [2,-1] and shift [-1,5]. ↳ Copy the result to (X) and (+) [7] Apply Scale/Sift ↳ Scale the noised latent by [2,-1] ↳ Shifted the scaled noised latent by [-1, 5] ↳ The result is "conditioned" noise latent. [8-10] Transformer [8] Self-Attention ↳ Feed the conditioned noised latent to Query-Key function to obtain a self-attention matrix ↳ Value is omitted for simplicity [9] Attention Pooling ↳ Multiply the conditioned noised latent with the self-attention matrix ↳ The result are attention weighted features [10] Pointwise Feed Forward Network ↳ Multiply the attention weighted features with weights and biases ↳ The result is the Predicted Noise 🏋️‍♂️ 𝗧𝗿𝗮𝗶𝗻 [11] ↳ Calculate MSE loss gradients by taking the different between the Predicted Noise and the Sampled Noise (ground truth). ↳ Use the loss gradients to kick off backpropagation to update all learnable parameters (red borders) ↳ Note the visual encoder and decoder's parameters are frozen (blue borders) 🎨 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗲 (𝗦𝗮𝗺𝗽𝗹𝗲) [12] Denoise ↳ Subtract the predicted noise from the noised latent to obtain the noise-free latent [13] Visual Decoder: Latent 🟩 → Pixels 🟨 ↳ Multiply the patches with weights and biases, followed by ReLU [14] Patches → Video ↳ Rearrange patches into a sequence of video frames.

Tom Yeh

238,303 просмотров • 2 лет назад

🎓Learn how to create a powerful Dynamic Embers smart material in a matter of seconds, in my latest video series (AAA) Pro Tips! This smart material can be used on virtually any 3D asset. ___________________________________________________ In this video, the steps are as follows: 1. Create a fill layer to then apply the base maps for our asset into it. 2. Bake out the world space, AO, Curvature, and Position maps. These maps will be crucial for the smart material to function. Also, be sure to add an emissive channel to the project. 3. Next, create another fill layer labeled (Fire animation gradient) with a black mask and inside the mask apply a paint layer. Select the paint layer and paint a simple pill shape using a soft brush. Add an anchorpoint to the mask and disable all channels in this fill layer. 4. Create a folder labeled (embers smart material) and set it to pass through. Add a paint layer inside this folder and set it to pass through as well followed by an anchorpoint. 5. Create another fill layer and apply the paint layers anchorpoint information into its color channel. From here an HSL filter should be added to desaturate and darken the colors. 6. Next, create an additional fill layer with an orange-colored emissive channel. Add a black mask to this fill layer and inside its mask apply a dirt generator. This will give us the illusion of the embers glowing from inside the charred tree. Duplicate this fill layer and tweak its emissive color to a yellowish hue and reduce its mask amount. Do this a second time but with a white-colored emissive channel. This will now create an effect of the embers appearing brighter the deeper they are inside the tree. 7. Lastly, add these layers into a folder with a black mask and apply the (Fire Animation Gradient) anchorpoint into its mask via a fill layer. Now we have a charred embers effect wherever we paint using the paint layer created inside the gradient mask! This gradient mask can also be exported into Unreal engine to create some stunning real-time dynamic animations or the new material can simply be used for in-engine vertex painting. ____________________________________________________ More AAA Game Dev Tips can be found on my YouTube channel here: Stay tuned for more weekly Tips! Happy Texturing!💚 #gamedev #gameart #3dmodeling #texturing #madewithsubstance #ue5 #unrealengine #unity #vfx
1:32

🎓Learn how to create a powerful Dynamic Embers smart material in a matter of seconds, in my latest video series (AAA) Pro Tips! This smart material can be used on virtually any 3D asset. ___________________________________________________ In this video, the steps are as follows: 1. Create a fill layer to then apply the base maps for our asset into it. 2. Bake out the world space, AO, Curvature, and Position maps. These maps will be crucial for the smart material to function. Also, be sure to add an emissive channel to the project. 3. Next, create another fill layer labeled (Fire animation gradient) with a black mask and inside the mask apply a paint layer. Select the paint layer and paint a simple pill shape using a soft brush. Add an anchorpoint to the mask and disable all channels in this fill layer. 4. Create a folder labeled (embers smart material) and set it to pass through. Add a paint layer inside this folder and set it to pass through as well followed by an anchorpoint. 5. Create another fill layer and apply the paint layers anchorpoint information into its color channel. From here an HSL filter should be added to desaturate and darken the colors. 6. Next, create an additional fill layer with an orange-colored emissive channel. Add a black mask to this fill layer and inside its mask apply a dirt generator. This will give us the illusion of the embers glowing from inside the charred tree. Duplicate this fill layer and tweak its emissive color to a yellowish hue and reduce its mask amount. Do this a second time but with a white-colored emissive channel. This will now create an effect of the embers appearing brighter the deeper they are inside the tree. 7. Lastly, add these layers into a folder with a black mask and apply the (Fire Animation Gradient) anchorpoint into its mask via a fill layer. Now we have a charred embers effect wherever we paint using the paint layer created inside the gradient mask! This gradient mask can also be exported into Unreal engine to create some stunning real-time dynamic animations or the new material can simply be used for in-engine vertex painting. ____________________________________________________ More AAA Game Dev Tips can be found on my YouTube channel here: Stay tuned for more weekly Tips! Happy Texturing!💚 #gamedev #gameart #3dmodeling #texturing #madewithsubstance #ue5 #unrealengine #unity #vfx

Cohen Brawley

46,031 просмотров • 2 лет назад

What if #AI became as decentralized as #Bitcoin? We sat down with our new friend 3700 from Bitcoin Virtual Machine to hear what their incredible team of anons are working on - "Truly Open AI." Full interview here:👇 1: What positive impact will Layer 2s have on Bitcoin? Layer 2s on Bitcoin open up opportunities for innovation, allowing developers to build dApps and smart contracts on top of Bitcoin, expanding its utility and use cases. By submitting transactions for final settlement on the Bitcoin network, Bitcoin Layer 2 networks claim to achieve the same (or close to) level of security and decentralization as the Bitcoin blockchain. Building a separate execution layer allows them the freedom to employ several technologies (such as rollups). Layer 2 can significantly improve Bitcoin's scalability by processing transactions off-chain, reducing congestion on the main blockchain. Overall, Layer 2s on Bitcoin have the potential to address some of Bitcoin's key limitations, making it more efficient, accessible, and versatile in the long run. 2: What does the ETF approval mean for Layer 2 on Bitcoin? The approval of ETF could potentially have several implications for Layer 2 on Bitcoin: Innovation and Development: With a growing interest in Bitcoin spurred by ETF approval, there could be a surge in research and development efforts focused on enhancing Layer 2. Developers and projects may be incentivized to create new and improved Layer 2 protocols to meet the evolving needs of the expanding Bitcoin ecosystem. An ETF approval could boost mainstream Bitcoin adoption and liquidity. This influx of users may also drive interest in Layer 2 on Bitcoin as a means to enhance the scalability and functionality of Bitcoin. 3: What are the primary challenges facing L2s on Bitcoin? The interoperability of different Layer 2s and their compatibility with Bitcoin's main blockchain can be a challenge. Ensuring seamless interaction between various Layer 2 networks and the Bitcoin blockchain is essential for a cohesive and efficient ecosystem. Some Layer 2s may introduce centralization risks if they rely heavily on centralized entities or trusted intermediaries. Maintaining decentralization and censorship resistance, which are core tenets of Bitcoin, while scaling with Layer 2s is a challenge. 4: What aspects of Layer 2 solutions for Bitcoin are you most enthusiastic about? AI represents one of the cornerstones of our modern era. However, achieving a decentralized AI infrastructure, owned and managed by users, has posed significant challenges. The primary obstacle has been the limited capacity to store and execute AI models due to size and computational limitations. To address this challenge, we propose a new blockchain architecture enabling developers to deploy their own Bitcoin Layer 2 solutions tailored specifically for AI tasks, called Truly Open AI. These Layer 2 blockchains are optimized to handle computationally intensive tasks, such as matrix multiplication, directly on-chain. These Bitcoin Layer 2 solutions offer exceptional throughput, minimal latency, and cost-effectiveness. AI dApps are programmed as Solidity smart contracts, ensuring they operate precisely as intended, free from interference or manipulation. Our BVM AI Contracts Library simplifies the integration of neural networks into dApps, empowering developers to embed AI seamlessly. In summary, I'm particularly enthusiastic about the potential of Layer 2 solutions for Bitcoin to revolutionize decentralized AI by providing scalability, security, and accessibility. 5: How is your Layer 2 different from others being built? BVM distinguishes itself as a Modular infrastructure that empowers thousands of distinct Bitcoin Layer 2 networks, spanning Gaming, DeFi, Social, and AI applications. We're continuously enriching the BVM Module Store with new modules to enhance its capabilities. With each new module, builders gain access to a wider array of tools to explore different use cases on the Bitcoin network. Recent additions include the Filecoin module for affordable storage and the AI Contracts Library for constructing AI-powered Bitcoin Layer 2 chains. We're also gearing up to release a ZK roll-up module in the coming weeks to offer an alternative to the standard optimistic roll-up. We aim to simplify the process of launching a Bitcoin Layer 2 network customized to specific requirements. Think of it as a SaaS offering with predefined best practices. Whether it's a DeFi Bitcoin Layer 2 or a GameFi Bitcoin Layer 2, we provide default solutions tailored to each use case. We're dedicated to expanding the BVM ecosystem by incentivizing more builders to join the Bitcoin network. Through various programs and grants, we support builders in covering their operational costs for Bitcoin Layer 2. Additionally, we offer rewards akin to 'L2 mining' to those who contribute to expanding the user base and total value locked on the network. In summary, BVM stands out with its modular infrastructure, tailored solutions, and efforts to grow the Bitcoin ecosystem.
1:16

What if #AI became as decentralized as #Bitcoin? We sat down with our new friend 3700 from Bitcoin Virtual Machine to hear what their incredible team of anons are working on - "Truly Open AI." Full interview here:👇 1: What positive impact will Layer 2s have on Bitcoin? Layer 2s on Bitcoin open up opportunities for innovation, allowing developers to build dApps and smart contracts on top of Bitcoin, expanding its utility and use cases. By submitting transactions for final settlement on the Bitcoin network, Bitcoin Layer 2 networks claim to achieve the same (or close to) level of security and decentralization as the Bitcoin blockchain. Building a separate execution layer allows them the freedom to employ several technologies (such as rollups). Layer 2 can significantly improve Bitcoin's scalability by processing transactions off-chain, reducing congestion on the main blockchain. Overall, Layer 2s on Bitcoin have the potential to address some of Bitcoin's key limitations, making it more efficient, accessible, and versatile in the long run. 2: What does the ETF approval mean for Layer 2 on Bitcoin? The approval of ETF could potentially have several implications for Layer 2 on Bitcoin: Innovation and Development: With a growing interest in Bitcoin spurred by ETF approval, there could be a surge in research and development efforts focused on enhancing Layer 2. Developers and projects may be incentivized to create new and improved Layer 2 protocols to meet the evolving needs of the expanding Bitcoin ecosystem. An ETF approval could boost mainstream Bitcoin adoption and liquidity. This influx of users may also drive interest in Layer 2 on Bitcoin as a means to enhance the scalability and functionality of Bitcoin. 3: What are the primary challenges facing L2s on Bitcoin? The interoperability of different Layer 2s and their compatibility with Bitcoin's main blockchain can be a challenge. Ensuring seamless interaction between various Layer 2 networks and the Bitcoin blockchain is essential for a cohesive and efficient ecosystem. Some Layer 2s may introduce centralization risks if they rely heavily on centralized entities or trusted intermediaries. Maintaining decentralization and censorship resistance, which are core tenets of Bitcoin, while scaling with Layer 2s is a challenge. 4: What aspects of Layer 2 solutions for Bitcoin are you most enthusiastic about? AI represents one of the cornerstones of our modern era. However, achieving a decentralized AI infrastructure, owned and managed by users, has posed significant challenges. The primary obstacle has been the limited capacity to store and execute AI models due to size and computational limitations. To address this challenge, we propose a new blockchain architecture enabling developers to deploy their own Bitcoin Layer 2 solutions tailored specifically for AI tasks, called Truly Open AI. These Layer 2 blockchains are optimized to handle computationally intensive tasks, such as matrix multiplication, directly on-chain. These Bitcoin Layer 2 solutions offer exceptional throughput, minimal latency, and cost-effectiveness. AI dApps are programmed as Solidity smart contracts, ensuring they operate precisely as intended, free from interference or manipulation. Our BVM AI Contracts Library simplifies the integration of neural networks into dApps, empowering developers to embed AI seamlessly. In summary, I'm particularly enthusiastic about the potential of Layer 2 solutions for Bitcoin to revolutionize decentralized AI by providing scalability, security, and accessibility. 5: How is your Layer 2 different from others being built? BVM distinguishes itself as a Modular infrastructure that empowers thousands of distinct Bitcoin Layer 2 networks, spanning Gaming, DeFi, Social, and AI applications. We're continuously enriching the BVM Module Store with new modules to enhance its capabilities. With each new module, builders gain access to a wider array of tools to explore different use cases on the Bitcoin network. Recent additions include the Filecoin module for affordable storage and the AI Contracts Library for constructing AI-powered Bitcoin Layer 2 chains. We're also gearing up to release a ZK roll-up module in the coming weeks to offer an alternative to the standard optimistic roll-up. We aim to simplify the process of launching a Bitcoin Layer 2 network customized to specific requirements. Think of it as a SaaS offering with predefined best practices. Whether it's a DeFi Bitcoin Layer 2 or a GameFi Bitcoin Layer 2, we provide default solutions tailored to each use case. We're dedicated to expanding the BVM ecosystem by incentivizing more builders to join the Bitcoin network. Through various programs and grants, we support builders in covering their operational costs for Bitcoin Layer 2. Additionally, we offer rewards akin to 'L2 mining' to those who contribute to expanding the user base and total value locked on the network. In summary, BVM stands out with its modular infrastructure, tailored solutions, and efforts to grow the Bitcoin ecosystem.

Supra

83,548 просмотров • 2 лет назад

CLIP by hand ✍️ ~ 13 steps walkthrough below CLIP, Contrastive Language-Image Pre-training, is OpenAI's answer to a question that sounds impossible: how do you put a sentence and a picture in the same space? CLIP shipped when OpenAI was still open, and those embeddings were shared far and wide. Almost every multimodal model you use today descends from them. How does it work? Goal: learn one shared embedding space for text and images. = 1. Given = A mini batch of three text-image pairs. OpenAI trained the original on 400 million. = 2. Text to vectors = Let us look up each word with word2vec. = 3. Image to vectors = We cut each image into two patches and flatten them. Now text and pixels are both just numbers. = 4. The other pairs = Repeat steps 2 and 3 for the rest of the batch. = 5. Encode = Let us push both sides through their encoders, a linear layer and a ReLU. In practice these are transformers, but the shape of the operation is the same. = 6. Mean pooling = We average across the columns, so each image and each sentence collapses to a single vector. = 7. Projection = The text vectors are 3D and the image vectors are 4D, so they cannot be compared at all. A linear layer projects both to 2D. That 2D space is the shared embedding space, and getting here is the whole point of the model. = 8. Prepare for matmul = Let us copy the text vectors down and the transposed image vectors across. = 9. MatMul = We multiply, which takes the dot product of every text vector with every image vector. Each cell is one estimate of how well a sentence matches a picture. = 10. Softmax, e to the power = Raise e to each cell. To keep it hand sized we approximate e with 3. = 11. Softmax, sum = Sum each row for image to text, each column for text to image. = 12. Softmax, normalize = Divide, and out come two similarity matrices, one per direction. = 13. Loss gradients = The targets are identity matrices: a pair that belongs together should score 1, every other cell 0. Subtract the target from the similarity and you have the gradients, in both directions. The takeaway: pairing a picture with a sentence comes down to a single dot product. Everything before step 9 is the work of getting them into one shared space, so that the dot product finally means something. 💾 Save this post!
0:36

CLIP by hand ✍️ ~ 13 steps walkthrough below CLIP, Contrastive Language-Image Pre-training, is OpenAI's answer to a question that sounds impossible: how do you put a sentence and a picture in the same space? CLIP shipped when OpenAI was still open, and those embeddings were shared far and wide. Almost every multimodal model you use today descends from them. How does it work? Goal: learn one shared embedding space for text and images. = 1. Given = A mini batch of three text-image pairs. OpenAI trained the original on 400 million. = 2. Text to vectors = Let us look up each word with word2vec. = 3. Image to vectors = We cut each image into two patches and flatten them. Now text and pixels are both just numbers. = 4. The other pairs = Repeat steps 2 and 3 for the rest of the batch. = 5. Encode = Let us push both sides through their encoders, a linear layer and a ReLU. In practice these are transformers, but the shape of the operation is the same. = 6. Mean pooling = We average across the columns, so each image and each sentence collapses to a single vector. = 7. Projection = The text vectors are 3D and the image vectors are 4D, so they cannot be compared at all. A linear layer projects both to 2D. That 2D space is the shared embedding space, and getting here is the whole point of the model. = 8. Prepare for matmul = Let us copy the text vectors down and the transposed image vectors across. = 9. MatMul = We multiply, which takes the dot product of every text vector with every image vector. Each cell is one estimate of how well a sentence matches a picture. = 10. Softmax, e to the power = Raise e to each cell. To keep it hand sized we approximate e with 3. = 11. Softmax, sum = Sum each row for image to text, each column for text to image. = 12. Softmax, normalize = Divide, and out come two similarity matrices, one per direction. = 13. Loss gradients = The targets are identity matrices: a pair that belongs together should score 1, every other cell 0. Subtract the target from the similarity and you have the gradients, in both directions. The takeaway: pairing a picture with a sentence comes down to a single dot product. Everything before step 9 is the work of getting them into one shared space, so that the dot product finally means something. 💾 Save this post!

Tom Yeh

20,750 просмотров • 17 дней назад

🟢Substance Painter Tip With "Anchor Points" you can link layers together (e.g. use one layer as a mask for another), which makes workflows much faster. Example: You have a shadow layer and want to add an outline around it. Instead of repainting the outline every time you change the shadows, use an Anchor Point: 1. Add an Anchor Point to the shadow layer 2. Add a Fill Filter for the shadow outline layer 3. For the Fill grayscale, select the Anchor Point 4. Add a "Mask Outline" filter 5. If you paint new shadows now, the outline will adjust automatically!
0:52

🟢Substance Painter Tip With "Anchor Points" you can link layers together (e.g. use one layer as a mask for another), which makes workflows much faster. Example: You have a shadow layer and want to add an outline around it. Instead of repainting the outline every time you change the shadows, use an Anchor Point: 1. Add an Anchor Point to the shadow layer 2. Add a Fill Filter for the shadow outline layer 3. For the Fill grayscale, select the Anchor Point 4. Add a "Mask Outline" filter 5. If you paint new shadows now, the outline will adjust automatically!

Keeki | 3D Artist

38,950 просмотров • 3 месяцев назад

Jensen Huang told a room of global investors that AI is not one industry. It is five stacked on top of each other. Most people are investing in layer four and ignoring layers one through three entirely. He called it the five-layer cake. Layer one is energy. Jensen said this is the single greatest opportunity for the energy industry in a hundred years. The first time in a century that the grid in most countries can actually attract serious capital. Nuclear, solar, wind, hydrogen, it does not matter what form. If it produces energy, it gets funded. Siemens, GE Vernova, Mitsubishi. That is why they are all doing so well right now. Layer two is chips, computers, networking, and silicon photonics. Everything that processes the intelligence. Layer three is infrastructure. Land, power, buildings, data center operations. Every single one in short supply today. Layer four is the model layer. OpenAI, Anthropic. The layer everyone talks about. Layer five is applications. Every startup applying AI to financial services, legal, healthcare, logistics, transportation. Last year alone, a hundred billion dollars of venture capital went into this layer. The single largest VC year in the history of humanity. Then he said the number that stopped me cold. We are putting one trillion dollars into this five-layer cake this year. That sounds enormous. Jensen thinks the AI industry will eventually run at twenty trillion dollars per year. We are one trillion in of a twenty trillion dollar per year ecosystem. Most people watching AI are staring at layer four. Jensen was describing layers one through five as a single compounding system where every layer feeds the one above it. The people who understand that will invest differently than the people who do not.
4:50

Jensen Huang told a room of global investors that AI is not one industry. It is five stacked on top of each other. Most people are investing in layer four and ignoring layers one through three entirely. He called it the five-layer cake. Layer one is energy. Jensen said this is the single greatest opportunity for the energy industry in a hundred years. The first time in a century that the grid in most countries can actually attract serious capital. Nuclear, solar, wind, hydrogen, it does not matter what form. If it produces energy, it gets funded. Siemens, GE Vernova, Mitsubishi. That is why they are all doing so well right now. Layer two is chips, computers, networking, and silicon photonics. Everything that processes the intelligence. Layer three is infrastructure. Land, power, buildings, data center operations. Every single one in short supply today. Layer four is the model layer. OpenAI, Anthropic. The layer everyone talks about. Layer five is applications. Every startup applying AI to financial services, legal, healthcare, logistics, transportation. Last year alone, a hundred billion dollars of venture capital went into this layer. The single largest VC year in the history of humanity. Then he said the number that stopped me cold. We are putting one trillion dollars into this five-layer cake this year. That sounds enormous. Jensen thinks the AI industry will eventually run at twenty trillion dollars per year. We are one trillion in of a twenty trillion dollar per year ecosystem. Most people watching AI are staring at layer four. Jensen was describing layers one through five as a single compounding system where every layer feeds the one above it. The people who understand that will invest differently than the people who do not.

Ihtesham Ali

98,071 просмотров • 2 месяцев назад