Video yükleniyor...
Video Yüklenemedi
[SVM] by Hand ✍️ Support Vector Machines (SVMs) reigned supreme in machine learning before the ascendancy of the deep learning revolution. This exercise compares Linear vs RBF SVMs---how they classify two test vectors, after SVMs are trained from six training vectors. [1] Given xi: Six training vectors (blue rows... show more
35,708 görüntüleme • 2 yıl önce •via X (Twitter)
0 Yorum
Yorum bulunmuyor
Orijinal gönderinin yorumları burada görünecek
![[Deep RNN] by Hand ✍️ A Deep Recurrent Neural Network (RNN) extends a basic single-layer RNN into multiple layers of hidden states, effectively incorporating deep learning into the RNN architecture. How does a Deep RNN work? [1] Given ↳ A sequence of four inputs X1, X2, X3, X4 ⬛️ ↳ Recurrent weights and biases for hidden layers a 🟩, b 🟧, c 🟪, and the output layer y 🟦. [2] Initialize Hidden States ↳ Set a0, b0, c0 to zeros — Process X1 (t = 1)— [3] First Hidden Layer (a) 🟩: a0 → a1 ↳ The transformation matrix is horizontal concatenation of input weights, hidden state weights and biases, visualized as [⬛️ | 🟩 | ⬜️] . ↳ The state matrix is vertical concatenation of input X1, previous hidden state a0, and an extra 1, visualized as [⬛️ ; 🟩 ; 1]. ↳ Multiply the two matrices to obtain new hidden state a1 = [0 ; 1]. [4] Second Hidden Layer (b) 🟪: b0 → b1 ↳ First layer a1 🟩 becomes the input. ↳ The transformation matrix is visualized as [🟩 | 🟪 | ⬜️]. ↳ The state matrix is the combination of a1, b0, and 1, visualized as [🟩; 🟪 ; 1]. ↳ Multiply the two matrices to obtain new hidden state b1 = [1; -1]. [5] Third Hidden Layer (c) 🟧: c0 → c1 ↳ Second layer b 🟪 becomes the input. ↳ The transformation matrix is visualized as [🟪 | 🟧 | ⬜️]. ↳ The state matrix is the combination of a1, b0, and 1, visualized as [🟪; 🟧; 1]. ↳ Multiply the two matrices to obtain new hidden state b1 = [1; -1]. [6] Output Layer (Y) 🟦 ↳ The transformation matrix is visualized as [🟧 | ⬜️]. ↳ The state matrix is the combination of c0 and , visualized as [🟧; 1]. ↳ Multiply the two matrices to obtain output Y1 = [3; 0; 3]. — Process X2 (t = 2)— [7] Previous Hidden States ↳ Copy the values of a1, b1, c1. [8] Hidden 🟩🟪🟧 + Output 🟦 ↳ Repeat [3]-[6] to obtain output Y2 = [5; 0; 4] — Process X3 (t = 3)— [9] Previous Hidden States ↳ Copy the values of a2, b2, c2. [10] Hidden 🟩🟪🟧 + Output 🟦 ↳ Repeat [3]-[6] to obtain output Y3 = [13; -1; 9] — Process X4 (t = 4)— [11] Previous Hidden States ↳ Copy the values of a3, b3, c3. [12] Hidden 🟩🟪🟧 + Output 🟦 ↳ Repeat [3]-[6] to obtain output Y4 = [15; 7; 2]](https://image.24vids.com/tw-1809362848054993158/ext_tw_video_thumb/1809362489563631616/pu/img/khsrKXTM6woSuL4P.jpg)
![[LSTM] by Hand ✍️ LSTMs have been the most effective architecture to process long sequences of data, until our world was taken over by the Transformers. LSTMs belong to the broader family of recurrent neural network (RNNs) that process data sequentially in a recurrent manner. Transformers, on the other hand, abandon recurrence and use self-attention instead to process data concurrently in parallel. Recently, there is renewed interest in recurrence as people realized self-attention doesn’t scale to extremely long sequences, like hundreds of thousands of tokens. Mamba is a good example to bring back recurrence. All of a sudden, it is cool to study LSTMs. How do LSTMs work? [1] Given ↳ 🟨 Input sequence X1, X2, X3 (d = 3) ↳ 🟩 Hidden state h (d = 2) ↳ 🟦 Memory C (d = 2) ↳ Weight matrices Wf, Wc, Wi, Wo Process t = 1 [2] Initialize ↳ Randomly set the previous hidden state h0 to [1, 1] and memory cells C0 to [0.3, -0.5] [3] Linear Transform ↳ Multiply the four weight matrices with the concatenation of current input (X1) and the previous hidden state (h0). ↳ The results are feature values, each is a linear combination of the current input and hidden state. [4] Non-linear Transform ↳ Apply sigmoid σ to obtain gate values (between 0 and 1). • Forget gate (f1): [-4, -6] → [0, 0] • Input gate (i1): [6, 4] → [1, 1] • Output gate (o1): [4, -5] → [1, 0] ↳ Apply tanh to obtain candidate memory values (between -1 and 1) • Candidate memory (C’1): [1, -6] → [0.8, -1] [5] Update Memory ↳ Forget (C0 .* f1): Element-wise multiply the current memory with forget gate values. ↳ Input (C’1 .* o1): Element-wise multiply the “candidate” memory with input gate values. ↳ Update the memory to C1 by adding the two terms above: C0 .* f1 + C’1 .* o1 = C1 [6] Candiate Output ↳ Apply tanh to the new memory C1 to obtain candidate output o’1. [0.8, -1] → [0.7, -0.8] [7] Update Hidden State ↳ Output (o’1 .* o1 → h1): Element-wise multiply the candidate output with the output gate. ↳ The result is updated hidden state h1 ↳ Also, it is the first output. Process t = 2 [8] Initialize ↳ Copy previous hidden state h1 and memory C1 [9] Linear Transform ↳ Repeat [3] [10] Update Memory (C2) ↳ Repeat [4] and [5] [11] Update Hidden State (h2) ↳ Repeat [6] and [7] Process t = 3 [12] Initialize ↳ Copy previous hidden state h2 and memory C2 [13] Linear Transform ↳ Repeat [3] [14] Update Memory (C3) ↳ Repeat [4] and [5] [15] Update Hidden State (h3) ↳ Repeat [6] and [7]](https://image.24vids.com/tw-1806285518126317877/ext_tw_video_thumb/1806285245903482880/pu/img/9-DANs-lC9ghBLvA.jpg)
![[VAE] by Hand ✍️ A Variational Auto Encoder (VAE) learns the structure (mean and variance) of hidden features and generates new data from the learned structure. In contrast, GANs only learn to generate new data to fool a discriminator; they may not necessarily know the underlying structure of the data. The International Conference on Learning Representations (ICLR) this year announced its first ever "Test of Time Award" to recognizes the VAE paper, published 10 years ago. This exercise demonstrates how to calculate a VAE by hand. [1] Given: ↳ Three training examples X1, X2, X3 ↳ Copy training examples to the bottom ↳ The purpose is to train the network to reconstruct the training examples. ↳ Since each target is a training example itself, we use the Greek word "auto" which means "self." This crucial step is what makes an autoencoder "auto." [2] Encoder: Layer 1 + ReLU ↳ Multiply inputs with weights and biases ↳ Apply ReLU, crossing out negative values (-1 -> 0) [3] Encoder: Mean and Variance ↳ Multiply features with two sets of weights and biases ↳ 🟩 The first set predicts the means (𝜇) of latent distributions ↳ 🟪 The second set predicts the standard deviation (𝜎) of latent distributions [4] Reparameterization Trick: Random Offset ↳ Sample epsilon ε from the normal distribution with mean = 0 and variance = 1. ↳ The purpose is to randomly pick a offset away from the mean. ↳ Multiply the standard deviation values with epsilon values. ↳ The purpose is to scale the offset by the standard deviation. [5] Reparameterization Trick: Mean + Offset ↳ Add the sampled offset to predicted mean ↳ The result are new parameters or features 🟨 as inputs to the Decoder. [6] Decoder: Layer 1 + ReLU ↳ Multiply input features with weights and biases ↳ Apply ReLU, crossing out negative values. Here, -4 is crossed out. [7] Decoder: Layer 2 ↳ Multiply features with weights and biases ↳ The output is Decoder's attempt to reconstruct the input data X from reparameterized distributions described by 𝜇 and 𝜎. [8]-[10] KL Divergence Loss [8] Loss Gradient: Mean 𝜇 ↳ We want 𝜇 to approach 0. ↳ A lot of math called SGVB simplifies the calculation of loss gradients to simply 𝜇 [9,10] Loss Gradient: Stdev 𝜎 ↳ We want 𝜎 to approach 1. ↳ A lot of math simplifies the calculation to 𝜎 - (1/ 𝜎) [11] Reconstruction Loss ↳ We want the reconstructed data Y (dark 🟧) to be the same as the input data X. ↳ Some math involving Mean Square Error simplifies the calculation to Y - X.](https://image.24vids.com/tw-1817192583191564496/ext_tw_video_thumb/1817192009591091200/pu/img/XAVt6rAbRS292En_.jpg)
![[Graph Convolutional Network] by hand ✍️ Graph Convolutional Networks (GCNs), introduced by Thomas Kipf and Max Welling in 2017, have emerged as a powerful tool in the analysis and interpretation of data structured as graphs. This exercise demonstrates how GCN works in a simple application: binary classification. -- Goal -- Predict if a node in a graph is X. -- Architecture -- 🟪 Graph Convolutional Network (GCN) 1. GCN1(4,3) 2. GCN2(3,3) 🟦 Fully Connected Network (FCN) 1. Linear1(3,5) 2. ReLU 3. Linear2(5,1) 4. Sigmoid Simplications: • Adjacent matrices are not normalized. • ReLU is applied to messages directly. -- Walkthrough -- [1] Given ↳ A graph with five nodes A, B, C, D, E [2] 🟩 Adjacency Matrix: Neighbors ↳ Add 1 for each edge to neighbors ↳ Repeat in both directions (e.g., A->C, C->A) ↳ Repeat for both GCN layers [3] 🟩 Adjacency Matrix: Self ↳ Add 1's for each self loop ↳ Equivalent to adding the identity matrix ↳ Repeat for both GCN layers [4] 🟪 GCN1: Messages ↳ Multiply the node embeddings 🟨 with weights and biases ↳ Apply ReLU (negatives → 0) ↳ The result is one message per node [5] 🟪 GCN1: Pooling ↳ Multiply the messages with the adjacent matrix ↳ The purpose is the pool messages from each node's neighbors as well as from the node itself. ↳ The result is a new feature per node [6] 🟪 GCN1: Visualize ↳ For node 1, visualize how messages are pooled to obtain a new feature for better understanding ↳ [3,0,1] + [1,0,0] = [4,0,1] [7] 🟪 GCN2: Messages ↳ Multiply the node features with weights and biases ↳ Apply ReLU (negatives → 0) ↳ The result is one message per node [8] 🟪 GCN2: Pooling ↳ Multiply the messages with the adjacent matrix ↳ The result is a new feature per node [9] 🟪 GCN2: Visualize ↳ For node 3, visualize how messages are pooled to obtain a new feature for better understanding ↳ [1,2,4] + [1,3,5] + [0,0,1] = [2,5,10] [10] 🟦 FCN: Linear 1 + ReLU ↳ Multiply node features with weights and biases ↳ Apply ReLU (negatives → 0) ↳ The result is a new feature per node ↳ Unlike in GCN layers, no messages from other nodes are included. [11] 🟦 FCN: Linear 2 ↳ Multiply node features with weights and biases [12] 🟦 FCN: Sigmoid ↳ Apply the Sigmoid activation function ↳ The purpose is to obtain a probability value for each node ↳ One way to calculate Sigmoid by hand ✍️ is to use the approximation below: • >= 3 → 1 • 0 → 0.5 • <= -3 → 0 -- Outputs -- A: 0 (Very unlikely) B: 1 (Very likely) C: 1 (Very likely) D: 1 (Very likely) E: 0.5 (Neutral)](https://image.24vids.com/tw-1828407238408438145/ext_tw_video_thumb/1828407079696039936/pu/img/mjnHB1FbGJngAUIw.jpg)
![Batch Normalization by hand ✍️ ~ 7 steps walkthrough below Batch normalization is common practice for improving training and achieving faster convergence. It sounds simple. But it is often misunderstood. 🤔 Does batch normalization involve trainable parameters, tunable hyper-parameters, or both? 🤔 Is batch normalization applied to inputs, features, weights, biases, or outputs? 🤔 How is batch normalization different from layer normalization? So I drew and calculated one entirely by hand. Goal: normalize a mini-batch of 4 examples to mean 0 and variance 1, then let the network scale it back. = 1. Given = A mini-batch of 4 training examples, each with 3 features. = 2. Linear layer = Let us multiply by the weights and add the biases. Batch norm sits after this, which answers the second question: what gets normalized is features, not inputs, weights or biases. = 3. ReLU = We apply the activation, and -2 becomes 0. Negative values are suppressed before any statistic is taken. = 4. Batch statistics = Let us compute the sum, mean, variance and standard deviation, one row at a time. A row is a feature and the four columns are the four examples, so every number here measures one feature against the rest of the batch. That is the "batch" in batch normalization, and it is exactly what layer normalization does not do. The statistics are rounded to whole numbers, which is what keeps the rest of the page doable in pen. = 5. Shift to mean 0 = We subtract the mean, in green. The four values in each feature now average to zero. = 6. Scale to variance 1 = Let us divide by the standard deviation, in orange. Each feature now has variance one, whatever scale it arrived at. = 7. Scale and shift = We multiply by a linear transformation and pass the result on. The diagonal and the last column are trainable, so having just forced every feature to mean 0 and variance 1, we hand the network the means to undo it. The outputs: Mean of each feature = [2, 1, 2] Std dev of each feature = [1, 1, 2] To the next layer = [2, -2, 2, 0], [-3, 3, 6, -3], [2, 0, 1, 2] The answers: 🤔 Both. The scale and shift are trainable, the statistics are not. Epsilon and the momentum on the running statistics are the hyper-parameters, and one mini-batch by hand needs neither. 🤔 Features, after the linear layer, not inputs, weights or biases. 🤔 Batch norm measures across the batch, one feature at a time. Layer norm measures across the features, one example at a time. 💾 Save this post!](https://image.24vids.com/tw-2079570896423911664/ext_tw_video_thumb/2079570876530298880/pu/img/w_TJ6G4bD69KBKuz.jpg)
![Vector Database by Hand ✍️ Vector databases are revolutionizing how we search and analyze complex data. They have become the backbone of Retrieval Augmented Generation (#RAG). How do vector databases work? [1] Given ↳ A dataset of three sentences, each has 3 words (or tokens) ↳ In practice, a dataset may contain millions or billions of sentences. The max number of tokens may be tens of thousands (e.g., 32,768 mistral-7b). Process "how are you" [2] 🟨 Word Embeddings ↳ For each word, look up corresponding word embedding vector from a table of 22 vectors, where 22 is the vocabulary size. ↳ In practice, the vocabulary size can be tens of thousands. The word embedding dimensions are in the thousands (e.g., 1024, 4096) [3] 🟩 Encoding ↳ Feed the sequence of word embeddings to an encoder to obtain a sequence of feature vectors, one per word. ↳ Here, the encoder is a simple one layer perceptron (linear layer + ReLU) ↳ In practice, the encoder is a transformer or one of its many variants. [4] 🟩 Mean Pooling ↳ Merge the sequence of feature vectors into a single vector using "mean pooling" which is to average across the columns. ↳ The result is a single vector. We often call it "text embeddings" or "sentence embeddings." ↳ Other pooling techniques are possible, such as CLS. But mean pooling is the most common. [5] 🟦 Indexing ↳ Reduce the dimensions of the text embedding vector by a projection matrix. The reduction rate is 50% (4->2). ↳ In practice, the values in this projection matrix is much more random. ↳ The purpose is similar to that of hashing, which is to obtain a short representation to allow faster comparison and retrieval. ↳ The resulting dimension-reduced index vector is saved in the vector storage. [6] Process "who are you" ↳ Repeat [2]-[5] [7] Process "who am I" ↳ Repeat [2]-[5] Now we have indexed our dataset in the vector database. [8] 🟥 Query: "am I you" ↳ Repeat [2]-[5] ↳ The result is a 2-d query vector. [9] 🟥 Dot Products ↳ Take dot product between the query vector and database vectors. They are all 2-d. ↳ The purpose is to use dot product to estimate similarity. ↳ By transposing the query vector, this step becomes a matrix multiplication. [10] 🟥 Nearest Neighbor ↳ Find the largest dot product by linear scan. ↳ The sentence with the highest dot product is "who am I" ↳ In practice, because scanning billions of vectors is slow, we use an Approximate Nearest Neighbor (ANN) algorithm like the Hierarchical Navigable Small Worlds (HNSW).](https://image.24vids.com/tw-1795076707386360227/ext_tw_video_thumb/1795076436019023872/pu/img/Ne2RjvPwQ1ouSr32.jpg)
![[Dropout] by hand ✍️ Dropout is a simple yet effective way of reducing overfitting and improving generalization. This by-hand exercise lets students practice calculating dropout, thereby gaining insight into its inner workings. As an additional bonus, students get to practice calculating the gradients of the Mean Square Error (MSE) loss. After the practice, students are often surprised by how simple it is. -- 𝗡𝗲𝘁𝘄𝗼𝗿𝗸 𝗔𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲 -- 1. Linear(2,4) 2. ReLU 3. Dropout(0.5) 4. Linear(4,3) 5. ReLU 6. Dropout(0.33) 7. Linear(3,2) -- 𝗪𝗮𝗹𝗸𝘁𝗵𝗿𝗼𝘂𝗴𝗵 -- 🏋️ Training [1] Given ↳ A training set of 2 examples X1, X2 [2] 🟧 Random: p > 0.5 ↳ Draw 4 random numbers ↳ For each random number, if it is above 0.5, we keep and denote it as ◯. Otherwise we drop and denote it as ╳. ↳ The result is [◯, ╳, ◯, ╳] [3] 🟧 Dropout: Matrix ↳ Calculate the scaling factor: 1 / (1-p) = 2 ↳ Set the diagonal based on [◯, ╳, ◯, ╳], where ◯ = 2 and ╳ = 0 ↳ The purpose is to drop the 2nd and the 4th nodes, and scale the remaining two nodes by 2. [4] 🟦 Random: p > 0.33 ↳ Draw 3 random numbers ↳ For each random number, if it is above 0.33, we keep and denote it as ◯. Otherwise we drop and denote it as ╳. ↳ The result is [◯, ◯, ╳] [5] 🟦 Dropout: Matrix ↳ Calculate the scaling factor: 1 / (1-p) = 1.5 ↳ Set the diagonal based on [◯, ◯, ╳], where ◯ = 1.5 and ╳ = 0 ↳ The purpose is to drop the 3rd node, and scale the remaining two nodes by 1.5. [6] Feed Forward ↳ Now we have all the matrices ready across the layers, perform the feed forward pass by calculating a series of matrix multiplications from the top to the bottom ↳ ReLU activation function is applied along the way to set negative feature values to zeroes, denoted by ╳. ↳ The outputs are Y. [7] 🟥 Loss Gradients of Mean Square Error (MSE) ↳ The formula is 2 * (Y - Y') ↳ First we calculate Outputs (Y) - Targets (Y') ↳ Second we multiply each element by 2 [8] Update Weights ↳ Use loss gradients to start back propagation ↳ Update some weights (light red) ↳ The values of the new weights are for demonstration purpose only, not based on real calculation. 🔍 Inference [9] Deactivate Dropout ↳ We set both dropout matrices to identity matrices ↳ The effect is to keep all the features as is. [10] Feed Forward ↳ Take the forward pass to make predictions about unseen data.](https://image.24vids.com/tw-1836044959020900392/ext_tw_video_thumb/1836044944273735680/pu/img/HAksFrOwf0Ct8B5_.jpg)
![[Discrete Fourier Transform] by Hand ✍️ In signal processing, the Discrete Fourier Transform (DFT) is no doubt the most important method. But the math involved is extremely complex, literally, involving a summation over a complex number term e^(-iwt). I developed this exercise to demonstrate that underneath such complexity, DFT is just a series of matrix multiplications you can calculate by hand. ✍️ Once you see that, it should not surprise you that a deep neural network, which is also a series of matrix multiplications, with activation functions in-between, can learn to perform DFT to process and analyze signals so effectively. How does DFT work? [1] Given ↳ Signals A, B, and C in the 🟧 frequency domain: ◦ A = cos(w) + 2cos(2w) ◦ B = cos(w) + cos(3w) + cos(4w) ◦ C = -cos(2w) + cos(3w) ◦ Each signal is a weighed sum of four cosine waves at frequencies 1w, 2w, 3w, and 4w. ◦ We will apply Inverse DFT to convert the signals to time domain representations, and then demonstrate DFT can convert back to their original frequency domain representations. ↳ Signal X in the 🟩 time domain. X is sampled at 10 time points 1t, 2t, …, 10t: ◦ X = [-2.5, -1.8, 3, -0.7, -1.0, -0.7, 3, -1.8, -2.5, 5] ◦ Suppose X is also a weighted sum of the same four cosine waves, but we don’t already know their weights. We will apply DFT to discover them. [2] 🟧 Frequency Matrix (F) ↳ Write the coefficients of A, B, C as a matrix F. Each signal is a row. Each frequency is a column. ↳ A → [1, 2, 0, 0] ↳ B → [1, 0, 1, 1] ↳ C → [0, 1-, 1, 0] [3] Cosine → Discrete ↳ Sample from the continuous cosine waves at discrete time points 1t, 2t, 3t, to 10t. [4] Cosine Matrix (W) ↳ Write the samples as a matrix, Each frequency is a row. Each time point is a column. [5] Inverse DFT: 🟧 Frequency → 🟩 Time ↳ Multiply the frequency matrix F and the cosine matrix W. ↳ The meaning of this multiplication is to linearly combine the four cosine waves (rows in W) into time-domain signals (rows in T) using the weights specified in F. ↳ The result is matrix T, which are signals A, B, C converted to the time domain. Each signal is a row. Each time point is a column. [6] Transpose ↳ Transpose T, converting each signal’s time domain representation from a row to a column. [7] DFT: 🟩 Time → 🟧 Frequency ↳ Multiply the cosine matrix W with the transpose of matrix T. ↳ The purpose of this multiplication is to take a dot-product between each time-domain signal (columns in the transpose of T) and each cosine wave (rows in W), which has the effect of projecting the signal onto a cosine wave to determine how much they are correlated. Zero means not correlated at all. ↳ The result is an intermediate version of the “recovered” frequency matrix where each column corresponds to a signal and each row corresponds to a frequency. ↳ Compared to the original frequency matrix F, this intermediate matrix has non-zero weights in the correct places, but scaled up by a factor of 5 (n/2, n=10). For example, signal A, originally [1,2,0,0], is recovered at [5,10,0,0]. [8] Scale ↳ Multiply each value by 2/n = 1/5 to scale down the intermediate matrix to match the magnitude of the original frequency matrix F. [9] Transpose ↳ Transpose the recovered frequency matrix back to the same orientation of the original frequency matrix F. ↳ Like magic 🪄, the result is identical to the original F, which means DFT successfully recovered the frequency components of signals A, B, C. [10] Apply DFT to X: 🟩 Time → 🟧 Frequency ↳ Now that we have some confidence in DFT’s ability to recover frequency components, we apply DFT to X’s time-domain representation by multiplying W with X. ↳ The result is the an intermediate matrix. [11] Scale ↳ Similarly, we scale down by a factor of 5 to obtain the recovered frequency components of X (a column). [12] Transpose ↳ Similarly, we transpose the recovered column to row to match the orientation of the frequency matrix. ↳ Using the coefficients [0,0,3,2], we can write the equation of X as 3cos(3w) + 2cos(4w). Notes: I hope this by hand exercise helps you understand the essence of DFT. But there is more technical details, such as: • Sine: The complete DFT math also includes sine waves that follow a similar calculation process. • Phase: Here, we assume all the cosine waves are aligned at the origin, namely, phase is 0. If a phase p is added, for example, cos(w+p), we will need to calculate the sine component and use their ratio to figure out what p is. • Magnitude: If phase is not zero, the magnitude will need to be calculated by combining both cosine and sine terms.](https://image.24vids.com/tw-1805694219031662799/ext_tw_video_thumb/1805693963464327168/pu/img/kOcpELm1j7wfqJlR.jpg)
![[Backpropagation] by Hand✍️ [1] Forward Pass ↳ Given a multi layer perceptron (3 levels), an input vector X, predictions Y^{Pred} = [0.5, 0.5, 0], and ground truth label Y^{Target} = [0, 1, 0]. [2] Backpropagation ↳ Insert cells to hold our calculations. [3] Layer 3 - Softmax (blue) ↳ Calculate ∂L / ∂z3 directly using the simple equation: Y^{Pred} - Y^{Target} = [0.5, -0.5, 0]. ↳ This simple equation is the benefit of using Softmax and Cross Entropy Loss together. [4] Layer 3 - Weights (orange) & Biases (black) ↳ Calculate ∂L / ∂W3 and ∂L / ∂b3 by multiplying ∂L / ∂z3 and [ a2 | 1 ]. [5] Layer 2 - Activations (green) ↳ Calculate ∂L / ∂a2 by multiplying ∂L / ∂z3 and W3. [6] Layer 2 - ReLU (blue) ↳ Calculate ∂L / ∂z2 by multiplying ∂L / ∂a2 with 1 for positive values and 0 otherwise. [7] Layer 2 - Weights (orange) & Biases (black) ↳ Calculate ∂L / ∂W2 and ∂L / ∂b2 by multiplying ∂L / ∂z2 and [ a1 | 1 ]. [8] Layer 1 - Activations (green) ↳ Calculate ∂L / ∂a1 by multiplying ∂L / ∂z2 and W2. [9] Layer 1 - ReLU (blue) ↳ Calculate ∂L / ∂z1 by multiplying ∂L / ∂a1 with 1 for positive values and 0 otherwise. [10] Layer 1 - Weights (orange) & Biases (black) ↳ Calculate ∂L / ∂W1 and ∂L / ∂b1 by multiplying ∂L / ∂z1 and [ x | 1 ]. [11] Gradient Descent ↳ Update weights and biases (typically a learning rate is applied here). 💡 Matrix Multiplication is All You Need: Just like in the forward pass, backpropagation is all about matrix multiplications. You can definitely do everything by hand as I demonstrated in this exercise, albeit slow and imperfect. This is why GPU's ability to multiply matrices efficiently plays such an important role in the deep learning evolution. This is why NVIDIA is now close to $1 trillion in valuation. 💡Exploding Gradients: We can already see the gradients are getting larger as we back-propagate up, even in this simple 3-layer network. This motivates using methods like skip connections to handle exploding (or diminishing) gradients as in the ResNet. I did the calculations entirely by hand. Please let me know if you spot any error or have any questions!](https://image.24vids.com/tw-1818652735535038715/ext_tw_video_thumb/1818652595235569664/pu/img/fBeCqhUByyXiDBbp.jpg)
![A rose curve is r = sin(kθ). Here k = b/a, with a = 1, 3, 7 down the rows and b = 2, 4, 8 across. With k = p/q in lowest terms and pq even, the rose has 2p petals and closes after θ = 2qπ. Each a is odd and each b a power of 2, so p = b and q = a: each column has 2b petals, 4, 8 and 16, and a only sets how far they wind: 2π, 6π, 14π. Swept over θ ∈ [0, 2aπ] in equal times, kθ = 2πb·t/T, so a column's three pens reach their petal tips together.](https://image.24vids.com/tw-2103170699208937579/amplify_video_thumb/2103170593545924608/img/BBR7KO2dNBMvv2LG.jpg)
![GUAVA LEAVES 🌿 TREATS / CURES ⬇️ 1. Type 2 Diabetes [4 wks, 1 TSPN, 2x daily] 2. Hypertension (High Blood Pressure) [6 wks, 1 TSPN, 2x daily] 3. Hyperlipidemia (High Cholesterol) [8 wks, 1 TSPN, 2x daily] 4. Prostate Enlargement (BPH) [12 wks, 1 TSPN, 2x daily] 5. Prostate Cancer [6 mos, 2 TSPN, 3x daily] 6. Breast Cancer [6 mos, 2 TSPN, 3x daily] 7. Colon Cancer [6 mos, 2 TSPN, 3x daily] 8. Osteoarthritis [8 wks, 1 TSPN, 2x daily] 9. Rheumatoid Arthritis [12 wks, 1.5 TSPN, 2x daily] 10. Acute Diarrhea [1 wk, 1 TSPN, 3x daily] 11. Dysentery [2 wks, 1 TSPN, 3x daily] 12. Gastroenteritis [2 wks, 1 TSPN, 3x daily] 13. Staphylococcus Aureus Infection [4 wks, 1 TSPN, 3x daily] 14. E. Coli Infection [3 wks, 1 TSPN, 3x daily] 15. Bronchitis [3 wks, 1 TSPN, 3x daily] 16. Asthma Attacks [8 wks, 1 TSPN, 2x daily] 17. Chronic Cough [2 wks, 1 TSPN, 3x daily] 18. Pulmonary Diseases [12 wks, 1 TSPN, 2x daily] 19. Dental Caries and Toothache [2 wks, 1 TSPN, 3x daily] 20. Liver Hepatotoxicity [8 wks, 1 TSPN, 2x daily] 21. Kidney Inflammation (Nephritis) [6 wks, 1 TSPN, 2x daily] 22. Peptic Ulcers [6 wks, 1 TSPN, 2x daily] 23. Metabolic Syndrome [12 wks, 1 TSPN, 2x daily] 24. Obesity and Weight Management [12 wks, 1 TSPN, 2x daily] 25. Malaria (Plasmodial Infection) [4 wks, 1.5 TSPN, 3x daily] 26. Dengue Fever [2 wks, 1 TSPN, 3x daily] 27. Menstrual Cramps (Dysmenorrhea) [1 wk, 1 TSPN, 3x daily] 28. Internal Wound Healing [4 wks, 1 TSPN, 2x daily] 29. Skin Micro-abscesses [3 wks, 1 TSPN, 2x daily] 30. Insomnia and Sleep Disorders [4 wks, 1 TSPN, 1x daily] Preparation & Dosage ⬇️ Get GUAVA LEAVES in Bulk, air-dry in a shade away from direct sunlight (in a shade / balcony) then blend bulk to POWDER when 100% dehydrated (Takes a week or Less) Inside half litre of already boiled water (or quantity of water you can consume at a go), STEEP / SOAK The measurement / dosage of GUAVA LEAVES Powder associated with the illness you're treating above. COVER it and allow infuse for 10 - 20 minutes or until warm enough to drink. DO NOT COOK IT! Filter / Sieve & drink whole liquid extract. Do this DAILY for the specified number of times & timeframe associated with the illness you intend to treat. KINDLY REPOST / SHARE & Follow Herbalist Henry Foundation 🌴🌿 on X (Twitter) 🙏](https://image.24vids.com/tw-2084509685009223963/amplify_video_thumb/2084509567107354624/img/gxZA9CQ6-hb6zTMr.jpg)
![Following many two-dimensional turbulence animations, it is time to share computer code! Implementation of a vorticity-streamfunction method with a pseudo-spectral discretization + third-order Runge-Kutta for time integration (our CFD class semester project). The MATLAB code: M = 256; % number of points N = M; Lx = 2*pi; Ly = 2*pi; nu = 5e-4; % kinematic viscosity Sc = 0.7; % Schmidt number beta = 0; % meridional gradient of Coriolis parameter ar = 0.02; %random number amplitude b = 1; % mean scalar gradient CFLmax = 0.8; tend = 200; % end time x=linspace(0,Lx,M+1); x(end)=[]; dx = Lx/M; kx=[0:M/2 -M/2+1:-1]*2*pi/Lx; y=linspace(0,Ly,N+1); y(end)=[]; ky=[0:N/2 -N/2+1:-1]*2*pi/Ly; dy = Ly/N; time = 0; index_kmax = ceil(M/3); kmax = kx(index_kmax); filter = ones(M,N); filter(index_kmax+1:2*index_kmax+3,index_kmax+1:2*index_kmax+3)=0; rng(64); [u, v, omega, psi, ddx, ddy, idel2, kk, k2]=deal(zeros(M,N)); for j=1:N ddx(:,j)=1i*kx; end for i=1:M ddy(i,:)=1i*ky; end for i=1:M for j=1:N idel2(i,j)=-kx(i)^2-ky(j)^2; end end idel2=1./idel2; idel2(1,1)=0; for i=1:M for j=1:N kk(i,j)=kx(i)^2+ky(j)^2; k2(i,j)=kx(i)^2+ky(j)^2; if kk(i,j) >= 6^2 && kk(i,j) <= 7^2 % forcing kk(i,j) = -kk(i,j); end if kk(i,j) <= 2^2 kk(i,j) = 8*kk(i,j); % large-scale dissipation end end end for i=1:M for j=1:N u(i,j) = cos(2*x(i))*sin(2*y(j))+ar*rand; v(i,j) = -sin(2*x(i))*cos(2*y(j))+ar*rand; end end uhat = fft2(u); vhat = fft2(v); omegahat = ddx.*vhat - ddy.*uhat; % make vorticity phi = rand(size(u)); phihat = fft2(phi); ncid = netcdf.create(' 'CLOBBER'); dimid_x = netcdf.defDim(ncid, 'x', M); dimid_y = netcdf.defDim(ncid, 'y', N); dimid_time = netcdf.defDim(ncid, 'time', netcdf.getConstant('NC_UNLIMITED')); varid_x = netcdf.defVar(ncid, 'x', 'NC_FLOAT', [dimid_x]); varid_y = netcdf.defVar(ncid, 'y', 'NC_FLOAT', [dimid_y]); varid_u = netcdf.defVar(ncid, 'u', 'NC_FLOAT', [dimid_x, dimid_y, dimid_time]); varid_v = netcdf.defVar(ncid, 'v', 'NC_FLOAT', [dimid_x, dimid_y, dimid_time]); varid_omega = netcdf.defVar(ncid, 'vorticity', 'NC_FLOAT', [dimid_x, dimid_y, dimid_time]); varid_phi = netcdf.defVar(ncid, 'scalar', 'NC_FLOAT', [dimid_x, dimid_y, dimid_time]); varid_dissipation = netcdf.defVar(ncid, 'dissipation', 'NC_FLOAT', [dimid_x, dimid_y, dimid_time]); varid_time = netcdf.defVar(ncid, 'time', 'NC_FLOAT', [dimid_time]); netcdf.endDef(ncid); netcdf.putVar(ncid, varid_time, 0, 1, time); netcdf.putVar(ncid, varid_x, 0, M, x); netcdf.putVar(ncid, varid_y, 0, N, y); netcdf.putVar(ncid, varid_phi, [0, 0, 0], [M, N, 1], phi); netcdf.putVar(ncid, varid_u, [0, 0, 0], [M, N, 1], u); netcdf.putVar(ncid, varid_v, [0, 0, 0], [M, N, 1], v); netcdf.putVar(ncid, varid_omega, [0, 0, 0], [M, N, 1], omega); netcdf.putVar(ncid, varid_dissipation, [0, 0, 0], [M, N, 1], 0*omega); dt = 0.5*min([dx dy]); nstep = 1; while time < tend psihat = -idel2.*omegahat; uhat = ddy.*psihat; vhat = -ddx.*psihat; u = real(ifft2(uhat)); v = real(ifft2(vhat)); omegadx = real(ifft2(ddx.*omegahat)); omegady = real(ifft2(ddy.*omegahat)); facto = exp(-nu*8/15*dt*kk); factp = exp(-nu/Sc*8/15*dt*k2); r0o = -fft2(u.*omegadx+v.*omegady)+beta*vhat; r0p = -fft2(u.*real(ifft2(ddx.*phihat))+v.*real(ifft2(ddy.*phihat)))+b*vhat; omegahat = facto.*(omegahat + dt*8/15*r0o); % update omega phihat = factp.*(phihat + dt*8/15*r0p); % update phi %%%% Substep 2 psihat = -idel2.*omegahat; uhat = ddy.*psihat; vhat = -ddx.*psihat; u = real(ifft2(uhat)); v = real(ifft2(vhat)); omegadx = real(ifft2(ddx.*omegahat)); omegady = real(ifft2(ddy.*omegahat)); r1o = -fft2(u.*omegadx+v.*omegady)+beta*vhat; r1p = -fft2(u.*real(ifft2(ddx.*phihat))+v.*real(ifft2(ddy.*phihat)))+b*vhat; omegahat = omegahat + dt*(-17/60*facto.*r0o + 5/12*r1o); phihat = phihat + dt*(-17/60*factp.*r0p + 5/12*r1p); facto = exp(-nu*(-17/60+5/12)*dt*kk); factp = exp(-nu/Sc*(-17/60+5/12)*dt*k2); omegahat = omegahat.*facto; phihat = phihat.*factp; %%%% Substep 3 psihat = -idel2.*omegahat; uhat = ddy.*psihat; vhat = -ddx.*psihat; % max(max(abs(real(ifft2(1i*ddx.*uhat+1i*ddy.*vhat))))) % divergence u = real(ifft2(uhat)); v = real(ifft2(vhat)); omegadx = real(ifft2(ddx.*omegahat)); omegady = real(ifft2(ddy.*omegahat)); r2o = -fft2(u.*omegadx+v.*omegady)+beta*vhat; r2p = -fft2(u.*real(ifft2(ddx.*phihat))+v.*real(ifft2(ddy.*phihat)))+b*vhat; omegahat = omegahat + dt*(-5/12*facto.*r1o + 3/4*r2o); phihat = phihat + dt*(-5/12*factp.*r1p + 3/4*r2p); facto = exp(-nu*(-5/12+3/4)*dt*kk); factp = exp(-nu/Sc*(-5/12+3/4)*dt*kk); omegahat = omegahat.*facto; phihat = phihat.*factp; phihat = filter.*phihat; omegahat = filter.*omegahat; time = time + dt; nstep = nstep + 1; CFL = max(max(abs(u)))/dx*dt+max(max(abs(v)))/dy*dt; if mod(nstep,20)==0 phi = real(ifft2(phihat)); omega = real(ifft2(omegahat)); dissipation = 2*nu*(real(ifft2(ddx.*uhat)).^2 + real(ifft2(ddy.*uhat)).^2 + real(ifft2(ddx.*vhat)).^2 + real(ifft2(ddy.*vhat)).^2); eta = (nu^3/mean(dissipation,'all'))^0.25; subplot(221); pcolor(x,y,omega'); title('Vorticity'); shading flat; axis equal tight; colorbar; drawnow subplot(222); pcolor(x,y,phi'); title('Scalar'); shading flat; axis equal tight; colorbar; drawnow subplot(223); pcolor(x,y,dissipation'); title('Dissipation'); shading flat; axis equal tight; colorbar; drawnow subplot(224); pcolor(x,y,u); title('u-velocity'); shading flat; axis equal tight; colorbar; drawnow fprintf(1,'step = %d time = %g dt = %g CFL = %g kmax*eta = %g %g\n', nstep, time, dt, CFL, eta.*kmax, eta.*kmax/sqrt(Sc)); [~, dim_time_len] = netcdf.inqDim(ncid,dimid_time); netcdf.putVar(ncid, varid_time, [dim_time_len], [1], time); netcdf.putVar(ncid, varid_phi, [0, 0, dim_time_len], [M, N, 1], phi); netcdf.putVar(ncid, varid_u, [0, 0, dim_time_len], [M, N, 1], u); netcdf.putVar(ncid, varid_v, [0, 0, dim_time_len], [M, N, 1], v); netcdf.putVar(ncid, varid_omega, [0, 0, dim_time_len], [M, N, 1], omega); netcdf.putVar(ncid, varid_dissipation, [0, 0, dim_time_len], [M, N, 1], dissipation); netcdf.sync(ncid); end dt = CFLmax/CFL*dt; %0.0005; end netcdf.close(ncid);](https://image.24vids.com/tw-1659560601478152193/ext_tw_video_thumb/1659556112872493056/pu/img/OfQDHPIG_usXyKjA.jpg)
![Health Benefits of MORINGA LEAVES 🌿 + How to Turn into Powder form! (MORINGA OLEIFERA) TREATS / CURES ⬇️ 1. Heart disease (cardioprotective) [1 TSPN 2x daily for 8-12 weeks] 2. Hypertension [1 TSPN 2x daily for 12 weeks] 3. High cholesterol [1-2 TSPN daily for 8-12 weeks] 4. Prostate enlargement/BPH [1 TSPN 2x daily for 12 weeks] 5. Prostate cancer support [1 TSPN 2x daily for 3-6 months] 6. Breast cancer support [1 TSPN 2x daily for 3-6 months] 7. Liver disease/hepatoprotection [1 TSPN 2x daily for 8-12 weeks] 8. Kidney disorders [1 TSPN 2x daily for 8-12 weeks] 9. Diabetes type 2 [1-1.5 TSPN 2x daily for 8-12 weeks] 10. Prediabetes [1 TSPN 2x daily for 12 weeks] 11. Inflammation (general) [1 TSPN 2x daily for 8-12 weeks] 12. Arthritis [1 TSPN 2x daily for 8-12 weeks] 13. Bacterial infections [1 TSPN 2x daily for 4-8 weeks] 14. Viral infections (incl. HIV support) [2-3 TSPN daily for 3-6 months] 15. Ulcers (gastric) [1 TSPN 2x daily for 4-8 weeks] 16. Anemia [1-2 TSPN daily for 8-12 weeks] 17. Malnutrition [1-2 TSPN daily for 8-12 weeks] 18. Asthma [1 TSPN 2x daily for 3-6 weeks] 19. Oxidative stress [1 TSPN 2x daily for 8-12 weeks] 20. Obesity/metabolic syndrome [1 TSPN 2x daily for 12 weeks] 21. Cancer (general support) [1 TSPN 2x daily for 3-6 months] 22. Colorectal cancer support [1 TSPN 2x daily for 3-6 months] 23. Lung cancer support [1 TSPN 2x daily for 3-6 months] 24. Thyroid imbalance [1 TSPN daily for 8-12 weeks] 25. Wound healing/support [1 TSPN 2x daily for 4-8 weeks] 26. Immune deficiency [1-2 TSPN daily for 8-12 weeks] 27. Neuroinflammation [1 TSPN 2x daily for 8-12 weeks] 28. Skin inflammation [1 TSPN 2x daily for 4-8 weeks] 29. Hyperglycemia [1 TSPN 2x daily for 8-12 weeks] 30. Dyslipidemia [1 TSPN 2x daily for 8-12 weeks] Preparation & Dosage ⬇️ Get MORINGA LEAVES in Bulk, air-dry in a shade away from direct sunlight (in a shade / balcony) then blend bulk to POWDER when 100% dehydrated (Takes a week or More) Inside half litre of already boiled water (or quantity of water you can consume at a go), STEEP / SOAK The measurement / dosage of MORINGA LEAVES Powder associated with the illness you're treating above. COVER it and allow infuse for 20 -30 minutes or until warm enough to drink. DO NOT COOK IT! Filter / Sieve & drink whole liquid extract. Do this DAILY for the specified number of times & timeframe associated with the illness you intend to treat. KINDLY REPOST / SHARE & Follow Herbalist Henry Foundation 🌴🌿 on X (Twitter) 🙏](https://image.24vids.com/tw-2109128751992717555/amplify_video_thumb/2109128697588449280/img/-7yk_Bc47RTD44T1.jpg)
![Behind the Scenes: How I prompted 3 scenes in 1 try on Seedance 2.0! Sharing my workflow of all my prompts for hopeless steve so you can see what were SD2 choices and what needed to be prompted. You can see how much more streamlined it is to go from one scene to the next when you can have 9 ref img's to work with. The prompt is only for the one that generated the most useful footage for each part. Scene 1, 2, 3 - 15s Ref: 1. House Ext. 2, Steve profile 3. Steve on the computer in the bedroom 4. mother cooking in kitchen. (The references a numbered so you know what the prompt is refering to.) Prompt: American sitcom style cartoon about Steve【@图片2】. Must maintain 100% fidelity to the art style and all character traits. No background music. [0:00-0:02] Wide establishing shot of the exterior of the house【@图片1】. Camera slowly zooms but remains in the wide shot. [Cut: 0:02-0:03] Interior wide shot of steve looking bored at computer 【@图片3】. No dialogue. [Cut: 0:03-0:05] Over the shoulder shot from behind steve. We see his monitor as he plays solitaire. We hear subtle mouse clicks and sound of cards as he drags and places them into the collumns. [Cut: 0:05-0:07] Side profile of Steve continuing to play solitaire on the computer. [Cut: 0:07-0:09] 【@图片4】Extreme close up of the mother's spatula flipping an egg over in the pan. [Cut: 0:09-0:11] Medium shot 【@图片4】 as the mother calls frustratedly, "Steve, breakfast is ready." [Cut: 0:11-0:12] Extreme close up in 【@图片4】of the mother's hand as she turns the stove off. [Cut: 0:12-0:13] 【@图片4】Extreme close up of the mother narrowing her eyes disgruntled. [Cut: 0:13-0:15] 【@图片4】 Medium shot as mother walks out of left of frame. Stationary camera.](https://image.24vids.com/tw-2024902234081997186/amplify_video_thumb/2024902013524606976/img/0lwVojYKObUAai_3.jpg)