Video wird geladen...
Video konnte nicht geladen werden
> 9 Terrorist Camps > 11 Air bases > 1 AEW&C/ELINT > 2 Command and Control center (Murid and Chaklala) > 6 Radars > 2 SAGW system (Lahore & Okara) > Runways of Sargodha and Rahim Yar Khan > 1 UAV hanger Sukkur > 1 AEW&C hanger Bholari, few... show more
161,464 Aufrufe • vor 1 Jahr •via X (Twitter)
0 Kommentare
Keine Kommentare verfügbar
Kommentare vom Original-Post werden hier angezeigt
![[LSTM] by Hand ✍️ LSTMs have been the most effective architecture to process long sequences of data, until our world was taken over by the Transformers. LSTMs belong to the broader family of recurrent neural network (RNNs) that process data sequentially in a recurrent manner. Transformers, on the other hand, abandon recurrence and use self-attention instead to process data concurrently in parallel. Recently, there is renewed interest in recurrence as people realized self-attention doesn’t scale to extremely long sequences, like hundreds of thousands of tokens. Mamba is a good example to bring back recurrence. All of a sudden, it is cool to study LSTMs. How do LSTMs work? [1] Given ↳ 🟨 Input sequence X1, X2, X3 (d = 3) ↳ 🟩 Hidden state h (d = 2) ↳ 🟦 Memory C (d = 2) ↳ Weight matrices Wf, Wc, Wi, Wo Process t = 1 [2] Initialize ↳ Randomly set the previous hidden state h0 to [1, 1] and memory cells C0 to [0.3, -0.5] [3] Linear Transform ↳ Multiply the four weight matrices with the concatenation of current input (X1) and the previous hidden state (h0). ↳ The results are feature values, each is a linear combination of the current input and hidden state. [4] Non-linear Transform ↳ Apply sigmoid σ to obtain gate values (between 0 and 1). • Forget gate (f1): [-4, -6] → [0, 0] • Input gate (i1): [6, 4] → [1, 1] • Output gate (o1): [4, -5] → [1, 0] ↳ Apply tanh to obtain candidate memory values (between -1 and 1) • Candidate memory (C’1): [1, -6] → [0.8, -1] [5] Update Memory ↳ Forget (C0 .* f1): Element-wise multiply the current memory with forget gate values. ↳ Input (C’1 .* o1): Element-wise multiply the “candidate” memory with input gate values. ↳ Update the memory to C1 by adding the two terms above: C0 .* f1 + C’1 .* o1 = C1 [6] Candiate Output ↳ Apply tanh to the new memory C1 to obtain candidate output o’1. [0.8, -1] → [0.7, -0.8] [7] Update Hidden State ↳ Output (o’1 .* o1 → h1): Element-wise multiply the candidate output with the output gate. ↳ The result is updated hidden state h1 ↳ Also, it is the first output. Process t = 2 [8] Initialize ↳ Copy previous hidden state h1 and memory C1 [9] Linear Transform ↳ Repeat [3] [10] Update Memory (C2) ↳ Repeat [4] and [5] [11] Update Hidden State (h2) ↳ Repeat [6] and [7] Process t = 3 [12] Initialize ↳ Copy previous hidden state h2 and memory C2 [13] Linear Transform ↳ Repeat [3] [14] Update Memory (C3) ↳ Repeat [4] and [5] [15] Update Hidden State (h3) ↳ Repeat [6] and [7]](https://image.24vids.com/tw-1806285518126317877/ext_tw_video_thumb/1806285245903482880/pu/img/9-DANs-lC9ghBLvA.jpg)
![[Deep RNN] by Hand ✍️ A Deep Recurrent Neural Network (RNN) extends a basic single-layer RNN into multiple layers of hidden states, effectively incorporating deep learning into the RNN architecture. How does a Deep RNN work? [1] Given ↳ A sequence of four inputs X1, X2, X3, X4 ⬛️ ↳ Recurrent weights and biases for hidden layers a 🟩, b 🟧, c 🟪, and the output layer y 🟦. [2] Initialize Hidden States ↳ Set a0, b0, c0 to zeros — Process X1 (t = 1)— [3] First Hidden Layer (a) 🟩: a0 → a1 ↳ The transformation matrix is horizontal concatenation of input weights, hidden state weights and biases, visualized as [⬛️ | 🟩 | ⬜️] . ↳ The state matrix is vertical concatenation of input X1, previous hidden state a0, and an extra 1, visualized as [⬛️ ; 🟩 ; 1]. ↳ Multiply the two matrices to obtain new hidden state a1 = [0 ; 1]. [4] Second Hidden Layer (b) 🟪: b0 → b1 ↳ First layer a1 🟩 becomes the input. ↳ The transformation matrix is visualized as [🟩 | 🟪 | ⬜️]. ↳ The state matrix is the combination of a1, b0, and 1, visualized as [🟩; 🟪 ; 1]. ↳ Multiply the two matrices to obtain new hidden state b1 = [1; -1]. [5] Third Hidden Layer (c) 🟧: c0 → c1 ↳ Second layer b 🟪 becomes the input. ↳ The transformation matrix is visualized as [🟪 | 🟧 | ⬜️]. ↳ The state matrix is the combination of a1, b0, and 1, visualized as [🟪; 🟧; 1]. ↳ Multiply the two matrices to obtain new hidden state b1 = [1; -1]. [6] Output Layer (Y) 🟦 ↳ The transformation matrix is visualized as [🟧 | ⬜️]. ↳ The state matrix is the combination of c0 and , visualized as [🟧; 1]. ↳ Multiply the two matrices to obtain output Y1 = [3; 0; 3]. — Process X2 (t = 2)— [7] Previous Hidden States ↳ Copy the values of a1, b1, c1. [8] Hidden 🟩🟪🟧 + Output 🟦 ↳ Repeat [3]-[6] to obtain output Y2 = [5; 0; 4] — Process X3 (t = 3)— [9] Previous Hidden States ↳ Copy the values of a2, b2, c2. [10] Hidden 🟩🟪🟧 + Output 🟦 ↳ Repeat [3]-[6] to obtain output Y3 = [13; -1; 9] — Process X4 (t = 4)— [11] Previous Hidden States ↳ Copy the values of a3, b3, c3. [12] Hidden 🟩🟪🟧 + Output 🟦 ↳ Repeat [3]-[6] to obtain output Y4 = [15; 7; 2]](https://image.24vids.com/tw-1809362848054993158/ext_tw_video_thumb/1809362489563631616/pu/img/khsrKXTM6woSuL4P.jpg)
![[Graph Convolutional Network] by hand ✍️ Graph Convolutional Networks (GCNs), introduced by Thomas Kipf and Max Welling in 2017, have emerged as a powerful tool in the analysis and interpretation of data structured as graphs. This exercise demonstrates how GCN works in a simple application: binary classification. -- Goal -- Predict if a node in a graph is X. -- Architecture -- 🟪 Graph Convolutional Network (GCN) 1. GCN1(4,3) 2. GCN2(3,3) 🟦 Fully Connected Network (FCN) 1. Linear1(3,5) 2. ReLU 3. Linear2(5,1) 4. Sigmoid Simplications: • Adjacent matrices are not normalized. • ReLU is applied to messages directly. -- Walkthrough -- [1] Given ↳ A graph with five nodes A, B, C, D, E [2] 🟩 Adjacency Matrix: Neighbors ↳ Add 1 for each edge to neighbors ↳ Repeat in both directions (e.g., A->C, C->A) ↳ Repeat for both GCN layers [3] 🟩 Adjacency Matrix: Self ↳ Add 1's for each self loop ↳ Equivalent to adding the identity matrix ↳ Repeat for both GCN layers [4] 🟪 GCN1: Messages ↳ Multiply the node embeddings 🟨 with weights and biases ↳ Apply ReLU (negatives → 0) ↳ The result is one message per node [5] 🟪 GCN1: Pooling ↳ Multiply the messages with the adjacent matrix ↳ The purpose is the pool messages from each node's neighbors as well as from the node itself. ↳ The result is a new feature per node [6] 🟪 GCN1: Visualize ↳ For node 1, visualize how messages are pooled to obtain a new feature for better understanding ↳ [3,0,1] + [1,0,0] = [4,0,1] [7] 🟪 GCN2: Messages ↳ Multiply the node features with weights and biases ↳ Apply ReLU (negatives → 0) ↳ The result is one message per node [8] 🟪 GCN2: Pooling ↳ Multiply the messages with the adjacent matrix ↳ The result is a new feature per node [9] 🟪 GCN2: Visualize ↳ For node 3, visualize how messages are pooled to obtain a new feature for better understanding ↳ [1,2,4] + [1,3,5] + [0,0,1] = [2,5,10] [10] 🟦 FCN: Linear 1 + ReLU ↳ Multiply node features with weights and biases ↳ Apply ReLU (negatives → 0) ↳ The result is a new feature per node ↳ Unlike in GCN layers, no messages from other nodes are included. [11] 🟦 FCN: Linear 2 ↳ Multiply node features with weights and biases [12] 🟦 FCN: Sigmoid ↳ Apply the Sigmoid activation function ↳ The purpose is to obtain a probability value for each node ↳ One way to calculate Sigmoid by hand ✍️ is to use the approximation below: • >= 3 → 1 • 0 → 0.5 • <= -3 → 0 -- Outputs -- A: 0 (Very unlikely) B: 1 (Very likely) C: 1 (Very likely) D: 1 (Very likely) E: 0.5 (Neutral)](https://image.24vids.com/tw-1828407238408438145/ext_tw_video_thumb/1828407079696039936/pu/img/mjnHB1FbGJngAUIw.jpg)
