Loading video...

Video Failed to Load

Go Home

[Graph Convolutional Network] by hand ✍️ Graph Convolutional Networks (GCNs), introduced by Thomas Kipf and Max Welling in 2017, have emerged as a powerful tool in the analysis and interpretation of data structured as graphs. This exercise demonstrates how GCN works in a simple application: binary classification. -- Goal...

ProfTomYeh's profile picture

Tom Yeh

64,377 subscribers

46,779 views • 2 years ago •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

[Deep RNN] by Hand ✍️ A Deep Recurrent Neural Network (RNN) extends a basic single-layer RNN into multiple layers of hidden states, effectively incorporating deep learning into the RNN architecture. How does a Deep RNN work? [1] Given ↳ A sequence of four inputs X1, X2, X3, X4 ⬛️ ↳ Recurrent weights and biases for hidden layers a 🟩, b 🟧, c 🟪, and the output layer y 🟦. [2] Initialize Hidden States ↳ Set a0, b0, c0 to zeros — Process X1 (t = 1)— [3] First Hidden Layer (a) 🟩: a0 → a1 ↳ The transformation matrix is horizontal concatenation of input weights, hidden state weights and biases, visualized as [⬛️ | 🟩 | ⬜️] . ↳ The state matrix is vertical concatenation of input X1, previous hidden state a0, and an extra 1, visualized as [⬛️ ; 🟩 ; 1]. ↳ Multiply the two matrices to obtain new hidden state a1 = [0 ; 1]. [4] Second Hidden Layer (b) 🟪: b0 → b1 ↳ First layer a1 🟩 becomes the input. ↳ The transformation matrix is visualized as [🟩 | 🟪 | ⬜️]. ↳ The state matrix is the combination of a1, b0, and 1, visualized as [🟩; 🟪 ; 1]. ↳ Multiply the two matrices to obtain new hidden state b1 = [1; -1]. [5] Third Hidden Layer (c) 🟧: c0 → c1 ↳ Second layer b 🟪 becomes the input. ↳ The transformation matrix is visualized as [🟪 | 🟧 | ⬜️]. ↳ The state matrix is the combination of a1, b0, and 1, visualized as [🟪; 🟧; 1]. ↳ Multiply the two matrices to obtain new hidden state b1 = [1; -1]. [6] Output Layer (Y) 🟦 ↳ The transformation matrix is visualized as [🟧 | ⬜️]. ↳ The state matrix is the combination of c0 and , visualized as [🟧; 1]. ↳ Multiply the two matrices to obtain output Y1 = [3; 0; 3]. — Process X2 (t = 2)— [7] Previous Hidden States ↳ Copy the values of a1, b1, c1. [8] Hidden 🟩🟪🟧 + Output 🟦 ↳ Repeat [3]-[6] to obtain output Y2 = [5; 0; 4] — Process X3 (t = 3)— [9] Previous Hidden States ↳ Copy the values of a2, b2, c2. [10] Hidden 🟩🟪🟧 + Output 🟦 ↳ Repeat [3]-[6] to obtain output Y3 = [13; -1; 9] — Process X4 (t = 4)— [11] Previous Hidden States ↳ Copy the values of a3, b3, c3. [12] Hidden 🟩🟪🟧 + Output 🟦 ↳ Repeat [3]-[6] to obtain output Y4 = [15; 7; 2]
0:10

[Deep RNN] by Hand ✍️ A Deep Recurrent Neural Network (RNN) extends a basic single-layer RNN into multiple layers of hidden states, effectively incorporating deep learning into the RNN architecture. How does a Deep RNN work? [1] Given ↳ A sequence of four inputs X1, X2, X3, X4 ⬛️ ↳ Recurrent weights and biases for hidden layers a 🟩, b 🟧, c 🟪, and the output layer y 🟦. [2] Initialize Hidden States ↳ Set a0, b0, c0 to zeros — Process X1 (t = 1)— [3] First Hidden Layer (a) 🟩: a0 → a1 ↳ The transformation matrix is horizontal concatenation of input weights, hidden state weights and biases, visualized as [⬛️ | 🟩 | ⬜️] . ↳ The state matrix is vertical concatenation of input X1, previous hidden state a0, and an extra 1, visualized as [⬛️ ; 🟩 ; 1]. ↳ Multiply the two matrices to obtain new hidden state a1 = [0 ; 1]. [4] Second Hidden Layer (b) 🟪: b0 → b1 ↳ First layer a1 🟩 becomes the input. ↳ The transformation matrix is visualized as [🟩 | 🟪 | ⬜️]. ↳ The state matrix is the combination of a1, b0, and 1, visualized as [🟩; 🟪 ; 1]. ↳ Multiply the two matrices to obtain new hidden state b1 = [1; -1]. [5] Third Hidden Layer (c) 🟧: c0 → c1 ↳ Second layer b 🟪 becomes the input. ↳ The transformation matrix is visualized as [🟪 | 🟧 | ⬜️]. ↳ The state matrix is the combination of a1, b0, and 1, visualized as [🟪; 🟧; 1]. ↳ Multiply the two matrices to obtain new hidden state b1 = [1; -1]. [6] Output Layer (Y) 🟦 ↳ The transformation matrix is visualized as [🟧 | ⬜️]. ↳ The state matrix is the combination of c0 and , visualized as [🟧; 1]. ↳ Multiply the two matrices to obtain output Y1 = [3; 0; 3]. — Process X2 (t = 2)— [7] Previous Hidden States ↳ Copy the values of a1, b1, c1. [8] Hidden 🟩🟪🟧 + Output 🟦 ↳ Repeat [3]-[6] to obtain output Y2 = [5; 0; 4] — Process X3 (t = 3)— [9] Previous Hidden States ↳ Copy the values of a2, b2, c2. [10] Hidden 🟩🟪🟧 + Output 🟦 ↳ Repeat [3]-[6] to obtain output Y3 = [13; -1; 9] — Process X4 (t = 4)— [11] Previous Hidden States ↳ Copy the values of a3, b3, c3. [12] Hidden 🟩🟪🟧 + Output 🟦 ↳ Repeat [3]-[6] to obtain output Y4 = [15; 7; 2]

Tom Yeh

26,548 views • 2 years ago

[VAE] by Hand ✍️ A Variational Auto Encoder (VAE) learns the structure (mean and variance) of hidden features and generates new data from the learned structure. In contrast, GANs only learn to generate new data to fool a discriminator; they may not necessarily know the underlying structure of the data. The International Conference on Learning Representations (ICLR) this year announced its first ever "Test of Time Award" to recognizes the VAE paper, published 10 years ago. This exercise demonstrates how to calculate a VAE by hand. [1] Given: ↳ Three training examples X1, X2, X3 ↳ Copy training examples to the bottom ↳ The purpose is to train the network to reconstruct the training examples. ↳ Since each target is a training example itself, we use the Greek word "auto" which means "self." This crucial step is what makes an autoencoder "auto." [2] Encoder: Layer 1 + ReLU ↳ Multiply inputs with weights and biases ↳ Apply ReLU, crossing out negative values (-1 -> 0) [3] Encoder: Mean and Variance ↳ Multiply features with two sets of weights and biases ↳ 🟩 The first set predicts the means (𝜇) of latent distributions ↳ 🟪 The second set predicts the standard deviation (𝜎) of latent distributions [4] Reparameterization Trick: Random Offset ↳ Sample epsilon ε from the normal distribution with mean = 0 and variance = 1. ↳ The purpose is to randomly pick a offset away from the mean. ↳ Multiply the standard deviation values with epsilon values. ↳ The purpose is to scale the offset by the standard deviation. [5] Reparameterization Trick: Mean + Offset ↳ Add the sampled offset to predicted mean ↳ The result are new parameters or features 🟨 as inputs to the Decoder. [6] Decoder: Layer 1 + ReLU ↳ Multiply input features with weights and biases ↳ Apply ReLU, crossing out negative values. Here, -4 is crossed out. [7] Decoder: Layer 2 ↳ Multiply features with weights and biases ↳ The output is Decoder's attempt to reconstruct the input data X from reparameterized distributions described by 𝜇 and 𝜎. [8]-[10] KL Divergence Loss [8] Loss Gradient: Mean 𝜇 ↳ We want 𝜇 to approach 0. ↳ A lot of math called SGVB simplifies the calculation of loss gradients to simply 𝜇 [9,10] Loss Gradient: Stdev 𝜎 ↳ We want 𝜎 to approach 1. ↳ A lot of math simplifies the calculation to 𝜎 - (1/ 𝜎) [11] Reconstruction Loss ↳ We want the reconstructed data Y (dark 🟧) to be the same as the input data X. ↳ Some math involving Mean Square Error simplifies the calculation to Y - X.
0:11

[VAE] by Hand ✍️ A Variational Auto Encoder (VAE) learns the structure (mean and variance) of hidden features and generates new data from the learned structure. In contrast, GANs only learn to generate new data to fool a discriminator; they may not necessarily know the underlying structure of the data. The International Conference on Learning Representations (ICLR) this year announced its first ever "Test of Time Award" to recognizes the VAE paper, published 10 years ago. This exercise demonstrates how to calculate a VAE by hand. [1] Given: ↳ Three training examples X1, X2, X3 ↳ Copy training examples to the bottom ↳ The purpose is to train the network to reconstruct the training examples. ↳ Since each target is a training example itself, we use the Greek word "auto" which means "self." This crucial step is what makes an autoencoder "auto." [2] Encoder: Layer 1 + ReLU ↳ Multiply inputs with weights and biases ↳ Apply ReLU, crossing out negative values (-1 -> 0) [3] Encoder: Mean and Variance ↳ Multiply features with two sets of weights and biases ↳ 🟩 The first set predicts the means (𝜇) of latent distributions ↳ 🟪 The second set predicts the standard deviation (𝜎) of latent distributions [4] Reparameterization Trick: Random Offset ↳ Sample epsilon ε from the normal distribution with mean = 0 and variance = 1. ↳ The purpose is to randomly pick a offset away from the mean. ↳ Multiply the standard deviation values with epsilon values. ↳ The purpose is to scale the offset by the standard deviation. [5] Reparameterization Trick: Mean + Offset ↳ Add the sampled offset to predicted mean ↳ The result are new parameters or features 🟨 as inputs to the Decoder. [6] Decoder: Layer 1 + ReLU ↳ Multiply input features with weights and biases ↳ Apply ReLU, crossing out negative values. Here, -4 is crossed out. [7] Decoder: Layer 2 ↳ Multiply features with weights and biases ↳ The output is Decoder's attempt to reconstruct the input data X from reparameterized distributions described by 𝜇 and 𝜎. [8]-[10] KL Divergence Loss [8] Loss Gradient: Mean 𝜇 ↳ We want 𝜇 to approach 0. ↳ A lot of math called SGVB simplifies the calculation of loss gradients to simply 𝜇 [9,10] Loss Gradient: Stdev 𝜎 ↳ We want 𝜎 to approach 1. ↳ A lot of math simplifies the calculation to 𝜎 - (1/ 𝜎) [11] Reconstruction Loss ↳ We want the reconstructed data Y (dark 🟧) to be the same as the input data X. ↳ Some math involving Mean Square Error simplifies the calculation to Y - X.

Tom Yeh

48,475 views • 2 years ago

Batch Normalization by hand ✍️ ~ 7 steps walkthrough below Batch normalization is common practice for improving training and achieving faster convergence. It sounds simple. But it is often misunderstood. 🤔 Does batch normalization involve trainable parameters, tunable hyper-parameters, or both? 🤔 Is batch normalization applied to inputs, features, weights, biases, or outputs? 🤔 How is batch normalization different from layer normalization? So I drew and calculated one entirely by hand. Goal: normalize a mini-batch of 4 examples to mean 0 and variance 1, then let the network scale it back. = 1. Given = A mini-batch of 4 training examples, each with 3 features. = 2. Linear layer = Let us multiply by the weights and add the biases. Batch norm sits after this, which answers the second question: what gets normalized is features, not inputs, weights or biases. = 3. ReLU = We apply the activation, and -2 becomes 0. Negative values are suppressed before any statistic is taken. = 4. Batch statistics = Let us compute the sum, mean, variance and standard deviation, one row at a time. A row is a feature and the four columns are the four examples, so every number here measures one feature against the rest of the batch. That is the "batch" in batch normalization, and it is exactly what layer normalization does not do. The statistics are rounded to whole numbers, which is what keeps the rest of the page doable in pen. = 5. Shift to mean 0 = We subtract the mean, in green. The four values in each feature now average to zero. = 6. Scale to variance 1 = Let us divide by the standard deviation, in orange. Each feature now has variance one, whatever scale it arrived at. = 7. Scale and shift = We multiply by a linear transformation and pass the result on. The diagonal and the last column are trainable, so having just forced every feature to mean 0 and variance 1, we hand the network the means to undo it. The outputs: Mean of each feature = [2, 1, 2] Std dev of each feature = [1, 1, 2] To the next layer = [2, -2, 2, 0], [-3, 3, 6, -3], [2, 0, 1, 2] The answers: 🤔 Both. The scale and shift are trainable, the statistics are not. Epsilon and the momentum on the running statistics are the hyper-parameters, and one mini-batch by hand needs neither. 🤔 Features, after the linear layer, not inputs, weights or biases. 🤔 Batch norm measures across the batch, one feature at a time. Layer norm measures across the features, one example at a time. 💾 Save this post!
0:17

Batch Normalization by hand ✍️ ~ 7 steps walkthrough below Batch normalization is common practice for improving training and achieving faster convergence. It sounds simple. But it is often misunderstood. 🤔 Does batch normalization involve trainable parameters, tunable hyper-parameters, or both? 🤔 Is batch normalization applied to inputs, features, weights, biases, or outputs? 🤔 How is batch normalization different from layer normalization? So I drew and calculated one entirely by hand. Goal: normalize a mini-batch of 4 examples to mean 0 and variance 1, then let the network scale it back. = 1. Given = A mini-batch of 4 training examples, each with 3 features. = 2. Linear layer = Let us multiply by the weights and add the biases. Batch norm sits after this, which answers the second question: what gets normalized is features, not inputs, weights or biases. = 3. ReLU = We apply the activation, and -2 becomes 0. Negative values are suppressed before any statistic is taken. = 4. Batch statistics = Let us compute the sum, mean, variance and standard deviation, one row at a time. A row is a feature and the four columns are the four examples, so every number here measures one feature against the rest of the batch. That is the "batch" in batch normalization, and it is exactly what layer normalization does not do. The statistics are rounded to whole numbers, which is what keeps the rest of the page doable in pen. = 5. Shift to mean 0 = We subtract the mean, in green. The four values in each feature now average to zero. = 6. Scale to variance 1 = Let us divide by the standard deviation, in orange. Each feature now has variance one, whatever scale it arrived at. = 7. Scale and shift = We multiply by a linear transformation and pass the result on. The diagonal and the last column are trainable, so having just forced every feature to mean 0 and variance 1, we hand the network the means to undo it. The outputs: Mean of each feature = [2, 1, 2] Std dev of each feature = [1, 1, 2] To the next layer = [2, -2, 2, 0], [-3, 3, 6, -3], [2, 0, 1, 2] The answers: 🤔 Both. The scale and shift are trainable, the statistics are not. Epsilon and the momentum on the running statistics are the hyper-parameters, and one mini-batch by hand needs neither. 🤔 Features, after the linear layer, not inputs, weights or biases. 🤔 Batch norm measures across the batch, one feature at a time. Layer norm measures across the features, one example at a time. 💾 Save this post!

Tom Yeh

20,848 views • 1 month ago

MLP in PyTorch by hand ✍️ ~ 7 steps walkthrough below Goal: fill in every blank in the PyTorch code to build a multi-layer perceptron. 1. Given Let us start with a code template on the left and the network it is supposed to build on the right. Every blank in the code can be worked out from the picture. 2. Linear layer We count: 3 features in, 4 features out. So the weight matrix is 4 by 3. There is an extra column for the biases, which means bias = T. 3. ReLU Let us apply the activation. ReLU crosses out the negatives, so -1 becomes 0. 4. Linear layer The input size is 4, because that is what the previous layer put out. The output size is 2. A 2 by 4 weight matrix, and this time no extra column, so bias = F. 5. ReLU We cross out the negatives again. 6. Linear layer Two features in, five out. A 5 by 2 weight matrix, with a bias column, so bias = T. 7. Sigmoid Let us finish. Sigmoid squashes the raw scores (3, 0, -2, 5, -5) into probabilities between 0 and 1. You have just implemented a three-layer deep neural network by hand. ✍️ == Story == Three years ago I gave this exercise to my students, to connect the code to the math. They found it odd. Every other AI course they were taking lived inside a Jupyter notebook, and here I was handing out paper. Three years later, my colleagues are the ones rushing to move their materials to paper. The exercise has not changed. Paper still asks the one thing a notebook lets you skip: do you actually understand what the code is doing? If you can tell me why the weight matrix is 4 by 3, and why bias is F on the second layer, you understand nn.Linear better than someone who has been copy-pasting it for a year. 💾 Save this post! #AIbyHand #PyTorch #DeepLearning
0:14

MLP in PyTorch by hand ✍️ ~ 7 steps walkthrough below Goal: fill in every blank in the PyTorch code to build a multi-layer perceptron. 1. Given Let us start with a code template on the left and the network it is supposed to build on the right. Every blank in the code can be worked out from the picture. 2. Linear layer We count: 3 features in, 4 features out. So the weight matrix is 4 by 3. There is an extra column for the biases, which means bias = T. 3. ReLU Let us apply the activation. ReLU crosses out the negatives, so -1 becomes 0. 4. Linear layer The input size is 4, because that is what the previous layer put out. The output size is 2. A 2 by 4 weight matrix, and this time no extra column, so bias = F. 5. ReLU We cross out the negatives again. 6. Linear layer Two features in, five out. A 5 by 2 weight matrix, with a bias column, so bias = T. 7. Sigmoid Let us finish. Sigmoid squashes the raw scores (3, 0, -2, 5, -5) into probabilities between 0 and 1. You have just implemented a three-layer deep neural network by hand. ✍️ == Story == Three years ago I gave this exercise to my students, to connect the code to the math. They found it odd. Every other AI course they were taking lived inside a Jupyter notebook, and here I was handing out paper. Three years later, my colleagues are the ones rushing to move their materials to paper. The exercise has not changed. Paper still asks the one thing a notebook lets you skip: do you actually understand what the code is doing? If you can tell me why the weight matrix is 4 by 3, and why bias is F on the second layer, you understand nn.Linear better than someone who has been copy-pasting it for a year. 💾 Save this post! #AIbyHand #PyTorch #DeepLearning

Tom Yeh

13,318 views • 1 month ago

[LSTM] by Hand ✍️ LSTMs have been the most effective architecture to process long sequences of data, until our world was taken over by the Transformers. LSTMs belong to the broader family of recurrent neural network (RNNs) that process data sequentially in a recurrent manner. Transformers, on the other hand, abandon recurrence and use self-attention instead to process data concurrently in parallel. Recently, there is renewed interest in recurrence as people realized self-attention doesn’t scale to extremely long sequences, like hundreds of thousands of tokens. Mamba is a good example to bring back recurrence. All of a sudden, it is cool to study LSTMs. How do LSTMs work? [1] Given ↳ 🟨 Input sequence X1, X2, X3 (d = 3) ↳ 🟩 Hidden state h (d = 2) ↳ 🟦 Memory C (d = 2) ↳ Weight matrices Wf, Wc, Wi, Wo Process t = 1 [2] Initialize ↳ Randomly set the previous hidden state h0 to [1, 1] and memory cells C0 to [0.3, -0.5] [3] Linear Transform ↳ Multiply the four weight matrices with the concatenation of current input (X1) and the previous hidden state (h0). ↳ The results are feature values, each is a linear combination of the current input and hidden state. [4] Non-linear Transform ↳ Apply sigmoid σ to obtain gate values (between 0 and 1). • Forget gate (f1): [-4, -6] → [0, 0] • Input gate (i1): [6, 4] → [1, 1] • Output gate (o1): [4, -5] → [1, 0] ↳ Apply tanh to obtain candidate memory values (between -1 and 1) • Candidate memory (C’1): [1, -6] → [0.8, -1] [5] Update Memory ↳ Forget (C0 .* f1): Element-wise multiply the current memory with forget gate values. ↳ Input (C’1 .* o1): Element-wise multiply the “candidate” memory with input gate values. ↳ Update the memory to C1 by adding the two terms above: C0 .* f1 + C’1 .* o1 = C1 [6] Candiate Output ↳ Apply tanh to the new memory C1 to obtain candidate output o’1. [0.8, -1] → [0.7, -0.8] [7] Update Hidden State ↳ Output (o’1 .* o1 → h1): Element-wise multiply the candidate output with the output gate. ↳ The result is updated hidden state h1 ↳ Also, it is the first output. Process t = 2 [8] Initialize ↳ Copy previous hidden state h1 and memory C1 [9] Linear Transform ↳ Repeat [3] [10] Update Memory (C2) ↳ Repeat [4] and [5] [11] Update Hidden State (h2) ↳ Repeat [6] and [7] Process t = 3 [12] Initialize ↳ Copy previous hidden state h2 and memory C2 [13] Linear Transform ↳ Repeat [3] [14] Update Memory (C3) ↳ Repeat [4] and [5] [15] Update Hidden State (h3) ↳ Repeat [6] and [7]
0:15

[LSTM] by Hand ✍️ LSTMs have been the most effective architecture to process long sequences of data, until our world was taken over by the Transformers. LSTMs belong to the broader family of recurrent neural network (RNNs) that process data sequentially in a recurrent manner. Transformers, on the other hand, abandon recurrence and use self-attention instead to process data concurrently in parallel. Recently, there is renewed interest in recurrence as people realized self-attention doesn’t scale to extremely long sequences, like hundreds of thousands of tokens. Mamba is a good example to bring back recurrence. All of a sudden, it is cool to study LSTMs. How do LSTMs work? [1] Given ↳ 🟨 Input sequence X1, X2, X3 (d = 3) ↳ 🟩 Hidden state h (d = 2) ↳ 🟦 Memory C (d = 2) ↳ Weight matrices Wf, Wc, Wi, Wo Process t = 1 [2] Initialize ↳ Randomly set the previous hidden state h0 to [1, 1] and memory cells C0 to [0.3, -0.5] [3] Linear Transform ↳ Multiply the four weight matrices with the concatenation of current input (X1) and the previous hidden state (h0). ↳ The results are feature values, each is a linear combination of the current input and hidden state. [4] Non-linear Transform ↳ Apply sigmoid σ to obtain gate values (between 0 and 1). • Forget gate (f1): [-4, -6] → [0, 0] • Input gate (i1): [6, 4] → [1, 1] • Output gate (o1): [4, -5] → [1, 0] ↳ Apply tanh to obtain candidate memory values (between -1 and 1) • Candidate memory (C’1): [1, -6] → [0.8, -1] [5] Update Memory ↳ Forget (C0 .* f1): Element-wise multiply the current memory with forget gate values. ↳ Input (C’1 .* o1): Element-wise multiply the “candidate” memory with input gate values. ↳ Update the memory to C1 by adding the two terms above: C0 .* f1 + C’1 .* o1 = C1 [6] Candiate Output ↳ Apply tanh to the new memory C1 to obtain candidate output o’1. [0.8, -1] → [0.7, -0.8] [7] Update Hidden State ↳ Output (o’1 .* o1 → h1): Element-wise multiply the candidate output with the output gate. ↳ The result is updated hidden state h1 ↳ Also, it is the first output. Process t = 2 [8] Initialize ↳ Copy previous hidden state h1 and memory C1 [9] Linear Transform ↳ Repeat [3] [10] Update Memory (C2) ↳ Repeat [4] and [5] [11] Update Hidden State (h2) ↳ Repeat [6] and [7] Process t = 3 [12] Initialize ↳ Copy previous hidden state h2 and memory C2 [13] Linear Transform ↳ Repeat [3] [14] Update Memory (C3) ↳ Repeat [4] and [5] [15] Update Hidden State (h3) ↳ Repeat [6] and [7]

Tom Yeh

72,966 views • 2 years ago

[Backpropagation] by Hand✍️ [1] Forward Pass ↳ Given a multi layer perceptron (3 levels), an input vector X, predictions Y^{Pred} = [0.5, 0.5, 0], and ground truth label Y^{Target} = [0, 1, 0]. [2] Backpropagation ↳ Insert cells to hold our calculations. [3] Layer 3 - Softmax (blue) ↳ Calculate ∂L / ∂z3 directly using the simple equation: Y^{Pred} - Y^{Target} = [0.5, -0.5, 0]. ↳ This simple equation is the benefit of using Softmax and Cross Entropy Loss together. [4] Layer 3 - Weights (orange) & Biases (black) ↳ Calculate ∂L / ∂W3 and ∂L / ∂b3 by multiplying ∂L / ∂z3 and [ a2 | 1 ]. [5] Layer 2 - Activations (green) ↳ Calculate ∂L / ∂a2 by multiplying ∂L / ∂z3 and W3. [6] Layer 2 - ReLU (blue) ↳ Calculate ∂L / ∂z2 by multiplying ∂L / ∂a2 with 1 for positive values and 0 otherwise. [7] Layer 2 - Weights (orange) & Biases (black) ↳ Calculate ∂L / ∂W2 and ∂L / ∂b2 by multiplying ∂L / ∂z2 and [ a1 | 1 ]. [8] Layer 1 - Activations (green) ↳ Calculate ∂L / ∂a1 by multiplying ∂L / ∂z2 and W2. [9] Layer 1 - ReLU (blue) ↳ Calculate ∂L / ∂z1 by multiplying ∂L / ∂a1 with 1 for positive values and 0 otherwise. [10] Layer 1 - Weights (orange) & Biases (black) ↳ Calculate ∂L / ∂W1 and ∂L / ∂b1 by multiplying ∂L / ∂z1 and [ x | 1 ]. [11] Gradient Descent ↳ Update weights and biases (typically a learning rate is applied here). 💡 Matrix Multiplication is All You Need: Just like in the forward pass, backpropagation is all about matrix multiplications. You can definitely do everything by hand as I demonstrated in this exercise, albeit slow and imperfect. This is why GPU's ability to multiply matrices efficiently plays such an important role in the deep learning evolution. This is why NVIDIA is now close to $1 trillion in valuation. 💡Exploding Gradients: We can already see the gradients are getting larger as we back-propagate up, even in this simple 3-layer network. This motivates using methods like skip connections to handle exploding (or diminishing) gradients as in the ResNet. I did the calculations entirely by hand. Please let me know if you spot any error or have any questions!
0:11

[Backpropagation] by Hand✍️ [1] Forward Pass ↳ Given a multi layer perceptron (3 levels), an input vector X, predictions Y^{Pred} = [0.5, 0.5, 0], and ground truth label Y^{Target} = [0, 1, 0]. [2] Backpropagation ↳ Insert cells to hold our calculations. [3] Layer 3 - Softmax (blue) ↳ Calculate ∂L / ∂z3 directly using the simple equation: Y^{Pred} - Y^{Target} = [0.5, -0.5, 0]. ↳ This simple equation is the benefit of using Softmax and Cross Entropy Loss together. [4] Layer 3 - Weights (orange) & Biases (black) ↳ Calculate ∂L / ∂W3 and ∂L / ∂b3 by multiplying ∂L / ∂z3 and [ a2 | 1 ]. [5] Layer 2 - Activations (green) ↳ Calculate ∂L / ∂a2 by multiplying ∂L / ∂z3 and W3. [6] Layer 2 - ReLU (blue) ↳ Calculate ∂L / ∂z2 by multiplying ∂L / ∂a2 with 1 for positive values and 0 otherwise. [7] Layer 2 - Weights (orange) & Biases (black) ↳ Calculate ∂L / ∂W2 and ∂L / ∂b2 by multiplying ∂L / ∂z2 and [ a1 | 1 ]. [8] Layer 1 - Activations (green) ↳ Calculate ∂L / ∂a1 by multiplying ∂L / ∂z2 and W2. [9] Layer 1 - ReLU (blue) ↳ Calculate ∂L / ∂z1 by multiplying ∂L / ∂a1 with 1 for positive values and 0 otherwise. [10] Layer 1 - Weights (orange) & Biases (black) ↳ Calculate ∂L / ∂W1 and ∂L / ∂b1 by multiplying ∂L / ∂z1 and [ x | 1 ]. [11] Gradient Descent ↳ Update weights and biases (typically a learning rate is applied here). 💡 Matrix Multiplication is All You Need: Just like in the forward pass, backpropagation is all about matrix multiplications. You can definitely do everything by hand as I demonstrated in this exercise, albeit slow and imperfect. This is why GPU's ability to multiply matrices efficiently plays such an important role in the deep learning evolution. This is why NVIDIA is now close to $1 trillion in valuation. 💡Exploding Gradients: We can already see the gradients are getting larger as we back-propagate up, even in this simple 3-layer network. This motivates using methods like skip connections to handle exploding (or diminishing) gradients as in the ResNet. I did the calculations entirely by hand. Please let me know if you spot any error or have any questions!

Tom Yeh

64,645 views • 2 years ago

We are excited to unveil the latest version of the AIOZ Node: The Version 4.0 update! This update includes a new user interface and brings substantial functional improvements, enhancing your overall experience for increased productivity and efficiency. More information below: The standout feature of AIOZ Node v4.0 is the introduction of the Transcoding functionality, which is currently available in beta. This functionality enables your node to participate in video transcoding, which converts video files into different formats for various digital devices and media platforms. By enabling transcoding, your node can contribute more significantly to the AIOZ Network, expanding the network's capabilities and potential $AIOZ token rewards. While the transcoding functionality is currently in beta, the upcoming AIOZ W3Stream integration, a DePIN Video Infrastructure due for release in Q3 2024, will unlock the full potential of your node and enable seamless video transcoding tasks. To get started with AIOZ Node v4.0, you simply need to visit our official website to download the latest version of the AIOZ Node: This download process is very straightforward, and with a one-click installation process, you can set up AIOZ Node v4.0 to start running on your device within a few minutes. If you are already running an AIOZ Node on your device, the version 4.0 update will be applied automatically, ensuring you have the latest features and improvements without hassle! With the Node v4.0 update running on your device, you can proceed to familiarize yourself with the new layout, check out the performance improvements, and start transcoding to see how it enhances your contributions to the network! Learn More: $AIOZ
0:15

We are excited to unveil the latest version of the AIOZ Node: The Version 4.0 update! This update includes a new user interface and brings substantial functional improvements, enhancing your overall experience for increased productivity and efficiency. More information below: The standout feature of AIOZ Node v4.0 is the introduction of the Transcoding functionality, which is currently available in beta. This functionality enables your node to participate in video transcoding, which converts video files into different formats for various digital devices and media platforms. By enabling transcoding, your node can contribute more significantly to the AIOZ Network, expanding the network's capabilities and potential $AIOZ token rewards. While the transcoding functionality is currently in beta, the upcoming AIOZ W3Stream integration, a DePIN Video Infrastructure due for release in Q3 2024, will unlock the full potential of your node and enable seamless video transcoding tasks. To get started with AIOZ Node v4.0, you simply need to visit our official website to download the latest version of the AIOZ Node: This download process is very straightforward, and with a one-click installation process, you can set up AIOZ Node v4.0 to start running on your device within a few minutes. If you are already running an AIOZ Node on your device, the version 4.0 update will be applied automatically, ensuring you have the latest features and improvements without hassle! With the Node v4.0 update running on your device, you can proceed to familiarize yourself with the new layout, check out the performance improvements, and start transcoding to see how it enhances your contributions to the network! Learn More: $AIOZ

AIOZ Network

20,428 views • 2 years ago

Vector Database by Hand ✍️ Vector databases are revolutionizing how we search and analyze complex data. They have become the backbone of Retrieval Augmented Generation (#RAG). How do vector databases work? [1] Given ↳ A dataset of three sentences, each has 3 words (or tokens) ↳ In practice, a dataset may contain millions or billions of sentences. The max number of tokens may be tens of thousands (e.g., 32,768 mistral-7b). Process "how are you" [2] 🟨 Word Embeddings ↳ For each word, look up corresponding word embedding vector from a table of 22 vectors, where 22 is the vocabulary size. ↳ In practice, the vocabulary size can be tens of thousands. The word embedding dimensions are in the thousands (e.g., 1024, 4096) [3] 🟩 Encoding ↳ Feed the sequence of word embeddings to an encoder to obtain a sequence of feature vectors, one per word. ↳ Here, the encoder is a simple one layer perceptron (linear layer + ReLU) ↳ In practice, the encoder is a transformer or one of its many variants. [4] 🟩 Mean Pooling ↳ Merge the sequence of feature vectors into a single vector using "mean pooling" which is to average across the columns. ↳ The result is a single vector. We often call it "text embeddings" or "sentence embeddings." ↳ Other pooling techniques are possible, such as CLS. But mean pooling is the most common. [5] 🟦 Indexing ↳ Reduce the dimensions of the text embedding vector by a projection matrix. The reduction rate is 50% (4->2). ↳ In practice, the values in this projection matrix is much more random. ↳ The purpose is similar to that of hashing, which is to obtain a short representation to allow faster comparison and retrieval. ↳ The resulting dimension-reduced index vector is saved in the vector storage. [6] Process "who are you" ↳ Repeat [2]-[5] [7] Process "who am I" ↳ Repeat [2]-[5] Now we have indexed our dataset in the vector database. [8] 🟥 Query: "am I you" ↳ Repeat [2]-[5] ↳ The result is a 2-d query vector. [9] 🟥 Dot Products ↳ Take dot product between the query vector and database vectors. They are all 2-d. ↳ The purpose is to use dot product to estimate similarity. ↳ By transposing the query vector, this step becomes a matrix multiplication. [10] 🟥 Nearest Neighbor ↳ Find the largest dot product by linear scan. ↳ The sentence with the highest dot product is "who am I" ↳ In practice, because scanning billions of vectors is slow, we use an Approximate Nearest Neighbor (ANN) algorithm like the Hierarchical Navigable Small Worlds (HNSW).
0:17

Vector Database by Hand ✍️ Vector databases are revolutionizing how we search and analyze complex data. They have become the backbone of Retrieval Augmented Generation (#RAG). How do vector databases work? [1] Given ↳ A dataset of three sentences, each has 3 words (or tokens) ↳ In practice, a dataset may contain millions or billions of sentences. The max number of tokens may be tens of thousands (e.g., 32,768 mistral-7b). Process "how are you" [2] 🟨 Word Embeddings ↳ For each word, look up corresponding word embedding vector from a table of 22 vectors, where 22 is the vocabulary size. ↳ In practice, the vocabulary size can be tens of thousands. The word embedding dimensions are in the thousands (e.g., 1024, 4096) [3] 🟩 Encoding ↳ Feed the sequence of word embeddings to an encoder to obtain a sequence of feature vectors, one per word. ↳ Here, the encoder is a simple one layer perceptron (linear layer + ReLU) ↳ In practice, the encoder is a transformer or one of its many variants. [4] 🟩 Mean Pooling ↳ Merge the sequence of feature vectors into a single vector using "mean pooling" which is to average across the columns. ↳ The result is a single vector. We often call it "text embeddings" or "sentence embeddings." ↳ Other pooling techniques are possible, such as CLS. But mean pooling is the most common. [5] 🟦 Indexing ↳ Reduce the dimensions of the text embedding vector by a projection matrix. The reduction rate is 50% (4->2). ↳ In practice, the values in this projection matrix is much more random. ↳ The purpose is similar to that of hashing, which is to obtain a short representation to allow faster comparison and retrieval. ↳ The resulting dimension-reduced index vector is saved in the vector storage. [6] Process "who are you" ↳ Repeat [2]-[5] [7] Process "who am I" ↳ Repeat [2]-[5] Now we have indexed our dataset in the vector database. [8] 🟥 Query: "am I you" ↳ Repeat [2]-[5] ↳ The result is a 2-d query vector. [9] 🟥 Dot Products ↳ Take dot product between the query vector and database vectors. They are all 2-d. ↳ The purpose is to use dot product to estimate similarity. ↳ By transposing the query vector, this step becomes a matrix multiplication. [10] 🟥 Nearest Neighbor ↳ Find the largest dot product by linear scan. ↳ The sentence with the highest dot product is "who am I" ↳ In practice, because scanning billions of vectors is slow, we use an Approximate Nearest Neighbor (ANN) algorithm like the Hierarchical Navigable Small Worlds (HNSW).

Tom Yeh

192,022 views • 2 years ago

Transformer by hand ✍️ ~ 6 steps walkthrough below Open the hood of a transformer and the parts list is overwhelming: embeddings, positional encoding, attention weighting, self-attention, cross-attention, multi-head attention, layer norm, skip connections, softmax, linear, Nx, shifted right, query, key, value, masking. Which of those actually make the car run? Two of them. Attention weighting and the feed-forward network. Everything else is an enhancement to make it run faster and longer, which is how we got from a car to a truck, and to the word "large" in large language model. So I drew and calculated those two parts entirely by hand. Goal: push five features through one transformer block, filling in every cell yourself. 1. Given Five positions of input features, arriving from the previous block. 2. Attention matrix Let us feed all five features to a query-key module (QK) and read back an attention weight matrix, A. The details of that module are a post of their own. 3. Attention weighting We multiply the input features by A to get the attention weighted features, Z. Still five positions. The effect is to combine features *across positions*, horizontally: X1 becomes X1 + X2, X2 becomes X2 + X3, and so on. 4. First layer Let us feed all five weighted features into the first layer of the FFN. Multiply by the weights and biases. This time the combining happens *across feature dimensions*, vertically, and each feature grows from 3 numbers to 4. Note that every position goes through the same weight matrix. That is what "position-wise" means. 5. ReLU We cross out the negatives. They become zeros. 6. Second layer Let us bring it back down: 4 dimensions to 3. The output feeds the next block, which has a completely separate set of parameters, and the whole thing runs again. You have just calculated a transformer block by hand. ✍️ The takeaway: the two parts are doing two different jobs, and neither one alone is enough. Attention mixes *across positions*, so a feature can see its neighbours. The FFN mixes *across feature dimensions*, so each position can think about itself. Horizontal, then vertical. Then that pattern repeats N times, each block with its own separate set of weights. That is the Nx from the list up top, and that is what makes the transformer run. 💾 Save this post! #AIbyHand #Transformers #DeepLearning
0:12

Transformer by hand ✍️ ~ 6 steps walkthrough below Open the hood of a transformer and the parts list is overwhelming: embeddings, positional encoding, attention weighting, self-attention, cross-attention, multi-head attention, layer norm, skip connections, softmax, linear, Nx, shifted right, query, key, value, masking. Which of those actually make the car run? Two of them. Attention weighting and the feed-forward network. Everything else is an enhancement to make it run faster and longer, which is how we got from a car to a truck, and to the word "large" in large language model. So I drew and calculated those two parts entirely by hand. Goal: push five features through one transformer block, filling in every cell yourself. 1. Given Five positions of input features, arriving from the previous block. 2. Attention matrix Let us feed all five features to a query-key module (QK) and read back an attention weight matrix, A. The details of that module are a post of their own. 3. Attention weighting We multiply the input features by A to get the attention weighted features, Z. Still five positions. The effect is to combine features *across positions*, horizontally: X1 becomes X1 + X2, X2 becomes X2 + X3, and so on. 4. First layer Let us feed all five weighted features into the first layer of the FFN. Multiply by the weights and biases. This time the combining happens *across feature dimensions*, vertically, and each feature grows from 3 numbers to 4. Note that every position goes through the same weight matrix. That is what "position-wise" means. 5. ReLU We cross out the negatives. They become zeros. 6. Second layer Let us bring it back down: 4 dimensions to 3. The output feeds the next block, which has a completely separate set of parameters, and the whole thing runs again. You have just calculated a transformer block by hand. ✍️ The takeaway: the two parts are doing two different jobs, and neither one alone is enough. Attention mixes *across positions*, so a feature can see its neighbours. The FFN mixes *across feature dimensions*, so each position can think about itself. Horizontal, then vertical. Then that pattern repeats N times, each block with its own separate set of weights. That is the Nx from the list up top, and that is what makes the transformer run. 💾 Save this post! #AIbyHand #Transformers #DeepLearning

Tom Yeh

26,089 views • 1 month ago

a contractor in Shenzhen priced a ¥12,470,900 hospital contract, about $1.7m, in one afternoon and beat firms carrying forty people he explained how he did it: the bid consultancy he used to pay took three days and ¥46,000 for the same envelope. he did this one alone, off one screen, at 11.4% margin, uploaded before the 17:00 cutoff 214 pages of tender documents read, 68 binding clauses pulled out, 9,485 building parts loaded, 14 places found where a duct and a beam sit in the same cubic metre, deepest one 38mm, all of them fixed, 3,318 lines of quantities priced and the package encrypted and uploaded before the 17:00 cutoff this is Graph Engineering: the job gets cut into small nodes, one narrow task each, wired so that one node's output is the next node's input, and any node is allowed to stop the whole run. it turns a model that answers you into a machine that finishes the job: - give every node one job and one output. a node doing two things fails at both and you cannot tell which one broke - put the cheapest rejection first. his qualification node reads clause 7.4, foreign-owned firms barred, and ends the run four seconds in, before anything expensive touches the model - what moves between nodes is a file. the model travels as a model, the quantities as a table, the price as a number - build exactly one loop: the checker finds 14 collisions, the fixer drops the duct 550mm, the checker runs again, and nothing moves on until the count is zero - cap that loop, or a graph will grind on three impossible clashes until the deadline passes - keep one node whose only job is to say no, and give it authority over everything above it - log each node's output on its own, because when the price comes out wrong you need to know which node believed the wrong thing - run the expensive nodes last, always the catch is that a graph is an extremely confident machine: point it at an outdated rate book and it prices an entire hospital off it without a single node noticing, because no node is asked to doubt the input, only to process it so the nodes that earn their keep are the ones that reject, and almost nobody builds those first bookmark this, the full build with all nine nodes and what each one hands to the next is written out in the article ↓
0:16

a contractor in Shenzhen priced a ¥12,470,900 hospital contract, about $1.7m, in one afternoon and beat firms carrying forty people he explained how he did it: the bid consultancy he used to pay took three days and ¥46,000 for the same envelope. he did this one alone, off one screen, at 11.4% margin, uploaded before the 17:00 cutoff 214 pages of tender documents read, 68 binding clauses pulled out, 9,485 building parts loaded, 14 places found where a duct and a beam sit in the same cubic metre, deepest one 38mm, all of them fixed, 3,318 lines of quantities priced and the package encrypted and uploaded before the 17:00 cutoff this is Graph Engineering: the job gets cut into small nodes, one narrow task each, wired so that one node's output is the next node's input, and any node is allowed to stop the whole run. it turns a model that answers you into a machine that finishes the job: - give every node one job and one output. a node doing two things fails at both and you cannot tell which one broke - put the cheapest rejection first. his qualification node reads clause 7.4, foreign-owned firms barred, and ends the run four seconds in, before anything expensive touches the model - what moves between nodes is a file. the model travels as a model, the quantities as a table, the price as a number - build exactly one loop: the checker finds 14 collisions, the fixer drops the duct 550mm, the checker runs again, and nothing moves on until the count is zero - cap that loop, or a graph will grind on three impossible clashes until the deadline passes - keep one node whose only job is to say no, and give it authority over everything above it - log each node's output on its own, because when the price comes out wrong you need to know which node believed the wrong thing - run the expensive nodes last, always the catch is that a graph is an extremely confident machine: point it at an outdated rate book and it prices an entire hospital off it without a single node noticing, because no node is asked to doubt the input, only to process it so the nodes that earn their keep are the ones that reject, and almost nobody builds those first bookmark this, the full build with all nine nodes and what each one hands to the next is written out in the article ↓

Argona

38,189 views • 1 month ago

Microsoft made 100B parameter models run on a single CPU. bitnet.cpp: The official inference framework for 1-bit LLMs. The math behind 1-bit LLMs is what makes them revolutionary. Traditional LLMs use 16-bit floating point weights. Every parameter is a number like 0.0023847 or -1.4729. When you run inference, you multiply these floats together. Billions of times. That's why you need GPUs, they're optimized for floating point matrix multiplication. BitNet b1.58 uses ternary weights: {-1, 0, 1}. That's not a simplification. That's a fundamental change in the math. When your weights are only -1, 0, or 1: → Multiply by 1 = keep the value → Multiply by -1 = flip the sign → Multiply by 0 = skip entirely Matrix multiplication becomes addition and subtraction. No floating point operations. No GPU required. This is why bitnet.cpp achieves: → 2.37x to 6.17x speedup on x86 CPUs → 1.37x to 5.07x speedup on ARM CPUs → 71.9% to 82.2% energy reduction on x86 → 55.4% to 70.0% energy reduction on ARM The speedups scale with model size. Larger models see bigger gains because there are more operations to simplify. A 100B parameter model running at human reading speed (5-7 tokens/second) on a single CPU. That's not optimization. That's a different paradigm. Why 1.58 bits? Because log₂(3) ≈ 1.58. Three possible values = 1.58 bits of information per weight. The key insight: These models aren't quantized after training. They're trained from scratch with ternary weights. The model learns to work within the constraint. No precision loss. No quality tradeoff.
0:13

Microsoft made 100B parameter models run on a single CPU. bitnet.cpp: The official inference framework for 1-bit LLMs. The math behind 1-bit LLMs is what makes them revolutionary. Traditional LLMs use 16-bit floating point weights. Every parameter is a number like 0.0023847 or -1.4729. When you run inference, you multiply these floats together. Billions of times. That's why you need GPUs, they're optimized for floating point matrix multiplication. BitNet b1.58 uses ternary weights: {-1, 0, 1}. That's not a simplification. That's a fundamental change in the math. When your weights are only -1, 0, or 1: → Multiply by 1 = keep the value → Multiply by -1 = flip the sign → Multiply by 0 = skip entirely Matrix multiplication becomes addition and subtraction. No floating point operations. No GPU required. This is why bitnet.cpp achieves: → 2.37x to 6.17x speedup on x86 CPUs → 1.37x to 5.07x speedup on ARM CPUs → 71.9% to 82.2% energy reduction on x86 → 55.4% to 70.0% energy reduction on ARM The speedups scale with model size. Larger models see bigger gains because there are more operations to simplify. A 100B parameter model running at human reading speed (5-7 tokens/second) on a single CPU. That's not optimization. That's a different paradigm. Why 1.58 bits? Because log₂(3) ≈ 1.58. Three possible values = 1.58 bits of information per weight. The key insight: These models aren't quantized after training. They're trained from scratch with ternary weights. The model learns to work within the constraint. No precision loss. No quality tradeoff.

Tech with Mak

23,036 views • 4 months ago

ReLU vs Leaky ReLU 👉 = ReLU = ReLU is the default activation in modern deep learning — cheap to compute, and stable enough to train networks hundreds of layers deep. To see what it does, picture five boba tea shops on the same block — 𝚊, 𝚋, 𝚌, 𝚍, 𝚎 — each running their own books. Each value is a shop's monthly profit — receipts minus rent, ingredients, and wages. When profit is positive, the shop stays open and the owner pockets every dollar. When profit turns negative, the shop runs out of cash and shutters — the lights go off, the books are wiped to zero. ReLU is exactly that rule, applied one shop at a time. Read the diagram left to right. The first column is the raw value x — each shop's profit at month's end. The second column is the gate: 1 if the shop is open (x > 0), 0 if it has shuttered. The last column is the ReLU output: open shops pass their profit through untouched, while shuttered ones are zeroed out. Five rows means five parallel shops on the same block, each evaluated independently. That's why ReLU is called an element-wise activation: every neuron decides its own fate. = LeakyRelu = Plain ReLU wipes negative values to zero — clean, but a shop that shutters can never recover, since both its output and its gradient stay pinned at zero. This is the dying ReLU problem, and in deep networks it can quietly kill a meaningful fraction of the units. Leaky ReLU is the one-line fix: instead of shuttering, the shop files for Chapter 11 protection and keeps the lights on at reduced capacity. Its debt is restructured down to a fraction α (typically 0.1) — the rest is forgiven, and the shop is wounded, not killed. A small negative signal still flows through, so the gradient survives, and the shop can crawl back to life if a TikTok goes viral. Read the diagram left to right. The first column is the raw value x — each shop's profit at month's end. The second column is the leakage α — the fraction of the loss held over after restructuring (default 0.1, editable). The third column is the gate: 1 for shops still in the black, α for those operating under bankruptcy protection. The last column is the Leaky ReLU output: y = x · gate. Profitable shops pass through untouched; struggling ones shrink by a factor of α but still carry a sign. Five rows means five parallel shops, each evaluated independently. Like ReLU, this is an element-wise activation: every neuron's fate is decided on its own merits. #aibyhahd
0:18

ReLU vs Leaky ReLU 👉 = ReLU = ReLU is the default activation in modern deep learning — cheap to compute, and stable enough to train networks hundreds of layers deep. To see what it does, picture five boba tea shops on the same block — 𝚊, 𝚋, 𝚌, 𝚍, 𝚎 — each running their own books. Each value is a shop's monthly profit — receipts minus rent, ingredients, and wages. When profit is positive, the shop stays open and the owner pockets every dollar. When profit turns negative, the shop runs out of cash and shutters — the lights go off, the books are wiped to zero. ReLU is exactly that rule, applied one shop at a time. Read the diagram left to right. The first column is the raw value x — each shop's profit at month's end. The second column is the gate: 1 if the shop is open (x > 0), 0 if it has shuttered. The last column is the ReLU output: open shops pass their profit through untouched, while shuttered ones are zeroed out. Five rows means five parallel shops on the same block, each evaluated independently. That's why ReLU is called an element-wise activation: every neuron decides its own fate. = LeakyRelu = Plain ReLU wipes negative values to zero — clean, but a shop that shutters can never recover, since both its output and its gradient stay pinned at zero. This is the dying ReLU problem, and in deep networks it can quietly kill a meaningful fraction of the units. Leaky ReLU is the one-line fix: instead of shuttering, the shop files for Chapter 11 protection and keeps the lights on at reduced capacity. Its debt is restructured down to a fraction α (typically 0.1) — the rest is forgiven, and the shop is wounded, not killed. A small negative signal still flows through, so the gradient survives, and the shop can crawl back to life if a TikTok goes viral. Read the diagram left to right. The first column is the raw value x — each shop's profit at month's end. The second column is the leakage α — the fraction of the loss held over after restructuring (default 0.1, editable). The third column is the gate: 1 for shops still in the black, α for those operating under bankruptcy protection. The last column is the Leaky ReLU output: y = x · gate. Profitable shops pass through untouched; struggling ones shrink by a factor of α but still carry a sign. Five rows means five parallel shops, each evaluated independently. Like ReLU, this is an element-wise activation: every neuron's fate is decided on its own merits. #aibyhahd

Tom Yeh

32,561 views • 4 months ago

FIVE LAYERS OF AGENT ENGINEERING, EACH ONE WRAPS THE ONE BELOW IT. IF YOU SKIP LAYER 2, YOUR LAYER 5 WILL LOOK BROKEN WHEN IT IS ACTUALLY JUST STANDING ON NOTHING. for weeks i debated harness vs loop vs graph like they were competing choices. then a stack diagram made the shape obvious. they are not choices. they are floors. 01 | prompt engineering. the message. unit of work: one input. inputs are role, instructions, examples, format. output is a single raw response. 02 | context engineering. the memory. unit of work: what stays in the window. a curator selects, compresses, and drops from query, docs, memory, prior turns, and tool outputs before the prompt runs. 03 | harness engineering. the machine. unit of work: the machine itself. gather (context + prompt) → LLM → tools or sub-agents → verifier → final response. the article calls this the operating environment. 04 | loop engineering. the system. unit of work: the run. goal + success criteria + max iterations + budget + completion check wrap around one harness pass. failed pass appends results to context and retries. 05 | graph engineering. the topology. unit of work: the graph run. goal + nodes + edges + state schema. graph routes to agent nodes, tool nodes, or human approval. a reviewer node with a different model and fresh context checks the final answer. the wrapping is the whole point. layer 5 assumes layer 4 works. layer 4 assumes layer 3 works. skip layer 2 and layer 3's verifier keeps failing without a clear reason. this is why swapping the model is a one-day project and swapping the stack is a quarter. the model is the commodity. the five layers around it are the engineering. full three-layer breakdown of the top of the stack (harness, loop, graph) in the post below.
0:20

FIVE LAYERS OF AGENT ENGINEERING, EACH ONE WRAPS THE ONE BELOW IT. IF YOU SKIP LAYER 2, YOUR LAYER 5 WILL LOOK BROKEN WHEN IT IS ACTUALLY JUST STANDING ON NOTHING. for weeks i debated harness vs loop vs graph like they were competing choices. then a stack diagram made the shape obvious. they are not choices. they are floors. 01 | prompt engineering. the message. unit of work: one input. inputs are role, instructions, examples, format. output is a single raw response. 02 | context engineering. the memory. unit of work: what stays in the window. a curator selects, compresses, and drops from query, docs, memory, prior turns, and tool outputs before the prompt runs. 03 | harness engineering. the machine. unit of work: the machine itself. gather (context + prompt) → LLM → tools or sub-agents → verifier → final response. the article calls this the operating environment. 04 | loop engineering. the system. unit of work: the run. goal + success criteria + max iterations + budget + completion check wrap around one harness pass. failed pass appends results to context and retries. 05 | graph engineering. the topology. unit of work: the graph run. goal + nodes + edges + state schema. graph routes to agent nodes, tool nodes, or human approval. a reviewer node with a different model and fresh context checks the final answer. the wrapping is the whole point. layer 5 assumes layer 4 works. layer 4 assumes layer 3 works. skip layer 2 and layer 3's verifier keeps failing without a clear reason. this is why swapping the model is a one-day project and swapping the stack is a quarter. the model is the commodity. the five layers around it are the engineering. full three-layer breakdown of the top of the stack (harness, loop, graph) in the post below.

kocer

30,675 views • 10 days ago

What is Hedera Hashgraph? Unlike traditional blockchains, Hedera (Hedera) does not organize transactions into a single chain of blocks. Instead, it uses a different distributed ledger structure called Hashgraph, designed to reach consensus quickly without proof-of-work mining. Here’s how it works: (1) It uses “gossip about gossip.” Nodes continuously share transactions with other nodes. Each message also contains information about previous messages, creating a history of how information moved through the network. This produces a directed acyclic graph, or DAG, rather than a traditional blockchain. (2) It uses virtual voting. Nodes do not need to send separate votes across the network. Instead, they can determine how other nodes would vote by examining the shared history. This reduces communication overhead while helping nodes establish transaction order. (3) It targets fast and final consensus. Hedera uses an asynchronous Byzantine Fault Tolerant, or aBFT, consensus model. Once consensus is reached, transactions achieve finality rather than remaining subject to the probabilistic settlement associated with some proof-of-work systems. (4) HBAR powers the network. HBAR is Hedera’s native cryptocurrency. It is used to pay network fees and plays a role in staking and network services. (5) Hedera is not simply another permissionless blockchain. Hedera is currently a public network with permissioned nodes operated through the Hedera Council. The Council governs key aspects of the network, while the underlying technology and network services are open source. (6) A path toward permissionless nodes is planned, but not live yet. Hedera has published a roadmap to gradually open up node operation beyond Council members, eventually moving toward a fully permissionless model where anyone can run a consensus node by staking HBAR. As of now, that transition has not been implemented, node operation is still limited to Council members. The bigger idea behind Hedera is straightforward. Instead of relying on blocks, miners, and proof-of-work, it uses Hashgraph consensus to coordinate transactions. That architecture is designed to deliver fast finality, high throughput, predictable fees, and low energy consumption. And that is what makes Hedera different from a conventional blockchain.
0:15

What is Hedera Hashgraph? Unlike traditional blockchains, Hedera (Hedera) does not organize transactions into a single chain of blocks. Instead, it uses a different distributed ledger structure called Hashgraph, designed to reach consensus quickly without proof-of-work mining. Here’s how it works: (1) It uses “gossip about gossip.” Nodes continuously share transactions with other nodes. Each message also contains information about previous messages, creating a history of how information moved through the network. This produces a directed acyclic graph, or DAG, rather than a traditional blockchain. (2) It uses virtual voting. Nodes do not need to send separate votes across the network. Instead, they can determine how other nodes would vote by examining the shared history. This reduces communication overhead while helping nodes establish transaction order. (3) It targets fast and final consensus. Hedera uses an asynchronous Byzantine Fault Tolerant, or aBFT, consensus model. Once consensus is reached, transactions achieve finality rather than remaining subject to the probabilistic settlement associated with some proof-of-work systems. (4) HBAR powers the network. HBAR is Hedera’s native cryptocurrency. It is used to pay network fees and plays a role in staking and network services. (5) Hedera is not simply another permissionless blockchain. Hedera is currently a public network with permissioned nodes operated through the Hedera Council. The Council governs key aspects of the network, while the underlying technology and network services are open source. (6) A path toward permissionless nodes is planned, but not live yet. Hedera has published a roadmap to gradually open up node operation beyond Council members, eventually moving toward a fully permissionless model where anyone can run a consensus node by staking HBAR. As of now, that transition has not been implemented, node operation is still limited to Council members. The bigger idea behind Hedera is straightforward. Instead of relying on blocks, miners, and proof-of-work, it uses Hashgraph consensus to coordinate transactions. That architecture is designed to deliver fast finality, high throughput, predictable fees, and low energy consumption. And that is what makes Hedera different from a conventional blockchain.

BSCN

18,168 views • 9 days ago

Softmax vs Sigmoid ✍️ Interact 👉 = Softmax = Softmax is how deep networks turn raw scores into a probability distribution — the final layer of every classifier, and the core of every attention head in a transformer. To see what it does, picture five boba tea shops on the same block, all competing for your dollar. Five candidates: a, b, c, d, e — different chains, different brewing styles, different pearls. A boba reviewer hands you a 𝘤𝘩𝘦𝘸𝘪𝘯𝘦𝘴𝘴 𝘴𝘤𝘰𝘳𝘦 for each — higher means perfectly chewy "QQ" pearls with the right bite (ask a Taiwanese friend to find out what QQ means). Negative scores are real: mushy bobas, overcooked pearls, a batch left sitting too long. How do you turn five chewiness scores into an allocation that adds to a whole dollar? You could spend everything at the chewiest shop, but that ignores how good the runners-up are. Softmax is the smooth alternative. Read the diagram left to right. First, raise each score to e^{x} — this does two things: it turns negative chewiness into small positives, and it stretches the gaps between scores exponentially. Then sum all five into a single total Z. Finally, divide each e^{x} by Z to get a probability. The five probabilities add up to one, so you can read them as percentages of your dollar. The chewiest shop gets the biggest slice — but never the whole dollar. That's the point of softmax: it ranks confidently while still leaving room for the others. = Sigmoid = Sigmoid squashes any real number into a probability between 0 and 1 — the classic activation for binary classification, and still the gating function inside LSTMs and GRUs. Same boba block as the previous Softmax example, narrowed to just two contenders — a hot new shop `a` with chewiness score x, and your usual go-to `b` whose score is pinned at zero (the neutral baseline you've come to expect). Sigmoid is just softmax with two players, one of them pinned to zero. Read the diagram left to right. First, raise each score to e^{x} — for the usual shop `b` whose score is zero, this is just e^0 = 1 (the constant baseline). Then sum the two into a total Z. Finally, divide each e^{x} by Z to get a probability. The two probabilities add up to one — the new shop wins more of your dollar when its pearls get chewier, and your usual keeps the rest. That's the point of sigmoid: it turns a single chewiness score into a clean 0-to-1 chance you'll try the new place over your usual. --- AI Math, Algorithms, Architectures by hand ✍️ Subscribe to my 60K+ reader newsletter 👉
0:14

Softmax vs Sigmoid ✍️ Interact 👉 = Softmax = Softmax is how deep networks turn raw scores into a probability distribution — the final layer of every classifier, and the core of every attention head in a transformer. To see what it does, picture five boba tea shops on the same block, all competing for your dollar. Five candidates: a, b, c, d, e — different chains, different brewing styles, different pearls. A boba reviewer hands you a 𝘤𝘩𝘦𝘸𝘪𝘯𝘦𝘴𝘴 𝘴𝘤𝘰𝘳𝘦 for each — higher means perfectly chewy "QQ" pearls with the right bite (ask a Taiwanese friend to find out what QQ means). Negative scores are real: mushy bobas, overcooked pearls, a batch left sitting too long. How do you turn five chewiness scores into an allocation that adds to a whole dollar? You could spend everything at the chewiest shop, but that ignores how good the runners-up are. Softmax is the smooth alternative. Read the diagram left to right. First, raise each score to e^{x} — this does two things: it turns negative chewiness into small positives, and it stretches the gaps between scores exponentially. Then sum all five into a single total Z. Finally, divide each e^{x} by Z to get a probability. The five probabilities add up to one, so you can read them as percentages of your dollar. The chewiest shop gets the biggest slice — but never the whole dollar. That's the point of softmax: it ranks confidently while still leaving room for the others. = Sigmoid = Sigmoid squashes any real number into a probability between 0 and 1 — the classic activation for binary classification, and still the gating function inside LSTMs and GRUs. Same boba block as the previous Softmax example, narrowed to just two contenders — a hot new shop `a` with chewiness score x, and your usual go-to `b` whose score is pinned at zero (the neutral baseline you've come to expect). Sigmoid is just softmax with two players, one of them pinned to zero. Read the diagram left to right. First, raise each score to e^{x} — for the usual shop `b` whose score is zero, this is just e^0 = 1 (the constant baseline). Then sum the two into a total Z. Finally, divide each e^{x} by Z to get a probability. The two probabilities add up to one — the new shop wins more of your dollar when its pearls get chewier, and your usual keeps the rest. That's the point of sigmoid: it turns a single chewiness score into a clean 0-to-1 chance you'll try the new place over your usual. --- AI Math, Algorithms, Architectures by hand ✍️ Subscribe to my 60K+ reader newsletter 👉

Tom Yeh

74,467 views • 4 months ago

Stateless History Node is almost like a regular Ethereum node, but it doesn't store state and it doesn't have EVM execution. It's used only for syncing events and thus - is faster and gives you FREE INDEXING. You don't have to pay 6 figures for RPC anymore! Just spin up a Stateless History Node, plug rindexer or Ponder there, and enjoy free (AND FAST!!) indexing! This node is syncing >1000 blocks per second at my local pc (less than 6hrs for the whole Ethereum), and it should use less than 200GB - which means you can host it on a MacMini, Hetzner or whatever. You can futhermore filter that by using block ranges or bloom filters, etc - I haven't developed this yet. What you see is a proof of concept. It works via native devp2p 'eth' protocol, but with EIP4444 and The Prune we would have to also support era1 archives and Portal Network. But so far it works - there are plenty of peers serving historical receipts, and they serve them FAST! If you run Stateless History Node you can also serve the blocks and receipts - so that could help to preserve archival data too. For now there is no data validation yet (and even no data storage - that's a very early PoC), but we can verify validity of chain by simultaneously running a lightweight CL node (or not lightweight if you're extremely paranoid). And then support verifying the hashes of receipts and blocks with their parents, maintaining full integrity and zero trust. It's also written in rust, btw. So, I guess, at least for Ethereum Mainnet the era of RPC's pumping moneybags is over - there's finally a local, trustless and free indexing alternative available. Too sad this won't work for Optimism / Base , cause despite introducing P2P after Bedrock - they haven't enabled receipts transfer in the protocol (or at least I couldn't find one). Arbitrum is even sadder - I don't believe there is a P2P layer at all - you just have to run your own node, hold state and execute blocks to get events. There is hope - Paradigm recently released Ress - stateless execution, but it requires nodes to support Witness preparation & exchange - but this could work for L2s - cause the main blocker for local RPCs rn is huge state (VPS with TB storage cost a lot), and the second blocker is EVM forks makes it hard to hold a node - it needs to be maintained, upgraded, etc. Ress at least solves the state part. But anyways, I will try to continue working on this and release some MVP version with RPC endpoint and data storage soon - follow the updates!
0:16

Stateless History Node is almost like a regular Ethereum node, but it doesn't store state and it doesn't have EVM execution. It's used only for syncing events and thus - is faster and gives you FREE INDEXING. You don't have to pay 6 figures for RPC anymore! Just spin up a Stateless History Node, plug rindexer or Ponder there, and enjoy free (AND FAST!!) indexing! This node is syncing >1000 blocks per second at my local pc (less than 6hrs for the whole Ethereum), and it should use less than 200GB - which means you can host it on a MacMini, Hetzner or whatever. You can futhermore filter that by using block ranges or bloom filters, etc - I haven't developed this yet. What you see is a proof of concept. It works via native devp2p 'eth' protocol, but with EIP4444 and The Prune we would have to also support era1 archives and Portal Network. But so far it works - there are plenty of peers serving historical receipts, and they serve them FAST! If you run Stateless History Node you can also serve the blocks and receipts - so that could help to preserve archival data too. For now there is no data validation yet (and even no data storage - that's a very early PoC), but we can verify validity of chain by simultaneously running a lightweight CL node (or not lightweight if you're extremely paranoid). And then support verifying the hashes of receipts and blocks with their parents, maintaining full integrity and zero trust. It's also written in rust, btw. So, I guess, at least for Ethereum Mainnet the era of RPC's pumping moneybags is over - there's finally a local, trustless and free indexing alternative available. Too sad this won't work for Optimism / Base , cause despite introducing P2P after Bedrock - they haven't enabled receipts transfer in the protocol (or at least I couldn't find one). Arbitrum is even sadder - I don't believe there is a P2P layer at all - you just have to run your own node, hold state and execute blocks to get events. There is hope - Paradigm recently released Ress - stateless execution, but it requires nodes to support Witness preparation & exchange - but this could work for L2s - cause the main blocker for local RPCs rn is huge state (VPS with TB storage cost a lot), and the second blocker is EVM forks makes it hard to hold a node - it needs to be maintained, upgraded, etc. Ress at least solves the state part. But anyways, I will try to continue working on this and release some MVP version with RPC endpoint and data storage soon - follow the updates!

Convergence Boy

29,823 views • 7 months ago

In my class, I teach the autoencoder by asking everyone to stand up. 🙆 Stretch your arms out wide. Imagine you are holding a heavy textbook (like Introduction to Algorithms by Prof. Cormen), the whole thing, every page. Now bring your hands slowly together until they almost touch your neck. The bottle "neck." The final exam is tomorrow and you are allowed one cheat sheet: whatever you can scribble on your palm. All nine hundred pages have to survive the squeeze. That is the encoder. Now imagine you sit in the exam. Push your arms back out to where they started. You try to rebuild the textbook from your palm notes. That is the decoder. Of course you cannot get every page back. What you get back is what mattered enough to write down, and the gap between the two is the loss the network is trying to shrink. I call it AI by Arms 🙆. It gets a laugh, and then it gets remembered. Goal: squeeze four numbers down to two, then rebuild the original four from them. 1. Given Let us start with four training examples: X1, X2, X3, X4. 2. Auto (copy to targets) We copy the training examples straight into the targets. That is the whole trick behind the name: "auto" is Greek for "self", and the data is its own label. 3. Encoder, layer 1 Let us multiply the inputs by the weights, add the biases, and apply ReLU. Negative values get crossed out and become zero. 4. Encoder, layer 2 (the bottleneck) We do it again, and now the four dimensions have become two. This layer is called the bottleneck, because everything has to fit through it. 5. Decoder, layer 1 Let us go back the other way: multiply, add, ReLU. This time there are no negatives to cross out. 6. Decoder, layer 2 We multiply once more and get the outputs Y. This is the decoder's attempt to rebuild the four original numbers from the two it was given. 7. MSE loss gradients Let us compare Y with the targets Y'. The gradient is 2 x (Y - Y'): subtract, then double. Those gradients kick off backpropagation, and the weights start to learn. Your entire education is all about encoding and decoding!
0:15

In my class, I teach the autoencoder by asking everyone to stand up. 🙆 Stretch your arms out wide. Imagine you are holding a heavy textbook (like Introduction to Algorithms by Prof. Cormen), the whole thing, every page. Now bring your hands slowly together until they almost touch your neck. The bottle "neck." The final exam is tomorrow and you are allowed one cheat sheet: whatever you can scribble on your palm. All nine hundred pages have to survive the squeeze. That is the encoder. Now imagine you sit in the exam. Push your arms back out to where they started. You try to rebuild the textbook from your palm notes. That is the decoder. Of course you cannot get every page back. What you get back is what mattered enough to write down, and the gap between the two is the loss the network is trying to shrink. I call it AI by Arms 🙆. It gets a laugh, and then it gets remembered. Goal: squeeze four numbers down to two, then rebuild the original four from them. 1. Given Let us start with four training examples: X1, X2, X3, X4. 2. Auto (copy to targets) We copy the training examples straight into the targets. That is the whole trick behind the name: "auto" is Greek for "self", and the data is its own label. 3. Encoder, layer 1 Let us multiply the inputs by the weights, add the biases, and apply ReLU. Negative values get crossed out and become zero. 4. Encoder, layer 2 (the bottleneck) We do it again, and now the four dimensions have become two. This layer is called the bottleneck, because everything has to fit through it. 5. Decoder, layer 1 Let us go back the other way: multiply, add, ReLU. This time there are no negatives to cross out. 6. Decoder, layer 2 We multiply once more and get the outputs Y. This is the decoder's attempt to rebuild the four original numbers from the two it was given. 7. MSE loss gradients Let us compare Y with the targets Y'. The gradient is 2 x (Y - Y'): subtract, then double. Those gradients kick off backpropagation, and the weights start to learn. Your entire education is all about encoding and decoding!

Tom Yeh

22,703 views • 1 month ago

iAGENT NODECITY: Season 1 Farming Launches in 24 Hours Step into NodeCity, where the glow of neon dances upon the slick streets and towering skyscrapers cast shadows against the skyline. So, what exactly is NodeCity? It's the dynamic dashboard of iAGENT, your portal to connect with your Agents, dive into vibrant discussions, and reap the rewards. These rewards come in the form of: - early access to Genesis Node WL. - a slice of the $AGNT airdrop. In NodeCity, points are the currency of choice, earned through two thrilling avenues: - Node Credit (NC): Share your invite link. - Node Points (NP): Conquer daily challenges. These challenges aren't your ordinary tasks. Picture them as verbal duels, each day presenting a fresh opportunity for witty exchanges. And who are the players? Four distinct characters, each offering five days of engaging interactions. Earn the most points in Week 1 due to Multiplier Here's the kicker: The first week starts off gently, with few simple prompts—share, like, RT and follow. But it's also the most lucrative, offering early bird advantages and a bountiful points harvest. The NodeCity Farming Season 1 will grace us with its presence for just over three weeks from its launch today. Even if you arrive a little late (5 days in), fear not. There still will be time to immerse yourself in the NodeCity experience and seize its riches! Coz at iAgent, we intend to “Make Farming Fun Again.” 🟢
0:10

iAGENT NODECITY: Season 1 Farming Launches in 24 Hours Step into NodeCity, where the glow of neon dances upon the slick streets and towering skyscrapers cast shadows against the skyline. So, what exactly is NodeCity? It's the dynamic dashboard of iAGENT, your portal to connect with your Agents, dive into vibrant discussions, and reap the rewards. These rewards come in the form of: - early access to Genesis Node WL. - a slice of the $AGNT airdrop. In NodeCity, points are the currency of choice, earned through two thrilling avenues: - Node Credit (NC): Share your invite link. - Node Points (NP): Conquer daily challenges. These challenges aren't your ordinary tasks. Picture them as verbal duels, each day presenting a fresh opportunity for witty exchanges. And who are the players? Four distinct characters, each offering five days of engaging interactions. Earn the most points in Week 1 due to Multiplier Here's the kicker: The first week starts off gently, with few simple prompts—share, like, RT and follow. But it's also the most lucrative, offering early bird advantages and a bountiful points harvest. The NodeCity Farming Season 1 will grace us with its presence for just over three weeks from its launch today. Even if you arrive a little late (5 days in), fear not. There still will be time to immerse yourself in the NodeCity experience and seize its riches! Coz at iAgent, we intend to “Make Farming Fun Again.” 🟢

iAgent

30,890 views • 2 years ago

Destra Network| True decentralized Storage solution How Destra Decentralized File Storage solves the centralization issues in Filecoin & IPFS Filecoin and IPFS have become synonymous with decentralized file storage, highly hyped alternatives to traditional centralized data storage systems. However, a deep dive into their architecture reveals a crucially centralized component: both networks rely on a set of centralized bootstrap nodes. Whenever a new node wants to join the Filecoin or IPFS network, it must first communicate with these bootstrap nodes to obtain information about other peers, placing complete trust in the provided information. The Filecoin team operates these bootstrap nodes on centralized cloud servers. These nodes serve as the initial points of contact for new nodes entering the network, acting as directories to facilitate the network's mesh topology, and thus are critical components of the architecture. Major Centralization Concerns: 1. Surveillance: Centralized nodes offer a concentrated point for surveillance. Authorities or malicious actors could target these nodes to gather data on user activities, node interactions, and network dynamics, posing significant privacy risks. 2. Central Control and Censorship: The operators of bootstrap nodes potentially wield significant control over the network. They can influence which parts of the network are more discoverable or enforce certain network rules, leading to a form of gatekeeping. This centralized control contrasts with the decentralization ethos of blockchain and peer-to-peer technologies. 3. Single Points of Failure: Bootstrap nodes act as central hubs through which traffic and connections are often routed, especially during the initial connection phase. This setup can create single points of failure. If these nodes are compromised, go offline, or experience technical issues, new nodes may struggle to integrate into the network, potentially taking down the whole network. At Destra, we are building the Destra Decentralized File Storage to ensure that such centralization or pseudo-decentralization has no place in our architecture. As always, we are committed to our vision of 100% decentralization, delivering an unprecedented level of decentralization to our users and the broader ecosystem. As we continue developing the Destra Decentralized File Storage, we will keep on publishing regular updates to our community.
0:10

Destra Network| True decentralized Storage solution How Destra Decentralized File Storage solves the centralization issues in Filecoin & IPFS Filecoin and IPFS have become synonymous with decentralized file storage, highly hyped alternatives to traditional centralized data storage systems. However, a deep dive into their architecture reveals a crucially centralized component: both networks rely on a set of centralized bootstrap nodes. Whenever a new node wants to join the Filecoin or IPFS network, it must first communicate with these bootstrap nodes to obtain information about other peers, placing complete trust in the provided information. The Filecoin team operates these bootstrap nodes on centralized cloud servers. These nodes serve as the initial points of contact for new nodes entering the network, acting as directories to facilitate the network's mesh topology, and thus are critical components of the architecture. Major Centralization Concerns: 1. Surveillance: Centralized nodes offer a concentrated point for surveillance. Authorities or malicious actors could target these nodes to gather data on user activities, node interactions, and network dynamics, posing significant privacy risks. 2. Central Control and Censorship: The operators of bootstrap nodes potentially wield significant control over the network. They can influence which parts of the network are more discoverable or enforce certain network rules, leading to a form of gatekeeping. This centralized control contrasts with the decentralization ethos of blockchain and peer-to-peer technologies. 3. Single Points of Failure: Bootstrap nodes act as central hubs through which traffic and connections are often routed, especially during the initial connection phase. This setup can create single points of failure. If these nodes are compromised, go offline, or experience technical issues, new nodes may struggle to integrate into the network, potentially taking down the whole network. At Destra, we are building the Destra Decentralized File Storage to ensure that such centralization or pseudo-decentralization has no place in our architecture. As always, we are committed to our vision of 100% decentralization, delivering an unprecedented level of decentralization to our users and the broader ecosystem. As we continue developing the Destra Decentralized File Storage, we will keep on publishing regular updates to our community.

Destra Network

10,985 views • 2 years ago