Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

ReLU vs Leaky ReLU 👉 = ReLU = ReLU is the default activation in modern deep learning — cheap to compute, and stable enough to train networks hundreds of layers deep. To see what it does, picture five boba tea shops on the same block — 𝚊, 𝚋, 𝚌,... 𝚍, 𝚎 — each running their own books. Each value is a shop's monthly profit — receipts minus rent, ingredients, and wages. When profit is positive, the shop stays open and the owner pockets every dollar. When profit turns negative, the shop runs out of cash and shutters — the lights go off, the books are wiped to zero. ReLU is exactly that rule, applied one shop at a time. Read the diagram left to right. The first column is the raw value x — each shop's profit at month's end. The second column is the gate: 1 if the shop is open (x > 0), 0 if it has shuttered. The last column is the ReLU output: open shops pass their profit through untouched, while shuttered ones are zeroed out. Five rows means five parallel shops on the same block, each evaluated independently. That's why ReLU is called an element-wise activation: every neuron decides its own fate. = LeakyRelu = Plain ReLU wipes negative values to zero — clean, but a shop that shutters can never recover, since both its output and its gradient stay pinned at zero. This is the dying ReLU problem, and in deep networks it can quietly kill a meaningful fraction of the units. Leaky ReLU is the one-line fix: instead of shuttering, the shop files for Chapter 11 protection and keeps the lights on at reduced capacity. Its debt is restructured down to a fraction α (typically 0.1) — the rest is forgiven, and the shop is wounded, not killed. A small negative signal still flows through, so the gradient survives, and the shop can crawl back to life if a TikTok goes viral. Read the diagram left to right. The first column is the raw value x — each shop's profit at month's end. The second column is the leakage α — the fraction of the loss held over after restructuring (default 0.1, editable). The third column is the gate: 1 for shops still in the black, α for those operating under bankruptcy protection. The last column is the Leaky ReLU output: y = x · gate. Profitable shops pass through untouched; struggling ones shrink by a factor of α but still carry a sign. Five rows means five parallel shops, each evaluated independently. Like ReLU, this is an element-wise activation: every neuron's fate is decided on its own merits. #aibyhahdshow more

Tom Yeh

64,669 subscribers

32,561 Aufrufe • vor 4 Monaten •via X (Twitter)

Bildung Wissenschaft & Technologie

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

$Softmax vs Sigmoid ✍️ Interact 👉 = Softmax = Softmax is how deep networks turn raw scores into a probability distribution — the final layer of every classifier, and the core of every attention head in a transformer. To see what it does, picture five boba tea shops on the same block, all competing for your dollar. Five candidates: a, b, c, d, e — different chains, different brewing styles, different pearls. A boba reviewer hands you a 𝘤𝘩𝘦𝘸𝘪𝘯𝘦𝘴𝘴 𝘴𝘤𝘰𝘳𝘦 for each — higher means perfectly chewy "QQ" pearls with the right bite (ask a Taiwanese friend to find out what QQ means). Negative scores are real: mushy bobas, overcooked pearls, a batch left sitting too long. How do you turn five chewiness scores into an allocation that adds to a whole dollar? You could spend everything at the chewiest shop, but that ignores how good the runners-up are. Softmax is the smooth alternative. Read the diagram left to right. First, raise each score to e^{x} — this does two things: it turns negative chewiness into small positives, and it stretches the gaps between scores exponentially. Then sum all five into a single total Z. Finally, divide each e^{x} by Z to get a probability. The five probabilities add up to one, so you can read them as percentages of your dollar. The chewiest shop gets the biggest slice — but never the whole dollar. That's the point of softmax: it ranks confidently while still leaving room for the others. = Sigmoid = Sigmoid squashes any real number into a probability between 0 and 1 — the classic activation for binary classification, and still the gating function inside LSTMs and GRUs. Same boba block as the previous Softmax example, narrowed to just two contenders — a hot new shop `a` with chewiness score x, and your usual go-to `b` whose score is pinned at zero (the neutral baseline you've come to expect). Sigmoid is just softmax with two players, one of them pinned to zero. Read the diagram left to right. First, raise each score to e^{x} — for the usual shop `b` whose score is zero, this is just e^0 = 1 (the constant baseline). Then sum the two into a total Z. Finally, divide each e^{x} by Z to get a probability. The two probabilities add up to one — the new shop wins more of your dollar when its pearls get chewier, and your usual keeps the rest. That's the point of sigmoid: it turns a single chewiness score into a clean 0-to-1 chance you'll try the new place over your usual. --- AI Math, Algorithms, Architectures by hand ✍️ Subscribe to my 60K+ reader newsletter 👉$

Softmax vs Sigmoid ✍️ Interact 👉 = Softmax = Softmax is how deep networks turn raw scores into a probability distribution — the final layer of every classifier, and the core of every attention head in a transformer. To see what it does, picture five boba tea shops on the same block, all competing for your dollar. Five candidates: a, b, c, d, e — different chains, different brewing styles, different pearls. A boba reviewer hands you a 𝘤𝘩𝘦𝘸𝘪𝘯𝘦𝘴𝘴 𝘴𝘤𝘰𝘳𝘦 for each — higher means perfectly chewy "QQ" pearls with the right bite (ask a Taiwanese friend to find out what QQ means). Negative scores are real: mushy bobas, overcooked pearls, a batch left sitting too long. How do you turn five chewiness scores into an allocation that adds to a whole dollar? You could spend everything at the chewiest shop, but that ignores how good the runners-up are. Softmax is the smooth alternative. Read the diagram left to right. First, raise each score to e^{x} — this does two things: it turns negative chewiness into small positives, and it stretches the gaps between scores exponentially. Then sum all five into a single total Z. Finally, divide each e^{x} by Z to get a probability. The five probabilities add up to one, so you can read them as percentages of your dollar. The chewiest shop gets the biggest slice — but never the whole dollar. That's the point of softmax: it ranks confidently while still leaving room for the others. = Sigmoid = Sigmoid squashes any real number into a probability between 0 and 1 — the classic activation for binary classification, and still the gating function inside LSTMs and GRUs. Same boba block as the previous Softmax example, narrowed to just two contenders — a hot new shop `a` with chewiness score x, and your usual go-to `b` whose score is pinned at zero (the neutral baseline you've come to expect). Sigmoid is just softmax with two players, one of them pinned to zero. Read the diagram left to right. First, raise each score to e^{x} — for the usual shop `b` whose score is zero, this is just e^0 = 1 (the constant baseline). Then sum the two into a total Z. Finally, divide each e^{x} by Z to get a probability. The two probabilities add up to one — the new shop wins more of your dollar when its pearls get chewier, and your usual keeps the rest. That's the point of sigmoid: it turns a single chewiness score into a clean 0-to-1 chance you'll try the new place over your usual. --- AI Math, Algorithms, Architectures by hand ✍️ Subscribe to my 60K+ reader newsletter 👉

Tom Yeh

74,410 Aufrufe • vor 4 Monaten

MLP in PyTorch by hand ✍️ ~ 7 steps walkthrough below Goal: fill in every blank in the PyTorch code to build a multi-layer perceptron. 1. Given Let us start with a code template on the left and the network it is supposed to build on the right. Every blank in the code can be worked out from the picture. 2. Linear layer We count: 3 features in, 4 features out. So the weight matrix is 4 by 3. There is an extra column for the biases, which means bias = T. 3. ReLU Let us apply the activation. ReLU crosses out the negatives, so -1 becomes 0. 4. Linear layer The input size is 4, because that is what the previous layer put out. The output size is 2. A 2 by 4 weight matrix, and this time no extra column, so bias = F. 5. ReLU We cross out the negatives again. 6. Linear layer Two features in, five out. A 5 by 2 weight matrix, with a bias column, so bias = T. 7. Sigmoid Let us finish. Sigmoid squashes the raw scores (3, 0, -2, 5, -5) into probabilities between 0 and 1. You have just implemented a three-layer deep neural network by hand. ✍️ == Story == Three years ago I gave this exercise to my students, to connect the code to the math. They found it odd. Every other AI course they were taking lived inside a Jupyter notebook, and here I was handing out paper. Three years later, my colleagues are the ones rushing to move their materials to paper. The exercise has not changed. Paper still asks the one thing a notebook lets you skip: do you actually understand what the code is doing? If you can tell me why the weight matrix is 4 by 3, and why bias is F on the second layer, you understand nn.Linear better than someone who has been copy-pasting it for a year. 💾 Save this post! #AIbyHand #PyTorch #DeepLearning

MLP in PyTorch by hand ✍️ ~ 7 steps walkthrough below Goal: fill in every blank in the PyTorch code to build a multi-layer perceptron. 1. Given Let us start with a code template on the left and the network it is supposed to build on the right. Every blank in the code can be worked out from the picture. 2. Linear layer We count: 3 features in, 4 features out. So the weight matrix is 4 by 3. There is an extra column for the biases, which means bias = T. 3. ReLU Let us apply the activation. ReLU crosses out the negatives, so -1 becomes 0. 4. Linear layer The input size is 4, because that is what the previous layer put out. The output size is 2. A 2 by 4 weight matrix, and this time no extra column, so bias = F. 5. ReLU We cross out the negatives again. 6. Linear layer Two features in, five out. A 5 by 2 weight matrix, with a bias column, so bias = T. 7. Sigmoid Let us finish. Sigmoid squashes the raw scores (3, 0, -2, 5, -5) into probabilities between 0 and 1. You have just implemented a three-layer deep neural network by hand. ✍️ == Story == Three years ago I gave this exercise to my students, to connect the code to the math. They found it odd. Every other AI course they were taking lived inside a Jupyter notebook, and here I was handing out paper. Three years later, my colleagues are the ones rushing to move their materials to paper. The exercise has not changed. Paper still asks the one thing a notebook lets you skip: do you actually understand what the code is doing? If you can tell me why the weight matrix is 4 by 3, and why bias is F on the second layer, you understand nn.Linear better than someone who has been copy-pasting it for a year. 💾 Save this post! #AIbyHand #PyTorch #DeepLearning

Tom Yeh

13,318 Aufrufe • vor 1 Monat

[VAE] by Hand ✍️ A Variational Auto Encoder (VAE) learns the structure (mean and variance) of hidden features and generates new data from the learned structure. In contrast, GANs only learn to generate new data to fool a discriminator; they may not necessarily know the underlying structure of the data. The International Conference on Learning Representations (ICLR) this year announced its first ever "Test of Time Award" to recognizes the VAE paper, published 10 years ago. This exercise demonstrates how to calculate a VAE by hand. [1] Given: ↳ Three training examples X1, X2, X3 ↳ Copy training examples to the bottom ↳ The purpose is to train the network to reconstruct the training examples. ↳ Since each target is a training example itself, we use the Greek word "auto" which means "self." This crucial step is what makes an autoencoder "auto." [2] Encoder: Layer 1 + ReLU ↳ Multiply inputs with weights and biases ↳ Apply ReLU, crossing out negative values (-1 -> 0) [3] Encoder: Mean and Variance ↳ Multiply features with two sets of weights and biases ↳ 🟩 The first set predicts the means (𝜇) of latent distributions ↳ 🟪 The second set predicts the standard deviation (𝜎) of latent distributions [4] Reparameterization Trick: Random Offset ↳ Sample epsilon ε from the normal distribution with mean = 0 and variance = 1. ↳ The purpose is to randomly pick a offset away from the mean. ↳ Multiply the standard deviation values with epsilon values. ↳ The purpose is to scale the offset by the standard deviation. [5] Reparameterization Trick: Mean + Offset ↳ Add the sampled offset to predicted mean ↳ The result are new parameters or features 🟨 as inputs to the Decoder. [6] Decoder: Layer 1 + ReLU ↳ Multiply input features with weights and biases ↳ Apply ReLU, crossing out negative values. Here, -4 is crossed out. [7] Decoder: Layer 2 ↳ Multiply features with weights and biases ↳ The output is Decoder's attempt to reconstruct the input data X from reparameterized distributions described by 𝜇 and 𝜎. [8]-[10] KL Divergence Loss [8] Loss Gradient: Mean 𝜇 ↳ We want 𝜇 to approach 0. ↳ A lot of math called SGVB simplifies the calculation of loss gradients to simply 𝜇 [9,10] Loss Gradient: Stdev 𝜎 ↳ We want 𝜎 to approach 1. ↳ A lot of math simplifies the calculation to 𝜎 - (1/ 𝜎) [11] Reconstruction Loss ↳ We want the reconstructed data Y (dark 🟧) to be the same as the input data X. ↳ Some math involving Mean Square Error simplifies the calculation to Y - X.

[VAE] by Hand ✍️ A Variational Auto Encoder (VAE) learns the structure (mean and variance) of hidden features and generates new data from the learned structure. In contrast, GANs only learn to generate new data to fool a discriminator; they may not necessarily know the underlying structure of the data. The International Conference on Learning Representations (ICLR) this year announced its first ever "Test of Time Award" to recognizes the VAE paper, published 10 years ago. This exercise demonstrates how to calculate a VAE by hand. [1] Given: ↳ Three training examples X1, X2, X3 ↳ Copy training examples to the bottom ↳ The purpose is to train the network to reconstruct the training examples. ↳ Since each target is a training example itself, we use the Greek word "auto" which means "self." This crucial step is what makes an autoencoder "auto." [2] Encoder: Layer 1 + ReLU ↳ Multiply inputs with weights and biases ↳ Apply ReLU, crossing out negative values (-1 -> 0) [3] Encoder: Mean and Variance ↳ Multiply features with two sets of weights and biases ↳ 🟩 The first set predicts the means (𝜇) of latent distributions ↳ 🟪 The second set predicts the standard deviation (𝜎) of latent distributions [4] Reparameterization Trick: Random Offset ↳ Sample epsilon ε from the normal distribution with mean = 0 and variance = 1. ↳ The purpose is to randomly pick a offset away from the mean. ↳ Multiply the standard deviation values with epsilon values. ↳ The purpose is to scale the offset by the standard deviation. [5] Reparameterization Trick: Mean + Offset ↳ Add the sampled offset to predicted mean ↳ The result are new parameters or features 🟨 as inputs to the Decoder. [6] Decoder: Layer 1 + ReLU ↳ Multiply input features with weights and biases ↳ Apply ReLU, crossing out negative values. Here, -4 is crossed out. [7] Decoder: Layer 2 ↳ Multiply features with weights and biases ↳ The output is Decoder's attempt to reconstruct the input data X from reparameterized distributions described by 𝜇 and 𝜎. [8]-[10] KL Divergence Loss [8] Loss Gradient: Mean 𝜇 ↳ We want 𝜇 to approach 0. ↳ A lot of math called SGVB simplifies the calculation of loss gradients to simply 𝜇 [9,10] Loss Gradient: Stdev 𝜎 ↳ We want 𝜎 to approach 1. ↳ A lot of math simplifies the calculation to 𝜎 - (1/ 𝜎) [11] Reconstruction Loss ↳ We want the reconstructed data Y (dark 🟧) to be the same as the input data X. ↳ Some math involving Mean Square Error simplifies the calculation to Y - X.

Tom Yeh

48,475 Aufrufe • vor 2 Jahren

Transformer by hand ✍️ ~ 6 steps walkthrough below Open the hood of a transformer and the parts list is overwhelming: embeddings, positional encoding, attention weighting, self-attention, cross-attention, multi-head attention, layer norm, skip connections, softmax, linear, Nx, shifted right, query, key, value, masking. Which of those actually make the car run? Two of them. Attention weighting and the feed-forward network. Everything else is an enhancement to make it run faster and longer, which is how we got from a car to a truck, and to the word "large" in large language model. So I drew and calculated those two parts entirely by hand. Goal: push five features through one transformer block, filling in every cell yourself. 1. Given Five positions of input features, arriving from the previous block. 2. Attention matrix Let us feed all five features to a query-key module (QK) and read back an attention weight matrix, A. The details of that module are a post of their own. 3. Attention weighting We multiply the input features by A to get the attention weighted features, Z. Still five positions. The effect is to combine features *across positions*, horizontally: X1 becomes X1 + X2, X2 becomes X2 + X3, and so on. 4. First layer Let us feed all five weighted features into the first layer of the FFN. Multiply by the weights and biases. This time the combining happens *across feature dimensions*, vertically, and each feature grows from 3 numbers to 4. Note that every position goes through the same weight matrix. That is what "position-wise" means. 5. ReLU We cross out the negatives. They become zeros. 6. Second layer Let us bring it back down: 4 dimensions to 3. The output feeds the next block, which has a completely separate set of parameters, and the whole thing runs again. You have just calculated a transformer block by hand. ✍️ The takeaway: the two parts are doing two different jobs, and neither one alone is enough. Attention mixes *across positions*, so a feature can see its neighbours. The FFN mixes *across feature dimensions*, so each position can think about itself. Horizontal, then vertical. Then that pattern repeats N times, each block with its own separate set of weights. That is the Nx from the list up top, and that is what makes the transformer run. 💾 Save this post! #AIbyHand #Transformers #DeepLearning

Transformer by hand ✍️ ~ 6 steps walkthrough below Open the hood of a transformer and the parts list is overwhelming: embeddings, positional encoding, attention weighting, self-attention, cross-attention, multi-head attention, layer norm, skip connections, softmax, linear, Nx, shifted right, query, key, value, masking. Which of those actually make the car run? Two of them. Attention weighting and the feed-forward network. Everything else is an enhancement to make it run faster and longer, which is how we got from a car to a truck, and to the word "large" in large language model. So I drew and calculated those two parts entirely by hand. Goal: push five features through one transformer block, filling in every cell yourself. 1. Given Five positions of input features, arriving from the previous block. 2. Attention matrix Let us feed all five features to a query-key module (QK) and read back an attention weight matrix, A. The details of that module are a post of their own. 3. Attention weighting We multiply the input features by A to get the attention weighted features, Z. Still five positions. The effect is to combine features across positions, horizontally: X1 becomes X1 + X2, X2 becomes X2 + X3, and so on. 4. First layer Let us feed all five weighted features into the first layer of the FFN. Multiply by the weights and biases. This time the combining happens across feature dimensions, vertically, and each feature grows from 3 numbers to 4. Note that every position goes through the same weight matrix. That is what "position-wise" means. 5. ReLU We cross out the negatives. They become zeros. 6. Second layer Let us bring it back down: 4 dimensions to 3. The output feeds the next block, which has a completely separate set of parameters, and the whole thing runs again. You have just calculated a transformer block by hand. ✍️ The takeaway: the two parts are doing two different jobs, and neither one alone is enough. Attention mixes across positions, so a feature can see its neighbours. The FFN mixes across feature dimensions, so each position can think about itself. Horizontal, then vertical. Then that pattern repeats N times, each block with its own separate set of weights. That is the Nx from the list up top, and that is what makes the transformer run. 💾 Save this post! #AIbyHand #Transformers #DeepLearning

Tom Yeh

26,089 Aufrufe • vor 1 Monat

[Graph Convolutional Network] by hand ✍️ Graph Convolutional Networks (GCNs), introduced by Thomas Kipf and Max Welling in 2017, have emerged as a powerful tool in the analysis and interpretation of data structured as graphs. This exercise demonstrates how GCN works in a simple application: binary classification. -- Goal -- Predict if a node in a graph is X. -- Architecture -- 🟪 Graph Convolutional Network (GCN) 1. GCN1(4,3) 2. GCN2(3,3) 🟦 Fully Connected Network (FCN) 1. Linear1(3,5) 2. ReLU 3. Linear2(5,1) 4. Sigmoid Simplications: • Adjacent matrices are not normalized. • ReLU is applied to messages directly. -- Walkthrough -- [1] Given ↳ A graph with five nodes A, B, C, D, E [2] 🟩 Adjacency Matrix: Neighbors ↳ Add 1 for each edge to neighbors ↳ Repeat in both directions (e.g., A->C, C->A) ↳ Repeat for both GCN layers [3] 🟩 Adjacency Matrix: Self ↳ Add 1's for each self loop ↳ Equivalent to adding the identity matrix ↳ Repeat for both GCN layers [4] 🟪 GCN1: Messages ↳ Multiply the node embeddings 🟨 with weights and biases ↳ Apply ReLU (negatives → 0) ↳ The result is one message per node [5] 🟪 GCN1: Pooling ↳ Multiply the messages with the adjacent matrix ↳ The purpose is the pool messages from each node's neighbors as well as from the node itself. ↳ The result is a new feature per node [6] 🟪 GCN1: Visualize ↳ For node 1, visualize how messages are pooled to obtain a new feature for better understanding ↳ [3,0,1] + [1,0,0] = [4,0,1] [7] 🟪 GCN2: Messages ↳ Multiply the node features with weights and biases ↳ Apply ReLU (negatives → 0) ↳ The result is one message per node [8] 🟪 GCN2: Pooling ↳ Multiply the messages with the adjacent matrix ↳ The result is a new feature per node [9] 🟪 GCN2: Visualize ↳ For node 3, visualize how messages are pooled to obtain a new feature for better understanding ↳ [1,2,4] + [1,3,5] + [0,0,1] = [2,5,10] [10] 🟦 FCN: Linear 1 + ReLU ↳ Multiply node features with weights and biases ↳ Apply ReLU (negatives → 0) ↳ The result is a new feature per node ↳ Unlike in GCN layers, no messages from other nodes are included. [11] 🟦 FCN: Linear 2 ↳ Multiply node features with weights and biases [12] 🟦 FCN: Sigmoid ↳ Apply the Sigmoid activation function ↳ The purpose is to obtain a probability value for each node ↳ One way to calculate Sigmoid by hand ✍️ is to use the approximation below: • >= 3 → 1 • 0 → 0.5 • <= -3 → 0 -- Outputs -- A: 0 (Very unlikely) B: 1 (Very likely) C: 1 (Very likely) D: 1 (Very likely) E: 0.5 (Neutral)

[Graph Convolutional Network] by hand ✍️ Graph Convolutional Networks (GCNs), introduced by Thomas Kipf and Max Welling in 2017, have emerged as a powerful tool in the analysis and interpretation of data structured as graphs. This exercise demonstrates how GCN works in a simple application: binary classification. -- Goal -- Predict if a node in a graph is X. -- Architecture -- 🟪 Graph Convolutional Network (GCN) 1. GCN1(4,3) 2. GCN2(3,3) 🟦 Fully Connected Network (FCN) 1. Linear1(3,5) 2. ReLU 3. Linear2(5,1) 4. Sigmoid Simplications: • Adjacent matrices are not normalized. • ReLU is applied to messages directly. -- Walkthrough -- [1] Given ↳ A graph with five nodes A, B, C, D, E [2] 🟩 Adjacency Matrix: Neighbors ↳ Add 1 for each edge to neighbors ↳ Repeat in both directions (e.g., A->C, C->A) ↳ Repeat for both GCN layers [3] 🟩 Adjacency Matrix: Self ↳ Add 1's for each self loop ↳ Equivalent to adding the identity matrix ↳ Repeat for both GCN layers [4] 🟪 GCN1: Messages ↳ Multiply the node embeddings 🟨 with weights and biases ↳ Apply ReLU (negatives → 0) ↳ The result is one message per node [5] 🟪 GCN1: Pooling ↳ Multiply the messages with the adjacent matrix ↳ The purpose is the pool messages from each node's neighbors as well as from the node itself. ↳ The result is a new feature per node [6] 🟪 GCN1: Visualize ↳ For node 1, visualize how messages are pooled to obtain a new feature for better understanding ↳ [3,0,1] + [1,0,0] = [4,0,1] [7] 🟪 GCN2: Messages ↳ Multiply the node features with weights and biases ↳ Apply ReLU (negatives → 0) ↳ The result is one message per node [8] 🟪 GCN2: Pooling ↳ Multiply the messages with the adjacent matrix ↳ The result is a new feature per node [9] 🟪 GCN2: Visualize ↳ For node 3, visualize how messages are pooled to obtain a new feature for better understanding ↳ [1,2,4] + [1,3,5] + [0,0,1] = [2,5,10] [10] 🟦 FCN: Linear 1 + ReLU ↳ Multiply node features with weights and biases ↳ Apply ReLU (negatives → 0) ↳ The result is a new feature per node ↳ Unlike in GCN layers, no messages from other nodes are included. [11] 🟦 FCN: Linear 2 ↳ Multiply node features with weights and biases [12] 🟦 FCN: Sigmoid ↳ Apply the Sigmoid activation function ↳ The purpose is to obtain a probability value for each node ↳ One way to calculate Sigmoid by hand ✍️ is to use the approximation below: • >= 3 → 1 • 0 → 0.5 • <= -3 → 0 -- Outputs -- A: 0 (Very unlikely) B: 1 (Very likely) C: 1 (Very likely) D: 1 (Very likely) E: 0.5 (Neutral)

Tom Yeh

46,779 Aufrufe • vor 2 Jahren

Batch Normalization by hand ✍️ ~ 7 steps walkthrough below Batch normalization is common practice for improving training and achieving faster convergence. It sounds simple. But it is often misunderstood. 🤔 Does batch normalization involve trainable parameters, tunable hyper-parameters, or both? 🤔 Is batch normalization applied to inputs, features, weights, biases, or outputs? 🤔 How is batch normalization different from layer normalization? So I drew and calculated one entirely by hand. Goal: normalize a mini-batch of 4 examples to mean 0 and variance 1, then let the network scale it back. = 1. Given = A mini-batch of 4 training examples, each with 3 features. = 2. Linear layer = Let us multiply by the weights and add the biases. Batch norm sits after this, which answers the second question: what gets normalized is features, not inputs, weights or biases. = 3. ReLU = We apply the activation, and -2 becomes 0. Negative values are suppressed before any statistic is taken. = 4. Batch statistics = Let us compute the sum, mean, variance and standard deviation, one row at a time. A row is a feature and the four columns are the four examples, so every number here measures one feature against the rest of the batch. That is the "batch" in batch normalization, and it is exactly what layer normalization does not do. The statistics are rounded to whole numbers, which is what keeps the rest of the page doable in pen. = 5. Shift to mean 0 = We subtract the mean, in green. The four values in each feature now average to zero. = 6. Scale to variance 1 = Let us divide by the standard deviation, in orange. Each feature now has variance one, whatever scale it arrived at. = 7. Scale and shift = We multiply by a linear transformation and pass the result on. The diagonal and the last column are trainable, so having just forced every feature to mean 0 and variance 1, we hand the network the means to undo it. The outputs: Mean of each feature = [2, 1, 2] Std dev of each feature = [1, 1, 2] To the next layer = [2, -2, 2, 0], [-3, 3, 6, -3], [2, 0, 1, 2] The answers: 🤔 Both. The scale and shift are trainable, the statistics are not. Epsilon and the momentum on the running statistics are the hyper-parameters, and one mini-batch by hand needs neither. 🤔 Features, after the linear layer, not inputs, weights or biases. 🤔 Batch norm measures across the batch, one feature at a time. Layer norm measures across the features, one example at a time. 💾 Save this post!

Batch Normalization by hand ✍️ ~ 7 steps walkthrough below Batch normalization is common practice for improving training and achieving faster convergence. It sounds simple. But it is often misunderstood. 🤔 Does batch normalization involve trainable parameters, tunable hyper-parameters, or both? 🤔 Is batch normalization applied to inputs, features, weights, biases, or outputs? 🤔 How is batch normalization different from layer normalization? So I drew and calculated one entirely by hand. Goal: normalize a mini-batch of 4 examples to mean 0 and variance 1, then let the network scale it back. = 1. Given = A mini-batch of 4 training examples, each with 3 features. = 2. Linear layer = Let us multiply by the weights and add the biases. Batch norm sits after this, which answers the second question: what gets normalized is features, not inputs, weights or biases. = 3. ReLU = We apply the activation, and -2 becomes 0. Negative values are suppressed before any statistic is taken. = 4. Batch statistics = Let us compute the sum, mean, variance and standard deviation, one row at a time. A row is a feature and the four columns are the four examples, so every number here measures one feature against the rest of the batch. That is the "batch" in batch normalization, and it is exactly what layer normalization does not do. The statistics are rounded to whole numbers, which is what keeps the rest of the page doable in pen. = 5. Shift to mean 0 = We subtract the mean, in green. The four values in each feature now average to zero. = 6. Scale to variance 1 = Let us divide by the standard deviation, in orange. Each feature now has variance one, whatever scale it arrived at. = 7. Scale and shift = We multiply by a linear transformation and pass the result on. The diagonal and the last column are trainable, so having just forced every feature to mean 0 and variance 1, we hand the network the means to undo it. The outputs: Mean of each feature = [2, 1, 2] Std dev of each feature = [1, 1, 2] To the next layer = [2, -2, 2, 0], [-3, 3, 6, -3], [2, 0, 1, 2] The answers: 🤔 Both. The scale and shift are trainable, the statistics are not. Epsilon and the momentum on the running statistics are the hyper-parameters, and one mini-batch by hand needs neither. 🤔 Features, after the linear layer, not inputs, weights or biases. 🤔 Batch norm measures across the batch, one feature at a time. Layer norm measures across the features, one example at a time. 💾 Save this post!

Tom Yeh

20,848 Aufrufe • vor 1 Monat

[Discrete Fourier Transform] by Hand ✍️ In signal processing, the Discrete Fourier Transform (DFT) is no doubt the most important method. But the math involved is extremely complex, literally, involving a summation over a complex number term e^(-iwt). I developed this exercise to demonstrate that underneath such complexity, DFT is just a series of matrix multiplications you can calculate by hand. ✍️ Once you see that, it should not surprise you that a deep neural network, which is also a series of matrix multiplications, with activation functions in-between, can learn to perform DFT to process and analyze signals so effectively. How does DFT work? [1] Given ↳ Signals A, B, and C in the 🟧 frequency domain: ◦ A = cos(w) + 2cos(2w) ◦ B = cos(w) + cos(3w) + cos(4w) ◦ C = -cos(2w) + cos(3w) ◦ Each signal is a weighed sum of four cosine waves at frequencies 1w, 2w, 3w, and 4w. ◦ We will apply Inverse DFT to convert the signals to time domain representations, and then demonstrate DFT can convert back to their original frequency domain representations. ↳ Signal X in the 🟩 time domain. X is sampled at 10 time points 1t, 2t, …, 10t: ◦ X = [-2.5, -1.8, 3, -0.7, -1.0, -0.7, 3, -1.8, -2.5, 5] ◦ Suppose X is also a weighted sum of the same four cosine waves, but we don’t already know their weights. We will apply DFT to discover them. [2] 🟧 Frequency Matrix (F) ↳ Write the coefficients of A, B, C as a matrix F. Each signal is a row. Each frequency is a column. ↳ A → [1, 2, 0, 0] ↳ B → [1, 0, 1, 1] ↳ C → [0, 1-, 1, 0] [3] Cosine → Discrete ↳ Sample from the continuous cosine waves at discrete time points 1t, 2t, 3t, to 10t. [4] Cosine Matrix (W) ↳ Write the samples as a matrix, Each frequency is a row. Each time point is a column. [5] Inverse DFT: 🟧 Frequency → 🟩 Time ↳ Multiply the frequency matrix F and the cosine matrix W. ↳ The meaning of this multiplication is to linearly combine the four cosine waves (rows in W) into time-domain signals (rows in T) using the weights specified in F. ↳ The result is matrix T, which are signals A, B, C converted to the time domain. Each signal is a row. Each time point is a column. [6] Transpose ↳ Transpose T, converting each signal’s time domain representation from a row to a column. [7] DFT: 🟩 Time → 🟧 Frequency ↳ Multiply the cosine matrix W with the transpose of matrix T. ↳ The purpose of this multiplication is to take a dot-product between each time-domain signal (columns in the transpose of T) and each cosine wave (rows in W), which has the effect of projecting the signal onto a cosine wave to determine how much they are correlated. Zero means not correlated at all. ↳ The result is an intermediate version of the “recovered” frequency matrix where each column corresponds to a signal and each row corresponds to a frequency. ↳ Compared to the original frequency matrix F, this intermediate matrix has non-zero weights in the correct places, but scaled up by a factor of 5 (n/2, n=10). For example, signal A, originally [1,2,0,0], is recovered at [5,10,0,0]. [8] Scale ↳ Multiply each value by 2/n = 1/5 to scale down the intermediate matrix to match the magnitude of the original frequency matrix F. [9] Transpose ↳ Transpose the recovered frequency matrix back to the same orientation of the original frequency matrix F. ↳ Like magic 🪄, the result is identical to the original F, which means DFT successfully recovered the frequency components of signals A, B, C. [10] Apply DFT to X: 🟩 Time → 🟧 Frequency ↳ Now that we have some confidence in DFT’s ability to recover frequency components, we apply DFT to X’s time-domain representation by multiplying W with X. ↳ The result is the an intermediate matrix. [11] Scale ↳ Similarly, we scale down by a factor of 5 to obtain the recovered frequency components of X (a column). [12] Transpose ↳ Similarly, we transpose the recovered column to row to match the orientation of the frequency matrix. ↳ Using the coefficients [0,0,3,2], we can write the equation of X as 3cos(3w) + 2cos(4w). Notes: I hope this by hand exercise helps you understand the essence of DFT. But there is more technical details, such as: • Sine: The complete DFT math also includes sine waves that follow a similar calculation process. • Phase: Here, we assume all the cosine waves are aligned at the origin, namely, phase is 0. If a phase p is added, for example, cos(w+p), we will need to calculate the sine component and use their ratio to figure out what p is. • Magnitude: If phase is not zero, the magnitude will need to be calculated by combining both cosine and sine terms.

[Discrete Fourier Transform] by Hand ✍️ In signal processing, the Discrete Fourier Transform (DFT) is no doubt the most important method. But the math involved is extremely complex, literally, involving a summation over a complex number term e^(-iwt). I developed this exercise to demonstrate that underneath such complexity, DFT is just a series of matrix multiplications you can calculate by hand. ✍️ Once you see that, it should not surprise you that a deep neural network, which is also a series of matrix multiplications, with activation functions in-between, can learn to perform DFT to process and analyze signals so effectively. How does DFT work? [1] Given ↳ Signals A, B, and C in the 🟧 frequency domain: ◦ A = cos(w) + 2cos(2w) ◦ B = cos(w) + cos(3w) + cos(4w) ◦ C = -cos(2w) + cos(3w) ◦ Each signal is a weighed sum of four cosine waves at frequencies 1w, 2w, 3w, and 4w. ◦ We will apply Inverse DFT to convert the signals to time domain representations, and then demonstrate DFT can convert back to their original frequency domain representations. ↳ Signal X in the 🟩 time domain. X is sampled at 10 time points 1t, 2t, …, 10t: ◦ X = [-2.5, -1.8, 3, -0.7, -1.0, -0.7, 3, -1.8, -2.5, 5] ◦ Suppose X is also a weighted sum of the same four cosine waves, but we don’t already know their weights. We will apply DFT to discover them. [2] 🟧 Frequency Matrix (F) ↳ Write the coefficients of A, B, C as a matrix F. Each signal is a row. Each frequency is a column. ↳ A → [1, 2, 0, 0] ↳ B → [1, 0, 1, 1] ↳ C → [0, 1-, 1, 0] [3] Cosine → Discrete ↳ Sample from the continuous cosine waves at discrete time points 1t, 2t, 3t, to 10t. [4] Cosine Matrix (W) ↳ Write the samples as a matrix, Each frequency is a row. Each time point is a column. [5] Inverse DFT: 🟧 Frequency → 🟩 Time ↳ Multiply the frequency matrix F and the cosine matrix W. ↳ The meaning of this multiplication is to linearly combine the four cosine waves (rows in W) into time-domain signals (rows in T) using the weights specified in F. ↳ The result is matrix T, which are signals A, B, C converted to the time domain. Each signal is a row. Each time point is a column. [6] Transpose ↳ Transpose T, converting each signal’s time domain representation from a row to a column. [7] DFT: 🟩 Time → 🟧 Frequency ↳ Multiply the cosine matrix W with the transpose of matrix T. ↳ The purpose of this multiplication is to take a dot-product between each time-domain signal (columns in the transpose of T) and each cosine wave (rows in W), which has the effect of projecting the signal onto a cosine wave to determine how much they are correlated. Zero means not correlated at all. ↳ The result is an intermediate version of the “recovered” frequency matrix where each column corresponds to a signal and each row corresponds to a frequency. ↳ Compared to the original frequency matrix F, this intermediate matrix has non-zero weights in the correct places, but scaled up by a factor of 5 (n/2, n=10). For example, signal A, originally [1,2,0,0], is recovered at [5,10,0,0]. [8] Scale ↳ Multiply each value by 2/n = 1/5 to scale down the intermediate matrix to match the magnitude of the original frequency matrix F. [9] Transpose ↳ Transpose the recovered frequency matrix back to the same orientation of the original frequency matrix F. ↳ Like magic 🪄, the result is identical to the original F, which means DFT successfully recovered the frequency components of signals A, B, C. [10] Apply DFT to X: 🟩 Time → 🟧 Frequency ↳ Now that we have some confidence in DFT’s ability to recover frequency components, we apply DFT to X’s time-domain representation by multiplying W with X. ↳ The result is the an intermediate matrix. [11] Scale ↳ Similarly, we scale down by a factor of 5 to obtain the recovered frequency components of X (a column). [12] Transpose ↳ Similarly, we transpose the recovered column to row to match the orientation of the frequency matrix. ↳ Using the coefficients [0,0,3,2], we can write the equation of X as 3cos(3w) + 2cos(4w). Notes: I hope this by hand exercise helps you understand the essence of DFT. But there is more technical details, such as: • Sine: The complete DFT math also includes sine waves that follow a similar calculation process. • Phase: Here, we assume all the cosine waves are aligned at the origin, namely, phase is 0. If a phase p is added, for example, cos(w+p), we will need to calculate the sine component and use their ratio to figure out what p is. • Magnitude: If phase is not zero, the magnitude will need to be calculated by combining both cosine and sine terms.

Tom Yeh

116,622 Aufrufe • vor 2 Jahren

Full Fine-tuning vs. Freezing Layers. Interact 👉 and == Full Fine-tuning == A real network has many — three layers in this example, billions of parameters in a production model. What does fine-tuning look like when you update all of them? That’s full fine-tuning: continue training every weight in the pretrained network on your new task. Every layer’s W gets its own ΔW. Nothing is frozen — every parameter is in play. Think of an MLP as a chain of prerequisites leading to an advanced course. Layer 1 might be Linear Algebra, layer 2 Probability, layer 3 Advanced Machine Learning — each one building on what came before. Fine-tuning is what happens during graduate study: the foundations are already there from undergrad, so you’re not re-learning. Full fine-tuning is reviewing every prerequisite to see what new topics have appeared and what discoveries the field has made since the last time you sat through them. Effective — but exhausting. This diagram shows the same three-layer MLP twice, side by side. On the left, the pretrained network runs on input X: three weight matrices W₁, W₂, W₃, each followed by a ReLU activation. Full fine-tuning gives the model the most freedom to specialize. Every parameter can move — and every parameter that can move must be stored. But not every prerequisite needs revisiting. The further you go back in the chain, the less the material has changed since pretraining — the linear-algebra basics under your computer-vision course are largely the same as they ever were. The next page does exactly that: freeze the prerequisites that haven’t moved, and only refresh the advanced one closest to your specialization. == Freezing Layers == Full fine-tuning reviewed every prerequisite — Linear Algebra, Probability, Advanced ML — to refresh each subject with the latest topics. Effective, but exhausting. Then you realize something. The prerequisites haven’t actually changed that much. Linear Algebra is still Linear Algebra; the matrix decompositions you learned still hold. Probability is still Probability; the distributions and Bayes’ rule haven’t moved. Almost all the new material — the new ideas, the recent discoveries — lives in the advanced layer at the top. That’s freezing layers: keep the prerequisite layers fixed at their pretrained state, and only update the advanced one. In the diagram below, W1 and W2 — the foundational prerequisites — stay frozen. Only W3 — the layer closest to your task-specific output — gets a ΔW.

Full Fine-tuning vs. Freezing Layers. Interact 👉 and == Full Fine-tuning == A real network has many — three layers in this example, billions of parameters in a production model. What does fine-tuning look like when you update all of them? That’s full fine-tuning: continue training every weight in the pretrained network on your new task. Every layer’s W gets its own ΔW. Nothing is frozen — every parameter is in play. Think of an MLP as a chain of prerequisites leading to an advanced course. Layer 1 might be Linear Algebra, layer 2 Probability, layer 3 Advanced Machine Learning — each one building on what came before. Fine-tuning is what happens during graduate study: the foundations are already there from undergrad, so you’re not re-learning. Full fine-tuning is reviewing every prerequisite to see what new topics have appeared and what discoveries the field has made since the last time you sat through them. Effective — but exhausting. This diagram shows the same three-layer MLP twice, side by side. On the left, the pretrained network runs on input X: three weight matrices W₁, W₂, W₃, each followed by a ReLU activation. Full fine-tuning gives the model the most freedom to specialize. Every parameter can move — and every parameter that can move must be stored. But not every prerequisite needs revisiting. The further you go back in the chain, the less the material has changed since pretraining — the linear-algebra basics under your computer-vision course are largely the same as they ever were. The next page does exactly that: freeze the prerequisites that haven’t moved, and only refresh the advanced one closest to your specialization. == Freezing Layers == Full fine-tuning reviewed every prerequisite — Linear Algebra, Probability, Advanced ML — to refresh each subject with the latest topics. Effective, but exhausting. Then you realize something. The prerequisites haven’t actually changed that much. Linear Algebra is still Linear Algebra; the matrix decompositions you learned still hold. Probability is still Probability; the distributions and Bayes’ rule haven’t moved. Almost all the new material — the new ideas, the recent discoveries — lives in the advanced layer at the top. That’s freezing layers: keep the prerequisite layers fixed at their pretrained state, and only update the advanced one. In the diagram below, W1 and W2 — the foundational prerequisites — stay frozen. Only W3 — the layer closest to your task-specific output — gets a ΔW.

Tom Yeh

27,587 Aufrufe • vor 4 Monaten

$i made $83,000 by looking at one number not the price not the volume the spread α = Δ / (V_h − μ) α is the fraction of traders in the market who already know the correct answer spread 0.05 → α = 5% safe to enter spread 0.15 → α = 15% be careful spread 0.25 → α = 20% one in five knows more than you before this i entered markets with wide spreads i thought illiquid, inefficient, that's where the money is turns out a wide spread means smart money is already there and i was paying their bill every single time once i started filtering markets by this formula stopped entering anything where α > 12% $83,000 in the next two months same polymarket same markets i just stopped being someone else's α i track these markets in real time through a bot it calculates α for every market automatically and only alerts when the entry is clean did you know the spread tells you exactly who else is in the game?$

i made $83,000 by looking at one number not the price not the volume the spread α = Δ / (V_h − μ) α is the fraction of traders in the market who already know the correct answer spread 0.05 → α = 5% safe to enter spread 0.15 → α = 15% be careful spread 0.25 → α = 20% one in five knows more than you before this i entered markets with wide spreads i thought illiquid, inefficient, that's where the money is turns out a wide spread means smart money is already there and i was paying their bill every single time once i started filtering markets by this formula stopped entering anything where α > 12% $83,000 in the next two months same polymarket same markets i just stopped being someone else's α i track these markets in real time through a bot it calculates α for every market automatically and only alerts when the entry is clean did you know the spread tells you exactly who else is in the game?

self.dll

42,399 Aufrufe • vor 6 Monaten

Macy's on 34th in NYC . It was reported that a group of migrant gang members were allegedly shoplifting at the name-brand shop when they said the person recording the attacked the person. This is an ongoing issue at Macy's and other shops across NYC. The way this works is they send out crews to shoplift, then the stolen products are them sold by a different group . This is very organized and are making tons of money. ( sent in by a source)

Viral News NYC

54,978 Aufrufe • vor 1 Jahr

whoever leaked this has bigger balls than sense SpaceXAI shipped five hireable workers for $200 a month, then wrote the catch into its own Grok Bot documentation and left the page up: all five run on one computer, so one sign-in hands the browser session, the files and the command-line credentials to every one of them the NSA, CISA and the cyber agencies of the UK, Canada, Australia and New Zealand had published the opposite instruction 103 days earlier: no broad or unrestricted access, low-risk and non-sensitive work only i ran four of mine on one account for a week, counting what each could reach: eleven signed-in apps, one browser profile, and deleting a bot left all of it standing Grok Bot is worth hiring five times over, and you can draw its blast radius before the second one exists: - sign in for the bot that needs the site, then open the others and see what they reach: that session is theirs the moment it exists - give each bot its own account on the app, since the docs tell you in writing to stop using separate bots as a security boundary - put the stop line in the description, as an approval controls the proposed action and leaves whatever already ran where it landed - cap the spend outside the product, because there is no bot-specific spend cap yet and the audit view of what they did is still coming - keep the money and the customer replies in your own hands, and let the other four start from scratch each morning on work that cannot bite one sign-in is also why this pays: five names finish inside your real tools instead of handing you drafts to paste my take, and it is the uncomfortable one: your real limit on Grok Bot is how many logins you will put on one machine, and the hiring was always the easy half bookmark this, the five descriptions that let bots hand work to each other and the one folder that survives an update are written out in the article ↓

Argona

694,236 Aufrufe • vor 19 Tagen

TESLA HALTED MODEL S AND MODEL X PRODUCTION TO BUILD AN ARMY OF OPTIMUS ROBOTS The Fremont assembly line was torn down in 46 days. In its place, Tesla is building a line for humanoid production, aiming for a million units a year A humanoid robot is a body shaped like a human. Physical AI is the intelligence that controls that body Walking and making coffee is often just imitation learning from a scripted routine. But once the environment shifts, the learned trick stops working Language models had the entire internet to train on. Robotics has nothing close to that scale of data, which is why one giant brain hasn't worked for anyone yet The industry is moving toward modularity instead - separate models for vision, movement, and planning, each improved on its own The real question is no longer whether a robot can move impressively. It's whether it can pull its sensors into one picture of the world and adapt to whatever wasn't scripted for it

TESLA HALTED MODEL S AND MODEL X PRODUCTION TO BUILD AN ARMY OF OPTIMUS ROBOTS The Fremont assembly line was torn down in 46 days. In its place, Tesla is building a line for humanoid production, aiming for a million units a year A humanoid robot is a body shaped like a human. Physical AI is the intelligence that controls that body Walking and making coffee is often just imitation learning from a scripted routine. But once the environment shifts, the learned trick stops working Language models had the entire internet to train on. Robotics has nothing close to that scale of data, which is why one giant brain hasn't worked for anyone yet The industry is moving toward modularity instead - separate models for vision, movement, and planning, each improved on its own The real question is no longer whether a robot can move impressively. It's whether it can pull its sensors into one picture of the world and adapt to whatever wasn't scripted for it

iamigorekk

22,205 Aufrufe • vor 18 Tagen

In my class, I teach the autoencoder by asking everyone to stand up. 🙆 Stretch your arms out wide. Imagine you are holding a heavy textbook (like Introduction to Algorithms by Prof. Cormen), the whole thing, every page. Now bring your hands slowly together until they almost touch your neck. The bottle "neck." The final exam is tomorrow and you are allowed one cheat sheet: whatever you can scribble on your palm. All nine hundred pages have to survive the squeeze. That is the encoder. Now imagine you sit in the exam. Push your arms back out to where they started. You try to rebuild the textbook from your palm notes. That is the decoder. Of course you cannot get every page back. What you get back is what mattered enough to write down, and the gap between the two is the loss the network is trying to shrink. I call it AI by Arms 🙆. It gets a laugh, and then it gets remembered. Goal: squeeze four numbers down to two, then rebuild the original four from them. 1. Given Let us start with four training examples: X1, X2, X3, X4. 2. Auto (copy to targets) We copy the training examples straight into the targets. That is the whole trick behind the name: "auto" is Greek for "self", and the data is its own label. 3. Encoder, layer 1 Let us multiply the inputs by the weights, add the biases, and apply ReLU. Negative values get crossed out and become zero. 4. Encoder, layer 2 (the bottleneck) We do it again, and now the four dimensions have become two. This layer is called the bottleneck, because everything has to fit through it. 5. Decoder, layer 1 Let us go back the other way: multiply, add, ReLU. This time there are no negatives to cross out. 6. Decoder, layer 2 We multiply once more and get the outputs Y. This is the decoder's attempt to rebuild the four original numbers from the two it was given. 7. MSE loss gradients Let us compare Y with the targets Y'. The gradient is 2 x (Y - Y'): subtract, then double. Those gradients kick off backpropagation, and the weights start to learn. Your entire education is all about encoding and decoding!

In my class, I teach the autoencoder by asking everyone to stand up. 🙆 Stretch your arms out wide. Imagine you are holding a heavy textbook (like Introduction to Algorithms by Prof. Cormen), the whole thing, every page. Now bring your hands slowly together until they almost touch your neck. The bottle "neck." The final exam is tomorrow and you are allowed one cheat sheet: whatever you can scribble on your palm. All nine hundred pages have to survive the squeeze. That is the encoder. Now imagine you sit in the exam. Push your arms back out to where they started. You try to rebuild the textbook from your palm notes. That is the decoder. Of course you cannot get every page back. What you get back is what mattered enough to write down, and the gap between the two is the loss the network is trying to shrink. I call it AI by Arms 🙆. It gets a laugh, and then it gets remembered. Goal: squeeze four numbers down to two, then rebuild the original four from them. 1. Given Let us start with four training examples: X1, X2, X3, X4. 2. Auto (copy to targets) We copy the training examples straight into the targets. That is the whole trick behind the name: "auto" is Greek for "self", and the data is its own label. 3. Encoder, layer 1 Let us multiply the inputs by the weights, add the biases, and apply ReLU. Negative values get crossed out and become zero. 4. Encoder, layer 2 (the bottleneck) We do it again, and now the four dimensions have become two. This layer is called the bottleneck, because everything has to fit through it. 5. Decoder, layer 1 Let us go back the other way: multiply, add, ReLU. This time there are no negatives to cross out. 6. Decoder, layer 2 We multiply once more and get the outputs Y. This is the decoder's attempt to rebuild the four original numbers from the two it was given. 7. MSE loss gradients Let us compare Y with the targets Y'. The gradient is 2 x (Y - Y'): subtract, then double. Those gradients kick off backpropagation, and the weights start to learn. Your entire education is all about encoding and decoding!

Tom Yeh

22,703 Aufrufe • vor 1 Monat

Love for humanity is the core of love for life. It arises from the recognition of life’s value and beauty, from faith in humanity’s ability for greatness, and from the awareness that you are its child — and that it is dear and close to you. : The author places love for humanity at the heart of love for life itself. It springs from a clear recognition of life’s immense value and beauty, from an unwavering faith in humanity’s capacity for greatness, and from the intimate awareness that each person is humanity’s own child, bound to it by blood, spirit, and destiny. This love is not distant admiration. It is the quiet, fierce tenderness a child feels for the parent who gave it existence. Humanity is dear and close because it is the living source from which we draw breath, meaning, and possibility. To love life is to love the collective that carries it forward. To doubt humanity is to doubt life’s own worth. When we see ourselves as its children, gratitude replaces judgment. Hope replaces despair. Love for humanity becomes the deepest form of self-love, because we are not separate from it. It is the recognition that every stranger carries a piece of the same miracle we carry in ourselves. That recognition turns strangers into kin, and the world into home.

Love for humanity is the core of love for life. It arises from the recognition of life’s value and beauty, from faith in humanity’s ability for greatness, and from the awareness that you are its child — and that it is dear and close to you. : The author places love for humanity at the heart of love for life itself. It springs from a clear recognition of life’s immense value and beauty, from an unwavering faith in humanity’s capacity for greatness, and from the intimate awareness that each person is humanity’s own child, bound to it by blood, spirit, and destiny. This love is not distant admiration. It is the quiet, fierce tenderness a child feels for the parent who gave it existence. Humanity is dear and close because it is the living source from which we draw breath, meaning, and possibility. To love life is to love the collective that carries it forward. To doubt humanity is to doubt life’s own worth. When we see ourselves as its children, gratitude replaces judgment. Hope replaces despair. Love for humanity becomes the deepest form of self-love, because we are not separate from it. It is the recognition that every stranger carries a piece of the same miracle we carry in ourselves. That recognition turns strangers into kin, and the world into home.

Zafar Mirzo | Quotes

287,911 Aufrufe • vor 8 Monaten

$The hard part of multi-agent systems is getting agents to stay quiet. Put five agents on one task, and they duplicate work and burn tokens talking to each other. Offloop trained a dispatcher model called D1 that decides which agent moves next and when the right move is to do nothing. They achieve state-of-the-art performance on GDPval at a fraction of the usual cost. You can bring your own AI subscription.$

The hard part of multi-agent systems is getting agents to stay quiet. Put five agents on one task, and they duplicate work and burn tokens talking to each other. Offloop trained a dispatcher model called D1 that decides which agent moves next and when the right move is to do nothing. They achieve state-of-the-art performance on GDPval at a fraction of the usual cost. You can bring your own AI subscription.

elvis

22,116 Aufrufe • vor 1 Monat

JUST IN: Bank of America just told its clients to take profits. About 70% of its bear-market signals are flashing, a level it typically reaches only near market tops. Weeks earlier, BofA's own fund manager survey showed the largest one-month jump into stocks ever recorded, with cash down to 3.9%, under the 4% line the bank treats as a sell signal. Read those together. Investors made their biggest dash into equities in the survey's history at almost the exact moment BofA's own indicators say the top is near. But the number that should actually stop you is buried in the note, and almost nobody is quoting it. The companies driving this entire rally, the AI hyperscalers, are on track to spend nearly 100% of their operating cash flow on capex by year-end. In 2023 that figure was 40%. Sit with that. Big tech used to throw off cash and hand it back through buybacks, which lifted the stocks. Now it is pouring almost every dollar it generates into chips and data centers. BofA notes buybacks have slowed and cash conversion has flat-lined. The engine of the rally is consuming the fuel that powered the stocks. It is the same $725 billion build that companies are now blaming for layoffs. The whole market is priced on one bet, and that bet has grown large enough to eat the cash that used to support the share prices. This is not a crash call. BofA's year-end target is 7,100, about 4% below today, and the median outcome after this cash signal since 2011 has been a 1% dip, not a collapse. The posts screaming sell everything are wrong. The real message is quieter. You are being paid less and less to stay, while the engine runs hotter and hotter.

JUST IN: Bank of America just told its clients to take profits. About 70% of its bear-market signals are flashing, a level it typically reaches only near market tops. Weeks earlier, BofA's own fund manager survey showed the largest one-month jump into stocks ever recorded, with cash down to 3.9%, under the 4% line the bank treats as a sell signal. Read those together. Investors made their biggest dash into equities in the survey's history at almost the exact moment BofA's own indicators say the top is near. But the number that should actually stop you is buried in the note, and almost nobody is quoting it. The companies driving this entire rally, the AI hyperscalers, are on track to spend nearly 100% of their operating cash flow on capex by year-end. In 2023 that figure was 40%. Sit with that. Big tech used to throw off cash and hand it back through buybacks, which lifted the stocks. Now it is pouring almost every dollar it generates into chips and data centers. BofA notes buybacks have slowed and cash conversion has flat-lined. The engine of the rally is consuming the fuel that powered the stocks. It is the same $725 billion build that companies are now blaming for layoffs. The whole market is priced on one bet, and that bet has grown large enough to eat the cash that used to support the share prices. This is not a crash call. BofA's year-end target is 7,100, about 4% below today, and the median outcome after this cash signal since 2011 has been a 1% dip, not a collapse. The posts screaming sell everything are wrong. The real message is quieter. You are being paid less and less to stay, while the engine runs hotter and hotter.

Shanaka Anslem Perera ⚡

17,270 Aufrufe • vor 2 Monaten

On June 22, the United States signed one executive order to build a quantum computer and, the same day, another to defend against what a quantum computer can do. The first establishes a national effort to build a machine powerful enough to open a new era of scientific discovery, delivered to a Department of Energy lab. The second orders an accelerated national migration to post-quantum cryptography, because the same physics that makes the machine useful makes today's encryption breakable. Read the two orders against each other and the asymmetry is the entire story. The order to build the machine sets no delivery date and is explicitly subject to the availability of appropriations. The order to defend the data sets hard deadlines: a migration pilot due December 31, 2027, and high-value systems moved to post-quantum cryptography by 2030 and 2031. The machine is a national aspiration. The migration is a national deadline. This is the part almost no one is pricing. Even the government is treating the regulatory clock as the binding one, while the market keeps watching the hardware clock. A product does not need to be broken to lose its value. It only needs to become uncertifiable, and the date it becomes uncertifiable is now written into federal policy. The machine can stay distant. The proofs are already on the clock.

On June 22, the United States signed one executive order to build a quantum computer and, the same day, another to defend against what a quantum computer can do. The first establishes a national effort to build a machine powerful enough to open a new era of scientific discovery, delivered to a Department of Energy lab. The second orders an accelerated national migration to post-quantum cryptography, because the same physics that makes the machine useful makes today's encryption breakable. Read the two orders against each other and the asymmetry is the entire story. The order to build the machine sets no delivery date and is explicitly subject to the availability of appropriations. The order to defend the data sets hard deadlines: a migration pilot due December 31, 2027, and high-value systems moved to post-quantum cryptography by 2030 and 2031. The machine is a national aspiration. The migration is a national deadline. This is the part almost no one is pricing. Even the government is treating the regulatory clock as the binding one, while the market keeps watching the hardware clock. A product does not need to be broken to lose its value. It only needs to become uncertifiable, and the date it becomes uncertifiable is now written into federal policy. The machine can stay distant. The proofs are already on the clock.

Shanaka Anslem Perera ⚡

35,284 Aufrufe • vor 2 Monaten

They did not take cursive from the schools because children no longer needed it. They took it because of what it was quietly building in them. Consider what the exercise actually is. A child, six years old, is handed a pen and asked to draw a single unbroken line that becomes a word. The wrist must float. The fingers must hold a living pressure, never quite the same twice, always correcting. The eye must follow the ink forward and trust the hand to finish what it has begun. There is no lifting, no stopping, no starting over mid-word. The loop must close. The ascender must rise and return. The sentence must travel from one margin to the other as a single continuous gesture, and at the end of it the hand must still be steady. Twelve years of this. Every day. Ten thousand small acts of sustained, self-correcting attention, carried out below the level of conscious thought, until the motion belongs to the body and the body belongs to the motion. This is not penmanship. It is the slow construction of an interior form. The hand that has learned to carry a line without breaking it is the hand of a mind that has learned to carry a thought without breaking it. The two are not metaphors for one another. They are the same faculty, trained in the same child, by the same daily discipline. Continuity of the stroke becomes continuity of the reasoning. The patience of the loop becomes the patience of the argument. The commitment to finish a word one has started becomes the commitment to finish a sentence, a paragraph, a life's idea, without reaching for the nearest distraction halfway through. Print is a different creature entirely. Print lifts. Print stops. Print assembles a word out of separate, stamped, interchangeable pieces, each one beginning and ending in isolation. A mind raised only on print learns to think the way print is made, in discrete tokens, in replaceable units, in fragments that can be recombined by any outside hand without the owner noticing the substitution. It is precisely the shape of thought a language model produces. It is precisely the shape of thought a language model can steer. Cursive is kata. This is the whole of it. A form repeated daily, for years, not for the sake of the form but for what the repetition lays down in the practitioner beneath the form. The swordsman does not train kata so that one day he may fight in kata. He trains it so that when the moment comes and there is no time to think, the movement is already inside him, older and deeper than thought, and it rises on its own. Cursive was the kata of the literate mind, the daily quiet drilling of continuity, of patience, of a line held steady under the long pressure of its own length. And the signature it produced at the end, that small flourished mark unique to a single human being on earth, was only the outward proof of an inward form no machine and no other hand could ever reproduce. Take the kata away and the practitioner is left with vocabulary in place of faculty. He can recognise a whole thought when he encounters one. He cannot carry one himself. He can admire a finished argument. He cannot sustain one long enough to close its loop. He begins books he does not finish, sentences he does not end, ideas he abandons the moment the screen in his palm offers him a brighter one. And when the machine begins feeding him tokens in the exact shape his schooling taught him to receive, he meets it with no interior resistance at all, because no interior form was ever built in him to push back with. They removed it quietly, across a generation, and they removed it in the last years before the machines arrived. Twelve years of daily practice in unbroken, embodied, self-authored thought, gone from the curriculum of almost every child in the Western world, just as the instruments designed to complete their sentences for them came online. The hand forgets. The mind, having never been taught the kata, forgets a thing it never knew it had. That is what cursive was. That is what was taken. And that is why the thought of anyone who still writes by hand, in long unlifted lines, remains, quietly, stubbornly, and without their ever needing to announce it, their own. Now the question stands open. What else has been banned, phased out, quietly retired from the curriculum and from common life over these same decades, under the same soft excuses? Mental arithmetic. Memorisation of poetry. Latin. Logic as a formal subject. Map reading. Knot work. The keeping of a commonplace book. The reading aloud of long passages in class. Singing in parts. What was each of those actually building in the child, beneath the surface of the lesson, and whose interest was served by its disappearance?

They did not take cursive from the schools because children no longer needed it. They took it because of what it was quietly building in them. Consider what the exercise actually is. A child, six years old, is handed a pen and asked to draw a single unbroken line that becomes a word. The wrist must float. The fingers must hold a living pressure, never quite the same twice, always correcting. The eye must follow the ink forward and trust the hand to finish what it has begun. There is no lifting, no stopping, no starting over mid-word. The loop must close. The ascender must rise and return. The sentence must travel from one margin to the other as a single continuous gesture, and at the end of it the hand must still be steady. Twelve years of this. Every day. Ten thousand small acts of sustained, self-correcting attention, carried out below the level of conscious thought, until the motion belongs to the body and the body belongs to the motion. This is not penmanship. It is the slow construction of an interior form. The hand that has learned to carry a line without breaking it is the hand of a mind that has learned to carry a thought without breaking it. The two are not metaphors for one another. They are the same faculty, trained in the same child, by the same daily discipline. Continuity of the stroke becomes continuity of the reasoning. The patience of the loop becomes the patience of the argument. The commitment to finish a word one has started becomes the commitment to finish a sentence, a paragraph, a life's idea, without reaching for the nearest distraction halfway through. Print is a different creature entirely. Print lifts. Print stops. Print assembles a word out of separate, stamped, interchangeable pieces, each one beginning and ending in isolation. A mind raised only on print learns to think the way print is made, in discrete tokens, in replaceable units, in fragments that can be recombined by any outside hand without the owner noticing the substitution. It is precisely the shape of thought a language model produces. It is precisely the shape of thought a language model can steer. Cursive is kata. This is the whole of it. A form repeated daily, for years, not for the sake of the form but for what the repetition lays down in the practitioner beneath the form. The swordsman does not train kata so that one day he may fight in kata. He trains it so that when the moment comes and there is no time to think, the movement is already inside him, older and deeper than thought, and it rises on its own. Cursive was the kata of the literate mind, the daily quiet drilling of continuity, of patience, of a line held steady under the long pressure of its own length. And the signature it produced at the end, that small flourished mark unique to a single human being on earth, was only the outward proof of an inward form no machine and no other hand could ever reproduce. Take the kata away and the practitioner is left with vocabulary in place of faculty. He can recognise a whole thought when he encounters one. He cannot carry one himself. He can admire a finished argument. He cannot sustain one long enough to close its loop. He begins books he does not finish, sentences he does not end, ideas he abandons the moment the screen in his palm offers him a brighter one. And when the machine begins feeding him tokens in the exact shape his schooling taught him to receive, he meets it with no interior resistance at all, because no interior form was ever built in him to push back with. They removed it quietly, across a generation, and they removed it in the last years before the machines arrived. Twelve years of daily practice in unbroken, embodied, self-authored thought, gone from the curriculum of almost every child in the Western world, just as the instruments designed to complete their sentences for them came online. The hand forgets. The mind, having never been taught the kata, forgets a thing it never knew it had. That is what cursive was. That is what was taken. And that is why the thought of anyone who still writes by hand, in long unlifted lines, remains, quietly, stubbornly, and without their ever needing to announce it, their own. Now the question stands open. What else has been banned, phased out, quietly retired from the curriculum and from common life over these same decades, under the same soft excuses? Mental arithmetic. Memorisation of poetry. Latin. Logic as a formal subject. Map reading. Knot work. The keeping of a commonplace book. The reading aloud of long passages in class. Singing in parts. What was each of those actually building in the child, beneath the surface of the lesson, and whose interest was served by its disappearance?

SiriusB

443,811 Aufrufe • vor 4 Monaten

Elon Musk is building a data center that no grid has to power and no county has to approve. Musk: “Think of it as a rack of compute in space.” Orbital hardware has always been custom. Each satellite is its own program, its own team, its own decade. SpaceX’s AI1 satellites are copies of each other. Each is built around a single Nvidia NVL72, the same rack-scale architecture running the largest data centers on the ground, redesigned to fly. Space stops being a program and becomes an inventory. Designing for orbit made the rack simpler and cheaper than the ground version, so SpaceX plans to run the space design in terrestrial data centers too. Racks alone have never made a data center. A thousand of them in a building with no network between them is a thousand computers. The interconnect is what turns them into one machine. Training a frontier model means moving enormous volumes of data between processors without pause, and the whole system runs at the speed of its slowest link. That is why the fabric between chips is fought over as hard as the chips themselves. Musk: “And then you can connect these racks of compute to either each other by the laser links, or directly to the Starlink constellation.” Light moves through glass at about two thirds of its speed in a vacuum, and fiber optic cable is glass. Every backbone route and undersea cable on Earth pays that tax. A laser between two satellites pays nothing. Over long distances, an orbital link arrives sooner than the same run in fiber. Orbit was supposed to be a trade, unlimited power in exchange for a worse network. The network is faster. The compute plugs into a delivery layer that is already finished. Thousands of satellites are up there right now carrying laser links, terminating at ground stations and dishes across most of the planet. The last mile was built years before the thing it delivers existed. Anyone else attempting this needs the rocket, the satellite bus, the solar manufacturing, the laser mesh, the ground network, and a supply line to a chipmaker. SpaceX had five of them before the project had a name. None of it was built for this. Starlink existed to sell internet subscriptions. The lasers existed so those subscriptions would work over open ocean. Starship existed for Mars. Nobody planned this. It assembled itself out of problems solved for other reasons. That is what twenty years of hard engineering leaves behind. Every previous expansion of human capability ran on a resource sitting inside somebody’s borders. Coal, oil, water, land. Sunlight in orbit belongs to nobody and never runs out. Compute still has an address. A building in Virginia, a substation, a county that had to approve it, a grid connection someone waited four years for. He is describing a version that only has an altitude.

Elon Musk is building a data center that no grid has to power and no county has to approve. Musk: “Think of it as a rack of compute in space.” Orbital hardware has always been custom. Each satellite is its own program, its own team, its own decade. SpaceX’s AI1 satellites are copies of each other. Each is built around a single Nvidia NVL72, the same rack-scale architecture running the largest data centers on the ground, redesigned to fly. Space stops being a program and becomes an inventory. Designing for orbit made the rack simpler and cheaper than the ground version, so SpaceX plans to run the space design in terrestrial data centers too. Racks alone have never made a data center. A thousand of them in a building with no network between them is a thousand computers. The interconnect is what turns them into one machine. Training a frontier model means moving enormous volumes of data between processors without pause, and the whole system runs at the speed of its slowest link. That is why the fabric between chips is fought over as hard as the chips themselves. Musk: “And then you can connect these racks of compute to either each other by the laser links, or directly to the Starlink constellation.” Light moves through glass at about two thirds of its speed in a vacuum, and fiber optic cable is glass. Every backbone route and undersea cable on Earth pays that tax. A laser between two satellites pays nothing. Over long distances, an orbital link arrives sooner than the same run in fiber. Orbit was supposed to be a trade, unlimited power in exchange for a worse network. The network is faster. The compute plugs into a delivery layer that is already finished. Thousands of satellites are up there right now carrying laser links, terminating at ground stations and dishes across most of the planet. The last mile was built years before the thing it delivers existed. Anyone else attempting this needs the rocket, the satellite bus, the solar manufacturing, the laser mesh, the ground network, and a supply line to a chipmaker. SpaceX had five of them before the project had a name. None of it was built for this. Starlink existed to sell internet subscriptions. The lasers existed so those subscriptions would work over open ocean. Starship existed for Mars. Nobody planned this. It assembled itself out of problems solved for other reasons. That is what twenty years of hard engineering leaves behind. Every previous expansion of human capability ran on a resource sitting inside somebody’s borders. Coal, oil, water, land. Sunlight in orbit belongs to nobody and never runs out. Compute still has an address. A building in Virginia, a substation, a county that had to approve it, a grid connection someone waited four years for. He is describing a version that only has an altitude.

Dustin

12,757 Aufrufe • vor 9 Tagen

Elon Musk gave the entire entertainment industry its expiration date, and he is the one building the thing that kills it. Musk: “My guess is that we see the first compelling half hour, pure AI show next year.” Next year. A complete show generated entirely by AI. No writers. No actors. No cameras. No sets. No crew. No studio. Just a prompt and enough compute to render a reality that never physically existed. And shows are the easy part. Musk: “I say probably we’re maybe three years away from AI does the whole video game.” A show plays the same way every time. A game has to generate a living world that reacts to every decision in real time across every single frame. That is a fundamentally harder class of problem. And Musk put three years on it. Right now a single AAA title takes seven years and half a billion dollars across thousands of engineers and artists just to ship it. Musk is describing a world where one person types a paragraph and gets something comparable. The entire value proposition of a multi-billion dollar industry lives inside that gap. And it closes in thirty-six months. But the prediction is not the story. The person making it is. This is not an analyst speculating from the sidelines. This is the man building the largest AI compute clusters on the planet. The man who built xAI from zero in under two years. The man stacking hundreds of thousands of GPUs into facilities designed to do exactly what he is describing. When Musk says three years, he is not guessing about what someone else might eventually ship. He is reading you a delivery date off his own roadmap. Every media company on Earth is valued on a single assumption. That quality content is expensive and difficult to produce at scale. That one assumption is the structural foundation underneath every studio, every network, and every publisher in existence. Musk is dismantling it with raw compute. The studios still parading thousand-person production teams are not demonstrating strength. They are advertising the exact cost structure that one person with a prompt and a GPU allocation is about to make irrelevant. And it does not stop at entertainment. If AI can generate an interactive world that responds to human input in real time, it can generate anything. Advertising. Architecture. Training simulations. Product design. Every industry built on humans manually constructing visual experiences frame by frame is sitting on the same countdown Musk just read out loud. Now zoom out. Because this is not just an industry story. For the entire history of human civilization, the distance between imagining a world and actually creating one required thousands of people, millions of hours, and billions of dollars. That distance built Hollywood. That distance built the gaming industry. That distance made content scarce and studios powerful. Musk is collapsing that distance to zero. When the gap between imagining something and it existing disappears, every business model built on the difficulty of creation disappears with it. That is not disruption. That is a full inversion of how human beings create. Musk did not make a casual prediction on that podcast. He told you what he is building. He told you the timeline. And he told you which industries do not survive it. The entertainment industry is still debating whether this future is real. Musk is not part of that debate. He is building. And he just told you the delivery date.

Elon Musk gave the entire entertainment industry its expiration date, and he is the one building the thing that kills it. Musk: “My guess is that we see the first compelling half hour, pure AI show next year.” Next year. A complete show generated entirely by AI. No writers. No actors. No cameras. No sets. No crew. No studio. Just a prompt and enough compute to render a reality that never physically existed. And shows are the easy part. Musk: “I say probably we’re maybe three years away from AI does the whole video game.” A show plays the same way every time. A game has to generate a living world that reacts to every decision in real time across every single frame. That is a fundamentally harder class of problem. And Musk put three years on it. Right now a single AAA title takes seven years and half a billion dollars across thousands of engineers and artists just to ship it. Musk is describing a world where one person types a paragraph and gets something comparable. The entire value proposition of a multi-billion dollar industry lives inside that gap. And it closes in thirty-six months. But the prediction is not the story. The person making it is. This is not an analyst speculating from the sidelines. This is the man building the largest AI compute clusters on the planet. The man who built xAI from zero in under two years. The man stacking hundreds of thousands of GPUs into facilities designed to do exactly what he is describing. When Musk says three years, he is not guessing about what someone else might eventually ship. He is reading you a delivery date off his own roadmap. Every media company on Earth is valued on a single assumption. That quality content is expensive and difficult to produce at scale. That one assumption is the structural foundation underneath every studio, every network, and every publisher in existence. Musk is dismantling it with raw compute. The studios still parading thousand-person production teams are not demonstrating strength. They are advertising the exact cost structure that one person with a prompt and a GPU allocation is about to make irrelevant. And it does not stop at entertainment. If AI can generate an interactive world that responds to human input in real time, it can generate anything. Advertising. Architecture. Training simulations. Product design. Every industry built on humans manually constructing visual experiences frame by frame is sitting on the same countdown Musk just read out loud. Now zoom out. Because this is not just an industry story. For the entire history of human civilization, the distance between imagining a world and actually creating one required thousands of people, millions of hours, and billions of dollars. That distance built Hollywood. That distance built the gaming industry. That distance made content scarce and studios powerful. Musk is collapsing that distance to zero. When the gap between imagining something and it existing disappears, every business model built on the difficulty of creation disappears with it. That is not disruption. That is a full inversion of how human beings create. Musk did not make a casual prediction on that podcast. He told you what he is building. He told you the timeline. And he told you which industries do not survive it. The entertainment industry is still debating whether this future is real. Musk is not part of that debate. He is building. And he just told you the delivery date.

Dustin

22,458 Aufrufe • vor 1 Monat