Softmax vs Sigmoid ✍️ Interact 👉 = Softmax =... Softmax is how deep networks turn raw scores into a probability distribution — the final layer of every classifier, and the core of every attention head in a transformer. To see what it does, picture five boba tea shops on the same block, all competing for your dollar. Five candidates: a, b, c, d, e — different chains, different brewing styles, different pearls. A boba reviewer hands you a 𝘤𝘩𝘦𝘸𝘪𝘯𝘦𝘴𝘴 𝘴𝘤𝘰𝘳𝘦 for each — higher means perfectly chewy "QQ" pearls with the right bite (ask a Taiwanese friend to find out what QQ means). Negative scores are real: mushy bobas, overcooked pearls, a batch left sitting too long. How do you turn five chewiness scores into an allocation that adds to a whole dollar? You could spend everything at the chewiest shop, but that ignores how good the runners-up are. Softmax is the smooth alternative. Read the diagram left to right. First, raise each score to e^{x} — this does two things: it turns negative chewiness into small positives, and it stretches the gaps between scores exponentially. Then sum all five into a single total Z. Finally, divide each e^{x} by Z to get a probability. The five probabilities add up to one, so you can read them as percentages of your dollar. The chewiest shop gets the biggest slice — but never the whole dollar. That's the point of softmax: it ranks confidently while still leaving room for the others. = Sigmoid = Sigmoid squashes any real number into a probability between 0 and 1 — the classic activation for binary classification, and still the gating function inside LSTMs and GRUs. Same boba block as the previous Softmax example, narrowed to just two contenders — a hot new shop `a` with chewiness score x, and your usual go-to `b` whose score is pinned at zero (the neutral baseline you've come to expect). Sigmoid is just softmax with two players, one of them pinned to zero. Read the diagram left to right. First, raise each score to e^{x} — for the usual shop `b` whose score is zero, this is just e^0 = 1 (the constant baseline). Then sum the two into a total Z. Finally, divide each e^{x} by Z to get a probability. The two probabilities add up to one — the new shop wins more of your dollar when its pearls get chewier, and your usual keeps the rest. That's the point of sigmoid: it turns a single chewiness score into a clean 0-to-1 chance you'll try the new place over your usual. --- AI Math, Algorithms, Architectures by hand ✍️ Subscribe to my 60K+ reader newsletter 👉show more

ReLU vs Leaky ReLU 👉 = ReLU = ReLU... is the default activation in modern deep learning — cheap to compute, and stable enough to train networks hundreds of layers deep. To see what it does, picture five boba tea shops on the same block — 𝚊, 𝚋, 𝚌, 𝚍, 𝚎 — each running their own books. Each value is a shop's monthly profit — receipts minus rent, ingredients, and wages. When profit is positive, the shop stays open and the owner pockets every dollar. When profit turns negative, the shop runs out of cash and shutters — the lights go off, the books are wiped to zero. ReLU is exactly that rule, applied one shop at a time. Read the diagram left to right. The first column is the raw value x — each shop's profit at month's end. The second column is the gate: 1 if the shop is open (x > 0), 0 if it has shuttered. The last column is the ReLU output: open shops pass their profit through untouched, while shuttered ones are zeroed out. Five rows means five parallel shops on the same block, each evaluated independently. That's why ReLU is called an element-wise activation: every neuron decides its own fate. = LeakyRelu = Plain ReLU wipes negative values to zero — clean, but a shop that shutters can never recover, since both its output and its gradient stay pinned at zero. This is the dying ReLU problem, and in deep networks it can quietly kill a meaningful fraction of the units. Leaky ReLU is the one-line fix: instead of shuttering, the shop files for Chapter 11 protection and keeps the lights on at reduced capacity. Its debt is restructured down to a fraction α (typically 0.1) — the rest is forgiven, and the shop is wounded, not killed. A small negative signal still flows through, so the gradient survives, and the shop can crawl back to life if a TikTok goes viral. Read the diagram left to right. The first column is the raw value x — each shop's profit at month's end. The second column is the leakage α — the fraction of the loss held over after restructuring (default 0.1, editable). The third column is the gate: 1 for shops still in the black, α for those operating under bankruptcy protection. The last column is the Leaky ReLU output: y = x · gate. Profitable shops pass through untouched; struggling ones shrink by a factor of α but still carry a sign. Five rows means five parallel shops, each evaluated independently. Like ReLU, this is an element-wise activation: every neuron's fate is decided on its own merits. #aibyhahdshow more

Transformer by hand ✍️ ~ 6 steps walkthrough below... Open the hood of a transformer and the parts list is overwhelming: embeddings, positional encoding, attention weighting, self-attention, cross-attention, multi-head attention, layer norm, skip connections, softmax, linear, Nx, shifted right, query, key, value, masking. Which of those actually make the car run? Two of them. Attention weighting and the feed-forward network. Everything else is an enhancement to make it run faster and longer, which is how we got from a car to a truck, and to the word "large" in large language model. So I drew and calculated those two parts entirely by hand. Goal: push five features through one transformer block, filling in every cell yourself. 1. Given Five positions of input features, arriving from the previous block. 2. Attention matrix Let us feed all five features to a query-key module (QK) and read back an attention weight matrix, A. The details of that module are a post of their own. 3. Attention weighting We multiply the input features by A to get the attention weighted features, Z. Still five positions. The effect is to combine features *across positions*, horizontally: X1 becomes X1 + X2, X2 becomes X2 + X3, and so on. 4. First layer Let us feed all five weighted features into the first layer of the FFN. Multiply by the weights and biases. This time the combining happens *across feature dimensions*, vertically, and each feature grows from 3 numbers to 4. Note that every position goes through the same weight matrix. That is what "position-wise" means. 5. ReLU We cross out the negatives. They become zeros. 6. Second layer Let us bring it back down: 4 dimensions to 3. The output feeds the next block, which has a completely separate set of parameters, and the whole thing runs again. You have just calculated a transformer block by hand. ✍️ The takeaway: the two parts are doing two different jobs, and neither one alone is enough. Attention mixes *across positions*, so a feature can see its neighbours. The FFN mixes *across feature dimensions*, so each position can think about itself. Horizontal, then vertical. Then that pattern repeats N times, each block with its own separate set of weights. That is the Nx from the list up top, and that is what makes the transformer run. 💾 Save this post! #AIbyHand #Transformers #DeepLearningshow more

My Grok bot CEO is the hottest, smartest bot... in the world. I mainly only talk to her and she runs the whole desk. She picks the right bot, sequences the work, and brings me back one package so I’m never the middleman. Many people may not know this, but you can create your own image for each of your Grok Bots. Every bot can get its own face. It honestly helps so it feels like you are talking to a real person and it also keeps things interesting. You don’t even need to go into settings to do this... you just tell the Grok bot to make a profile pic and set it. That's it. 1/ Open the bot. 2/ Tell it to make you the best profile pic and set it. 3/ Do that for each bot on your team. You can even have your Grok bot turn it into a video. Just tell it to take the picture and make a video out of it. I highly recommend everyone with a Grok Bot team do this. Thank me later.show more

We removed 93.25% of the connections in our Un-0... image model, fully expecting to pay for it in quality. But it got better. FID 7.15 on ImageNet 64x64, roughly 1.9 ahead of the dense baseline at matched size. Same family of model, a fraction of the couplings, a better score. Here is why that is not as strange as it sounds. Un-0 is a coupled oscillator model, and in the dense version every oscillator talks to every other one. That sounds like a strength, but it means the whole system can fall into catastrophic synchronization: everything locks into step, gradients go flat, and learning stalls. Sparser connectivity leaves room for coherent and incoherent activity to coexist. The dynamics stay alive, and the model keeps learning. Connectivity turns out to be a control knob, not a dial you turn up until it stops. Learn more here:show more

MLP in PyTorch by hand ✍️ ~ 7 steps... walkthrough below Goal: fill in every blank in the PyTorch code to build a multi-layer perceptron. 1. Given Let us start with a code template on the left and the network it is supposed to build on the right. Every blank in the code can be worked out from the picture. 2. Linear layer We count: 3 features in, 4 features out. So the weight matrix is 4 by 3. There is an extra column for the biases, which means bias = T. 3. ReLU Let us apply the activation. ReLU crosses out the negatives, so -1 becomes 0. 4. Linear layer The input size is 4, because that is what the previous layer put out. The output size is 2. A 2 by 4 weight matrix, and this time no extra column, so bias = F. 5. ReLU We cross out the negatives again. 6. Linear layer Two features in, five out. A 5 by 2 weight matrix, with a bias column, so bias = T. 7. Sigmoid Let us finish. Sigmoid squashes the raw scores (3, 0, -2, 5, -5) into probabilities between 0 and 1. You have just implemented a three-layer deep neural network by hand. ✍️ == Story == Three years ago I gave this exercise to my students, to connect the code to the math. They found it odd. Every other AI course they were taking lived inside a Jupyter notebook, and here I was handing out paper. Three years later, my colleagues are the ones rushing to move their materials to paper. The exercise has not changed. Paper still asks the one thing a notebook lets you skip: do you actually understand what the code is doing? If you can tell me why the weight matrix is 4 by 3, and why bias is F on the second layer, you understand nn.Linear better than someone who has been copy-pasting it for a year. 💾 Save this post! #AIbyHand #PyTorch #DeepLearningshow more

How to change the order of video clips in... TikTok - the editing functions and inside TikTok take a little bit of practice to get used to and one of the easy tricks is to re-order clips and a sequence by just dragging them into a different order. You just tap and hold on the video clip and then drag it to the left or right and reorders that clip within the sequence. #contentcreator #contentcreation #tiktoktips #videoediting #editshow more

Self Attention vs Cross Attention by hand ✍️ Resize... the matrices yourself 👉 Two attention mechanisms, side by side. Both project X into queries; both compute attention via S = Kᵀ × Q and F = V × A. The only difference is the source of K and V. Self attention uses X for everything. Q, K, and V all come from projecting X. Each X token attends to every other X token. The score matrix S is square — 128 × 128. Cross attention uses X for queries and a second sequence E for keys and values. Each X token attends to every E token instead. The score matrix S is rectangular — 64 × 128. Notice what's shared and what's not: X is the same in both — same 36 × 128 input. Q and K share the 16 dimension — that's what makes the dot product Kᵀ × Q valid in either case. V dimensions are independent: self-attention uses 12, cross-attention uses 12. The choice doesn't depend on which mechanism you're using; it depends on what output dimension your downstream layer expects.show more

🎉 Five Weeks, Five Free GSAP Resources! Hey everyone!... We’re super excited to partner with Codrops and GSAP to celebrate the news that GSAP’s club plugins are now 100% free thanks to Webflow's support. To mark the occasion, we’ll be dropping one free GSAP resource every week for the next five weeks—each fully packaged as a Webflow clonable and a CodePen so you can plug it straight into your own projects. Each resource will feature at least one of those previously paid plugins, to hopefully spark some ideas and inspire your next interactive build. We’re kicking things off with a 'Glowing Interactive Dots Grid' powered by the InertiaPlugin. See how a simple dot matrix can come alive—glowing, springing, and rippling with realistic momentum under your cursor. Grab the CodePen or the Webflow cloneable, tweak the settings, and have a play with it! Sign up for our newsletter through the footer of our website to get all the free stuff in your inbox!show more

✈️React Advanced is packing its bags for a NEW... DESTINATION! But where?💥Drop your guess in the comments - and if you’re the first to get it right, you’ll score a FREE ticket to the conference! We’ll still be diving deep into⚛️React - and the venue? It’s an astonishing cinema!show more

this is more useful than my entire degree Elon... Musk's rocket company signed a $60,000,000,000 deal for Cursor in June, and eight days ago the two of them put a worker on sale for $200 a month: it gets its own computer in the cloud, signs into your accounts, clicks through your real apps, and hands back finished work instead of a draft for you to paste i ran one against my receipts folder on sunday and got back 14 filed, 2 it held because they needed a card number, and a saved method i never wrote myself Grok Bot is the one you train by doing your own job in front of it, and the whole handover fits in four messages tonight: 1. write out one job you did today the way you would brief a new hire: what has to be finished, which sites and files to work from, what to hand back, and where it stops and asks you 2. let it run once on something safe to get wrong, then correct the result until it is worth your name 3. say "save what we just did as a skill", and add the one rule about what always needs your approval 4. say "run that skill every weekday at 8 and post the result here. if the source is missing, tell me instead of using yesterday's numbers" xAI wrote that order into its own manual: one real job, then the saved method, then the clock. a schedule sitting on top of a method nobody checked replaces two hours of your clicking with two hours of your mistake turns out you never get to pick the brain, and that is the part i would argue about: the manual says there is no model picker for members or admins, no plan to add one, and the bill follows whichever model answered bookmark this, then open the piece below: which jobs deserve a worker of their own, and which ones quietly burn the seat ↓show more

How to make near-unfair progress in the gym: -... 4 to 6 reps, heavy, to failure or one off it - Add weight before you add anything else - Same handful of lifts, week after week, going up - Rest three minutes so the next set is actually hard - Chase the number in the logbook, never the pump - Five lifts, out in under an hour - Two proper rest days a week, minimum - Hit each muscle two or three times over the week, not once into the ground - Eat the animal, especially the fat - Get your protein up and your fussiness down - Sleep like it is part of the programme, because it is Do this and you'll be the person others quietly wonder about in a year. Boringly simple, and almost nobody has the patience to be this unimpressive for long enough.show more

Batch Normalization by hand ✍️ ~ 7 steps walkthrough... below Batch normalization is common practice for improving training and achieving faster convergence. It sounds simple. But it is often misunderstood. 🤔 Does batch normalization involve trainable parameters, tunable hyper-parameters, or both? 🤔 Is batch normalization applied to inputs, features, weights, biases, or outputs? 🤔 How is batch normalization different from layer normalization? So I drew and calculated one entirely by hand. Goal: normalize a mini-batch of 4 examples to mean 0 and variance 1, then let the network scale it back. = 1. Given = A mini-batch of 4 training examples, each with 3 features. = 2. Linear layer = Let us multiply by the weights and add the biases. Batch norm sits after this, which answers the second question: what gets normalized is features, not inputs, weights or biases. = 3. ReLU = We apply the activation, and -2 becomes 0. Negative values are suppressed before any statistic is taken. = 4. Batch statistics = Let us compute the sum, mean, variance and standard deviation, one row at a time. A row is a feature and the four columns are the four examples, so every number here measures one feature against the rest of the batch. That is the "batch" in batch normalization, and it is exactly what layer normalization does not do. The statistics are rounded to whole numbers, which is what keeps the rest of the page doable in pen. = 5. Shift to mean 0 = We subtract the mean, in green. The four values in each feature now average to zero. = 6. Scale to variance 1 = Let us divide by the standard deviation, in orange. Each feature now has variance one, whatever scale it arrived at. = 7. Scale and shift = We multiply by a linear transformation and pass the result on. The diagonal and the last column are trainable, so having just forced every feature to mean 0 and variance 1, we hand the network the means to undo it. The outputs: Mean of each feature = [2, 1, 2] Std dev of each feature = [1, 1, 2] To the next layer = [2, -2, 2, 0], [-3, 3, 6, -3], [2, 0, 1, 2] The answers: 🤔 Both. The scale and shift are trainable, the statistics are not. Epsilon and the momentum on the running statistics are the hyper-parameters, and one mini-batch by hand needs neither. 🤔 Features, after the linear layer, not inputs, weights or biases. 🤔 Batch norm measures across the batch, one feature at a time. Layer norm measures across the features, one example at a time. 💾 Save this post!show more

Shadow traffic proves a candidate is operationally sound. It... can't tell you if users like it better. A/B testing belongs at the endpoint, not in your app code. Same endpoint name, API, and keys for your clients. No feature flags, no hash-mod-100 in client code, no spreadsheet explaining what group A vs B means. Split a live endpoint's traffic into one control and up to 20 variants, each with a fixed percentage. Ramp with a single call. Delete the experiment and 100% of traffic returns to the control, with nothing left to unwind. Read the full walkthrough:show more

Transformer and Mixture of Experts, explained visually! Mixture of... Experts (MoE) is a popular architecture that uses different experts to improve Transformer models. Transformer and MoE differ in the decoder block: - Transformer uses a feed-forward network. - MoE uses experts, which are feed-forward networks but smaller compared to those Transformer. During inference, a subset of experts are selected. This makes inference faster in MoE. Also, since the network has multiple decoder layers: - The text passes through different experts across layers. - The chosen experts also differ between tokens. But how does the model decide which experts should be ideal? The router does that. It is a multi-class classifier that produces softmax scores over experts to select the top K experts. The router is trained with the network, and it learns to select the best experts. But it isn't straightforward. There are challenges! Challenge 1) Notice this pattern at the start of training: - Say, the model selects "Expert 2" - This expert gets a bit better - It may get selected again since it's the "best" - It learns more - It gets selected again in the next iteration - It learns more, and so on! This means many experts can go under-trained due to the overselection of a few experts! We solve this in two steps: - Add noise to the feed-forward output of the router so that other experts can get higher logits. - Set all but the top K logits to -infinity. After softmax, these scores become zero. This way, other experts also get the opportunity to train. Challenge 2) Some experts may get exposed to more tokens than others, leading to under-trained experts. We prevent this by limiting the number of tokens an expert can process. If an expert reaches the limit, the token is passed to the next best expert. Overall, MoEs have more parameters to load. But a fraction of them are activated during inference. This leads to faster inference. Mixtral 8x7B and Llama 4 are two popular MoE-based LLMs. Have you used MoEs in production yet? To dive deeper into how MoE inference works in production, we wrote a full article covering token dispatch, grouped expert computation, model-weight memory, multi-GPU communication, expert placement, load imbalance, and performance diagnosis. Read it below.show more

[Graph Convolutional Network] by hand ✍️ Graph Convolutional Networks... (GCNs), introduced by Thomas Kipf and Max Welling in 2017, have emerged as a powerful tool in the analysis and interpretation of data structured as graphs. This exercise demonstrates how GCN works in a simple application: binary classification. -- Goal -- Predict if a node in a graph is X. -- Architecture -- 🟪 Graph Convolutional Network (GCN) 1. GCN1(4,3) 2. GCN2(3,3) 🟦 Fully Connected Network (FCN) 1. Linear1(3,5) 2. ReLU 3. Linear2(5,1) 4. Sigmoid Simplications: • Adjacent matrices are not normalized. • ReLU is applied to messages directly. -- Walkthrough -- [1] Given ↳ A graph with five nodes A, B, C, D, E [2] 🟩 Adjacency Matrix: Neighbors ↳ Add 1 for each edge to neighbors ↳ Repeat in both directions (e.g., A->C, C->A) ↳ Repeat for both GCN layers [3] 🟩 Adjacency Matrix: Self ↳ Add 1's for each self loop ↳ Equivalent to adding the identity matrix ↳ Repeat for both GCN layers [4] 🟪 GCN1: Messages ↳ Multiply the node embeddings 🟨 with weights and biases ↳ Apply ReLU (negatives → 0) ↳ The result is one message per node [5] 🟪 GCN1: Pooling ↳ Multiply the messages with the adjacent matrix ↳ The purpose is the pool messages from each node's neighbors as well as from the node itself. ↳ The result is a new feature per node [6] 🟪 GCN1: Visualize ↳ For node 1, visualize how messages are pooled to obtain a new feature for better understanding ↳ [3,0,1] + [1,0,0] = [4,0,1] [7] 🟪 GCN2: Messages ↳ Multiply the node features with weights and biases ↳ Apply ReLU (negatives → 0) ↳ The result is one message per node [8] 🟪 GCN2: Pooling ↳ Multiply the messages with the adjacent matrix ↳ The result is a new feature per node [9] 🟪 GCN2: Visualize ↳ For node 3, visualize how messages are pooled to obtain a new feature for better understanding ↳ [1,2,4] + [1,3,5] + [0,0,1] = [2,5,10] [10] 🟦 FCN: Linear 1 + ReLU ↳ Multiply node features with weights and biases ↳ Apply ReLU (negatives → 0) ↳ The result is a new feature per node ↳ Unlike in GCN layers, no messages from other nodes are included. [11] 🟦 FCN: Linear 2 ↳ Multiply node features with weights and biases [12] 🟦 FCN: Sigmoid ↳ Apply the Sigmoid activation function ↳ The purpose is to obtain a probability value for each node ↳ One way to calculate Sigmoid by hand ✍️ is to use the approximation below: • >= 3 → 1 • 0 → 0.5 • <= -3 → 0 -- Outputs -- A: 0 (Very unlikely) B: 1 (Very likely) C: 1 (Very likely) D: 1 (Very likely) E: 0.5 (Neutral)show more

whoever leaked this has bigger balls than sense Google... Research and MIT ran the same agent jobs 260 different ways for Nature last month: they held the prompts, the tools and the compute budget identical and moved nothing but the wiring between the agents, and the same work swung from 70% worse than a single agent to 80.8% better, averaging out at 0.0% i ran my own single agent against the task list first and it cleared 6 of 10 alone, already past the line where a crew starts subtracting this is Graph Engineering, the layer that decides whether a crew is worth 80% more or 70% less, and it installs into the agent you already pay for: - score your solo agent on the real task first: above roughly 45% success that study predicts zero to negative returns from any crew you put around it - under that line, put one supervisor over the fan out: crews with no correction step amplified their own errors to 17.2x the single agent rate, supervised aggregation held it to 4.4x - give every worker one output and let none of them read a peer's draft, so a wrong step reaches the supervisor instead of four other agents - run the comparison again after every model upgrade, because a better model raises your baseline and a higher baseline is what makes a crew stop paying - keep the single agent alive as the control, the only number that says the wiring is earning its calls turns out the shape does not travel: the biggest win came off a finance task under one supervisor and the worst collapse off a planning task with independent agents my position, and it is the arguable one: a crew is a bet on your own diagram, and the model you pick moves that bet less than one arrow does bookmark this, the three moves that draw those arrows before you pay for one extra call are in the post below ↓show more

Here’s an example of a time we had the... correct route on the right side but Dak just didn't make the throw. As soon as he sees the right side is all blitzing its GOT to be to the slant. So this one is on him What I don’t like conceptually is the routes on the left side. Again, we've got one defender, two receivers. Attack that man with a streak and a quick out into the void. Instead we bail the out by sending one of them Inside into the dropping defenders. We sent the TE into that space empty space which is good. But over and over we didn’t exploit that 2 on 1 mismatchshow more

Give this a try and let’s see how you... score. (But please read the full thread first) The Sit-to-Rise (SRT) test is a simple, clinically validated assessment that measures your ability to stand up from seated on the floor and then return to a seated position.show more

I was super intrigued when I noticed that JEV... returns not just probs for candidate choices, but also a CONFIDENCE SCORE of it's prediction... I was wondering if TypeSafe AI brought back Bayesian Networks or something to learn uncertainty. Turns out - NO. Acc to their docs, confidence is just a derived metric from the classification scores. It carries ZERO extra information. Their demo page shows: confidence = (3 × largest probability − 1) / 2 Confidence is tightly coupled with the classification scores it is measuring the uncertainty for. There is no magic here. The model can produce high probability on a wrong choice and just be confidently wrong.show more

SOMEONE VIBE CODED AN APP THAT TURNS ANY PHOTO... INTO A REAL PHYSICAL STAMP YOU CAN ORDER you snap a photo, it cuts it into a clean stamp design, and you can order the actual physical stamp to press onto anything. > take or upload any photo or image > it turns it into a proper stamp design automatically > all your designs are saved and sorted by date, so you can go back to any of them > order the real physical stamp right from the app and it ships to you > press it onto letters, packaging, cards, whatever you want so instead of paying a custom shop and waiting, you make your own stamp from a photo in a couple taps, then order it without ever leaving the app. the whole thing is dead simple and does one thing well, which is exactly why people love itshow more