Today we report a startling pretraining finding: Attention circuits... STOP meaningfully evolving much sooner than we expect. In other words, the model has decided where to look very early on. It is known that different seeds create starkly different results; here we isolate a specific mechanism that causes it. The setup: We perform two identical training runs with the same data order and different initializations and find that by the first thousand steps each run’s attention resembles its own step-10k pattern MORE than the individual finished runs resemble each other. The initialization plays a greater role in the attention patterns than training dynamics, which implies that the final pattern is randomly chosen. NOTE: Randomly chosen, not from the space of all the possible attention patterns that it could be, but from the different attention pattern basins that it could converge to—which one it ends up in appears to be init dependent. Some differences in the loss are attributable to these different attention patterns. This means the final performance is somewhat dependent on a random choice. Depending on how lucky with pattern formation one gets in the first thousand steps, we could end up much better or worse in different contexts. This further implies training could simply find the optimal patterns, or at least better ones, if the attention could evolve instead of being stuck. A story for why attention evolves less than we’d hope falls out of the softmax derivative. We can observe a positive feedback loop we call the credit ratcheting. Let oᵢ be the head output and δᵢ = ∂L/∂oᵢ the loss gradient at that output. Define key k’s advantage for query i as Aᵢₖ = −(vₖ − oᵢ)ᵀδᵢ i.e. how much the loss drops if oᵢ moves toward vₖ, relative to the current attention-weighted mix. Thus, the gradient with respect to sᵢₖ = kₖᵀqᵢ / √d is ∂L/∂sᵢₖ = −aᵢₖAᵢₖ A key can have a huge advantage and have its score still barely move, because the advantage is multiplied by the attention it already gets. It’s the same form as a softmax policy gradient, π(a)·A(a): attention is the policy, and it mostly learns about the actions it already takes. In aggregate, the query gradient is gᵢ = Jᵢᵀδᵢ where Jᵢ = (1/√d) ∑ₖ aᵢₖ(vₖ − oᵢ)kₖᵀ. Across 166,656 Jacobians in Qwen2.5-1.5B, ≥99% of tokens in each of the 3 heads we probed had squared-spectrum effective rank <10, in a 128-dim head. The update to the query is being filtered through an exceedingly narrow channel. The values have the same issue. Key k’s share of the value gradient is Gᵢₖ = aᵢₖδᵢxₖᵀ, so at similar δᵢxₖᵀ, 10× the attention ≈ 10× the credit. This means the values mostly adapt to positions already looking at them. This compounds! Attention allocates credit, credit updates values, and values can pull attention further toward the winners. The most natural approach to fix this is to precondition the query gradient with the Fisher of the attention distribution (NATQ): Fᵢ = Covₐᵢ(k) / d uᵢ = (Fᵢ + λᵢI)⁻¹gᵢ g̃ᵢ = uᵢ · (‖gᵢ‖₂ / ‖uᵢ‖₂) Even this blunt version does a lot. At 124M, entropy rose from 3.75 to 4.69 nats, top-key attention fell from 0.238 to 0.149, and in layers 4–7, keys the baseline gave under 0.1% attention got 67% of the gain. Validation loss rose by 0.082 nats. A more targeted intervention need not make this concession. Here’s the complete blog: We frame this as an open problem that offers a stark contrast in the way we perceive pretraining. If we can find a way to allow for attention to evolve throughout training, opposed to just the very start, we may find inordinate gains on model capability. Internally we half joke that the alternative read is that attention might simply not be real past the first few steps, and it’s all the MLPs. If training is able to work as well as it does without really changing the attention patterns much, maybe the MLP is the real workhorse that’s driving the learning.show more

Transformer by hand ✍️ ~ 6 steps walkthrough below... Open the hood of a transformer and the parts list is overwhelming: embeddings, positional encoding, attention weighting, self-attention, cross-attention, multi-head attention, layer norm, skip connections, softmax, linear, Nx, shifted right, query, key, value, masking. Which of those actually make the car run? Two of them. Attention weighting and the feed-forward network. Everything else is an enhancement to make it run faster and longer, which is how we got from a car to a truck, and to the word "large" in large language model. So I drew and calculated those two parts entirely by hand. Goal: push five features through one transformer block, filling in every cell yourself. 1. Given Five positions of input features, arriving from the previous block. 2. Attention matrix Let us feed all five features to a query-key module (QK) and read back an attention weight matrix, A. The details of that module are a post of their own. 3. Attention weighting We multiply the input features by A to get the attention weighted features, Z. Still five positions. The effect is to combine features *across positions*, horizontally: X1 becomes X1 + X2, X2 becomes X2 + X3, and so on. 4. First layer Let us feed all five weighted features into the first layer of the FFN. Multiply by the weights and biases. This time the combining happens *across feature dimensions*, vertically, and each feature grows from 3 numbers to 4. Note that every position goes through the same weight matrix. That is what "position-wise" means. 5. ReLU We cross out the negatives. They become zeros. 6. Second layer Let us bring it back down: 4 dimensions to 3. The output feeds the next block, which has a completely separate set of parameters, and the whole thing runs again. You have just calculated a transformer block by hand. ✍️ The takeaway: the two parts are doing two different jobs, and neither one alone is enough. Attention mixes *across positions*, so a feature can see its neighbours. The FFN mixes *across feature dimensions*, so each position can think about itself. Horizontal, then vertical. Then that pattern repeats N times, each block with its own separate set of weights. That is the Nx from the list up top, and that is what makes the transformer run. 💾 Save this post! #AIbyHand #Transformers #DeepLearningshow more

So I saw an Alex jones clip earlier and... this was playing on the other corner and I feel like it has not gotten enough attention. If you pay attention to the timing. As the people are reacting to the sound, he’s already up and running on the roof! How could he have a rifle at all, much less disassemble it in that amount of time??show more

Pay really close attention to the sound that we... are supposed to think is a gunshot in that exact moment. It has a very different sound than a gunshot.show more

Had courtside tickets for the NBA Playoffs tonight but... unfortunately $ATTENTION coin already has too much attention and was not allowed in stadium. Not joking, the fucking FBI was there 😭 Made the best of it and Jake Ryan and a few others got some content for his IG, but obviously a bit of a let down. However this is the beauty of $ATTENTION coin. We have so many bagworkers and people willing to put their future on the line for this coin that every single day there is potential for a gigaviral clip with no risk of burnout. And to be honest I think this will be good long run. People who would've sold the news are out. The floor keeps rising due to the 70% lock and a new opportunity for a major catalyst comes each day. EVERYBODY WILL BE ONBOARDED TO $ATTENTION THIS YEARshow more

World Models are the path for some AI Models... in the future. But how can we efficiently train these models to not only see the world the way humans do but to see the world in a new and unique way. By visualizing, what is normally sequenced audio patterns, we can derive much more insights. Here we see Paganini in a visual form that can than be described and transcribed into a World Model. We can observe connections in a manner that may not have been clear prior to the digitalization of music and sound in this way. The company with the most valuable potential in building a World Model is Tesla. Not that this type of visualization is being used, but that the mechanisms are in place, and the technology is in place for the company to thrive in this new form of AI.show more

Sometimes Flow is refered to as a silent film,... which isn't true. Dialogue free would be a better description. Without dialogue we can use the other tools of cinema in a much more expressive way. The music, the sound, the camera, lighting and animation don't have to compete with the dialogue which can tend to attract all the attention.show more

So, my opinion on what the Antarctic (Antarctica) Anomaly... is that it's a type of frequency technology. It must be way more powerful than HAARP, as many have claimed it to be, because we would see these anomalies at other HAARP sites, and we don't, not like this. With that said, and I'm very much trying to avoid letting what I want it to be not play a part here, I think it is a technology that is being used either off the coast of Antarctica itself or Bouvet Island. A third possibility is an area just to the northwest of the island that looks odd. It's possible it is a sonar scan from a ship, but why in that remote location? It looks like an antenna set up or rows of something that is out of place. I also believe that the weather events and fires that have taken place in Africa could possibly have been because of this. Each time we saw the anomaly, it was followed by a destructive weather event in Africa. A weird connection to that is we have been told and warned of a very busy 2024 Atlantic hurricane season. This is in part because of the above-average Atlantic ocean temperatures, which is the fuel to Hurricanes. With all this info, it's possible to see how the Anomaly could be a frequency tech that can manipulate or create weather, And or WARM up the Ocean temps to purposely enhance the Hurricane season and Storm growth. Keep in mind that many of our hurricanes and many of the biggest hurricanes have come from the west coast of Africa and form over the Cape Verde islands before heading towards the Caribbean and the United States. This is all of course speculation, and I'm learning many new things every day, so this idea may morph over time as we learn more. In the end, it is very hard to ignore all these findings. #antarctica #anonaly #AntarcticaAnomaly #BouvetIslandshow more

Today we are happy to share one of our... first open research releases: Latent Sparse Attention (LSA), a step towards subquadratic attention that matches accuracy of full attention at ultra-long context. Sparse attention attends to only a few tokens per query, but it has to find the right ones. At 10M tokens, that search takes 98% of the attention layer's time. LSA makes sparse attention up to 22.7× faster and full attention 72.4x faster. With Ales Kubicek Andrea Caciolai Liangzu Peng Sukrit Sumant Ben Hoover Fraser Ruairidh Battleday Radim Urban Torsten Hoefler 🇨🇭 James Whittingtonshow more

Self Attention vs Cross Attention by hand ✍️ Resize... the matrices yourself 👉 Two attention mechanisms, side by side. Both project X into queries; both compute attention via S = Kᵀ × Q and F = V × A. The only difference is the source of K and V. Self attention uses X for everything. Q, K, and V all come from projecting X. Each X token attends to every other X token. The score matrix S is square — 128 × 128. Cross attention uses X for queries and a second sequence E for keys and values. Each X token attends to every E token instead. The score matrix S is rectangular — 64 × 128. Notice what's shared and what's not: X is the same in both — same 36 × 128 input. Q and K share the 16 dimension — that's what makes the dot product Kᵀ × Q valid in either case. V dimensions are independent: self-attention uses 12, cross-attention uses 12. The choice doesn't depend on which mechanism you're using; it depends on what output dimension your downstream layer expects.show more

The final part of the three part "Profiling in... PyTorch" series is out now! In Part 1 we profiled addition and multiplication In Part 2 we profiled a torch Linear layer (and also MLP) In this part we profile Attention. > a naive implementation of attention > talk about the torch sdpa API > math backend > xformers backend > flash backend > cuDNN backend PS: This is a long read, bookmark this and read it over the weekends. 🤗show more

👀👀👀👀 "Ripple is the company to be paying attention... to. I believe that the United States will be looking at this as a Swift dollar, a way of transacting and doing a lot of different things...so XRP is the ticker on that." David Stryzewski strikes again! Her reaction is priceless.show more

Hard to believe this is a controversial topic. You... think Caitlin Clark, the best basketball player in the history of women’s basketball &amp; the most recognized athlete in college right now, is flopping for attention? You may be the one seeking attention. Watch it real time.show more

😂 But seriously… in ABA, a reinforcer can be... ANYTHING! A reinforcer is something that increases the likelihood of a behavior happening again. It could be a favorite song, a game, attention, snacks, or an activity. The key? It has to actually be reinforcing to THAT individual! #ABAtherapy #Reinforcement #BehaviorAnalysisshow more

Blind cats don't know they're supposed to be helpless.... That's the first thing nobody tells you. We look at an animal whose eyes have clouded over and we feel the floor drop out from under our own feet, because we imagine our own darkness: the stumbling, the panic, the hands out in front of us. We project. A cat does not. A cat has a completely different relationship with the room it lives in, and that gap between what we fear for them and what they actually experience is where this whole story lives. Here is the part that makes me stop scrolling every time I see it. Sight is not the foundation of a cat's world. It's one tool in a kit, and for a creature built to hunt in half-light, it's not even the sharpest one. Cats see poorly up close. They struggle to focus on anything inside a few inches of their nose. Their eyes are built for movement at dusk, not for reading the fine print of a hallway. So when the sight goes, the cat doesn't lose its world. It loses one translator of it. And then comes the question that this video is quietly built around: what happens when someone decides that one translator isn't enough? What do you build for an animal that can't tell you what it needs, can't read instructions, and has strong opinions about anything that touches its face? I'm not going to tell you the answer. I want to tell you why the question matters more than it looks. Think about how a cat actually moves through a house. Not how we imagine it. How it truly does it. The whiskers, for starters. Those aren't decoration. Each one is rooted far deeper in the skin than an ordinary hair, wrapped in a dense network of nerves, and they work as living radar. A cat reads air currents bouncing off a wall before its nose ever reaches it. It reads the width of a gap before committing its shoulders to the squeeze. Whiskers also sit on the backs of the front legs, which is why a cat knows what's under its paws without looking down. It is a body wired for touch at a distance. Then the nose. Cats carry on the order of two hundred million scent receptors, versus roughly five million in a person. Every doorway has a smell. Every chair leg has a history. Your cat doesn't just know where the couch is. It knows who sat there yesterday, what they ate, and whether they were stressed. Then the ears. Each one swivels independently, driven by a small army of muscles, and the hearing range runs far beyond ours. A cat can locate a rustle to within a few inches in total darkness. Now stack those three together and add the thing that quietly matters most: memory. Cats build a map of their home the way a sailor learns a coastline. Where the table legs are. How many steps it takes from the water bowl to the window. Which floorboard is cold in the morning. A map like that survives the loss of eyesight. Which is exactly why so many blind cats seem fine for a while. Sometimes for years. But "fine" is doing a lot of work in that sentence. Because a map is only as good as the world staying put. Move one chair and the cat's confidence doesn't just dip, it can collapse. Carry a laundry basket into a hallway that was empty all week. Leave a shoe where a shoe has never been. The cat's whole internal model, the thing it trusted more than any single sense, gets a hole punched in it. And what does a smart, careful animal do when its map can't be trusted? It slows down. It starts keeping to the walls. It stops jumping to the places it loved. That's the quiet cost nobody measures. Not the bump on the head. The shrinking. A cat that used to cross a room like it was late for an appointment now crosses it like it's walking on ice. And the people who love that cat watch the shrinking happen and don't know what to do with their hands. Let me tell you about the thing that makes this so hard to see coming. Cats are professionals at hiding weakness. It's an old survival habit, written deep into the species: in the wild, the animal that looks vulnerable becomes the animal that gets picked off. So a cat that is losing its sight will often behave as though nothing is happening. It will time its steps to familiar rhythms. It will follow the sound of the kibble bag. It will bluff. Vets often point out that owners of senior cats discover vision loss late, sometimes only when the house changes and the cat suddenly can't cope. A new rug. A guest's suitcase. A cat who walks into a door that has been open a hundred times before. And here's the part that deserves to be said plainly, because it's the one place in this story where waiting can cost something. Vision loss in a cat is not always just "getting old." Conditions like high blood pressure, retinal problems, glaucoma, cataracts, infections, and injuries can all take sight, and some of them are treatable if they're caught early. Sudden changes, such as eyes that look cloudy overnight, pupils that look unusually wide, a cat suddenly bumping into things, or a cat that seems disoriented, are worth a prompt call to a veterinarian. Not next month. This week. That's the medical truth. It's not the interesting part of this story. The interesting part starts after the diagnosis, in the long stretch of ordinary days where an animal has to decide whether to trust the world again. Because this is what separates a blind cat who thrives from a blind cat who withdraws: not the eyes. The nerve. I keep returning to that word because it shows up in both senses. The optic nerve, the thing that carries the picture. And the other kind. The willingness to step forward when you can't see where your foot is going. Every blind animal is running a constant, silent negotiation with risk. Do I go to the food bowl or wait for it to be brought? Do I jump onto the windowsill or sit in the hallway? Each step is a bet. Lose enough bets and an animal stops placing them. So the people who share their homes with blind animals end up becoming amateur engineers almost by accident. They tape down a loose rug corner. They learn to leave the furniture exactly where it is, even when a friend says the room would look better rearranged. They sprinkle a little vanilla by a doorway, a little lavender by the stairs, so the nose can add labels to the map. They clap softly before they walk in, because a startled blind cat is a cat who just lost a few weeks of progress. None of it is dramatic. All of it is love expressed as logistics. And then, somewhere in that world of tiny adaptations, somebody asks a question that sounds almost silly until you think about it. What if the cat could have a way to feel an obstacle before it arrives? Not a gadget with a screen. Not a leash. Something that works with an animal's own instincts instead of fighting them. Because there's a trap in every idea like this, and it's the trap that kills most well-meaning pet inventions: humans design for what makes humans feel better. We buy the thing that calms our worry. Then the cat looks at it, looks at us, and quietly decides that it will be wearing nothing at all, thank you. Anyone who has tried to put a collar on a cat that didn't want a collar knows exactly what I mean. Cats don't negotiate. They don't pretend. If something is annoying, they will find the one position on the planet in which it becomes impossible to ignore, freeze in it, and stare at you with the disappointment of a retired judge. So the real test of any aid for a blind cat isn't whether it looks clever to us. It's whether the cat will walk in it. Walk. Not tolerate. Not sit rigid like a statue until someone takes it off. Actually move, the way it moved before, with the kind of looseness that you can't fake and can't coach. That's an extremely high bar. Hold on to it, because we'll come back to it. First, I want to talk about the other side of the equation. The humans. There is a particular kind of person who ends up with a blind senior cat. Not always, but often. Many of them didn't plan it. They adopted a healthy animal, years ago, and the animal aged beside them through apartment moves and job changes and bad winters. Or they took in an older cat precisely because nobody else wanted one. Senior animals sit in shelters far longer than kittens. A cat with a medical label sits longer still. Anyone who has walked past a row of cages knows the quiet arithmetic that happens: small, soft, and young goes first. A blind cat, on paper, is a hard sell. "Special needs." Two words that make a lot of people take a half step backward. And yet. Ask anyone who lives with one and you'll hear a version of the same sentence, delivered with a slightly embarrassed shrug: they're honestly easier than I expected. They don't pace the window waiting for the bird. They don't get distracted. They are tuned to your voice in a way that sighted cats sometimes aren't, because your voice is a landmark. Your footsteps are a landmark. The sound of the can opener is practically a lighthouse. A blind cat doesn't just live in your house. It lives in your presence, mapping its safety to the sound of your life. That's a heavy kind of trust, if you stop and feel its weight. You become the fixed point in somebody's world. And I think that's what pulls people toward videos like this one. We're not really moved by the cat. Or not only. We're moved by the glimpse of the other half of the relationship: a person at a kitchen table at night, doing research, comparing materials, trying things that don't work, and then trying again. Taking an animal's limitation personally, and then refusing to leave it alone. There's a quiet radicalism in that. The world has a very clear opinion about which lives are worth the extra effort. It is expressed in the price of a procedure, in the line at the shelter, in the sentence that begins with "well, at his age." Most of us have heard that sentence or said it. It's not cruelty. It's a reflex. It's what we say to make a hard thing feel finished. The person who builds something instead is answering that sentence. Without arguing. Without a speech. By sitting down with a problem and refusing the reflex. I find that more persuasive than any slogan. Now, let's be honest about the limits, because a story that only flatters is a story that deserves less of your trust. Not every blind animal adapts at the same speed. Some take weeks to rebuild their confidence, some take months, some never quite return to the person they were. A cat who loses sight slowly over years has often had time to build a map in advance. A cat who loses sight suddenly may be in shock, and shock looks like withdrawal. It would be a lie to say that a clever piece of equipment fixes all of it. Nothing fixes all of it. And there's a risk in the other direction, too. The more we celebrate a good solution, the more pressure builds on the owners who haven't found theirs. People watch these clips late at night, feeling like they're failing an animal who is perfectly fine, and who might be perfectly happy with a stable room and a patient person. So I'm careful with the claim I'm making. I'm not telling you this video solves blindness. I'm telling you it shows something about what happens when imagination meets an animal's real, specific, unpredictable needs. And what happens next isn't the part I can summarize for you, because the whole point is that you have to see how the cat answers. Because that's the thing about animals. They are the only honest critics left. A person will tell you your idea is wonderful to spare your feelings. A cat will simply refuse to move. When a cat does move, that is the review. Let me widen the lens for a minute, because there's something older than pet products hiding inside this story. Every tool ever invented for people who can't see is, at its root, an attempt to do one thing: push the reach of touch farther than the body allows. A cane is a very long fingertip. It lets a person feel a curb two steps before the curb arrives. Some blind people go further and learn to click their tongues and listen to the echoes, building a sound-picture of the room. It sounds like a superpower and it's really a discipline. Brains that lose one channel tend to lean harder on the others, and with practice they can read in them what the missing channel used to say. A cat already owns the original version of that technology. It grows it out of its face. Whiskers are, in effect, a cat's built-in cane. They're believed to be roughly as wide as the cat's own body, which is part of why a cat can look at a gap and know without calculation whether it fits. They fan forward when a cat is curious and press back when it's cornered. In the dark they are doing math the rest of us can't see. So here's the question that I can't stop turning over. If whiskers already do that work, how much room is there for anything else? If a cat has spent its entire life reading the world with its own face, what could possibly be added that wouldn't just be noise? The answer, I suspect, is: only something that respects the face. Something that gives the whiskers, the head, the body one more small, clear signal at the exact moment the world has turned unreliable. A signal that comes early enough to matter. A signal that doesn't feel like a restraint. A signal that the cat can ignore the second it doesn't need it. Easy to say. Brutally hard to build. And I'm not going to pretend I know how it was solved, because I'd rather you see the answer arrive on its own terms than hear me announce it. What I can tell you is what it means when it works. It means a cat that has been told for months, by a dozen small bumps, that the world is dangerous, receives its first quiet piece of evidence that it might not be. And an animal's confidence doesn't come back in a rush. It comes back the way a tide comes in: you stand there and think nothing is happening, and then you notice your feet are wet. There's something else I want to say about that, because it's the part I think about long after the clip is over. When we mourn for a blind animal, we are mostly mourning for ourselves. For the loss of the cat that watched the birds. For the end of the version of the pet that we first met. We are grieving a story, and the cat is not in the story. The cat is in the room. As far as anyone can tell, animals live remarkably close to the present tense. A cat doesn't lie awake missing the way the light used to fall across the carpet. It wakes up, it listens, it smells, it works out what is happening now, and it starts moving. There's a lesson in that which is almost embarrassing to say out loud: the cat is not suffering from the loss of sight nearly as much as it is suffering from the loss of predictability. Give the animal back predictability, and a large part of the suffering lifts. Which is exactly the kind of insight that makes you look at your own life sideways. How much of what we call grief is really the loss of predictability? How much of what we call fear is really a missing landmark? It's not only cats, of course. Dogs lose their sight too, often in the same slow way. Many sniff their way through the transition, trusting their noses and their people. Dogs tend to be louder about the confusion: they bump, they startle, they stand still in the middle of a hallway as if someone paused the world. But they seem to rebuild the same way cats do, one familiar object at a time, until the house becomes a place they can read again. The pattern across species is surprisingly consistent. Remove one sense and the animal doesn't collapse. It reorganizes. It's the same quiet stubbornness that gets any living thing through winter. And the work of the humans around them is to make that reorganization less lonely. That's all. Not to fix. To accompany. There's a version of caring that wants to fix, and a version that wants to accompany, and they are very different animals. The fixer needs the problem to disappear so the discomfort disappears with it. The companion can sit inside a difficult thing and still be useful. Most of the people who thrive with a blind or aging pet end up learning to be the second kind, usually by failing as the first. Let me clear away a few myths, because they do real damage, and because a story about blindness is a good place to say them out loud. Myth one: a blind pet is a suffering pet. Blindness on its own is not necessarily painful, and many animals adjust to it far better than their owners expect. Some causes of vision loss, glaucoma for one, can be painful, which is one more reason a veterinarian should look at any change in the eyes. But the loss of sight itself is not the same thing as misery. Plenty of blind cats eat, groom, play, purr, and demand attention at four in the morning with undiminished authority. Myth two: it's kinder not to try. This is the one that worries me most. It's usually said by someone trying to be gentle, and it lands on an owner who is already exhausted. But "kinder" is a judgment about the animal's inner life, and we are not in a position to make that call by looking at the eyes alone. The better questions are the ordinary ones. Is the animal eating? Moving? Engaging? Responding to affection? Free from pain? A veterinarian can help weigh those. Blindness alone is not the answer to any of them. Myth three: you have to rearrange your life. Not really. You mostly have to stop rearranging it. The great gift you can give a blind animal is consistency. Keep the water where it was. Keep the litter where it was. Keep the chair where the chair was. Let the house become a stable language again. Myth four: they can't play. Anyone who has watched a blind cat hunt a toy that makes noise knows how quickly that one falls apart. The brain does not care which sense delivers the chase. Myth five, and the quietest: there's nothing you can do. That's the myth I wish more people would examine, because it's the one that turns love into helplessness. There's almost always something. It might be small. It might be a smell on a doorframe, a rug that grips, a word you say before you touch. The size of the thing is not the point. The point is that the cat can feel somebody has been thinking about it. And here's what strikes me: that last sentence is true for people, too. Think about the people in your life who are going quiet at the edges. A grandparent whose hearing is going. A friend who has stopped driving at night. A parent who has started saying "I'm fine" a little too fast. What they need, more often than a solution, is evidence that someone has been thinking about the problem from their side of it. Designers call this empathy and put it in a slide deck. Animals call it Tuesday. I think that's why content like this travels. Not because it's cute. Plenty of cute things come and go in a day. It travels because it hands the viewer a feeling that's hard to find anywhere else on a screen: the feeling that attention, applied patiently to a small problem, can change what a day feels like for another living being. Most of the feed is built to do the opposite. It's built to make you react, then move on. This is the rare thing that makes you want to do something, even if the something is only to look closer at the animal sleeping on your own couch and notice that it is, actually, still here. I want to be careful, though, because there's a lazy way to talk about this and I'd like to avoid it. The lazy way says: look how inspiring. A cat overcame its disability. A human saved the day. Neither of those framings is quite right. The cat didn't overcome anything in the dramatic, cinematic sense. It adapted, the way animals have always adapted, because adapting is the only option on offer. And the human didn't save the day. The human made a day a little easier. Those are smaller claims than the internet likes. They are also truer. And true things have a strange staying power. You can feel the difference. So what is this video, exactly? It isn't a rescue in the sense of fire trucks and rope. Nobody dangles from a cliff. There's no villain, no clock, no swelling music. Whatever is at stake is quieter than that and, I'd argue, more universal: the stake is one animal's willingness to take the next step into a room it can't see. That's it. That's the whole drama. And yet I've watched people who are completely immune to dramatic footage go very still during this kind of moment. I think it's because everyone has, at some point, stood at the edge of a dark room, holding something that might help or might not, wondering whether to trust it. If you've ever lived with an animal that's getting old, you know a particular quiet fear. It doesn't arrive as a crisis. It arrives as an accumulation. A little less jumping. A little more sleeping. A little hesitation at the top of the stairs. One day you realize you've been planning your whole house around a smaller and smaller version of the animal you adopted, and you haven't said it to anyone. That fear is one of the most common and least discussed experiences of loving a pet. People carry it alone because it feels disloyal to say it out loud. As if naming the decline might speed it up. So when someone shows a way to push that decline back, even by an inch, it matters far beyond the product or the platform. It says: you are allowed to try. You are allowed to be inventive about this. The ending isn't written on the first bad day. I want to say something about senior animals specifically, because they've been on my mind. We celebrate beginnings in a way that's almost ridiculous. The kitten photo gets a thousand likes before breakfast. But the animals who've been with us for ten, fourteen, sixteen years have earned something the kittens haven't yet: a shared history. They know the sound of your key. They know which of your moods is the one to avoid. They've watched you through a breakup, a move, a layoff, a loss. They are, in a very real sense, witnesses. And witnesses deserve care when their own eyes begin to fail. That's the emotional math underneath all of this, and it's why these small clips of small solutions hit harder than they should. We aren't only watching a cat. We are watching a debt being paid. The animal gave years of presence, and now, finally, someone is giving something back in the only currency it can read. Is it sentimental? Of course. I'm not going to pretend otherwise. But sentiment isn't the enemy of truth. Sometimes it's the only instrument sensitive enough to detect it. Now, a few practical words, because it would be strange to talk this much about blind pets and say nothing useful. If your cat or dog seems to be losing vision, see a vet. Soon. Some causes are treatable and time-sensitive, and only an exam can tell the difference. If vision loss is confirmed, consistency is your best tool. Don't move the furniture. Keep food, water, and litter in the same places. Talk as you approach so you never surprise them. Use scent and texture as signposts: a different mat at the top of the stairs, a dab of a safe, pet-friendly scent near a doorway. Block off stairs and drops until you're sure they've learned them. Give them time. Then give them more time. And let them be braver than you expect. Over-protecting an animal can be its own kind of cage. That's the advice. The rest is attention. It's worth noticing how little time this takes. The whole thing runs about fourteen seconds, shorter than most of the ads you scrolled past this morning. Yet it carries a full arc: a problem, a risk, a decision, and a result that no caption can fully hold. Short videos usually trade depth for speed. Every so often, one manages to keep both, and those are the ones worth sending to the one person who will understand exactly why you sent it. Which brings me back to the bar I mentioned earlier, the one I said we'd return to. The bar of whether an animal will actually walk, not merely tolerate. Whether it will move with the looseness you can't fake. I think that's the most honest test of any act of care, human or otherwise. Not whether the person receiving it says thank you. Whether they move differently afterward. Whether their shoulders drop. Whether they take the step. Each step, taken freely, can say more about trust than anything an animal could ever put into words. I won't tell you how this one ends. The video is short, and anything I said would only stand between you and the moment itself. But I'll tell you what to watch for, and it's not what's on the surface. Watch the way the animal decides. Not the object. Not the hands. The deciding. There is a flicker in it, a half-second where trust is either extended or withheld, and I haven't been able to shake it. It's the same flicker that happens in all of us when we've been hurt by the dark before and someone offers us a light we've never tried. Some of us don't take it. Some of us do. Go and see which kind of animal this is.show more

AI has had exactly two scaling axes that worked... so far, and the second one is starting to look finite too the first one was pretraining: with scaling parameters and data, we got world knowledge (i.e. ChatGPT had read enough to know things), but it started saturating a while ago the second one was RL, and people had been doing RL the whole time before that: RLHF is RL but it never scaled far because it was trying to control the exact output, which tokens come out, how the text reads, but you can only push that so far before you’re just polishing RLVR dropped that constraint: giving the model a task, then checking whether the final answer is right, and ignoring everything in between -- so the model does whatever it wants in the middle and only the endpoint gets graded, and that’s much closer to actual RL and it’s what bought us planning and reasoning (arguably, tool use sits around 2.5 on this list -- while useful, it's not a different kind of thing) so one axis gave knowledge, the other gave reasoning, and both of them are one model working alone the next axis is how many models you can get working on the same problem, which is a different kind of axis than the previous two we know that multi-agent RL has always been the harder problem: I spent years in that literature and the gap between single-agent and multi-agent is definitely not incremental -- it’s a whole different class of difficulty! which is also why the derivatives are steep at the start, nobody has picked the easy wins yet... and the thing that gates this multi-agent coordination is communication: models can only coordinate as well as they can exchange information, and right now they do that by writing sentences to each other imagine what could we possibly achieve if we properly open that third axis development by letting models to exchange information in their native "language" without loosing any computational data that they produce during inferenceshow more

There is only one reason to share truth. And... no it has nothing to do with money- and everything to do about helping God move. Peace is the prize- a better world for everyone is the prize. We make SACRIFICES for things we want most in this world. Christ made the ultimate sacrifice If He can do that for us FREE of charge- we owe it to each other to work TOGETHER to make the world a better place- despite the toll and sacrifice. A level playing field- where we can all work hard and do our best to succeed- that is the payout. A better world for all We’re not here for just ourselves- otherwise the second commandment wouldn’t be like the first- love your neighbor as yourself. This world gets better the sooner we all wake up. Information is free- we collect and share it for free in hopes that it keeps spreading. Only working TOGETHER can we win. We are meant to move these messages for God so that humanity can wake up- this plan is designed that way- working together The Lord is ONE. We are bound as a family by Our Father in Heaven. We are apart of Him and He is apart of all of us. We are billions of little pieces of the Lords gift of life. We’ve been separated and pushed apart because it makes us weak. We were designed to be powerful by the love we draw from Creator and push through creation. We are ONE with Our Holy Father 🙏💗 Be EXCELLENT to each other frensshow more

Transformer: Multi-Head Attention ~ Math vs Code 🔢💻 ~... I made this visualization to show you how to implement the multi-head attention math in PyTorch within 50 LoC. Multi-Head Attention is what makes the Transformer's performance outstanding. It captures and represents more diverse linguistic relationships and patterns, and attends to different learned input embedding spaces. The parallel computing design also makes the model more efficient.show more

Sharing some more UI previews for Superlogical, this time... the "tab peek." If you three-finger swipe down, you can "peek" at the other session tabs. These are live-updating, metal-rendered previews including splits. If you keep dragging, you enter a "mission control" style mode. This is one of those UI patterns that once you get used to, you mindlessly find yourself doing it ALL THE TIME. It's a great way to quickly take a look at everything thats going on. You aren't reading the mini text of course, but the shape of it tells you a lot about where things are. The peek pushes the terminal down off screen, it doesn't resize it, so it doesn't force any weird reflow or anything either. There's a lot more to talk about, especially about how this interacts with how we expose what needs attention so you can easily find the terminals/sessions/etc. that require your attention. Of course, you'll be able to drive this with the keyboard too. :) As always, a disclaimer: this is all early work subject to change, just trying to share some of what we're building in public. A lot of the UI experiential design is almonk so follow him for more updates too!show more

the order of these songs is so random, should... we pay attention to it???!!show more

One of the odd things about learning mathematics is... how long you can manipulate an equation without having much of a picture of it in your head. You learn the notation, follow the derivation, get the answer, and move on. Then somebody shows you the same idea visually and you realize there was a geometric intuition hiding underneath the symbols the whole time. A Fourier sum becomes rotating vectors added head-to-tail. Attention becomes a weighted combination you can watch being formed. SVD becomes a sequence of transformations you can follow one step at a time. Even the equation behind a diffusion process feels different when you can see a point following a changing vector field. None of this replaces the mathematics. The equations are still where the precision lives. But once you have a picture in your head, the notation has something to attach itself to. I think this is why some mathematical ideas suddenly become much easier years after we first learned them. We finally stop seeing the equation as a line of symbols to manipulate and start seeing the process it describes. I wish more mathematics had been taught that way.show more