正在加载视频...

视频加载失败

返回首页

For years, AdamW has been the default optimizer behind every LLM training run on the planet. A new optimizer called Muon is twice as computationally efficient - and the core idea is embarrassingly elegant: orthogonalize the momentum matrix. The same singular value decomposition that powers every matrix operation inside...

0xDeliriumm's profile picture

Deliriumm

4,790 subscribers

14,292 次观看 • 12 天前 •via X (Twitter)

科学技术教育
LIVE

Anya Rossi• Live Now

Private livecam show

10 条评论

Alexandru-Andrei Rotaru 的头像
Alexandru-Andrei Rotaru11 天前

I am genuinely impressed by the solution's technical ingenuity. However, I want to point out a broader issue that has been bothering me for a while: the math is decoupled from real-world practicalities, and that has profound drawbacks. How is an algorithm (more efficient) genuinely better when I spend most of my time figuring out regression and improvements in the verticals of the task domain? No amount of clever math will fix the fact that your dataset underrepresents certain cases. To make an analogy, if you make one flight more fuel-efficient to cut CO2 emissions, you end up optimising a local element, while more and more people travel because of "less costly flights", cancelling the savings or even increasing overall CO2 emissions. From what I can tell, we keep making "one flight more fuel efficient" (a.k.a. smarter training algos), while the overall trustworthiness and commercial viability of the products don't really improve, and may even get worse.

Michael 的头像
Michael12 天前

New? muon has been around for a while

Jia-Bin Huang 的头像
Jia-Bin Huang11 天前

Bro…

Dev Patel 的头像
Dev Patel12 天前

i learnt so much in these 17 minutes though it will take time to full absorb in , great video !!

Hudson Gouge 的头像
Hudson Gouge11 天前

lol. Are you from 2024?

karter 的头像
karter11 天前

the part that gets left out: muon only covers the 2d matrices. embeddings, biases, norms still run on adamw in practice. so it's a hybrid, not a replacement. that changes what the 2x claim can mean.

ReignSwords 的头像
ReignSwords12 天前

The SVD angle is what makes it click for me — instead of letting momentum drift in every direction, Muon keeps the update close to orthogonal. Curious how much of that 2x holds up outside large transformer runs, though — has anyone seen it shine on smaller models too?

Alex 的头像
Alex11 天前

Awesome! 🔥🔥🔥

shubh 的头像
shubh11 天前

I can understand every bit of this .

AI Quanting 的头像
AI Quanting12 天前

Is the 2x figure from the original small runs, or has anyone shown it holding at frontier scale? That's the part I keep looking for. Optimizer wins have a habit of shrinking as the token budget grows.

相关视频

Eigenvalues are why Google's PageRank converges, why PCA compresses 1,000-dimensional embeddings into 2D without losing structure, and why quantum mechanics gives discrete energy levels instead of a continuous blur. The same matrix operation that finds the most important directions in a dataset is what transformer attention heads use to rotate and project token embeddings into query-key space. Every LLM response you've ever read passed through dozens of eigenvalue decompositions. 22 minutes. Bookmark & watch today. The most useful operation in all of applied mathematics.
20:56

Eigenvalues are why Google's PageRank converges, why PCA compresses 1,000-dimensional embeddings into 2D without losing structure, and why quantum mechanics gives discrete energy levels instead of a continuous blur. The same matrix operation that finds the most important directions in a dataset is what transformer attention heads use to rotate and project token embeddings into query-key space. Every LLM response you've ever read passed through dozens of eigenvalue decompositions. 22 minutes. Bookmark & watch today. The most useful operation in all of applied mathematics.

Eron

21,712 次观看 • 2 个月前

Every fundamental law of physics - Newton, Einstein, Quantum Field Theory - is derived from a single equation. Physicists don't propose new laws by writing forces anymore. They write a new Lagrangian and the math produces the physics automatically. The same principle that governs a swinging pendulum governs every quantum field in the universe - and the optimizer inside every LLM training run is solving the same underlying problem. The mechanism is stationary action. A system takes the path where the total action stops changing for any small variation. That condition produces the Euler-Lagrange equation. One equation, any number of variables, any theory of physics. In Quantum Field Theory the variables are fields across all of space and the gradient must vanish at every point simultaneously - exactly the convergence condition gradient descent hunts for across billions of LLM parameters during backpropagation. 20 minutes. Bookmark & watch today. The single equation that every law of physics is secretly a special case of.
18:21

Every fundamental law of physics - Newton, Einstein, Quantum Field Theory - is derived from a single equation. Physicists don't propose new laws by writing forces anymore. They write a new Lagrangian and the math produces the physics automatically. The same principle that governs a swinging pendulum governs every quantum field in the universe - and the optimizer inside every LLM training run is solving the same underlying problem. The mechanism is stationary action. A system takes the path where the total action stops changing for any small variation. That condition produces the Euler-Lagrange equation. One equation, any number of variables, any theory of physics. In Quantum Field Theory the variables are fields across all of space and the gradient must vanish at every point simultaneously - exactly the convergence condition gradient descent hunts for across billions of LLM parameters during backpropagation. 20 minutes. Bookmark & watch today. The single equation that every law of physics is secretly a special case of.

Eron

24,223 次观看 • 28 天前

1/9 Current optimizers are static during training. What if the optimizer could think and react while the network trains? We let an agent watch the training dynamics and rewrite the optimizer's source code as training goes: a reactive optimizer. In every language and image setting we tried that has no tuned recipe, it matches or beats the best recipe one can one-shot, and it guards against loss spikes and overfitting by itself. In the near future, optimization decisions may belong to agents, not humans. 📝 💻 Try it on your task: 🧵
0:24

1/9 Current optimizers are static during training. What if the optimizer could think and react while the network trains? We let an agent watch the training dynamics and rewrite the optimizer's source code as training goes: a reactive optimizer. In every language and image setting we tried that has no tuned recipe, it matches or beats the best recipe one can one-shot, and it guards against loss spikes and overfitting by itself. In the near future, optimization decisions may belong to agents, not humans. 📝 💻 Try it on your task: 🧵

Xingjian Bai

216,932 次观看 • 15 天前

MIT FILMED THE LECTURE WHERE THEIR MOST LEGENDARY LINEAR ALGEBRA PROFESSOR EXPLAINS WHY EVERY GPS SIGNAL EVERY MEDICAL SCAN AND EVERY AI MODEL ON EARTH IS SECRETLY SOLVING A PROBLEM THAT HAS NO SOLUTION This is Gilbert Strang, MIT, 18.06 Linear Algebra, lecture 15. The professor whose textbooks sit in over 3,000 universities across 135 countries and whose course has been watched by millions of engineers and mathematicians worldwide. He opens with one claim - when equations have no solution, you do not give up. You solve the closest problem that can be solved. He starts with a vector b and a line through a. The projection of b onto that line is the point closest to b that actually lives on the line. The key is a right angle. The error - the gap between b and its projection - must be perpendicular to a. That one geometric fact produces the entire formula. The projection is a times a transpose b divided by a transpose a. One fraction. The numerator is a matrix. The denominator is a number. Then the projection matrix. Every projection is carried out by a matrix P that multiplies b and lands on the line. That matrix has two properties that define all projections everywhere. First, P transpose equals P - the matrix is symmetric. Second, P squared equals P - project twice and you stay where you are. The first projection moves you onto the line. The second leaves you there. Then higher dimensions. Instead of projecting onto a line, project onto a plane. Instead of one vector a, you have a matrix A with two columns spanning the plane. The error vector must now be perpendicular to both columns - perpendicular to the entire plane. That condition produces one matrix equation. A transpose times the error equals zero. The error lives in the null space of A transpose, which is exactly perpendicular to the column space of A. The four fundamental subspaces appear without being called. Then the formula for n dimensions. The projection matrix is A times A transpose A inverse times A transpose. Strang warns the room not to simplify it by canceling A inverse against A - A is not square, it has no inverse, and the formula cannot be pulled apart. The only case where it simplifies to the identity is when A spans the whole space and the projection is b itself. Watch the moment he applies the formula to fitting a line through three points that no line can pass through. Three equations, two unknowns, no solution. He writes the matrix, writes the right-hand side, and shows that multiplying both sides by A transpose produces an equation that always has a solution. That equation is the foundation of least squares, GPS triangulation, and every regression model trained in 2026. A data scientist I know rewatched this lecture before building a recommendation system from scratch. Said it was the first time the normal equations felt like geometry rather than a formula someone handed down to memorize. Free on YouTube, MIT OpenCourseWare, filmed over 20 years ago. bookmark this and watch later - after this lecture every unsolvable equation you meet will feel like a projection waiting to be found
48:19

MIT FILMED THE LECTURE WHERE THEIR MOST LEGENDARY LINEAR ALGEBRA PROFESSOR EXPLAINS WHY EVERY GPS SIGNAL EVERY MEDICAL SCAN AND EVERY AI MODEL ON EARTH IS SECRETLY SOLVING A PROBLEM THAT HAS NO SOLUTION This is Gilbert Strang, MIT, 18.06 Linear Algebra, lecture 15. The professor whose textbooks sit in over 3,000 universities across 135 countries and whose course has been watched by millions of engineers and mathematicians worldwide. He opens with one claim - when equations have no solution, you do not give up. You solve the closest problem that can be solved. He starts with a vector b and a line through a. The projection of b onto that line is the point closest to b that actually lives on the line. The key is a right angle. The error - the gap between b and its projection - must be perpendicular to a. That one geometric fact produces the entire formula. The projection is a times a transpose b divided by a transpose a. One fraction. The numerator is a matrix. The denominator is a number. Then the projection matrix. Every projection is carried out by a matrix P that multiplies b and lands on the line. That matrix has two properties that define all projections everywhere. First, P transpose equals P - the matrix is symmetric. Second, P squared equals P - project twice and you stay where you are. The first projection moves you onto the line. The second leaves you there. Then higher dimensions. Instead of projecting onto a line, project onto a plane. Instead of one vector a, you have a matrix A with two columns spanning the plane. The error vector must now be perpendicular to both columns - perpendicular to the entire plane. That condition produces one matrix equation. A transpose times the error equals zero. The error lives in the null space of A transpose, which is exactly perpendicular to the column space of A. The four fundamental subspaces appear without being called. Then the formula for n dimensions. The projection matrix is A times A transpose A inverse times A transpose. Strang warns the room not to simplify it by canceling A inverse against A - A is not square, it has no inverse, and the formula cannot be pulled apart. The only case where it simplifies to the identity is when A spans the whole space and the projection is b itself. Watch the moment he applies the formula to fitting a line through three points that no line can pass through. Three equations, two unknowns, no solution. He writes the matrix, writes the right-hand side, and shows that multiplying both sides by A transpose produces an equation that always has a solution. That equation is the foundation of least squares, GPS triangulation, and every regression model trained in 2026. A data scientist I know rewatched this lecture before building a recommendation system from scratch. Said it was the first time the normal equations felt like geometry rather than a formula someone handed down to memorize. Free on YouTube, MIT OpenCourseWare, filmed over 20 years ago. bookmark this and watch later - after this lecture every unsolvable equation you meet will feel like a projection waiting to be found

Lupen

24,677 次观看 • 3 个月前

In 1830, Liouville proved the area under the most important curve in mathematics has no formula. Not because nobody was clever enough - because the formula was never there to be found. Then somebody squared the problem, lifted it into a second dimension, and a circle walked into the exponent entirely on its own. The area is the square root of π - the same √π that normalizes every probability distribution inside every LLM and every diffusion model on every GPU cluster today. The curve is e to the minus x squared - the bell curve De Moivre discovered in 1733 solving a coin-flip problem. One dimension and every integration method slides off it. Square the integral, rename one variable to y, and x² + y² appears in the exponent - Pythagoras hiding inside a function with no circles anywhere. Switch to polar coordinates and geometry hands you a free factor of r. The infinite hill collapses to π. That factor now sits inside every softmax normalization, every Gaussian noise layer in stable diffusion training, every loss function a transformer optimizes during backpropagation. 20 minutes. Bookmark & watch today. The proof that a circle was hiding inside a straight line the entire time.
19:47

In 1830, Liouville proved the area under the most important curve in mathematics has no formula. Not because nobody was clever enough - because the formula was never there to be found. Then somebody squared the problem, lifted it into a second dimension, and a circle walked into the exponent entirely on its own. The area is the square root of π - the same √π that normalizes every probability distribution inside every LLM and every diffusion model on every GPU cluster today. The curve is e to the minus x squared - the bell curve De Moivre discovered in 1733 solving a coin-flip problem. One dimension and every integration method slides off it. Square the integral, rename one variable to y, and x² + y² appears in the exponent - Pythagoras hiding inside a function with no circles anywhere. Switch to polar coordinates and geometry hands you a free factor of r. The infinite hill collapses to π. That factor now sits inside every softmax normalization, every Gaussian noise layer in stable diffusion training, every loss function a transformer optimizes during backpropagation. 20 minutes. Bookmark & watch today. The proof that a circle was hiding inside a straight line the entire time.

Eron

14,069 次观看 • 15 天前

Self Attention by hand ✍️ ~ 9 steps walkthrough below Self-attention is what enables LLMs to understand context. How does it work? So I drew and calculated one entirely by hand. Goal: turn four 6D features into four 3D attention weighted features, filling in every cell yourself. = 1. Given = Four feature vectors, six dimensions each, one per position. = 2. Query, key, value = Let us multiply the features by WQ, WK and WV. Queries, keys and values all come out of the same four features, and that is what the word "self" is doing in self-attention. = 3. Prepare for MatMul = We copy the queries across the top and the transposed keys down the side. Lining the two up is half the work. = 4. MatMul = Let us multiply K transpose by Q. Every cell is the dot product of one key with one query, which we use as a matching score. That works because the dot product is the numerator of cosine similarity: it is how alike two vectors are, before anyone divides by their lengths. = 5. Scale = We divide by the square root of dk, the dimension of a key vector, here 3. Without it the scores grow with the dimension and a 64-wide head would swamp the softmax. To keep the page doable in pen, the drawing approximates dividing by root 3 with halving. = 6. e to the power = Let us raise e to the power of each score. This is the first half of softmax, and the drawing uses 3 in place of e, which is close enough to do in your head. = 7. Sum = We add up each column: 16, 6, 7 and 12. = 8. Normalize = Let us divide every cell by its column sum. That gives the attention weight matrix in yellow, and each of its four columns is now a probability distribution over the four positions. The decimals are nudged as they are rounded, so every column still sums to exactly 1. = 9. MatMul = We multiply the value vectors by those weights. Each output is a blend of all four values, mixed in the proportion the attention matrix just decided, and it goes to the position-wise feed forward network in the next layer: the FFN box at the bottom of the page. The outputs: Attention weights (A), by column = [.2, .6, 0, .2], [.2, .4, .2, .2], [.4, .2, 0, .4], [.1, .7, .1, .1] Attention weighted features (Z) = [8, 2, 6], [8, 4, 4], [16, 4, 2], [4, 2, 7] The takeaway: attention is a weighted average, and everything before step 9 exists to decide the weights. Compare every position with every other, turn the scores into one distribution per position, then blend. 💾 Save this post!
0:22

Self Attention by hand ✍️ ~ 9 steps walkthrough below Self-attention is what enables LLMs to understand context. How does it work? So I drew and calculated one entirely by hand. Goal: turn four 6D features into four 3D attention weighted features, filling in every cell yourself. = 1. Given = Four feature vectors, six dimensions each, one per position. = 2. Query, key, value = Let us multiply the features by WQ, WK and WV. Queries, keys and values all come out of the same four features, and that is what the word "self" is doing in self-attention. = 3. Prepare for MatMul = We copy the queries across the top and the transposed keys down the side. Lining the two up is half the work. = 4. MatMul = Let us multiply K transpose by Q. Every cell is the dot product of one key with one query, which we use as a matching score. That works because the dot product is the numerator of cosine similarity: it is how alike two vectors are, before anyone divides by their lengths. = 5. Scale = We divide by the square root of dk, the dimension of a key vector, here 3. Without it the scores grow with the dimension and a 64-wide head would swamp the softmax. To keep the page doable in pen, the drawing approximates dividing by root 3 with halving. = 6. e to the power = Let us raise e to the power of each score. This is the first half of softmax, and the drawing uses 3 in place of e, which is close enough to do in your head. = 7. Sum = We add up each column: 16, 6, 7 and 12. = 8. Normalize = Let us divide every cell by its column sum. That gives the attention weight matrix in yellow, and each of its four columns is now a probability distribution over the four positions. The decimals are nudged as they are rounded, so every column still sums to exactly 1. = 9. MatMul = We multiply the value vectors by those weights. Each output is a blend of all four values, mixed in the proportion the attention matrix just decided, and it goes to the position-wise feed forward network in the next layer: the FFN box at the bottom of the page. The outputs: Attention weights (A), by column = [.2, .6, 0, .2], [.2, .4, .2, .2], [.4, .2, 0, .4], [.1, .7, .1, .1] Attention weighted features (Z) = [8, 2, 6], [8, 4, 4], [16, 4, 2], [4, 2, 7] The takeaway: attention is a weighted average, and everything before step 9 exists to decide the weights. Compare every position with every other, turn the scores into one distribution per position, then blend. 💾 Save this post!

Tom Yeh

27,383 次观看 • 3 个月前

Attention is a lookup. Each token builds a query, compares it against every key in the sequence, and pulls value vectors weighted by the match. Stack that 96 layers deep and you get a frontier model. Video covers the full pipeline: Q/K/V, attention scores, encoder blocks.
17:32

Attention is a lookup. Each token builds a query, compares it against every key in the sequence, and pulls value vectors weighted by the match. Stack that 96 layers deep and you get a frontier model. Video covers the full pipeline: Q/K/V, attention scores, encoder blocks.

tetsuo

106,557 次观看 • 4 个月前

In 1957, Stephen Smale proved you can turn a sphere inside out without cutting, tearing, or creasing it - even though you can't do it with a circle. The proof was so abstract it took 7 years before anyone found a practical method. The key is a topological invariant called the turning number - the same class of invariants that every LLM uses when its embedding layers preserve structure across high-dimensional transformations on GPU clusters. The trick is corrugation. Divide the sphere into guide strips connected by wavy segments - the waviness gives enough flexibility for strips to twist past each other without creasing. Push the polar caps through, twist them in opposite directions to convert loops into twisting, then push the equator back through the center. 16 rotated copies of one building block construct the entire eversion. A circle can't do this because its turning number changes from +1 to -1 when reversed. Spheres have an extra dimension where saddles, bowls, and domes interact - their invariant stays the same regardless of which face is out. The same dimensional escape that lets attention heads resolve contradictions in higher-dimensional latent space that look impossible in any lower projection. 19 minutes. Bookmark & watch today. The proof that spheres break a rule circles never can.
19:13

In 1957, Stephen Smale proved you can turn a sphere inside out without cutting, tearing, or creasing it - even though you can't do it with a circle. The proof was so abstract it took 7 years before anyone found a practical method. The key is a topological invariant called the turning number - the same class of invariants that every LLM uses when its embedding layers preserve structure across high-dimensional transformations on GPU clusters. The trick is corrugation. Divide the sphere into guide strips connected by wavy segments - the waviness gives enough flexibility for strips to twist past each other without creasing. Push the polar caps through, twist them in opposite directions to convert loops into twisting, then push the equator back through the center. 16 rotated copies of one building block construct the entire eversion. A circle can't do this because its turning number changes from +1 to -1 when reversed. Spheres have an extra dimension where saddles, bowls, and domes interact - their invariant stays the same regardless of which face is out. The same dimensional escape that lets attention heads resolve contradictions in higher-dimensional latent space that look impossible in any lower projection. 19 minutes. Bookmark & watch today. The proof that spheres break a rule circles never can.

Eron

28,367 次观看 • 8 天前

In 1747, Euler spent 133 pages proving that logarithms have infinitely many solutions - and in doing so accidentally revealed why every LLM, every signal processor, and every Fourier transform works the way it does. The imaginary unit I doesn't break exponential functions. It rotates them, turning exponential growth into the sine and cosine waves that live inside every transformer attention layer. The proof runs through Briggs' 1617 zooming algorithm, complex polar coordinates, and a thin-triangle argument that connects imaginary exponents to angles in radians. When you plug I into the exponential function, you don't get growth - you get rotation confined to the unit circle. The same unit circle that softmax maps every attention score onto before the GPU multiplies them. 27 minutes. Bookmark & watch today. The most beautiful equation in mathematics and why it powers every model you have ever used.
26:53

In 1747, Euler spent 133 pages proving that logarithms have infinitely many solutions - and in doing so accidentally revealed why every LLM, every signal processor, and every Fourier transform works the way it does. The imaginary unit I doesn't break exponential functions. It rotates them, turning exponential growth into the sine and cosine waves that live inside every transformer attention layer. The proof runs through Briggs' 1617 zooming algorithm, complex polar coordinates, and a thin-triangle argument that connects imaginary exponents to angles in radians. When you plug I into the exponential function, you don't get growth - you get rotation confined to the unit circle. The same unit circle that softmax maps every attention score onto before the GPU multiplies them. 27 minutes. Bookmark & watch today. The most beautiful equation in mathematics and why it powers every model you have ever used.

Deliriumm

128,190 次观看 • 1 个月前

Kimi CEO Zhilin Yang: "a smart agent isn't enough - it has to run long Kimi 3 new attention makes the context 10x faster, so the agent never has to stop" here's how built it: step 1 → chase token efficiency, not compute - Muon optimizer, 2x more learning per token step 2 → make context cheap, not just long - new attention runs 10x faster, so the agent never stalls step 3 → let it run 200-300 steps alone - K2 already solves problems its own builders can't step 4 → give it a strong prior - the agent searches less and finds the answer faster step 5 → stop chasing a smarter model - chase one that never has to stop in 26-min talk is him revealing the exact architecture behind K3 and how to properly build agents on it watch & bookmark - start with the article below ↓
18:00

Kimi CEO Zhilin Yang: "a smart agent isn't enough - it has to run long Kimi 3 new attention makes the context 10x faster, so the agent never has to stop" here's how built it: step 1 → chase token efficiency, not compute - Muon optimizer, 2x more learning per token step 2 → make context cheap, not just long - new attention runs 10x faster, so the agent never stalls step 3 → let it run 200-300 steps alone - K2 already solves problems its own builders can't step 4 → give it a strong prior - the agent searches less and finds the answer faster step 5 → stop chasing a smarter model - chase one that never has to stop in 26-min talk is him revealing the exact architecture behind K3 and how to properly build agents on it watch & bookmark - start with the article below ↓

codila

12,037 次观看 • 3 个月前

A 91-year-old professor is why Nvidia is worth $4 trillion. His name is Gilbert Strang. He teaches linear algebra at MIT. Every AI model on Earth runs on his course. The course has been free on YouTube since 2005. The videos have earned him nothing. MIT 18.06 opens with "The Geometry of Linear Equations." No advanced math. Strang takes a system of two equations, draws it two ways, and shows the class that a matrix is a picture, not an abstraction. The row picture is two lines that cross. The column picture is two arrows that sum to a target. Every neural network on Earth operates on the column picture. Strang first taught linear algebra at MIT in 1962. He wrote the textbook in 1976. It is on every serious engineer's shelf. Every quant fund, every ML lab, every rendering engine at Pixar is running his math. His central insight is that most people are taught matrices as bookkeeping. That is the first thing to unlearn. A matrix is a linear transformation. A linear transformation is a way of moving space. Once you see the space move, the math stops being algebra and becomes geometry. The Kalman filter is a linear system. PCA is a linear system. Every gradient step in a neural net is a matrix-vector product. GPT is a stack of matrix-vector products, each one a scene from MIT 18.06 running on a Blackwell GPU. He retired in 2023 after 61 years at MIT. The course is still up. Watched tens of millions of times. The chip is $40,000. Strang never asked for a royalty.
37:24

A 91-year-old professor is why Nvidia is worth $4 trillion. His name is Gilbert Strang. He teaches linear algebra at MIT. Every AI model on Earth runs on his course. The course has been free on YouTube since 2005. The videos have earned him nothing. MIT 18.06 opens with "The Geometry of Linear Equations." No advanced math. Strang takes a system of two equations, draws it two ways, and shows the class that a matrix is a picture, not an abstraction. The row picture is two lines that cross. The column picture is two arrows that sum to a target. Every neural network on Earth operates on the column picture. Strang first taught linear algebra at MIT in 1962. He wrote the textbook in 1976. It is on every serious engineer's shelf. Every quant fund, every ML lab, every rendering engine at Pixar is running his math. His central insight is that most people are taught matrices as bookkeeping. That is the first thing to unlearn. A matrix is a linear transformation. A linear transformation is a way of moving space. Once you see the space move, the math stops being algebra and becomes geometry. The Kalman filter is a linear system. PCA is a linear system. Every gradient step in a neural net is a matrix-vector product. GPT is a stack of matrix-vector products, each one a scene from MIT 18.06 running on a Blackwell GPU. He retired in 2023 after 61 years at MIT. The course is still up. Watched tens of millions of times. The chip is $40,000. Strang never asked for a royalty.

Ochob

131,686 次观看 • 3 个月前

Ever wondered what neural networks are and how they work? Systems like ChatGPT use neural networks to work as well as they do. Neural networks are composed of "layers" of neurons, layers with different functions; connections between layers called "weights"; and mathematical functions called "activation functions". If you’re interested in learning about these systems, check the comments. Ultimately, the neural network structure of the model serves to visually demonstrate that it is, in fact, a complex mathematical equation. When companies release the model's weights, they are releasing a key component needed to run the model's complete equation. Without the weights, the equation is incomplete. For the math-minded: the weights of a model are the learned numbers (they are variables during training) that are then used as constants in the mathematical functions that make up the model. Neural networks are ultimately just one big, hyper-complex mathematical function, and when a model is trained, it learns the constants associated with the high-dimensional input.
2:37

Ever wondered what neural networks are and how they work? Systems like ChatGPT use neural networks to work as well as they do. Neural networks are composed of "layers" of neurons, layers with different functions; connections between layers called "weights"; and mathematical functions called "activation functions". If you’re interested in learning about these systems, check the comments. Ultimately, the neural network structure of the model serves to visually demonstrate that it is, in fact, a complex mathematical equation. When companies release the model's weights, they are releasing a key component needed to run the model's complete equation. Without the weights, the equation is incomplete. For the math-minded: the weights of a model are the learned numbers (they are variables during training) that are then used as constants in the mathematical functions that make up the model. Neural networks are ultimately just one big, hyper-complex mathematical function, and when a model is trained, it learns the constants associated with the high-dimensional input.

Harper Carroll

50,905 次观看 • 1 年前

In 1958, the US Navy stood Frank Rosenblatt in front of a $2 million IBM computer and told reporters it was the embryo of a machine that would someday be conscious of its own existence. His entire idea fit on one line of math. In 1969, two MIT researchers published a single puzzle that killed the funding for a decade. The line of math survived anyway - it runs inside every neuron in every LLM deployed today, copied billions of times and stacked in layers. The formula is a weighted sum followed by a nonlinearity: step of W dot X plus B. Draw it geometrically and it's a straight line dividing space into ones and zeros. The learning rule nudges the weights whenever it's wrong. Minsky and Papert proved it could never solve XOR - four points on a square where no straight line separates the classes. The fix was stacking layers and training them with backpropagation, published in Nature in 1986 by Rumelhart, Hinton, and Williams. Hinton shared the 2024 Nobel Prize in Physics for it. Rosenblatt drowned in a boating accident in 1971 on his 43rd birthday, 15 years before the solution arrived. 5 minutes. Bookmark & watch today. The one line of 1958 math that every transformer, every LLM, and every GPU on earth still runs on.
5:23

In 1958, the US Navy stood Frank Rosenblatt in front of a $2 million IBM computer and told reporters it was the embryo of a machine that would someday be conscious of its own existence. His entire idea fit on one line of math. In 1969, two MIT researchers published a single puzzle that killed the funding for a decade. The line of math survived anyway - it runs inside every neuron in every LLM deployed today, copied billions of times and stacked in layers. The formula is a weighted sum followed by a nonlinearity: step of W dot X plus B. Draw it geometrically and it's a straight line dividing space into ones and zeros. The learning rule nudges the weights whenever it's wrong. Minsky and Papert proved it could never solve XOR - four points on a square where no straight line separates the classes. The fix was stacking layers and training them with backpropagation, published in Nature in 1986 by Rumelhart, Hinton, and Williams. Hinton shared the 2024 Nobel Prize in Physics for it. Rosenblatt drowned in a boating accident in 1971 on his 43rd birthday, 15 years before the solution arrived. 5 minutes. Bookmark & watch today. The one line of 1958 math that every transformer, every LLM, and every GPU on earth still runs on.

Deliriumm

11,638 次观看 • 25 天前

Gilbert Strang taught linear algebra at MIT for fifty years. His last lecture is on YouTube. It has fewer views than his worst lecture. Nobody told students it was the last one. He just walked in, said "the final class in linear algebra at MIT," and started reviewing old exams. Google built a $2,000,000,000,000 company on one concept from this course. He is 88 years old. He still answers emails. This is MIT 18.06. Linear Algebra. The most watched mathematics course in history. Then the Markov matrix. Strang writes a transition matrix on the board. Three states. People moving between them every step. After enough steps - the system locks into a steady state that never changes. That steady state is an eigenvector. Google's PageRank works the same way. Websites are states. Clicks are transitions. The importance of every page on the internet is one eigenvector of one enormous matrix. Then least squares. Three data points. No line passes through all three. So you find the line that minimizes total error. That is how every AI model on earth is trained - including the ones running inside Goldman Sachs trading desks. Then the projection. The closest point on a plane to a vector outside it. Strang draws it in 30 seconds. That same operation is how Netflix decides what to recommend to 280,000,000 subscribers. Watch the moment he gives back the final exam answer and asks students to work backwards to the question - the room solves it in silence faster than any other lecture all semester. A software engineer told me 18.06 was the course that got her from $90,000 to $200,000 in two years. Same company. Different title. Bookmark this and watch later - after this lecture every dataset you touch will feel like a geometry problem waiting to be solved. MIT 18.06 Lecture 34 | Linear Algebra Final Review | Gilbert Strang
43:06

Gilbert Strang taught linear algebra at MIT for fifty years. His last lecture is on YouTube. It has fewer views than his worst lecture. Nobody told students it was the last one. He just walked in, said "the final class in linear algebra at MIT," and started reviewing old exams. Google built a $2,000,000,000,000 company on one concept from this course. He is 88 years old. He still answers emails. This is MIT 18.06. Linear Algebra. The most watched mathematics course in history. Then the Markov matrix. Strang writes a transition matrix on the board. Three states. People moving between them every step. After enough steps - the system locks into a steady state that never changes. That steady state is an eigenvector. Google's PageRank works the same way. Websites are states. Clicks are transitions. The importance of every page on the internet is one eigenvector of one enormous matrix. Then least squares. Three data points. No line passes through all three. So you find the line that minimizes total error. That is how every AI model on earth is trained - including the ones running inside Goldman Sachs trading desks. Then the projection. The closest point on a plane to a vector outside it. Strang draws it in 30 seconds. That same operation is how Netflix decides what to recommend to 280,000,000 subscribers. Watch the moment he gives back the final exam answer and asks students to work backwards to the question - the room solves it in silence faster than any other lecture all semester. A software engineer told me 18.06 was the course that got her from $90,000 to $200,000 in two years. Same company. Different title. Bookmark this and watch later - after this lecture every dataset you touch will feel like a geometry problem waiting to be solved. MIT 18.06 Lecture 34 | Linear Algebra Final Review | Gilbert Strang

Zyphor

56,603 次观看 • 2 个月前

Complex numbers aren't imaginary. They're the only way to describe a wave that has both amplitude and phase at the same time - and every LLM processes language as exactly that kind of wave through its attention layers. The same math that describes quantum wave interference is what makes Fourier transforms work, which is what lets GPT process tokens in parallel instead of one at a time. Descartes named them "imaginary" before anyone understood what they were. 20 minutes. Bookmark & watch today. The confusion clears up faster than any textbook manages.
19:29

Complex numbers aren't imaginary. They're the only way to describe a wave that has both amplitude and phase at the same time - and every LLM processes language as exactly that kind of wave through its attention layers. The same math that describes quantum wave interference is what makes Fourier transforms work, which is what lets GPT process tokens in parallel instead of one at a time. Descartes named them "imaginary" before anyone understood what they were. 20 minutes. Bookmark & watch today. The confusion clears up faster than any textbook manages.

Eron

43,247 次观看 • 2 个月前

Heinz Hopf discovered this in 1931. Roger Penrose called it "an element of the architecture of our world." It shows up in at least 8 areas of physics, yet most mathematicians spend entire careers never touching it. The structure that defines every qubit in quantum computing, every rotation a robotics controller computes, and the geometry inside every LLM embedding space operating on GPU clusters today - and nobody talks about it. The Hopf fibration maps a 4D hypersphere onto a regular sphere using circles that never intersect but each links through every other exactly once. The Bloch sphere every quantum computer operates on, the fiber bundles underlying gauge theory, and the quaternion math inside every 3D graphics engine and every embedding rotation on the planet. Bookmark & watch today. The most important shape nobody talks about.
2:11

Heinz Hopf discovered this in 1931. Roger Penrose called it "an element of the architecture of our world." It shows up in at least 8 areas of physics, yet most mathematicians spend entire careers never touching it. The structure that defines every qubit in quantum computing, every rotation a robotics controller computes, and the geometry inside every LLM embedding space operating on GPU clusters today - and nobody talks about it. The Hopf fibration maps a 4D hypersphere onto a regular sphere using circles that never intersect but each links through every other exactly once. The Bloch sphere every quantum computer operates on, the fiber bundles underlying gauge theory, and the quaternion math inside every 3D graphics engine and every embedding rotation on the planet. Bookmark & watch today. The most important shape nobody talks about.

Eron

27,256 次观看 • 3 天前

New short course: Attention in Transformers: Concepts and Code in PyTorch. Last week we released a course on how LLM transformers work. This week, go deeper and learn about the technical ideas behind the attention mechanism, and see how to code it in PyTorch. This course is built with Joshua Starmer, Founder and CEO of StatQuest. The attention mechanism was a breakthrough that led to transformers, the architecture powering large language models like ChatGPT. Transformers, introduced in the 2017 paper: "Attention is All You Need" by Viswani and others, took off because of its highly scalable design. In this course, you’ll learn how the attention mechanism, a key element of transformer-based LLMs, works and implement it in PyTorch. You'll develop deep intuition about building reliable, functional, and scalable AI applications. What you will do: - Understand the evolution of the attention mechanism, a key breakthrough that led to transformers. - Learn the relationships between word embeddings, positional embeddings, and attention. - Learn about the Query, Key, and Value matrices, and how to produce and use them in attention. - Walk through the math required to calculate self-attention and masked self-attention to learn why and how they work. - Understand the difference between self-attention and masked self-attention and how one is used in the encoder to build context-aware embeddings and the other is used in the decoder for generative outputs. - Learn the details of the encoder-decoder architecture, cross-attention, and multi-head attention and how they are all incorporated into a transformer. - Use PyTorch to code a class that implements self-attention, masked self-attention, and multi-head attention. There're lots of exciting technical details in this course. Please sign up here:
2:41

New short course: Attention in Transformers: Concepts and Code in PyTorch. Last week we released a course on how LLM transformers work. This week, go deeper and learn about the technical ideas behind the attention mechanism, and see how to code it in PyTorch. This course is built with Joshua Starmer, Founder and CEO of StatQuest. The attention mechanism was a breakthrough that led to transformers, the architecture powering large language models like ChatGPT. Transformers, introduced in the 2017 paper: "Attention is All You Need" by Viswani and others, took off because of its highly scalable design. In this course, you’ll learn how the attention mechanism, a key element of transformer-based LLMs, works and implement it in PyTorch. You'll develop deep intuition about building reliable, functional, and scalable AI applications. What you will do: - Understand the evolution of the attention mechanism, a key breakthrough that led to transformers. - Learn the relationships between word embeddings, positional embeddings, and attention. - Learn about the Query, Key, and Value matrices, and how to produce and use them in attention. - Walk through the math required to calculate self-attention and masked self-attention to learn why and how they work. - Understand the difference between self-attention and masked self-attention and how one is used in the encoder to build context-aware embeddings and the other is used in the decoder for generative outputs. - Learn the details of the encoder-decoder architecture, cross-attention, and multi-head attention and how they are all incorporated into a transformer. - Use PyTorch to code a class that implements self-attention, masked self-attention, and multi-head attention. There're lots of exciting technical details in this course. Please sign up here:

Andrew Ng

132,792 次观看 • 1 年前

SVM by hand ✍️ ~ 19 steps walkthrough below (Linear vs RBF) Support Vector Machines reigned supreme in machine learning before the deep learning revolution. An SVM predicts with dot products, the same matrix multiplication every model uses. What it does not do is train by backpropagation: it is fitted by convex optimization, so there is no matrix-multiplication backward pass for a GPU to accelerate. I drew and calculated two SVMs by hand: a linear one (top) and an RBF one (bottom), classifying the same two test vectors. Goal: turn six training vectors and their learned coefficients into a prediction, and see what changing the kernel actually changes. = 1. Given = Six training vectors, their labels, and the coefficients and bias already learned. A coefficient of zero means that vector is not a support vector: too far from the boundary to matter. = 2. Linear kernel, test vector 1 = Let us take the dot product of the test vector with every training vector. The dot product stands in for cosine similarity, and the column of results is the first column of the kernel matrix K. = 3. Linear kernel, test vector 2 = We do the same for the second, and K is complete. = 4. Signed weights = Let us multiply each coefficient by its label. The second training vector drops out here, because its coefficient is 0. = 5. Weighted combination = We multiply the signed weights through K and add the bias b. The result is a signed distance to the decision boundary: 17 and 5. = 6. Classify = Let us take the sign. Both are positive. = 7 to 11. RBF kernel, test vector 1 = Now the same picture with a different kernel, in five moves: square the differences, sum them, take the square root for the L2 distance, multiply by minus gamma, and raise e to that power. The negation is what turns a distance into a similarity, and gamma controls how far a single training vector's influence reaches. = 12 to 16. RBF kernel, test vector 2 = We repeat all five. The numbers change, the moves do not. = 17 to 19. Decision boundary, again = Signed weights, weighted combination, sign. Identical arithmetic to steps 4 through 6, on a K that was built a completely different way. The outputs: Linear K, first column = [13, 25, 12, 15, 19, 27] Linear decision values = 17 and 5, both positive RBF decision values = -2 and 1, so negative and positive The takeaway: the kernel is the only thing that changed, and it changed the answer. The linear SVM calls both test vectors positive; the RBF one splits them. Everything after the kernel matrix, the signed weights and the weighted combination and the sign, is the same page of arithmetic twice. 💾 Save this post!
0:44

SVM by hand ✍️ ~ 19 steps walkthrough below (Linear vs RBF) Support Vector Machines reigned supreme in machine learning before the deep learning revolution. An SVM predicts with dot products, the same matrix multiplication every model uses. What it does not do is train by backpropagation: it is fitted by convex optimization, so there is no matrix-multiplication backward pass for a GPU to accelerate. I drew and calculated two SVMs by hand: a linear one (top) and an RBF one (bottom), classifying the same two test vectors. Goal: turn six training vectors and their learned coefficients into a prediction, and see what changing the kernel actually changes. = 1. Given = Six training vectors, their labels, and the coefficients and bias already learned. A coefficient of zero means that vector is not a support vector: too far from the boundary to matter. = 2. Linear kernel, test vector 1 = Let us take the dot product of the test vector with every training vector. The dot product stands in for cosine similarity, and the column of results is the first column of the kernel matrix K. = 3. Linear kernel, test vector 2 = We do the same for the second, and K is complete. = 4. Signed weights = Let us multiply each coefficient by its label. The second training vector drops out here, because its coefficient is 0. = 5. Weighted combination = We multiply the signed weights through K and add the bias b. The result is a signed distance to the decision boundary: 17 and 5. = 6. Classify = Let us take the sign. Both are positive. = 7 to 11. RBF kernel, test vector 1 = Now the same picture with a different kernel, in five moves: square the differences, sum them, take the square root for the L2 distance, multiply by minus gamma, and raise e to that power. The negation is what turns a distance into a similarity, and gamma controls how far a single training vector's influence reaches. = 12 to 16. RBF kernel, test vector 2 = We repeat all five. The numbers change, the moves do not. = 17 to 19. Decision boundary, again = Signed weights, weighted combination, sign. Identical arithmetic to steps 4 through 6, on a K that was built a completely different way. The outputs: Linear K, first column = [13, 25, 12, 15, 19, 27] Linear decision values = 17 and 5, both positive RBF decision values = -2 and 1, so negative and positive The takeaway: the kernel is the only thing that changed, and it changed the answer. The linear SVM calls both test vectors positive; the RBF one splits them. Everything after the kernel matrix, the signed weights and the weighted combination and the sign, is the same page of arithmetic twice. 💾 Save this post!

Tom Yeh

16,916 次观看 • 3 个月前

Graph Convolutional Network by hand ✍️ ~ 12 steps walkthrough below Graph Convolutional Networks (GCNs), introduced by Thomas Kipf and Max Welling in 2017, are the tool for data shaped like a graph: social networks, recommendations, biological networks, drug discovery, molecular chemistry. I drew and calculated a simple GCN entirely by hand. Goal: run a two-layer GCN, then a small classifier, on a five-node graph, filling in every cell yourself. 1. Given A graph of five nodes, A to E, with edges between some of them. 2. Adjacency matrix (neighbors) Put a 1 wherever two nodes share an edge, in both directions. 3. Adjacency matrix (self) Add 1s down the diagonal, one self-loop per node. That is just adding the identity matrix. 4. Messages Multiply each node's embedding by the weights and biases, then ReLU. Negatives become 0. 5. Pooling Multiply the messages by the adjacency matrix. Each node gathers the messages of its neighbours and itself. 6. Visualize Node A pools [3,0,1] + [1,0,0] = [4,0,1]. 7. Second GCN layer Messages again: weights, biases, ReLU. 8. Pooling again Pool over each node and its neighbours, once more. 9. Visualize Node C pools [1,2,4] + [1,3,5] + [0,0,1] = [2,5,10]. 10. Fully connected layer Weights, biases, ReLU. This time there are no neighbours to pool, just the node itself. 11. Linear layer One more: weights and biases. 12. Sigmoid Squash each score to a probability (≥ 3 → 1, 0 → 0.5, ≤ -3 → 0). That is the classification for each node. You have just classified every node in the graph by hand. ✍️ The outputs: A: 0 (very unlikely) B: 1 (very likely) C: 1 (very likely) D: 1 (very likely) E: 0.5 (neutral) The takeaway: a GCN layer is two parts. The top part pools each node with its neighbours through the adjacency matrix. The bottom part is an MLP that transforms each node on its own. A transformer layer has the same two parts, with an attention matrix where the adjacency matrix was. Both matrices do one job, mixing across positions: attention over tokens, adjacency over nodes. In my class I call the GCN the transformer's little cousin: a bit more stubborn, because its attention is fixed by the graph rather than computed from Q, K, and V. Draw the two side by side and the resemblance is hard to miss. 💾 Save this post! #AIbyHand #GraphNeuralNetworks #DeepLearning
0:23

Graph Convolutional Network by hand ✍️ ~ 12 steps walkthrough below Graph Convolutional Networks (GCNs), introduced by Thomas Kipf and Max Welling in 2017, are the tool for data shaped like a graph: social networks, recommendations, biological networks, drug discovery, molecular chemistry. I drew and calculated a simple GCN entirely by hand. Goal: run a two-layer GCN, then a small classifier, on a five-node graph, filling in every cell yourself. 1. Given A graph of five nodes, A to E, with edges between some of them. 2. Adjacency matrix (neighbors) Put a 1 wherever two nodes share an edge, in both directions. 3. Adjacency matrix (self) Add 1s down the diagonal, one self-loop per node. That is just adding the identity matrix. 4. Messages Multiply each node's embedding by the weights and biases, then ReLU. Negatives become 0. 5. Pooling Multiply the messages by the adjacency matrix. Each node gathers the messages of its neighbours and itself. 6. Visualize Node A pools [3,0,1] + [1,0,0] = [4,0,1]. 7. Second GCN layer Messages again: weights, biases, ReLU. 8. Pooling again Pool over each node and its neighbours, once more. 9. Visualize Node C pools [1,2,4] + [1,3,5] + [0,0,1] = [2,5,10]. 10. Fully connected layer Weights, biases, ReLU. This time there are no neighbours to pool, just the node itself. 11. Linear layer One more: weights and biases. 12. Sigmoid Squash each score to a probability (≥ 3 → 1, 0 → 0.5, ≤ -3 → 0). That is the classification for each node. You have just classified every node in the graph by hand. ✍️ The outputs: A: 0 (very unlikely) B: 1 (very likely) C: 1 (very likely) D: 1 (very likely) E: 0.5 (neutral) The takeaway: a GCN layer is two parts. The top part pools each node with its neighbours through the adjacency matrix. The bottom part is an MLP that transforms each node on its own. A transformer layer has the same two parts, with an attention matrix where the adjacency matrix was. Both matrices do one job, mixing across positions: attention over tokens, adjacency over nodes. In my class I call the GCN the transformer's little cousin: a bit more stubborn, because its attention is fixed by the graph rather than computed from Q, K, and V. Draw the two side by side and the resemblance is hard to miss. 💾 Save this post! #AIbyHand #GraphNeuralNetworks #DeepLearning

Tom Yeh

16,956 次观看 • 3 个月前