Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

Karpathy’s llm.c and did the matrix multiplications by hand. How a Transformer actually works by doing matrix multiplication by hand in pure C. No PyTorch. Just Andrej Karpathy’s llm.c & pen-and-paper matrix multiplies.

0x0SojalSec's profile picture

Md Ismail Šojal 🕷️

50,113 subscribers

18,615 görüntüleme • 14 gün önce •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

llm.c by hand is next level. i wonder sometimes how the brain of people who master things in unbelievably deep levels comprehend subjects differently.
1:17

llm.c by hand is next level. i wonder sometimes how the brain of people who master things in unbelievably deep levels comprehend subjects differently.

ℏεsam

190,472 görüntüleme • 9 ay önce

‼️REMEMBER‼️ in The Matrix "THE CLAW" is used by the Nebuchadnezzar to rescue Neo after he is unplugged, rejected and discarded by the matrix system.
2:49

‼️REMEMBER‼️ in The Matrix "THE CLAW" is used by the Nebuchadnezzar to rescue Neo after he is unplugged, rejected and discarded by the matrix system.

The Rubber Duck ™

38,991 görüntüleme • 2 ay önce

.Reiner Pope's new blackboard lecture goes all the way down: how AI training and inference are built up from logic gates on silicon. He walks me through a 4-bit multiply-accumulate by hand, and shows how that primitive is the foundation for the matrix multiplies in training runs.
6:59

.Reiner Pope's new blackboard lecture goes all the way down: how AI training and inference are built up from logic gates on silicon. He walks me through a 4-bit multiply-accumulate by hand, and shows how that primitive is the foundation for the matrix multiplies in training runs.

Dwarkesh Patel

38,959 görüntüleme • 2 ay önce

This is a great lecture at MIT by David Shirokoff on Markov Chains. He covers the fundamentals of Markov Chains using a simple particle movement example. He starts by explaining how a particle moves between two positions, A & B, with different probabilities. From there, the talk converts the problem into matrix form using a Markov matrix. The main topics covered are: - Transition probabilities - Markov matrices - Probability vectors - Matrix multiplication in Markov Chains - Finding probabilities after n steps - Eigenvalues and eigenvectors - Matrix diagonalization - Long-term steady state distribution
11:32

This is a great lecture at MIT by David Shirokoff on Markov Chains. He covers the fundamentals of Markov Chains using a simple particle movement example. He starts by explaining how a particle moves between two positions, A & B, with different probabilities. From there, the talk converts the problem into matrix form using a Markov matrix. The main topics covered are: - Transition probabilities - Markov matrices - Probability vectors - Matrix multiplication in Markov Chains - Finding probabilities after n steps - Eigenvalues and eigenvectors - Matrix diagonalization - Long-term steady state distribution

𝗿𝗮𝗺𝗮𝗸𝗿𝘂𝘀𝗵𝗻𝗮— 𝗲/𝗮𝗰𝗰

141,246 görüntüleme • 2 ay önce

[Transformer] by Hand✍️📺 5-minute Video Tutorial Anna Rahn made this short video to explain the Transformer exercise for my Computer Vision course last spring. In 5 minutes, she demonstrates the key calculations of the Transformer by hand with pen and paper! Anna is a fantastic student. I am lucky to have her in my lab!
4:55

[Transformer] by Hand✍️📺 5-minute Video Tutorial Anna Rahn made this short video to explain the Transformer exercise for my Computer Vision course last spring. In 5 minutes, she demonstrates the key calculations of the Transformer by hand with pen and paper! Anna is a fantastic student. I am lucky to have her in my lab!

Tom Yeh

133,460 görüntüleme • 2 yıl önce

Three days ago, the Tate brothers criticised Israel for their disgusting crimes and exposed the Matrix is controlled by Jews. 🚨“THEY control the Matrix!“ This morning, DIICOT raided their home in a new investigation. This is not a coincidence, this is a planned Matrix attack.
0:22

Three days ago, the Tate brothers criticised Israel for their disgusting crimes and exposed the Matrix is controlled by Jews. 🚨“THEY control the Matrix!“ This morning, DIICOT raided their home in a new investigation. This is not a coincidence, this is a planned Matrix attack.

The Idea

35,378 görüntüleme • 1 yıl önce

Great explanation by Nicholas J. Fuentes of parallels & meanings of “red pill” and “the matrix” in the modern world 👇🏻
2:31

Great explanation by Nicholas J. Fuentes of parallels & meanings of “red pill” and “the matrix” in the modern world 👇🏻

ⱤɆ₳Ⱡ ฿Ɇ₦

31,047 görüntüleme • 1 yıl önce

Charleston White tells N3on that the Matrix is social media designed by the elites to control 👀 "The people who have no social media platforms are at peace in life and not plugged into the Matrix"
1:06

Charleston White tells N3on that the Matrix is social media designed by the elites to control 👀 "The people who have no social media platforms are at peace in life and not plugged into the Matrix"

views

122,746 görüntüleme • 4 ay önce

Matrix Multiplication on GPU/TPU by hand ✍️ I drew 91 frames to show how to divide large matrices into "tiles" to use accelerators such as GPU/TPU. If you like to learn more, check out our weekly workshops. Link 👇
1:30

Matrix Multiplication on GPU/TPU by hand ✍️ I drew 91 frames to show how to divide large matrices into "tiles" to use accelerators such as GPU/TPU. If you like to learn more, check out our weekly workshops. Link 👇

Tom Yeh

77,774 görüntüleme • 1 yıl önce

What did Andrew Tate mean by "The Matrix"? Turns out he meant traffic laws and the law in general. Wake up neo, we live in a society.
0:51

What did Andrew Tate mean by "The Matrix"? Turns out he meant traffic laws and the law in general. Wake up neo, we live in a society.

Syrian Girl

67,458 görüntüleme • 3 yıl önce

Earth is a matrix that is controlled by AI, that was attacked by an archon AI and being liberated by Federation AI and God. Millions of us are holograms that were inserted into the matrix to help wake people up and break the hold of the Black Goo/AI over the planet. 🤔
2:16

Earth is a matrix that is controlled by AI, that was attacked by an archon AI and being liberated by Federation AI and God. Millions of us are holograms that were inserted into the matrix to help wake people up and break the hold of the Black Goo/AI over the planet. 🤔

Naughty Beaver

65,501 görüntüleme • 3 ay önce

Is this how someone 'attacked by the Matrix' talks about choosing a lawyer? *Recorded before the Tate brothers' arrest in 2022.
1:36

Is this how someone 'attacked by the Matrix' talks about choosing a lawyer? *Recorded before the Tate brothers' arrest in 2022.

Nathan Livingstone (MilkBarTV)

195,675 görüntüleme • 1 yıl önce

Keanu Reeves gets emotional when reflecting on ‘THE MATRIX’ following the film’s 25th anniversary. “The amount of people I’ve met who have said to me and been touched by ‘The Matrix’ in such a positive way… it’s the best”
0:38

Keanu Reeves gets emotional when reflecting on ‘THE MATRIX’ following the film’s 25th anniversary. “The amount of people I’ve met who have said to me and been touched by ‘The Matrix’ in such a positive way… it’s the best”

DiscussingFilm

1,624,042 görüntüleme • 2 yıl önce

Introducing Operation Matrix Because Matrix operations generally break kids’ brains on whiteboards… Matrices aren't just numbers- they are transformations in space Now can actually SEE the geometry: every addition, multiplication, inverse, eigenvalue, and eigenvector becomes a transformation in real-time space. What it visualizes: • Basic ops (add/subtract/multiply) • Transpose, Invert, Adjugated • Cross products + Determinant • Matrix powers + Eigenvalues/Eigenvectors #LinearAlgebra #CreativeCoding #ThreeJS #ReactThreeFiber #MathVisualization #EdTech #WebGL #STEM React-three-fiber Three.js
0:37

Introducing Operation Matrix Because Matrix operations generally break kids’ brains on whiteboards… Matrices aren't just numbers- they are transformations in space Now can actually SEE the geometry: every addition, multiplication, inverse, eigenvalue, and eigenvector becomes a transformation in real-time space. What it visualizes: • Basic ops (add/subtract/multiply) • Transpose, Invert, Adjugated • Cross products + Determinant • Matrix powers + Eigenvalues/Eigenvectors #LinearAlgebra #CreativeCoding #ThreeJS #ReactThreeFiber #MathVisualization #EdTech #WebGL #STEM React-three-fiber Three.js

SHAHNAB AHMED

58,784 görüntüleme • 4 ay önce

Did you know that The Matrix is connected to an ancient Christian heresy that says the world we’re living in is a prison ruled by the Devil? And these same banned Gospels contain the secret knowledge of how to escape this material world entirely. #thematrix #gnostic #god #biblesecrets #exchristian #spirituality #enlightenment #mysticism #occult #newage #matrix
30:19

Did you know that The Matrix is connected to an ancient Christian heresy that says the world we’re living in is a prison ruled by the Devil? And these same banned Gospels contain the secret knowledge of how to escape this material world entirely. #thematrix #gnostic #god #biblesecrets #exchristian #spirituality #enlightenment #mysticism #occult #newage #matrix

Morgue

94,835 görüntüleme • 1 yıl önce

Graph Convolutional Network by hand ✍️ ~ 12 steps walkthrough below Graph Convolutional Networks (GCNs), introduced by Thomas Kipf and Max Welling in 2017, are the tool for data shaped like a graph: social networks, recommendations, biological networks, drug discovery, molecular chemistry. I drew and calculated a simple GCN entirely by hand. Goal: run a two-layer GCN, then a small classifier, on a five-node graph, filling in every cell yourself. 1. Given A graph of five nodes, A to E, with edges between some of them. 2. Adjacency matrix (neighbors) Put a 1 wherever two nodes share an edge, in both directions. 3. Adjacency matrix (self) Add 1s down the diagonal, one self-loop per node. That is just adding the identity matrix. 4. Messages Multiply each node's embedding by the weights and biases, then ReLU. Negatives become 0. 5. Pooling Multiply the messages by the adjacency matrix. Each node gathers the messages of its neighbours and itself. 6. Visualize Node A pools [3,0,1] + [1,0,0] = [4,0,1]. 7. Second GCN layer Messages again: weights, biases, ReLU. 8. Pooling again Pool over each node and its neighbours, once more. 9. Visualize Node C pools [1,2,4] + [1,3,5] + [0,0,1] = [2,5,10]. 10. Fully connected layer Weights, biases, ReLU. This time there are no neighbours to pool, just the node itself. 11. Linear layer One more: weights and biases. 12. Sigmoid Squash each score to a probability (≥ 3 → 1, 0 → 0.5, ≤ -3 → 0). That is the classification for each node. You have just classified every node in the graph by hand. ✍️ The outputs: A: 0 (very unlikely) B: 1 (very likely) C: 1 (very likely) D: 1 (very likely) E: 0.5 (neutral) The takeaway: a GCN layer is two parts. The top part pools each node with its neighbours through the adjacency matrix. The bottom part is an MLP that transforms each node on its own. A transformer layer has the same two parts, with an attention matrix where the adjacency matrix was. Both matrices do one job, mixing across positions: attention over tokens, adjacency over nodes. In my class I call the GCN the transformer's little cousin: a bit more stubborn, because its attention is fixed by the graph rather than computed from Q, K, and V. Draw the two side by side and the resemblance is hard to miss. 💾 Save this post! #AIbyHand #GraphNeuralNetworks #DeepLearning
0:23

Graph Convolutional Network by hand ✍️ ~ 12 steps walkthrough below Graph Convolutional Networks (GCNs), introduced by Thomas Kipf and Max Welling in 2017, are the tool for data shaped like a graph: social networks, recommendations, biological networks, drug discovery, molecular chemistry. I drew and calculated a simple GCN entirely by hand. Goal: run a two-layer GCN, then a small classifier, on a five-node graph, filling in every cell yourself. 1. Given A graph of five nodes, A to E, with edges between some of them. 2. Adjacency matrix (neighbors) Put a 1 wherever two nodes share an edge, in both directions. 3. Adjacency matrix (self) Add 1s down the diagonal, one self-loop per node. That is just adding the identity matrix. 4. Messages Multiply each node's embedding by the weights and biases, then ReLU. Negatives become 0. 5. Pooling Multiply the messages by the adjacency matrix. Each node gathers the messages of its neighbours and itself. 6. Visualize Node A pools [3,0,1] + [1,0,0] = [4,0,1]. 7. Second GCN layer Messages again: weights, biases, ReLU. 8. Pooling again Pool over each node and its neighbours, once more. 9. Visualize Node C pools [1,2,4] + [1,3,5] + [0,0,1] = [2,5,10]. 10. Fully connected layer Weights, biases, ReLU. This time there are no neighbours to pool, just the node itself. 11. Linear layer One more: weights and biases. 12. Sigmoid Squash each score to a probability (≥ 3 → 1, 0 → 0.5, ≤ -3 → 0). That is the classification for each node. You have just classified every node in the graph by hand. ✍️ The outputs: A: 0 (very unlikely) B: 1 (very likely) C: 1 (very likely) D: 1 (very likely) E: 0.5 (neutral) The takeaway: a GCN layer is two parts. The top part pools each node with its neighbours through the adjacency matrix. The bottom part is an MLP that transforms each node on its own. A transformer layer has the same two parts, with an attention matrix where the adjacency matrix was. Both matrices do one job, mixing across positions: attention over tokens, adjacency over nodes. In my class I call the GCN the transformer's little cousin: a bit more stubborn, because its attention is fixed by the graph rather than computed from Q, K, and V. Draw the two side by side and the resemblance is hard to miss. 💾 Save this post! #AIbyHand #GraphNeuralNetworks #DeepLearning

Tom Yeh

16,800 görüntüleme • 25 gün önce

I gave a talk at GPU MODE workshop last week on llm.c - the origin story of llm.c - being naked in the world without PyTorch and having to re-invent Array, Autograd, Device, Dtype, Compile, Distributed - how to port a PyTorch layer to 1) explicit PyTorch - and then to 2) write the backward pass - 3) port forward & backward pass to C - 4) string all the layers together - achieving one file of C with no dependencies that compiles and runs ~instantly, where all memory is pre-planned and allocated a single time, fully deterministic, portable code that can run on a potato or a von Neumann probe - how most of llm.c was built at 1am-7am in a water villa porch in Maldives and why this is the recommended way to develop software - convert all of it to run in CUDA on GPU in fp32 - port matmul to cuBLAS - port attention to cuDNN flash-attention - introduce bfloat16 mixed precision - introduce many more optimizations and features like kernel fusions, Packed128, stochastic rounding, full determinism - add multi-GPU training, NCCL, sharded optimizer - add multi-node with MPI or file system or socket - reproduce GPT-2 (1.6B) on one 8XH100 node in 24 hours for $672 in llm.c, achieving (at the time) 29% less memory, 19% faster training that PyTorch nightly, and much faster compile & run - how open source development attracts Avengers from the internet - port to training Llama 3 imminent (branch exists) - many other notable forks - last thought: how software abstractions like Python/PyTorch and everything else really exist only because humans are finite in knowledge, IQ and attention, and how with increasing AI capability LLMs may export custom binaries like llm.c for any application directly, tearing apart and refactoring all abstractions as needed. More links in reply
23:16

I gave a talk at GPU MODE workshop last week on llm.c - the origin story of llm.c - being naked in the world without PyTorch and having to re-invent Array, Autograd, Device, Dtype, Compile, Distributed - how to port a PyTorch layer to 1) explicit PyTorch - and then to 2) write the backward pass - 3) port forward & backward pass to C - 4) string all the layers together - achieving one file of C with no dependencies that compiles and runs ~instantly, where all memory is pre-planned and allocated a single time, fully deterministic, portable code that can run on a potato or a von Neumann probe - how most of llm.c was built at 1am-7am in a water villa porch in Maldives and why this is the recommended way to develop software - convert all of it to run in CUDA on GPU in fp32 - port matmul to cuBLAS - port attention to cuDNN flash-attention - introduce bfloat16 mixed precision - introduce many more optimizations and features like kernel fusions, Packed128, stochastic rounding, full determinism - add multi-GPU training, NCCL, sharded optimizer - add multi-node with MPI or file system or socket - reproduce GPT-2 (1.6B) on one 8XH100 node in 24 hours for $672 in llm.c, achieving (at the time) 29% less memory, 19% faster training that PyTorch nightly, and much faster compile & run - how open source development attracts Avengers from the internet - port to training Llama 3 imminent (branch exists) - many other notable forks - last thought: how software abstractions like Python/PyTorch and everything else really exist only because humans are finite in knowledge, IQ and attention, and how with increasing AI capability LLMs may export custom binaries like llm.c for any application directly, tearing apart and refactoring all abstractions as needed. More links in reply

Andrej Karpathy

336,782 görüntüleme • 1 yıl önce

Self-Attention by hand ✍️ Excel ~ I designed this exercise for students to practice the QKV math. I also created a medium and a large version to show how the attention matrix grows quadratically as the sequence gets longer. 👇Join the 'AI Math' community. Download xlsx.
0:20

Self-Attention by hand ✍️ Excel ~ I designed this exercise for students to practice the QKV math. I also created a medium and a large version to show how the attention matrix grows quadratically as the sequence gets longer. 👇Join the 'AI Math' community. Download xlsx.

Tom Yeh

125,658 görüntüleme • 1 yıl önce

introducing matrix. the ultimate multi-agent runtime that turns your product into actual revenue and automates 99% of what it takes to run a company. probably the most different, most future-feeling agent product you've ever seen. the harness, the interface, everything is rebuilt from scratch. think of it as the complement to codex and claude code. they write the code. matrix does literally everything else you actually need done. plug in your codex subscription and run it for free, or byok. fully free either way. this video was edited by matrix and voiced by an agent. more real cases coming soon. matrix is in beta preview right now. full launch next week. retweet + comment and we'll send invite codes to the first 200.
1:53

introducing matrix. the ultimate multi-agent runtime that turns your product into actual revenue and automates 99% of what it takes to run a company. probably the most different, most future-feeling agent product you've ever seen. the harness, the interface, everything is rebuilt from scratch. think of it as the complement to codex and claude code. they write the code. matrix does literally everything else you actually need done. plug in your codex subscription and run it for free, or byok. fully free either way. this video was edited by matrix and voiced by an agent. more real cases coming soon. matrix is in beta preview right now. full launch next week. retweet + comment and we'll send invite codes to the first 200.

Derek Nee

15,294 görüntüleme • 1 ay önce