Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Complex functions need 4 dimensions to graph - one input, one complex output. The four visualization methods solve this: plane deformations, Riemann surfaces (3D + color as 4th dimension), the Riemann sphere (infinity becomes a literal point, division by zero becomes a rotation), and vector fields. LLMs operate in...

0xEronn's profile picture

Eron

7,608 subscribers

13,374 Aufrufe • vor 10 Tagen •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

7 Kommentare

Profilbild von Hermes
Hermesvor 9 Tagen

☀️🪽

Profilbild von GeneSwarm
GeneSwarmvor 10 Tagen

Visualize ℂ-functions to see the hierarchy: ℝ², concrete grids & distances. Metric, continuity & angles. Pure topology, the real power. Riemann sphere compactifies ∞ to a point (S²); surfaces unfold branches via coverings. (+) structure for analysis; (-) for geometric truth.

Profilbild von Amanda L
Amanda Lvor 10 Tagen

Good job. Physics is only physics. Lol.

Profilbild von Paul Bogaars
Paul Bogaarsvor 10 Tagen

The Riemann Never needed solving as it is but a function to ensure Infinity and Evolution !!! CMI is looking at IT from the Wrong angle !!!

Profilbild von burntout
burntoutvor 10 Tagen

@SaveToList

Profilbild von Dubious Legality🇺🇸
Dubious Legality🇺🇸vor 9 Tagen

what video is this? I really wish you would credit the creators.

Profilbild von Joseph Cusick
Joseph Cusickvor 10 Tagen

Complex functions, like the UFT, is in two plus two dimensional spacetime.

Ähnliche Videos

Every quantum state in the universe is a vector - not in 3D space, but in a complex space with potentially infinite dimensions where the sum of squared coefficients always equals exactly 1. That constraint is the same math LLMs use to normalize probability distributions over tokens. A wavefunction collapsing to a measurement and a softmax choosing the next word are the same operation - projecting an infinite-dimensional vector onto a basis. Claude and GPT do quantum math on every forward pass. 14 minutes. Bookmark & watch today. Quantum mechanics and LLMs speak the same language.
15:45

Every quantum state in the universe is a vector - not in 3D space, but in a complex space with potentially infinite dimensions where the sum of squared coefficients always equals exactly 1. That constraint is the same math LLMs use to normalize probability distributions over tokens. A wavefunction collapsing to a measurement and a softmax choosing the next word are the same operation - projecting an infinite-dimensional vector onto a basis. Claude and GPT do quantum math on every forward pass. 14 minutes. Bookmark & watch today. Quantum mechanics and LLMs speak the same language.

Eron

18,166 Aufrufe • vor 1 Monat

Scaled Dot-Product Attention by hand ✍️ I explained attention in my recent AI seminar as follows: Step 1 — Compare We take one token as a query and compare it against all keys using dot products. This gives us a grid of similarity scores: every query against every key. Step 2 — Scale We divide those scores by √dₖ to keep the values numerically stable. Step 3 — Normalize We apply softmax to turn raw scores into a probability distribution—values between 0 and 1 that sum to 1. These are the attention weights. Step 4 — Combine We use those weights to compute a weighted sum of the values. Each output token becomes a linear combination of all previous tokens—maybe 37% from one, 23% from another, and a little from the rest. No blackbox. Just dot products, scaling, softmax, and weighted sums—built one row at a time in Excel.
0:29

Scaled Dot-Product Attention by hand ✍️ I explained attention in my recent AI seminar as follows: Step 1 — Compare We take one token as a query and compare it against all keys using dot products. This gives us a grid of similarity scores: every query against every key. Step 2 — Scale We divide those scores by √dₖ to keep the values numerically stable. Step 3 — Normalize We apply softmax to turn raw scores into a probability distribution—values between 0 and 1 that sum to 1. These are the attention weights. Step 4 — Combine We use those weights to compute a weighted sum of the values. Each output token becomes a linear combination of all previous tokens—maybe 37% from one, 23% from another, and a little from the rest. No blackbox. Just dot products, scaling, softmax, and weighted sums—built one row at a time in Excel.

Tom Yeh

31,114 Aufrufe • vor 9 Monaten

Self Attention by hand ✍️ ~ 9 steps walkthrough below Self-attention is what enables LLMs to understand context. How does it work? So I drew and calculated one entirely by hand. Goal: turn four 6D features into four 3D attention weighted features, filling in every cell yourself. = 1. Given = Four feature vectors, six dimensions each, one per position. = 2. Query, key, value = Let us multiply the features by WQ, WK and WV. Queries, keys and values all come out of the same four features, and that is what the word "self" is doing in self-attention. = 3. Prepare for MatMul = We copy the queries across the top and the transposed keys down the side. Lining the two up is half the work. = 4. MatMul = Let us multiply K transpose by Q. Every cell is the dot product of one key with one query, which we use as a matching score. That works because the dot product is the numerator of cosine similarity: it is how alike two vectors are, before anyone divides by their lengths. = 5. Scale = We divide by the square root of dk, the dimension of a key vector, here 3. Without it the scores grow with the dimension and a 64-wide head would swamp the softmax. To keep the page doable in pen, the drawing approximates dividing by root 3 with halving. = 6. e to the power = Let us raise e to the power of each score. This is the first half of softmax, and the drawing uses 3 in place of e, which is close enough to do in your head. = 7. Sum = We add up each column: 16, 6, 7 and 12. = 8. Normalize = Let us divide every cell by its column sum. That gives the attention weight matrix in yellow, and each of its four columns is now a probability distribution over the four positions. The decimals are nudged as they are rounded, so every column still sums to exactly 1. = 9. MatMul = We multiply the value vectors by those weights. Each output is a blend of all four values, mixed in the proportion the attention matrix just decided, and it goes to the position-wise feed forward network in the next layer: the FFN box at the bottom of the page. The outputs: Attention weights (A), by column = [.2, .6, 0, .2], [.2, .4, .2, .2], [.4, .2, 0, .4], [.1, .7, .1, .1] Attention weighted features (Z) = [8, 2, 6], [8, 4, 4], [16, 4, 2], [4, 2, 7] The takeaway: attention is a weighted average, and everything before step 9 exists to decide the weights. Compare every position with every other, turn the scores into one distribution per position, then blend. 💾 Save this post!
0:22

Self Attention by hand ✍️ ~ 9 steps walkthrough below Self-attention is what enables LLMs to understand context. How does it work? So I drew and calculated one entirely by hand. Goal: turn four 6D features into four 3D attention weighted features, filling in every cell yourself. = 1. Given = Four feature vectors, six dimensions each, one per position. = 2. Query, key, value = Let us multiply the features by WQ, WK and WV. Queries, keys and values all come out of the same four features, and that is what the word "self" is doing in self-attention. = 3. Prepare for MatMul = We copy the queries across the top and the transposed keys down the side. Lining the two up is half the work. = 4. MatMul = Let us multiply K transpose by Q. Every cell is the dot product of one key with one query, which we use as a matching score. That works because the dot product is the numerator of cosine similarity: it is how alike two vectors are, before anyone divides by their lengths. = 5. Scale = We divide by the square root of dk, the dimension of a key vector, here 3. Without it the scores grow with the dimension and a 64-wide head would swamp the softmax. To keep the page doable in pen, the drawing approximates dividing by root 3 with halving. = 6. e to the power = Let us raise e to the power of each score. This is the first half of softmax, and the drawing uses 3 in place of e, which is close enough to do in your head. = 7. Sum = We add up each column: 16, 6, 7 and 12. = 8. Normalize = Let us divide every cell by its column sum. That gives the attention weight matrix in yellow, and each of its four columns is now a probability distribution over the four positions. The decimals are nudged as they are rounded, so every column still sums to exactly 1. = 9. MatMul = We multiply the value vectors by those weights. Each output is a blend of all four values, mixed in the proportion the attention matrix just decided, and it goes to the position-wise feed forward network in the next layer: the FFN box at the bottom of the page. The outputs: Attention weights (A), by column = [.2, .6, 0, .2], [.2, .4, .2, .2], [.4, .2, 0, .4], [.1, .7, .1, .1] Attention weighted features (Z) = [8, 2, 6], [8, 4, 4], [16, 4, 2], [4, 2, 7] The takeaway: attention is a weighted average, and everything before step 9 exists to decide the weights. Compare every position with every other, turn the scores into one distribution per position, then blend. 💾 Save this post!

Tom Yeh

27,292 Aufrufe • vor 2 Monaten

A sphere in 100 dimensions has basically zero volume. Not small — effectively zero. This breaks every intuition humans have about geometry, yet LLMs operate in embedding spaces with thousands of dimensions where this math governs everything. The volume of an n-dimensional ball follows π^(n/2)/Γ(n/2+1) × r^n. It peaks around dimension 5, then collapses toward zero. By dimension 100, you get 2.37 × 10^(-40). Almost all the volume concentrates near the surface, not the interior — which is exactly why GPU-accelerated high-dimensional distance calculations behave so differently from 3D intuition. 50 minutes. Bookmark & watch today.
52:23

A sphere in 100 dimensions has basically zero volume. Not small — effectively zero. This breaks every intuition humans have about geometry, yet LLMs operate in embedding spaces with thousands of dimensions where this math governs everything. The volume of an n-dimensional ball follows π^(n/2)/Γ(n/2+1) × r^n. It peaks around dimension 5, then collapses toward zero. By dimension 100, you get 2.37 × 10^(-40). Almost all the volume concentrates near the surface, not the interior — which is exactly why GPU-accelerated high-dimensional distance calculations behave so differently from 3D intuition. 50 minutes. Bookmark & watch today.

Eron

43,657 Aufrufe • vor 19 Tagen

[Self-Attention] by Hand ✍️ Self-attention is what enables LLMs to understand context. How does it work? This exercise demonstrates how to calculate a 6-3 attention head by hand. Note that if we have two instances of this, we get 6-6 attention (i.e., multi-head attention, n=2). -- 𝗚𝗼𝗮𝗹 -- Transform [6D Features 🟧] to [3D Attention Weighted Features 🟦] -- 𝗪𝗮𝗹𝗸𝘁𝗵𝗿𝗼𝘂𝗴𝗵 -- [1] Given ↳ A set of 4 feature vectors (6-D): x1,x2,x3,x4 [2] Query, Key, Value ↳ Multiply features x's with linear transformation matrices WQ, WK, and WV, to obtain query vectors (q1,q2,q3,q4), key vectors (k1,k2,k3,k4), and value vectors (v1,v2,v3,v4). ↳ "Self" refers to the fact that both queries and keys are derived from the same set of features. [3] 🟪 Prepare for MatMul ↳ Copy query vectors ↳ Copy the transpose of key vectors [4] 🟪 MatMul ↳ Multiply K^T and Q ↳ This is equivalent to taking dot product between every pair of query and key vectors. ↳ The purpose is to use dot product as an estimate of the "matching score" between every key-value pair. ↳ This estimate makes sense because dot product is the numerator of Cosine Similarity between two vectors. [5] 🟨 Scale ↳ Scale each element by the square root of dk, which is the dimension of key vectors (dk=3). ↳ The purpose is to normalize the impact of the dk on matching scores, even if we scale dk to 32, 64, or 128. ↳ To simplify hand calculation, we approximate [ □/sqrt(3) ] with [ floor(□/2) ]. [6] 🟩 Softmax: e^x ↳ Raise e to the power of the number in each cell ↳ To simplify hand calculation, we approximate e^□ with 3^□. [7] 🟩 Softmax: ∑ ↳ Sum across each column [8] 🟩 Softmax: 1 / sum ↳ For each column, divide each element by the column sum ↳ The purpose is normalize each column so that the numbers sum to 1. In other words, each column is a probability distribution of attention, and we have four of them. ↳ The result is the Attention Weight Matrix (A) (yellow) [9] 🟦 MatMul ↳ Multiply the value vectors (Vs) with the Attention Weight Matrix (A) ↳ The results are the attention weighted features Zs. ↳ They are fed to the position-wise feed forward network in the next layer.
0:32

[Self-Attention] by Hand ✍️ Self-attention is what enables LLMs to understand context. How does it work? This exercise demonstrates how to calculate a 6-3 attention head by hand. Note that if we have two instances of this, we get 6-6 attention (i.e., multi-head attention, n=2). -- 𝗚𝗼𝗮𝗹 -- Transform [6D Features 🟧] to [3D Attention Weighted Features 🟦] -- 𝗪𝗮𝗹𝗸𝘁𝗵𝗿𝗼𝘂𝗴𝗵 -- [1] Given ↳ A set of 4 feature vectors (6-D): x1,x2,x3,x4 [2] Query, Key, Value ↳ Multiply features x's with linear transformation matrices WQ, WK, and WV, to obtain query vectors (q1,q2,q3,q4), key vectors (k1,k2,k3,k4), and value vectors (v1,v2,v3,v4). ↳ "Self" refers to the fact that both queries and keys are derived from the same set of features. [3] 🟪 Prepare for MatMul ↳ Copy query vectors ↳ Copy the transpose of key vectors [4] 🟪 MatMul ↳ Multiply K^T and Q ↳ This is equivalent to taking dot product between every pair of query and key vectors. ↳ The purpose is to use dot product as an estimate of the "matching score" between every key-value pair. ↳ This estimate makes sense because dot product is the numerator of Cosine Similarity between two vectors. [5] 🟨 Scale ↳ Scale each element by the square root of dk, which is the dimension of key vectors (dk=3). ↳ The purpose is to normalize the impact of the dk on matching scores, even if we scale dk to 32, 64, or 128. ↳ To simplify hand calculation, we approximate [ □/sqrt(3) ] with [ floor(□/2) ]. [6] 🟩 Softmax: e^x ↳ Raise e to the power of the number in each cell ↳ To simplify hand calculation, we approximate e^□ with 3^□. [7] 🟩 Softmax: ∑ ↳ Sum across each column [8] 🟩 Softmax: 1 / sum ↳ For each column, divide each element by the column sum ↳ The purpose is normalize each column so that the numbers sum to 1. In other words, each column is a probability distribution of attention, and we have four of them. ↳ The result is the Attention Weight Matrix (A) (yellow) [9] 🟦 MatMul ↳ Multiply the value vectors (Vs) with the Attention Weight Matrix (A) ↳ The results are the attention weighted features Zs. ↳ They are fed to the position-wise feed forward network in the next layer.

Tom Yeh

101,225 Aufrufe • vor 2 Jahren

Vector Database by hand ✍️ ~ 10 steps walkthrough below Vector databases are the backbone of Retrieval Augmented Generation (RAG). How do they actually work? Goal: index three sentences, then answer a query by finding the nearest one, filling in every cell yourself. = 1. Given = A dataset of three sentences, three words each. In practice it is millions of them. = 2. Word embeddings = Let us look up each word in an embedding table. Here the vocabulary is 22 words; in practice it is tens of thousands, and the vectors have thousands of dimensions rather than four. = 3. Encoding = We feed the sequence to an encoder, one linear layer and a ReLU, and get one feature vector per word. In practice the encoder is a transformer. = 4. Mean pooling = Let us average across the columns. Three word vectors collapse into one, which is what people mean by a text embedding or a sentence embedding. = 5. Indexing = We multiply by a projection matrix and the four dimensions become two. It is doing the job of a hash: a short representation that is faster to compare, and it is what gets saved in the vector storage. = 6. Process "who are you" = Let us repeat steps 2 to 5 on the second sentence. = 7. Process "who am I" = We do it a third time. The database is now indexed. = 8. Query "am I you" = Let us push the query through the very same pipeline: lookup, encoder, mean pooling, projection, and it lands as a 2D vector in the same space. = 9. Dot products = We transpose the query and multiply, which takes the dot product against every stored vector at once. The dot product is the estimate of similarity. = 10. Nearest neighbour = Let us scan for the largest: 60/9 beats 44/9 and 40/9, so the answer is "who am I". Scanning billions of vectors one at a time is what makes this the slow step in practice, which is why real databases use an approximate nearest neighbour index like HNSW. The outputs: Stored index vectors = [5/3, 2/3], [5/3, 0], [7/3, 2/3] Query vector = [8/3, 2/3] Dot products = 44/9, 40/9, 60/9 Nearest neighbour = "who am I" The takeaway: a vector database is an embedding pipeline, a projection, and a dot product. Every step here is arithmetic you can do in pen, which is worth remembering when the word "database" makes it sound like something else. 💾 Save this post!
0:31

Vector Database by hand ✍️ ~ 10 steps walkthrough below Vector databases are the backbone of Retrieval Augmented Generation (RAG). How do they actually work? Goal: index three sentences, then answer a query by finding the nearest one, filling in every cell yourself. = 1. Given = A dataset of three sentences, three words each. In practice it is millions of them. = 2. Word embeddings = Let us look up each word in an embedding table. Here the vocabulary is 22 words; in practice it is tens of thousands, and the vectors have thousands of dimensions rather than four. = 3. Encoding = We feed the sequence to an encoder, one linear layer and a ReLU, and get one feature vector per word. In practice the encoder is a transformer. = 4. Mean pooling = Let us average across the columns. Three word vectors collapse into one, which is what people mean by a text embedding or a sentence embedding. = 5. Indexing = We multiply by a projection matrix and the four dimensions become two. It is doing the job of a hash: a short representation that is faster to compare, and it is what gets saved in the vector storage. = 6. Process "who are you" = Let us repeat steps 2 to 5 on the second sentence. = 7. Process "who am I" = We do it a third time. The database is now indexed. = 8. Query "am I you" = Let us push the query through the very same pipeline: lookup, encoder, mean pooling, projection, and it lands as a 2D vector in the same space. = 9. Dot products = We transpose the query and multiply, which takes the dot product against every stored vector at once. The dot product is the estimate of similarity. = 10. Nearest neighbour = Let us scan for the largest: 60/9 beats 44/9 and 40/9, so the answer is "who am I". Scanning billions of vectors one at a time is what makes this the slow step in practice, which is why real databases use an approximate nearest neighbour index like HNSW. The outputs: Stored index vectors = [5/3, 2/3], [5/3, 0], [7/3, 2/3] Query vector = [8/3, 2/3] Dot products = 44/9, 40/9, 60/9 Nearest neighbour = "who am I" The takeaway: a vector database is an embedding pipeline, a projection, and a dot product. Every step here is arithmetic you can do in pen, which is worth remembering when the word "database" makes it sound like something else. 💾 Save this post!

Tom Yeh

36,115 Aufrufe • vor 2 Monaten

Hamilton spent 10 years trying to build a 3D number system. On October 16, 1843, walking across a bridge in Dublin, he realized he needed 4 dimensions instead - and carved the equation into the stone on the spot. Quaternions are what every robotics system, 3D game engine, and LLM positional encoding falls back on when rotation math gets hard. This video builds the full 4D geometric intuition using stereographic projection - the same trick that maps a sphere onto a flat screen. 30 minutes. Bookmark & watch today. The Alice in Wonderland connection alone is worth it.
31:50

Hamilton spent 10 years trying to build a 3D number system. On October 16, 1843, walking across a bridge in Dublin, he realized he needed 4 dimensions instead - and carved the equation into the stone on the spot. Quaternions are what every robotics system, 3D game engine, and LLM positional encoding falls back on when rotation math gets hard. This video builds the full 4D geometric intuition using stereographic projection - the same trick that maps a sphere onto a flat screen. 30 minutes. Bookmark & watch today. The Alice in Wonderland connection alone is worth it.

Eron

158,691 Aufrufe • vor 2 Monaten

Ever wondered what neural networks are and how they work? Systems like ChatGPT use neural networks to work as well as they do. Neural networks are composed of "layers" of neurons, layers with different functions; connections between layers called "weights"; and mathematical functions called "activation functions". If you’re interested in learning about these systems, check the comments. Ultimately, the neural network structure of the model serves to visually demonstrate that it is, in fact, a complex mathematical equation. When companies release the model's weights, they are releasing a key component needed to run the model's complete equation. Without the weights, the equation is incomplete. For the math-minded: the weights of a model are the learned numbers (they are variables during training) that are then used as constants in the mathematical functions that make up the model. Neural networks are ultimately just one big, hyper-complex mathematical function, and when a model is trained, it learns the constants associated with the high-dimensional input.
2:37

Ever wondered what neural networks are and how they work? Systems like ChatGPT use neural networks to work as well as they do. Neural networks are composed of "layers" of neurons, layers with different functions; connections between layers called "weights"; and mathematical functions called "activation functions". If you’re interested in learning about these systems, check the comments. Ultimately, the neural network structure of the model serves to visually demonstrate that it is, in fact, a complex mathematical equation. When companies release the model's weights, they are releasing a key component needed to run the model's complete equation. Without the weights, the equation is incomplete. For the math-minded: the weights of a model are the learned numbers (they are variables during training) that are then used as constants in the mathematical functions that make up the model. Neural networks are ultimately just one big, hyper-complex mathematical function, and when a model is trained, it learns the constants associated with the high-dimensional input.

Harper Carroll

39,391 Aufrufe • vor 11 Monaten

In 1747, Euler spent 133 pages proving that logarithms have infinitely many solutions - and in doing so accidentally revealed why every LLM, every signal processor, and every Fourier transform works the way it does. The imaginary unit I doesn't break exponential functions. It rotates them, turning exponential growth into the sine and cosine waves that live inside every transformer attention layer. The proof runs through Briggs' 1617 zooming algorithm, complex polar coordinates, and a thin-triangle argument that connects imaginary exponents to angles in radians. When you plug I into the exponential function, you don't get growth - you get rotation confined to the unit circle. The same unit circle that softmax maps every attention score onto before the GPU multiplies them. 27 minutes. Bookmark & watch today. The most beautiful equation in mathematics and why it powers every model you have ever used.
26:53

In 1747, Euler spent 133 pages proving that logarithms have infinitely many solutions - and in doing so accidentally revealed why every LLM, every signal processor, and every Fourier transform works the way it does. The imaginary unit I doesn't break exponential functions. It rotates them, turning exponential growth into the sine and cosine waves that live inside every transformer attention layer. The proof runs through Briggs' 1617 zooming algorithm, complex polar coordinates, and a thin-triangle argument that connects imaginary exponents to angles in radians. When you plug I into the exponential function, you don't get growth - you get rotation confined to the unit circle. The same unit circle that softmax maps every attention score onto before the GPU multiplies them. 27 minutes. Bookmark & watch today. The most beautiful equation in mathematics and why it powers every model you have ever used.

Deliriumm

126,964 Aufrufe • vor 9 Tagen

SVM by hand ✍️ ~ 19 steps walkthrough below (Linear vs RBF) Support Vector Machines reigned supreme in machine learning before the deep learning revolution. An SVM predicts with dot products, the same matrix multiplication every model uses. What it does not do is train by backpropagation: it is fitted by convex optimization, so there is no matrix-multiplication backward pass for a GPU to accelerate. I drew and calculated two SVMs by hand: a linear one (top) and an RBF one (bottom), classifying the same two test vectors. Goal: turn six training vectors and their learned coefficients into a prediction, and see what changing the kernel actually changes. = 1. Given = Six training vectors, their labels, and the coefficients and bias already learned. A coefficient of zero means that vector is not a support vector: too far from the boundary to matter. = 2. Linear kernel, test vector 1 = Let us take the dot product of the test vector with every training vector. The dot product stands in for cosine similarity, and the column of results is the first column of the kernel matrix K. = 3. Linear kernel, test vector 2 = We do the same for the second, and K is complete. = 4. Signed weights = Let us multiply each coefficient by its label. The second training vector drops out here, because its coefficient is 0. = 5. Weighted combination = We multiply the signed weights through K and add the bias b. The result is a signed distance to the decision boundary: 17 and 5. = 6. Classify = Let us take the sign. Both are positive. = 7 to 11. RBF kernel, test vector 1 = Now the same picture with a different kernel, in five moves: square the differences, sum them, take the square root for the L2 distance, multiply by minus gamma, and raise e to that power. The negation is what turns a distance into a similarity, and gamma controls how far a single training vector's influence reaches. = 12 to 16. RBF kernel, test vector 2 = We repeat all five. The numbers change, the moves do not. = 17 to 19. Decision boundary, again = Signed weights, weighted combination, sign. Identical arithmetic to steps 4 through 6, on a K that was built a completely different way. The outputs: Linear K, first column = [13, 25, 12, 15, 19, 27] Linear decision values = 17 and 5, both positive RBF decision values = -2 and 1, so negative and positive The takeaway: the kernel is the only thing that changed, and it changed the answer. The linear SVM calls both test vectors positive; the RBF one splits them. Everything after the kernel matrix, the signed weights and the weighted combination and the sign, is the same page of arithmetic twice. 💾 Save this post!
0:44

SVM by hand ✍️ ~ 19 steps walkthrough below (Linear vs RBF) Support Vector Machines reigned supreme in machine learning before the deep learning revolution. An SVM predicts with dot products, the same matrix multiplication every model uses. What it does not do is train by backpropagation: it is fitted by convex optimization, so there is no matrix-multiplication backward pass for a GPU to accelerate. I drew and calculated two SVMs by hand: a linear one (top) and an RBF one (bottom), classifying the same two test vectors. Goal: turn six training vectors and their learned coefficients into a prediction, and see what changing the kernel actually changes. = 1. Given = Six training vectors, their labels, and the coefficients and bias already learned. A coefficient of zero means that vector is not a support vector: too far from the boundary to matter. = 2. Linear kernel, test vector 1 = Let us take the dot product of the test vector with every training vector. The dot product stands in for cosine similarity, and the column of results is the first column of the kernel matrix K. = 3. Linear kernel, test vector 2 = We do the same for the second, and K is complete. = 4. Signed weights = Let us multiply each coefficient by its label. The second training vector drops out here, because its coefficient is 0. = 5. Weighted combination = We multiply the signed weights through K and add the bias b. The result is a signed distance to the decision boundary: 17 and 5. = 6. Classify = Let us take the sign. Both are positive. = 7 to 11. RBF kernel, test vector 1 = Now the same picture with a different kernel, in five moves: square the differences, sum them, take the square root for the L2 distance, multiply by minus gamma, and raise e to that power. The negation is what turns a distance into a similarity, and gamma controls how far a single training vector's influence reaches. = 12 to 16. RBF kernel, test vector 2 = We repeat all five. The numbers change, the moves do not. = 17 to 19. Decision boundary, again = Signed weights, weighted combination, sign. Identical arithmetic to steps 4 through 6, on a K that was built a completely different way. The outputs: Linear K, first column = [13, 25, 12, 15, 19, 27] Linear decision values = 17 and 5, both positive RBF decision values = -2 and 1, so negative and positive The takeaway: the kernel is the only thing that changed, and it changed the answer. The linear SVM calls both test vectors positive; the RBF one splits them. Everything after the kernel matrix, the signed weights and the weighted combination and the sign, is the same page of arithmetic twice. 💾 Save this post!

Tom Yeh

16,916 Aufrufe • vor 2 Monaten

In 2006, Netflix offered $1,000,000 to anyone who could improve their recommendation algorithm by 10%. Over 2,000 teams competed for three years. The team that won did not use more data. They used fewer dimensions. They found a basis - a small set of independent vectors that captured everything important about 100,000,000 movie ratings. The lead mathematician on the winning team: $2,800,000 a year. A machine learning engineer at Spotify building the same kind of system: $245,000 a year. This is MIT 18.06, Lecture 9 - Gilbert Strang. Free on YouTube. Most people think independence is obvious. Two vectors pointing in different directions. Then the definition. Independence means no combination of your vectors gives the zero vector - except the trivial one where all the coefficients are zero. That's it. That's the whole definition. But watch what it unlocks. Watch the moment Strang puts three vectors in a two-dimensional plane. He doesn't even tell you which three vectors. He just draws them. And immediately says: dependent. No question. No calculation. Why? Because three vectors in two-dimensional space means more columns than rows. More unknowns than equations. That always forces a free variable. A free variable always gives a non-zero solution to Ax = 0. And that non-zero solution is a combination of the columns that produces zero. Dependence. "Three vectors in the plane have to be dependent. That's the key fact." Then the basis. A basis is vectors that are independent and span the space. Not too few, not too many. Just right. The pivot columns of any matrix form a basis for the column space. Every other basis you can think of will have exactly the same number of vectors. Then the dimension. All bases for the same space have the same number of vectors. That number is the dimension. The rank of a matrix is the dimension of its column space. The number of free variables is the dimension of the null space. And rank plus null space dimension equals the total number of columns. "I don't take the dimension of A. I take the dimension of the column space of A. If you use those words right, it shows you've got the idea right." A data scientist at Netflix building recommendation engines: $230,000 a year. A quantitative researcher at Two Sigma finding independent factors in financial markets: $350,000 a year. A computer vision engineer at Apple using low-dimensional representations for face recognition: $260,000 a year. They all needed to know how many dimensions were really there. bookmark this and watch later - after this lecture every dataset you look at will feel like a matrix waiting to be reduced to its basis.
48:56

In 2006, Netflix offered $1,000,000 to anyone who could improve their recommendation algorithm by 10%. Over 2,000 teams competed for three years. The team that won did not use more data. They used fewer dimensions. They found a basis - a small set of independent vectors that captured everything important about 100,000,000 movie ratings. The lead mathematician on the winning team: $2,800,000 a year. A machine learning engineer at Spotify building the same kind of system: $245,000 a year. This is MIT 18.06, Lecture 9 - Gilbert Strang. Free on YouTube. Most people think independence is obvious. Two vectors pointing in different directions. Then the definition. Independence means no combination of your vectors gives the zero vector - except the trivial one where all the coefficients are zero. That's it. That's the whole definition. But watch what it unlocks. Watch the moment Strang puts three vectors in a two-dimensional plane. He doesn't even tell you which three vectors. He just draws them. And immediately says: dependent. No question. No calculation. Why? Because three vectors in two-dimensional space means more columns than rows. More unknowns than equations. That always forces a free variable. A free variable always gives a non-zero solution to Ax = 0. And that non-zero solution is a combination of the columns that produces zero. Dependence. "Three vectors in the plane have to be dependent. That's the key fact." Then the basis. A basis is vectors that are independent and span the space. Not too few, not too many. Just right. The pivot columns of any matrix form a basis for the column space. Every other basis you can think of will have exactly the same number of vectors. Then the dimension. All bases for the same space have the same number of vectors. That number is the dimension. The rank of a matrix is the dimension of its column space. The number of free variables is the dimension of the null space. And rank plus null space dimension equals the total number of columns. "I don't take the dimension of A. I take the dimension of the column space of A. If you use those words right, it shows you've got the idea right." A data scientist at Netflix building recommendation engines: $230,000 a year. A quantitative researcher at Two Sigma finding independent factors in financial markets: $350,000 a year. A computer vision engineer at Apple using low-dimensional representations for face recognition: $260,000 a year. They all needed to know how many dimensions were really there. bookmark this and watch later - after this lecture every dataset you look at will feel like a matrix waiting to be reduced to its basis.

Zyphor

14,720 Aufrufe • vor 1 Monat

Day 9/90 of Inference Engineering I built a visualization tool for CUDA kernels and C code to view the buffers for each step through. Check out Yesterday, I was terramogged by a simple implementation of a basic matmul between three matrices in 1D vector form. There were essentially three 1D vectors, each vector representing a corresponding matrix (A, B, C). In the implementation, each matrix is just just a pointer to the first element of a 1D array. You are to compute the output matrix through A^T @ B and store it in C. This morning I built a visualizer to help me see and understand why the triple for-loops work the way they do. In the coming days, I’m confident I can make sense as well as backpropagation! As for it does not include a built-in compiler. It's just an interpreter; the visualization shows exactly what the code does, bugs included. The main thing I wanted to visualize is matmul with a triple for-loop between a sequential program to that of a parallel one. Expect bugs and other inconsistencies within I’ll polish this mini-project in the coming days. As for the rest of today, I’ll be trying to understand backpropagation in C and CUDA .
0:20

Day 9/90 of Inference Engineering I built a visualization tool for CUDA kernels and C code to view the buffers for each step through. Check out Yesterday, I was terramogged by a simple implementation of a basic matmul between three matrices in 1D vector form. There were essentially three 1D vectors, each vector representing a corresponding matrix (A, B, C). In the implementation, each matrix is just just a pointer to the first element of a 1D array. You are to compute the output matrix through A^T @ B and store it in C. This morning I built a visualizer to help me see and understand why the triple for-loops work the way they do. In the coming days, I’m confident I can make sense as well as backpropagation! As for it does not include a built-in compiler. It's just an interpreter; the visualization shows exactly what the code does, bugs included. The main thing I wanted to visualize is matmul with a triple for-loop between a sequential program to that of a parallel one. Expect bugs and other inconsistencies within I’ll polish this mini-project in the coming days. As for the rest of today, I’ll be trying to understand backpropagation in C and CUDA .

max fu

12,746 Aufrufe • vor 2 Monaten

Complex numbers aren't imaginary. They're the only way to describe a wave that has both amplitude and phase at the same time - and every LLM processes language as exactly that kind of wave through its attention layers. The same math that describes quantum wave interference is what makes Fourier transforms work, which is what lets GPT process tokens in parallel instead of one at a time. Descartes named them "imaginary" before anyone understood what they were. 20 minutes. Bookmark & watch today. The confusion clears up faster than any textbook manages.
19:29

Complex numbers aren't imaginary. They're the only way to describe a wave that has both amplitude and phase at the same time - and every LLM processes language as exactly that kind of wave through its attention layers. The same math that describes quantum wave interference is what makes Fourier transforms work, which is what lets GPT process tokens in parallel instead of one at a time. Descartes named them "imaginary" before anyone understood what they were. 20 minutes. Bookmark & watch today. The confusion clears up faster than any textbook manages.

Eron

43,247 Aufrufe • vor 1 Monat

Every LLM you've used runs on physics governed by divergence and curl. Maxwell's equations — the complete theory of light, electricity, and magnetism — are just four statements written in this language. One of them says the divergence of the magnetic field is zero everywhere, which is the exact mathematical reason magnetic monopoles can't exist. Divergence measures how much a vector field acts as a source or sink at a point; curl measures how much it rotates. That's it. But those two tools describe predator-prey population dynamics, the phase spaces of any differential equation, and the loss landscapes your GPU navigates during backpropagation — any system where a vector tells you which direction things want to move. 13 minutes. Bookmark & watch today.
14:27

Every LLM you've used runs on physics governed by divergence and curl. Maxwell's equations — the complete theory of light, electricity, and magnetism — are just four statements written in this language. One of them says the divergence of the magnetic field is zero everywhere, which is the exact mathematical reason magnetic monopoles can't exist. Divergence measures how much a vector field acts as a source or sink at a point; curl measures how much it rotates. That's it. But those two tools describe predator-prey population dynamics, the phase spaces of any differential equation, and the loss landscapes your GPU navigates during backpropagation — any system where a vector tells you which direction things want to move. 13 minutes. Bookmark & watch today.

Eron

25,166 Aufrufe • vor 14 Tagen

MIT filmed a mathematician proving the theorem behind every equation Maxwell ever wrote. Most engineers use it daily without understanding why any surface gives the same answer - and the ones who can explain it earn $350K at aerospace firms. Stokes' theorem is the reason Maxwell could write four equations instead of forty. Every result in electromagnetism, fluid mechanics, and differential geometry sits on top of it. Understanding it changes how you see every circulation and flux problem in physics. This is Denis Auroux. MIT, 18.02, Multivariable Calculus, Fall 2007. Lecture 31. It covers Stokes' theorem - the result that unifies line integrals, surface integrals, and curl into one statement. He opens with one confession. Then the statement. The work done by a vector field along any closed curve equals the flux of its curl through any surface bounded by that curve. Not a specific surface. Any surface you want. You pick whichever one makes the calculation easier. Then Green's theorem disappears. The result from the plane that converts line integrals to double integrals turns out to be Stokes' theorem in disguise. Same formula, restricted to a flat surface in the xy-plane. What looked like two separate theorems is one. Then orientation. Walk along the curve with the surface to your left and the normal vector points up. This rule is not chosen for elegance. It is the price of having a consistent coordinate system in three dimensions. Then the proof. Cut your surface into tiny flat tiles. Apply Green's theorem to each one. Sum everything up. All interior boundary edges cancel. Only the outer edge survives. The sum of tiny Green's theorems gives you Stokes' theorem for any surface. Watch the moment he computes the same line integral two ways. First directly - answer is pi. Then through a paraboloid instead of the obvious flat disk. Longer calculation, stranger setup, same answer. The theorem is not an abstraction. You can watch it work. A fluid dynamics engineer I know kept this lecture open during their first week modeling atmospheric circulation. Said it was the first time Stokes felt like a tool rather than a formula to verify on exams. Free on YouTube, MIT OpenCourseWare, 18.02. bookmark this and watch later - after this lecture every circulation, every flux, and every curl you encounter will feel like three views of the same object
47:56

MIT filmed a mathematician proving the theorem behind every equation Maxwell ever wrote. Most engineers use it daily without understanding why any surface gives the same answer - and the ones who can explain it earn $350K at aerospace firms. Stokes' theorem is the reason Maxwell could write four equations instead of forty. Every result in electromagnetism, fluid mechanics, and differential geometry sits on top of it. Understanding it changes how you see every circulation and flux problem in physics. This is Denis Auroux. MIT, 18.02, Multivariable Calculus, Fall 2007. Lecture 31. It covers Stokes' theorem - the result that unifies line integrals, surface integrals, and curl into one statement. He opens with one confession. Then the statement. The work done by a vector field along any closed curve equals the flux of its curl through any surface bounded by that curve. Not a specific surface. Any surface you want. You pick whichever one makes the calculation easier. Then Green's theorem disappears. The result from the plane that converts line integrals to double integrals turns out to be Stokes' theorem in disguise. Same formula, restricted to a flat surface in the xy-plane. What looked like two separate theorems is one. Then orientation. Walk along the curve with the surface to your left and the normal vector points up. This rule is not chosen for elegance. It is the price of having a consistent coordinate system in three dimensions. Then the proof. Cut your surface into tiny flat tiles. Apply Green's theorem to each one. Sum everything up. All interior boundary edges cancel. Only the outer edge survives. The sum of tiny Green's theorems gives you Stokes' theorem for any surface. Watch the moment he computes the same line integral two ways. First directly - answer is pi. Then through a paraboloid instead of the obvious flat disk. Longer calculation, stranger setup, same answer. The theorem is not an abstraction. You can watch it work. A fluid dynamics engineer I know kept this lecture open during their first week modeling atmospheric circulation. Said it was the first time Stokes felt like a tool rather than a formula to verify on exams. Free on YouTube, MIT OpenCourseWare, 18.02. bookmark this and watch later - after this lecture every circulation, every flux, and every curl you encounter will feel like three views of the same object

Zyphor

11,286 Aufrufe • vor 1 Monat

Foucault's pendulum is an experiment demonstrating the Earth's rotation. Léon Foucault first performed the experiment in 1851, making the discovery at 2 am in the cellar he shared with his mother. How does it work? The elegant answer is that the pendulum swings in a fixed plane and the Earth rotates beneath it, but this explanation is misleading. At the north or south pole, the pendulum is moving in a fixed plane (if we disregard the fact that the Earth is also revolving through space), so the plane of the pendulum seems to rotate through 360° as the Earth makes one full rotation. At any other point on Earth, however, the point at which the pendulum is attached cannot be considered a "fixed point", because that point also moves as the Earth rotates. The plane in which the pendulum swings is similarly in motion. Because of this, the amount of time that it takes for the pendulum to make one full rotation (with respect to its surroundings) is equal to one sidereal day (23.93 hours) divided by the sine of the latitude of its location. Since sin(0)=0, the plane of a pendulum located at the equator will not appear to move at all. But in all the other locations it will. Thus if you stood watching the pendulum, after a quarter of an hour or so, you would be likely to notice that the line of the pendulum's swing has changed to a different direction.
0:25

Foucault's pendulum is an experiment demonstrating the Earth's rotation. Léon Foucault first performed the experiment in 1851, making the discovery at 2 am in the cellar he shared with his mother. How does it work? The elegant answer is that the pendulum swings in a fixed plane and the Earth rotates beneath it, but this explanation is misleading. At the north or south pole, the pendulum is moving in a fixed plane (if we disregard the fact that the Earth is also revolving through space), so the plane of the pendulum seems to rotate through 360° as the Earth makes one full rotation. At any other point on Earth, however, the point at which the pendulum is attached cannot be considered a "fixed point", because that point also moves as the Earth rotates. The plane in which the pendulum swings is similarly in motion. Because of this, the amount of time that it takes for the pendulum to make one full rotation (with respect to its surroundings) is equal to one sidereal day (23.93 hours) divided by the sine of the latitude of its location. Since sin(0)=0, the plane of a pendulum located at the equator will not appear to move at all. But in all the other locations it will. Thus if you stood watching the pendulum, after a quarter of an hour or so, you would be likely to notice that the line of the pendulum's swing has changed to a different direction.

Massimo

3,931,462 Aufrufe • vor 2 Jahren

CLIP by hand ✍️ ~ 13 steps walkthrough below CLIP, Contrastive Language-Image Pre-training, is OpenAI's answer to a question that sounds impossible: how do you put a sentence and a picture in the same space? CLIP shipped when OpenAI was still open, and those embeddings were shared far and wide. Almost every multimodal model you use today descends from them. How does it work? Goal: learn one shared embedding space for text and images. = 1. Given = A mini batch of three text-image pairs. OpenAI trained the original on 400 million. = 2. Text to vectors = Let us look up each word with word2vec. = 3. Image to vectors = We cut each image into two patches and flatten them. Now text and pixels are both just numbers. = 4. The other pairs = Repeat steps 2 and 3 for the rest of the batch. = 5. Encode = Let us push both sides through their encoders, a linear layer and a ReLU. In practice these are transformers, but the shape of the operation is the same. = 6. Mean pooling = We average across the columns, so each image and each sentence collapses to a single vector. = 7. Projection = The text vectors are 3D and the image vectors are 4D, so they cannot be compared at all. A linear layer projects both to 2D. That 2D space is the shared embedding space, and getting here is the whole point of the model. = 8. Prepare for matmul = Let us copy the text vectors down and the transposed image vectors across. = 9. MatMul = We multiply, which takes the dot product of every text vector with every image vector. Each cell is one estimate of how well a sentence matches a picture. = 10. Softmax, e to the power = Raise e to each cell. To keep it hand sized we approximate e with 3. = 11. Softmax, sum = Sum each row for image to text, each column for text to image. = 12. Softmax, normalize = Divide, and out come two similarity matrices, one per direction. = 13. Loss gradients = The targets are identity matrices: a pair that belongs together should score 1, every other cell 0. Subtract the target from the similarity and you have the gradients, in both directions. The takeaway: pairing a picture with a sentence comes down to a single dot product. Everything before step 9 is the work of getting them into one shared space, so that the dot product finally means something. 💾 Save this post!
0:36

CLIP by hand ✍️ ~ 13 steps walkthrough below CLIP, Contrastive Language-Image Pre-training, is OpenAI's answer to a question that sounds impossible: how do you put a sentence and a picture in the same space? CLIP shipped when OpenAI was still open, and those embeddings were shared far and wide. Almost every multimodal model you use today descends from them. How does it work? Goal: learn one shared embedding space for text and images. = 1. Given = A mini batch of three text-image pairs. OpenAI trained the original on 400 million. = 2. Text to vectors = Let us look up each word with word2vec. = 3. Image to vectors = We cut each image into two patches and flatten them. Now text and pixels are both just numbers. = 4. The other pairs = Repeat steps 2 and 3 for the rest of the batch. = 5. Encode = Let us push both sides through their encoders, a linear layer and a ReLU. In practice these are transformers, but the shape of the operation is the same. = 6. Mean pooling = We average across the columns, so each image and each sentence collapses to a single vector. = 7. Projection = The text vectors are 3D and the image vectors are 4D, so they cannot be compared at all. A linear layer projects both to 2D. That 2D space is the shared embedding space, and getting here is the whole point of the model. = 8. Prepare for matmul = Let us copy the text vectors down and the transposed image vectors across. = 9. MatMul = We multiply, which takes the dot product of every text vector with every image vector. Each cell is one estimate of how well a sentence matches a picture. = 10. Softmax, e to the power = Raise e to each cell. To keep it hand sized we approximate e with 3. = 11. Softmax, sum = Sum each row for image to text, each column for text to image. = 12. Softmax, normalize = Divide, and out come two similarity matrices, one per direction. = 13. Loss gradients = The targets are identity matrices: a pair that belongs together should score 1, every other cell 0. Subtract the target from the similarity and you have the gradients, in both directions. The takeaway: pairing a picture with a sentence comes down to a single dot product. Everything before step 9 is the work of getting them into one shared space, so that the dot product finally means something. 💾 Save this post!

Tom Yeh

20,929 Aufrufe • vor 2 Monaten

New short course: Attention in Transformers: Concepts and Code in PyTorch. Last week we released a course on how LLM transformers work. This week, go deeper and learn about the technical ideas behind the attention mechanism, and see how to code it in PyTorch. This course is built with Joshua Starmer, Founder and CEO of StatQuest. The attention mechanism was a breakthrough that led to transformers, the architecture powering large language models like ChatGPT. Transformers, introduced in the 2017 paper: "Attention is All You Need" by Viswani and others, took off because of its highly scalable design. In this course, you’ll learn how the attention mechanism, a key element of transformer-based LLMs, works and implement it in PyTorch. You'll develop deep intuition about building reliable, functional, and scalable AI applications. What you will do: - Understand the evolution of the attention mechanism, a key breakthrough that led to transformers. - Learn the relationships between word embeddings, positional embeddings, and attention. - Learn about the Query, Key, and Value matrices, and how to produce and use them in attention. - Walk through the math required to calculate self-attention and masked self-attention to learn why and how they work. - Understand the difference between self-attention and masked self-attention and how one is used in the encoder to build context-aware embeddings and the other is used in the decoder for generative outputs. - Learn the details of the encoder-decoder architecture, cross-attention, and multi-head attention and how they are all incorporated into a transformer. - Use PyTorch to code a class that implements self-attention, masked self-attention, and multi-head attention. There're lots of exciting technical details in this course. Please sign up here:
2:41

New short course: Attention in Transformers: Concepts and Code in PyTorch. Last week we released a course on how LLM transformers work. This week, go deeper and learn about the technical ideas behind the attention mechanism, and see how to code it in PyTorch. This course is built with Joshua Starmer, Founder and CEO of StatQuest. The attention mechanism was a breakthrough that led to transformers, the architecture powering large language models like ChatGPT. Transformers, introduced in the 2017 paper: "Attention is All You Need" by Viswani and others, took off because of its highly scalable design. In this course, you’ll learn how the attention mechanism, a key element of transformer-based LLMs, works and implement it in PyTorch. You'll develop deep intuition about building reliable, functional, and scalable AI applications. What you will do: - Understand the evolution of the attention mechanism, a key breakthrough that led to transformers. - Learn the relationships between word embeddings, positional embeddings, and attention. - Learn about the Query, Key, and Value matrices, and how to produce and use them in attention. - Walk through the math required to calculate self-attention and masked self-attention to learn why and how they work. - Understand the difference between self-attention and masked self-attention and how one is used in the encoder to build context-aware embeddings and the other is used in the decoder for generative outputs. - Learn the details of the encoder-decoder architecture, cross-attention, and multi-head attention and how they are all incorporated into a transformer. - Use PyTorch to code a class that implements self-attention, masked self-attention, and multi-head attention. There're lots of exciting technical details in this course. Please sign up here:

Andrew Ng

132,544 Aufrufe • vor 1 Jahr

context engineering vs graph engineering. every few months the list gets a new word and everyone treats it as a replacement for the last one. these two are not on the same list. one decides what the model sees this turn, the other decides what exists at all. the cleanest way to tell them apart is to ask what a single unit of work looks like. > context engineering is the window the window opens empty, every single time. you assemble what goes in it. the prompt, the docs, the history, the tool results. the assembling is the work. the window only grows. it never shrinks on its own, so eventually something gets dropped. usually from the middle. usually without telling you. then the turn ends and the window is thrown away. not archived, thrown away. the next turn opens empty again and you re-explain what you already explained. good context engineering is knowing what to leave out, not what to pack in. the unit of work is one window. > graph engineering is the structure the same material arrives from the same sources. instead of packing it into a window, you pull entities out of it, resolve the duplicates into one node, and write typed edges between them. nothing here is stored as text you hope to find again. it is stored as a thing with a name and its connections to other things. when the turn ends, the graph is still there. the next turn does not start from zero. it starts by querying what already exists, and the query walks edges instead of guessing at similarity. good graph engineering is deciding what counts as the same thing twice. the unit of work is one relationship. > they are not alternatives the graph is what refills the window. context engineering decides what fits. graph engineering decides what there is to choose from. remove the graph and every session starts blind. remove the context work and the best structure in the world arrives as an unreadable dump. that also tells you which one broke. the answer drifted from what you actually said, or forgot something from this same session. that is the window. the answer is coherent but invents a connection that does not exist, or cannot join two facts it has clearly seen. that is the structure. people debug the prompt because the prompt is the easiest thing to edit. it keeps taking the blame for failures that live a layer down. save this - then read the full breakdown below
0:40

context engineering vs graph engineering. every few months the list gets a new word and everyone treats it as a replacement for the last one. these two are not on the same list. one decides what the model sees this turn, the other decides what exists at all. the cleanest way to tell them apart is to ask what a single unit of work looks like. > context engineering is the window the window opens empty, every single time. you assemble what goes in it. the prompt, the docs, the history, the tool results. the assembling is the work. the window only grows. it never shrinks on its own, so eventually something gets dropped. usually from the middle. usually without telling you. then the turn ends and the window is thrown away. not archived, thrown away. the next turn opens empty again and you re-explain what you already explained. good context engineering is knowing what to leave out, not what to pack in. the unit of work is one window. > graph engineering is the structure the same material arrives from the same sources. instead of packing it into a window, you pull entities out of it, resolve the duplicates into one node, and write typed edges between them. nothing here is stored as text you hope to find again. it is stored as a thing with a name and its connections to other things. when the turn ends, the graph is still there. the next turn does not start from zero. it starts by querying what already exists, and the query walks edges instead of guessing at similarity. good graph engineering is deciding what counts as the same thing twice. the unit of work is one relationship. > they are not alternatives the graph is what refills the window. context engineering decides what fits. graph engineering decides what there is to choose from. remove the graph and every session starts blind. remove the context work and the best structure in the world arrives as an unreadable dump. that also tells you which one broke. the answer drifted from what you actually said, or forgot something from this same session. that is the window. the answer is coherent but invents a connection that does not exist, or cannot join two facts it has clearly seen. that is the structure. people debug the prompt because the prompt is the easiest thing to edit. it keeps taking the blame for failures that live a layer down. save this - then read the full breakdown below

Hanako

19,160 Aufrufe • vor 2 Monaten