Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

Scaled Dot-Product Attention by hand ✍️ I explained attention in my recent AI seminar as follows: Step 1 — Compare We take one token as a query and compare it against all keys using dot products. This gives us a grid of similarity scores: every query against every key....

ProfTomYeh's profile picture

Tom Yeh

56,031 subscribers

31,114 görüntüleme • 6 ay önce •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

[Self-Attention] by Hand ✍️ Self-attention is what enables LLMs to understand context. How does it work? This exercise demonstrates how to calculate a 6-3 attention head by hand. Note that if we have two instances of this, we get 6-6 attention (i.e., multi-head attention, n=2). -- 𝗚𝗼𝗮𝗹 -- Transform [6D Features 🟧] to [3D Attention Weighted Features 🟦] -- 𝗪𝗮𝗹𝗸𝘁𝗵𝗿𝗼𝘂𝗴𝗵 -- [1] Given ↳ A set of 4 feature vectors (6-D): x1,x2,x3,x4 [2] Query, Key, Value ↳ Multiply features x's with linear transformation matrices WQ, WK, and WV, to obtain query vectors (q1,q2,q3,q4), key vectors (k1,k2,k3,k4), and value vectors (v1,v2,v3,v4). ↳ "Self" refers to the fact that both queries and keys are derived from the same set of features. [3] 🟪 Prepare for MatMul ↳ Copy query vectors ↳ Copy the transpose of key vectors [4] 🟪 MatMul ↳ Multiply K^T and Q ↳ This is equivalent to taking dot product between every pair of query and key vectors. ↳ The purpose is to use dot product as an estimate of the "matching score" between every key-value pair. ↳ This estimate makes sense because dot product is the numerator of Cosine Similarity between two vectors. [5] 🟨 Scale ↳ Scale each element by the square root of dk, which is the dimension of key vectors (dk=3). ↳ The purpose is to normalize the impact of the dk on matching scores, even if we scale dk to 32, 64, or 128. ↳ To simplify hand calculation, we approximate [ □/sqrt(3) ] with [ floor(□/2) ]. [6] 🟩 Softmax: e^x ↳ Raise e to the power of the number in each cell ↳ To simplify hand calculation, we approximate e^□ with 3^□. [7] 🟩 Softmax: ∑ ↳ Sum across each column [8] 🟩 Softmax: 1 / sum ↳ For each column, divide each element by the column sum ↳ The purpose is normalize each column so that the numbers sum to 1. In other words, each column is a probability distribution of attention, and we have four of them. ↳ The result is the Attention Weight Matrix (A) (yellow) [9] 🟦 MatMul ↳ Multiply the value vectors (Vs) with the Attention Weight Matrix (A) ↳ The results are the attention weighted features Zs. ↳ They are fed to the position-wise feed forward network in the next layer.
0:32

[Self-Attention] by Hand ✍️ Self-attention is what enables LLMs to understand context. How does it work? This exercise demonstrates how to calculate a 6-3 attention head by hand. Note that if we have two instances of this, we get 6-6 attention (i.e., multi-head attention, n=2). -- 𝗚𝗼𝗮𝗹 -- Transform [6D Features 🟧] to [3D Attention Weighted Features 🟦] -- 𝗪𝗮𝗹𝗸𝘁𝗵𝗿𝗼𝘂𝗴𝗵 -- [1] Given ↳ A set of 4 feature vectors (6-D): x1,x2,x3,x4 [2] Query, Key, Value ↳ Multiply features x's with linear transformation matrices WQ, WK, and WV, to obtain query vectors (q1,q2,q3,q4), key vectors (k1,k2,k3,k4), and value vectors (v1,v2,v3,v4). ↳ "Self" refers to the fact that both queries and keys are derived from the same set of features. [3] 🟪 Prepare for MatMul ↳ Copy query vectors ↳ Copy the transpose of key vectors [4] 🟪 MatMul ↳ Multiply K^T and Q ↳ This is equivalent to taking dot product between every pair of query and key vectors. ↳ The purpose is to use dot product as an estimate of the "matching score" between every key-value pair. ↳ This estimate makes sense because dot product is the numerator of Cosine Similarity between two vectors. [5] 🟨 Scale ↳ Scale each element by the square root of dk, which is the dimension of key vectors (dk=3). ↳ The purpose is to normalize the impact of the dk on matching scores, even if we scale dk to 32, 64, or 128. ↳ To simplify hand calculation, we approximate [ □/sqrt(3) ] with [ floor(□/2) ]. [6] 🟩 Softmax: e^x ↳ Raise e to the power of the number in each cell ↳ To simplify hand calculation, we approximate e^□ with 3^□. [7] 🟩 Softmax: ∑ ↳ Sum across each column [8] 🟩 Softmax: 1 / sum ↳ For each column, divide each element by the column sum ↳ The purpose is normalize each column so that the numbers sum to 1. In other words, each column is a probability distribution of attention, and we have four of them. ↳ The result is the Attention Weight Matrix (A) (yellow) [9] 🟦 MatMul ↳ Multiply the value vectors (Vs) with the Attention Weight Matrix (A) ↳ The results are the attention weighted features Zs. ↳ They are fed to the position-wise feed forward network in the next layer.

Tom Yeh

101,010 görüntüleme • 2 yıl önce

[CLIP] by Hand ✍️ The CLIP (Contrastive Language–Image Pre-training) model, a groundbreaking work by OpenAI, redefines the intersection of computer vision and natural language processing. It is the basis of all the multi-modal foundation models we see today. How does CLIP work? Goal: 🟨 Learn a shared embedding space for text and image [1] Given ↳ A mini batch of 3 text-image pairs ↳ OpenAI used 400 million text-image pairs to train its original CLIP model. Process 1st pair: "big table" [2] 🟪 Text → 2 Vectors (3D) ↳ Look up word embedding vectors using word2vec. [3] 🟩 Image → 2 Vectors (4D) ↳ Divide the image into two patches. ↳ Flatten each patch [4] Process other pairs ↳ Repeat [2]-[3] [5] 🟪 Text Encoder & 🟩 Image Encoder ↳ Encode input vectors into feature vectors ↳ Here, both encoders are simple one layer perceptron (linear + ReLU) ↳ In practice, the encoders are usually transformer models. [6] 🟪 🟩 Mean Pooling: 2 → 1 vector ↳ Average 2 feature vectors into a single vector by averaging across the columns ↳ The goal is to have one vector to represent each image or text [7] 🟪 🟩 -> 🟨 Projection ↳ Note that the text and image feature vectors from the encoders have different dimensions (3D vs. 4D). ↳ Use a linear layer to project image and text vectors to a 2D shared embedding space. 🏋️ Contrastive Pre-training 🏋️ [8] Prepare for MatMul ↳ Copy text vectors (T1,T2,T3) ↳ Copy the transpose of image vectors (I1,I2,I3) ↳ They are all in the 2D shared embedding space. [9] 🟦 MatMul ↳ Multiply T and I matrices. ↳ This is equivalent to taking dot product between every pair of image and text vectors. ↳ The purpose is to use dot product to estimate the similarity between a pair of image-text. [10] 🟦 Softmax: e^x ↳ Raise e to the power of the number in each cell ↳ To simplify hand calculation, we approximate e^□ with 3^□. [11] 🟦 Softmax: ∑ ↳ Sum each row for 🟩 image→🟪 text ↳ Sum each column for 🟪 text→ 🟩 image [12] 🟦 Softmax: 1 / sum ↳ Divide each element by the column sum to obtain a similarity matrix for 🟪 text→🟩 image ↳ Divide each element by the row sum to obtain a similarity matrix for 🟩 image→🟪 text [13] 🟥 Loss Gradients ↳ The "Targets" for the similarity matrices are Identity Matrices. ↳ Why? If I and T come from the same pair (i=j), we want the highest value, which is 1, and 0 otherwise. ↳ Apply the simple equation of [Similarity - Target] to compute gradients of for both directions. ↳ Why so simple? Because when Softmax and Cross-Entropy Loss are used together, the math magically works out that way. ↳ These gradients kick off the backpropagation process to update weights and biases of the encoders and projection layers (red borders).
1:20

[CLIP] by Hand ✍️ The CLIP (Contrastive Language–Image Pre-training) model, a groundbreaking work by OpenAI, redefines the intersection of computer vision and natural language processing. It is the basis of all the multi-modal foundation models we see today. How does CLIP work? Goal: 🟨 Learn a shared embedding space for text and image [1] Given ↳ A mini batch of 3 text-image pairs ↳ OpenAI used 400 million text-image pairs to train its original CLIP model. Process 1st pair: "big table" [2] 🟪 Text → 2 Vectors (3D) ↳ Look up word embedding vectors using word2vec. [3] 🟩 Image → 2 Vectors (4D) ↳ Divide the image into two patches. ↳ Flatten each patch [4] Process other pairs ↳ Repeat [2]-[3] [5] 🟪 Text Encoder & 🟩 Image Encoder ↳ Encode input vectors into feature vectors ↳ Here, both encoders are simple one layer perceptron (linear + ReLU) ↳ In practice, the encoders are usually transformer models. [6] 🟪 🟩 Mean Pooling: 2 → 1 vector ↳ Average 2 feature vectors into a single vector by averaging across the columns ↳ The goal is to have one vector to represent each image or text [7] 🟪 🟩 -> 🟨 Projection ↳ Note that the text and image feature vectors from the encoders have different dimensions (3D vs. 4D). ↳ Use a linear layer to project image and text vectors to a 2D shared embedding space. 🏋️ Contrastive Pre-training 🏋️ [8] Prepare for MatMul ↳ Copy text vectors (T1,T2,T3) ↳ Copy the transpose of image vectors (I1,I2,I3) ↳ They are all in the 2D shared embedding space. [9] 🟦 MatMul ↳ Multiply T and I matrices. ↳ This is equivalent to taking dot product between every pair of image and text vectors. ↳ The purpose is to use dot product to estimate the similarity between a pair of image-text. [10] 🟦 Softmax: e^x ↳ Raise e to the power of the number in each cell ↳ To simplify hand calculation, we approximate e^□ with 3^□. [11] 🟦 Softmax: ∑ ↳ Sum each row for 🟩 image→🟪 text ↳ Sum each column for 🟪 text→ 🟩 image [12] 🟦 Softmax: 1 / sum ↳ Divide each element by the column sum to obtain a similarity matrix for 🟪 text→🟩 image ↳ Divide each element by the row sum to obtain a similarity matrix for 🟩 image→🟪 text [13] 🟥 Loss Gradients ↳ The "Targets" for the similarity matrices are Identity Matrices. ↳ Why? If I and T come from the same pair (i=j), we want the highest value, which is 1, and 0 otherwise. ↳ Apply the simple equation of [Similarity - Target] to compute gradients of for both directions. ↳ Why so simple? Because when Softmax and Cross-Entropy Loss are used together, the math magically works out that way. ↳ These gradients kick off the backpropagation process to update weights and biases of the encoders and projection layers (red borders).

Tom Yeh

67,834 görüntüleme • 2 yıl önce

Watch as our Guardian-1 interceptors launch remotely from a container off a USV in Ukraine. At Powerus, we continue to innovate and stay one step ahead, all in support of the warfighter. Powerus
0:24

Watch as our Guardian-1 interceptors launch remotely from a container off a USV in Ukraine. At Powerus, we continue to innovate and stay one step ahead, all in support of the warfighter. Powerus

Brett Velicovich

191,471 görüntüleme • 17 gün önce

A special message for the Choir from Eric. “I’ve been involved in every step of restoring this historic building into a place we can call our own. And because you have been with me every step of my career, I'm proud to dedicate a physical brick of the Chief's Building to each and every one of you.” More news coming soon.
0:41

A special message for the Choir from Eric. “I’ve been involved in every step of restoring this historic building into a place we can call our own. And because you have been with me every step of my career, I'm proud to dedicate a physical brick of the Chief's Building to each and every one of you.” More news coming soon.

Eric Church

144,837 görüntüleme • 2 yıl önce

A tricky LLM interview question: You're serving a reasoning model on vLLM, and it keeps running out of GPU memory on long traces. So you add KV cache compression and evict 90% of the cached tokens. VRAM usage stays as is and GPU still runs out of memory. Why? (answer below) Evicting 90% of the KV cache can free almost none of the memory it was using. This sounds counterintuitive, but it follows directly from how production servers store the cache today. The KV cache grows with every token a model generates. Each token appends its key and value vectors across every layer, and nothing is freed while generation continues. This is the dominant memory cost for reasoning models. If a 32K-token CoT caches ~32K tokens of KV vectors, a Qwen3-32B with 4-bit weights will run out-of-memory around 24K tokens on a 24GB GPU. One obvious solution is to keep the important tokens and drop the rest, since attention is sparse enough to allow it. But this does not solve the memory problem yet. The reason is paged attention, which is the memory manager behind vLLM and most production servers. Under the hood, it splits GPU memory into fixed physical blocks, each one holds the KV for about 16 tokens. This block returns to the allocator only when every slot inside it is empty. Since the eviction logic selects tokens by importance, and such tokens are scattered across blocks... ...so despite eviction, almost every block is left with at least some survivor tokens. For instance, if the logic evicts 14k of 16k tokens across 1,000 blocks, most likely every block will still have a token. This means the allocator frees almost nothing. Placing the new tokens into those freed slots is not ideal because it breaks the cache's layout. Say token 16,001 arrives, and it's placed in the slot the 40th token used to hold. The cache now reads position 38, then 16,001, then 41, so the cache is no longer in token order. Attention can still compute the right answer from that, but only if every slot now carries a separate note recording which position it actually holds. This introduces another bookkeeping cost that an in-order layout inherently avoids. So the cache is logically 90% smaller and still physically the same size. Many compression results miss this because they measure on pre-allocated contiguous tensors rather than a paged server. There's another problem. Eviction methods pick which tokens to keep by looking at the attention scores themselves (as expected). But fast attention kernels used in production, like FlashAttention, never save those scores. They compute attention in small pieces and throw the full score grid away as they go, which is also why they're fast. So the exact signal eviction methods need isn't available in memory. The workaround is to fall back to eager attention and build the full matrix, which gives up the speed FlashAttention was there to provide. NVIDIA published a method called TriAttention to solve both these problems. It never needs attention scores. Instead, it scores tokens from the geometry of the model's key and query vectors before RoPE is applied, where those vectors sit in stable clusters. For the memory problem, it runs a compaction pass every 128 decoded tokens. The surviving tokens slide forward to close the holes eviction creates, so whole blocks empty out and return to the allocator while the cache stays in token order. On long reasoning traces, the approach matches full-attention accuracy while decoding 2.5x faster and using 10.7x less KV memory. KV cache compression is a big infrastructure problem. The number that decides whether it works is the count of freed blocks, not the count of evicted tokens. You can find the NVIDIA write-up here: I wrote a first-principles breakdown of how the KV cache works. It walks through why the model stores keys and values at all, why the cache grows with every token, and a comparison of LLM generation speed with and without KV caching. Read it below.
0:23

A tricky LLM interview question: You're serving a reasoning model on vLLM, and it keeps running out of GPU memory on long traces. So you add KV cache compression and evict 90% of the cached tokens. VRAM usage stays as is and GPU still runs out of memory. Why? (answer below) Evicting 90% of the KV cache can free almost none of the memory it was using. This sounds counterintuitive, but it follows directly from how production servers store the cache today. The KV cache grows with every token a model generates. Each token appends its key and value vectors across every layer, and nothing is freed while generation continues. This is the dominant memory cost for reasoning models. If a 32K-token CoT caches ~32K tokens of KV vectors, a Qwen3-32B with 4-bit weights will run out-of-memory around 24K tokens on a 24GB GPU. One obvious solution is to keep the important tokens and drop the rest, since attention is sparse enough to allow it. But this does not solve the memory problem yet. The reason is paged attention, which is the memory manager behind vLLM and most production servers. Under the hood, it splits GPU memory into fixed physical blocks, each one holds the KV for about 16 tokens. This block returns to the allocator only when every slot inside it is empty. Since the eviction logic selects tokens by importance, and such tokens are scattered across blocks... ...so despite eviction, almost every block is left with at least some survivor tokens. For instance, if the logic evicts 14k of 16k tokens across 1,000 blocks, most likely every block will still have a token. This means the allocator frees almost nothing. Placing the new tokens into those freed slots is not ideal because it breaks the cache's layout. Say token 16,001 arrives, and it's placed in the slot the 40th token used to hold. The cache now reads position 38, then 16,001, then 41, so the cache is no longer in token order. Attention can still compute the right answer from that, but only if every slot now carries a separate note recording which position it actually holds. This introduces another bookkeeping cost that an in-order layout inherently avoids. So the cache is logically 90% smaller and still physically the same size. Many compression results miss this because they measure on pre-allocated contiguous tensors rather than a paged server. There's another problem. Eviction methods pick which tokens to keep by looking at the attention scores themselves (as expected). But fast attention kernels used in production, like FlashAttention, never save those scores. They compute attention in small pieces and throw the full score grid away as they go, which is also why they're fast. So the exact signal eviction methods need isn't available in memory. The workaround is to fall back to eager attention and build the full matrix, which gives up the speed FlashAttention was there to provide. NVIDIA published a method called TriAttention to solve both these problems. It never needs attention scores. Instead, it scores tokens from the geometry of the model's key and query vectors before RoPE is applied, where those vectors sit in stable clusters. For the memory problem, it runs a compaction pass every 128 decoded tokens. The surviving tokens slide forward to close the holes eviction creates, so whole blocks empty out and return to the allocator while the cache stays in token order. On long reasoning traces, the approach matches full-attention accuracy while decoding 2.5x faster and using 10.7x less KV memory. KV cache compression is a big infrastructure problem. The number that decides whether it works is the count of freed blocks, not the count of evicted tokens. You can find the NVIDIA write-up here: I wrote a first-principles breakdown of how the KV cache works. It walks through why the model stores keys and values at all, why the cache grows with every token, and a comparison of LLM generation speed with and without KV caching. Read it below.

Avi Chawla

267,206 görüntüleme • 19 gün önce

As we step into the new year, I take a moment to reflect on the journey we have shared in 2024. This year has tested our endurance, pushing us to confront every challenge head-on and emerge stronger, unyielding in our commitment to stand against those who seek to divide and deprive us. 2024 has been a year of profound personal and professional growth, a time when we braved all obstacles and adversities to serve people selflessly. What has truly stood out is the love and blessings of Gonodebota. Without their grace, we would be incomplete. I am deeply grateful to each and every one of you for making this year memorable.
9:32

As we step into the new year, I take a moment to reflect on the journey we have shared in 2024. This year has tested our endurance, pushing us to confront every challenge head-on and emerge stronger, unyielding in our commitment to stand against those who seek to divide and deprive us. 2024 has been a year of profound personal and professional growth, a time when we braved all obstacles and adversities to serve people selflessly. What has truly stood out is the love and blessings of Gonodebota. Without their grace, we would be incomplete. I am deeply grateful to each and every one of you for making this year memorable.

Abhishek Banerjee

16,810 görüntüleme • 1 yıl önce

This Python script helps you better understand how embeddings work for SEO. Input a sentence + a query and BERT will calculate a content "Similarity Score": Search engines use embedding models to translate your content into numeric values. This is how they're able to mathematically determine whether a page on your site is actually relevant to a query someone is searching for. Once both the content and query are run through embeddings - they can calculate the "cosine similarity" or how similar the two entities are to each one. With this Python script, you can actually visualize how cosine similarity works for yourself. To you use you'll simply: 1. Save the Python script to a text file 2. Use your Terminal to run the script 3. You'll be prompted for both a "Sentence" and "Query" 4. Once entered, the script will calculate a "Similarity Score" between the text and query. This is how relevant your sentence is for the target keyword. By running this, you'll see that search engines are able to come with an calculation of content relevance. You'll need to think about the content on your site the same way - which sections have high scores and which ones have low ones? I've linked the Python script in the comments below. No coding knowledge is required and it walks you step by step on how to implement it.
2:37

This Python script helps you better understand how embeddings work for SEO. Input a sentence + a query and BERT will calculate a content "Similarity Score": Search engines use embedding models to translate your content into numeric values. This is how they're able to mathematically determine whether a page on your site is actually relevant to a query someone is searching for. Once both the content and query are run through embeddings - they can calculate the "cosine similarity" or how similar the two entities are to each one. With this Python script, you can actually visualize how cosine similarity works for yourself. To you use you'll simply: 1. Save the Python script to a text file 2. Use your Terminal to run the script 3. You'll be prompted for both a "Sentence" and "Query" 4. Once entered, the script will calculate a "Similarity Score" between the text and query. This is how relevant your sentence is for the target keyword. By running this, you'll see that search engines are able to come with an calculation of content relevance. You'll need to think about the content on your site the same way - which sections have high scores and which ones have low ones? I've linked the Python script in the comments below. No coding knowledge is required and it walks you step by step on how to implement it.

Chris Long

13,185 görüntüleme • 2 yıl önce

The tweet I’m quoting is the list method. In this tweet I’m going to show you the MAP method. Step 1. Load up a game, I’ve loaded Burnley for this example. Step 2. Refresh the page every now and again until you see the map show a section blue. Step 3. Zoom in, see if the dot has loaded in (if not refresh till it does, but try not to refresh a lot) Step 4. When the dot loads, click on and off till it shows prices Here’s a video example of me missing a Burnley one.
0:30

The tweet I’m quoting is the list method. In this tweet I’m going to show you the MAP method. Step 1. Load up a game, I’ve loaded Burnley for this example. Step 2. Refresh the page every now and again until you see the map show a section blue. Step 3. Zoom in, see if the dot has loaded in (if not refresh till it does, but try not to refresh a lot) Step 4. When the dot loads, click on and off till it shows prices Here’s a video example of me missing a Burnley one.

Ballot News

135,702 görüntüleme • 2 ay önce

SORA by Hand ✍️ OpenAI’s #SORA took over the Internet when it was announced earlier this year. The technology behind Sora is the Diffusion Transformer (DiT) developed by William Peebles and Shining Xie. How does DiT work? 𝗚𝗼𝗮𝗹: Generate a video conditioned by a text prompt and a series of diffusion steps [1] Given ↳ Video ↳ Prompt: "sora is sky" ↳ Diffusion step: t = 3 [2] Video → Patches ↳ Divide all pixels in all frames into 4 spacetime patches [3] Visual Encoder: Pixels 🟨 → Latent 🟩 ↳ Multiply the patches with weights and biases, followed by ReLU ↳ The result is a latent feature vector per patch ↳ The purpose is dimension reduction from 4 (2x2x1) to 2 (2x1). ↳ In the paper, the reduction is 196,608 (256x256x3)→ 4096 (32x32x4) [4] ⬛ Add Noise ↳ Sample a noise according to the diffusion time step t. Typically, the larger the t, the smaller the noise. ↳ Add the Sampled Noise to latent features to obtain Noised Latent. ↳ The goal is to purposely add noise to a video and ask the model to guess what that noise is. ↳ This is analogous to training a language model by purposely deleting a word in a sentence and ask the model to guess what the deleted word was. [5-7] 🟪 Conditioning by Adaptive Layer Norm [5] Encode Conditions ↳ Encode "sora is sky" into a text embedding vector [0,1,-1]. ↳ Encode t = 3 to as a binary vector [1,1]. ↳ Concatenate the two vectors in to a 5D column vector. [6] Estimate Scale/Shift ↳ Multiply the combined vector with weights and biases ↳ The goal is to estimate the scale [2,-1] and shift [-1,5]. ↳ Copy the result to (X) and (+) [7] Apply Scale/Sift ↳ Scale the noised latent by [2,-1] ↳ Shifted the scaled noised latent by [-1, 5] ↳ The result is "conditioned" noise latent. [8-10] Transformer [8] Self-Attention ↳ Feed the conditioned noised latent to Query-Key function to obtain a self-attention matrix ↳ Value is omitted for simplicity [9] Attention Pooling ↳ Multiply the conditioned noised latent with the self-attention matrix ↳ The result are attention weighted features [10] Pointwise Feed Forward Network ↳ Multiply the attention weighted features with weights and biases ↳ The result is the Predicted Noise 🏋️‍♂️ 𝗧𝗿𝗮𝗶𝗻 [11] ↳ Calculate MSE loss gradients by taking the different between the Predicted Noise and the Sampled Noise (ground truth). ↳ Use the loss gradients to kick off backpropagation to update all learnable parameters (red borders) ↳ Note the visual encoder and decoder's parameters are frozen (blue borders) 🎨 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗲 (𝗦𝗮𝗺𝗽𝗹𝗲) [12] Denoise ↳ Subtract the predicted noise from the noised latent to obtain the noise-free latent [13] Visual Decoder: Latent 🟩 → Pixels 🟨 ↳ Multiply the patches with weights and biases, followed by ReLU [14] Patches → Video ↳ Rearrange patches into a sequence of video frames.
0:43

SORA by Hand ✍️ OpenAI’s #SORA took over the Internet when it was announced earlier this year. The technology behind Sora is the Diffusion Transformer (DiT) developed by William Peebles and Shining Xie. How does DiT work? 𝗚𝗼𝗮𝗹: Generate a video conditioned by a text prompt and a series of diffusion steps [1] Given ↳ Video ↳ Prompt: "sora is sky" ↳ Diffusion step: t = 3 [2] Video → Patches ↳ Divide all pixels in all frames into 4 spacetime patches [3] Visual Encoder: Pixels 🟨 → Latent 🟩 ↳ Multiply the patches with weights and biases, followed by ReLU ↳ The result is a latent feature vector per patch ↳ The purpose is dimension reduction from 4 (2x2x1) to 2 (2x1). ↳ In the paper, the reduction is 196,608 (256x256x3)→ 4096 (32x32x4) [4] ⬛ Add Noise ↳ Sample a noise according to the diffusion time step t. Typically, the larger the t, the smaller the noise. ↳ Add the Sampled Noise to latent features to obtain Noised Latent. ↳ The goal is to purposely add noise to a video and ask the model to guess what that noise is. ↳ This is analogous to training a language model by purposely deleting a word in a sentence and ask the model to guess what the deleted word was. [5-7] 🟪 Conditioning by Adaptive Layer Norm [5] Encode Conditions ↳ Encode "sora is sky" into a text embedding vector [0,1,-1]. ↳ Encode t = 3 to as a binary vector [1,1]. ↳ Concatenate the two vectors in to a 5D column vector. [6] Estimate Scale/Shift ↳ Multiply the combined vector with weights and biases ↳ The goal is to estimate the scale [2,-1] and shift [-1,5]. ↳ Copy the result to (X) and (+) [7] Apply Scale/Sift ↳ Scale the noised latent by [2,-1] ↳ Shifted the scaled noised latent by [-1, 5] ↳ The result is "conditioned" noise latent. [8-10] Transformer [8] Self-Attention ↳ Feed the conditioned noised latent to Query-Key function to obtain a self-attention matrix ↳ Value is omitted for simplicity [9] Attention Pooling ↳ Multiply the conditioned noised latent with the self-attention matrix ↳ The result are attention weighted features [10] Pointwise Feed Forward Network ↳ Multiply the attention weighted features with weights and biases ↳ The result is the Predicted Noise 🏋️‍♂️ 𝗧𝗿𝗮𝗶𝗻 [11] ↳ Calculate MSE loss gradients by taking the different between the Predicted Noise and the Sampled Noise (ground truth). ↳ Use the loss gradients to kick off backpropagation to update all learnable parameters (red borders) ↳ Note the visual encoder and decoder's parameters are frozen (blue borders) 🎨 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗲 (𝗦𝗮𝗺𝗽𝗹𝗲) [12] Denoise ↳ Subtract the predicted noise from the noised latent to obtain the noise-free latent [13] Visual Decoder: Latent 🟩 → Pixels 🟨 ↳ Multiply the patches with weights and biases, followed by ReLU [14] Patches → Video ↳ Rearrange patches into a sequence of video frames.

Tom Yeh

238,166 görüntüleme • 2 yıl önce

No client can say no to you if you learn how to do this. I went from discovery call to finished spanish client deck in under 2 minutes. Here's how I did it: Step 1: Open Claude Step 2: Connect Gmail, Notion, and gammaapp via MCP (one click each, no API keys) Step 3: "Pull the key points from my discovery call notes in Gmail" Step 4: "Grab the client overview from Notion" Step 5: "Build a campaign proposal deck in Gamma using both" A full deck appeared Check here to start:
0:58

No client can say no to you if you learn how to do this. I went from discovery call to finished spanish client deck in under 2 minutes. Here's how I did it: Step 1: Open Claude Step 2: Connect Gmail, Notion, and gammaapp via MCP (one click each, no API keys) Step 3: "Pull the key points from my discovery call notes in Gmail" Step 4: "Grab the client overview from Notion" Step 5: "Build a campaign proposal deck in Gamma using both" A full deck appeared Check here to start:

Oluwatimileyin✨🦋

89,017 görüntüleme • 4 ay önce

Tough times have made us strong. And the strong bring the times of victory closer. Step by step. Today, tomorrow, every day, every minute. No one should or will be able to "turn off" our resilience, endurance, grit, and fortitude. Neither on a regular nor an emergency basis. None of them have an "expiration date" an "end date", or a final point after which we would stop resisting and fighting. There is only one such point: our victory. As we bring it closer every day, we say, "We will fight for as long as it takes”. We were doing it in the first minutes of February 24th; we have been doing it for all these 585 days, and we will keep doing it. Glory to all those who are fighting and defending the homeland! Glory to our defenders! Glory to the Ukrainian people! Glory to Ukraine! 🇺🇦🇺🇦🇺🇦
14:02

Tough times have made us strong. And the strong bring the times of victory closer. Step by step. Today, tomorrow, every day, every minute. No one should or will be able to "turn off" our resilience, endurance, grit, and fortitude. Neither on a regular nor an emergency basis. None of them have an "expiration date" an "end date", or a final point after which we would stop resisting and fighting. There is only one such point: our victory. As we bring it closer every day, we say, "We will fight for as long as it takes”. We were doing it in the first minutes of February 24th; we have been doing it for all these 585 days, and we will keep doing it. Glory to all those who are fighting and defending the homeland! Glory to our defenders! Glory to the Ukrainian people! Glory to Ukraine! 🇺🇦🇺🇦🇺🇦

Volodymyr Zelenskyy / Володимир Зеленський

946,467 görüntüleme • 2 yıl önce

PART TWO: We walked step by step from Matumbato Road through ngong rd to the City Mortuary. Every step carried pain. It’s still early. The families haven’t arrived yet to collect Baby Blessing, 7, and Baby Shiny, 1. So we remain here walking in silence because our hearts refuse to leave. This is not just a walk. It is grief. It is love. From day one, we have stood with this family through every tear, every painful moment. And even now, we are not just here we are part of their pain, their strength, their hope. 🕊️
11:10

PART TWO: We walked step by step from Matumbato Road through ngong rd to the City Mortuary. Every step carried pain. It’s still early. The families haven’t arrived yet to collect Baby Blessing, 7, and Baby Shiny, 1. So we remain here walking in silence because our hearts refuse to leave. This is not just a walk. It is grief. It is love. From day one, we have stood with this family through every tear, every painful moment. And even now, we are not just here we are part of their pain, their strength, their hope. 🕊️

Mike Sonko

34,315 görüntüleme • 2 ay önce

They told us the future of the internet was built by tech giants. We decided it should be built by us. 2025 was the year we moved from blueprint to backbone, successfully launching our product and token proving that enterprise-grade infra can be truly open. As we move into 2026, our focus shifts to massive service expansion. We are unlocking GPU and AI compute capabilities, onboarding a new wave of partners, and expanding our footprint into every major global market. The foundation is laid. 2026 is the year of global scale.
0:40

They told us the future of the internet was built by tech giants. We decided it should be built by us. 2025 was the year we moved from blueprint to backbone, successfully launching our product and token proving that enterprise-grade infra can be truly open. As we move into 2026, our focus shifts to massive service expansion. We are unlocking GPU and AI compute capabilities, onboarding a new wave of partners, and expanding our footprint into every major global market. The foundation is laid. 2026 is the year of global scale.

Impossible Cloud Network

20,233 görüntüleme • 6 ay önce

ATTENTION We are thrilled to announce the successful launch of the BETA VERSION of our app! This is a significant milestone, bringing us one step closer to becoming a key player in the DeFi ecosystem. Our app will empower users to purchase real-world products using CTF tokens across both the XRP Ledger and Polygon chain, in addition to a wide range of other cryptocurrencies. Moreover, users will have the exciting opportunity to upload receipts from partnered vendors and earn direct CTF token rewards! We’ve now entered the pilot phase, where our team will conduct rigorous internal testing to refine both the app and reward framework. For more details, visit
0:50

ATTENTION We are thrilled to announce the successful launch of the BETA VERSION of our app! This is a significant milestone, bringing us one step closer to becoming a key player in the DeFi ecosystem. Our app will empower users to purchase real-world products using CTF tokens across both the XRP Ledger and Polygon chain, in addition to a wide range of other cryptocurrencies. Moreover, users will have the exciting opportunity to upload receipts from partnered vendors and earn direct CTF token rewards! We’ve now entered the pilot phase, where our team will conduct rigorous internal testing to refine both the app and reward framework. For more details, visit

CryptoTradingFund

169,865 görüntüleme • 1 yıl önce

To everyone who wrote, called, prayed, and shared memories of Siddhartha — thank you. Your words have been a source of strength and comfort for the family, and for all of us at Aequitas. We have read every message — each one carrying love, respect, and the warmth of a life that touched so many. His legacy lives on — in the people he shaped, and the values he lived by. We pledge to honor him by carrying those values forward, every day. Sharing a glimpse of the beautiful journey he led. Remembering him, and celebrating him — always. With gratitude, 𝗧𝗲𝗮𝗺 𝗔𝗲𝗾𝘂𝗶𝘁𝗮𝘀
2:15

To everyone who wrote, called, prayed, and shared memories of Siddhartha — thank you. Your words have been a source of strength and comfort for the family, and for all of us at Aequitas. We have read every message — each one carrying love, respect, and the warmth of a life that touched so many. His legacy lives on — in the people he shaped, and the values he lived by. We pledge to honor him by carrying those values forward, every day. Sharing a glimpse of the beautiful journey he led. Remembering him, and celebrating him — always. With gratitude, 𝗧𝗲𝗮𝗺 𝗔𝗲𝗾𝘂𝗶𝘁𝗮𝘀

Aequitas Investments

31,807 görüntüleme • 5 ay önce

🌌 From Earth to Mars, just a step away—each step carries endless possibilities and dreams! The future is right in front of us. 🚀 MARS Project makes interstellar travel not just a sci-fi dream, but a reality within reach! ✨ By staking FSP & STP-X, we pave the way for you to journey to Mars and step into a new era of interstellar civilization. 💫 Every MARS token carries the power to cross galaxies and bring you closer to the future. Let’s embark on this journey together, from Earth to Mars, creating our own interstellar legacy! The future is here, and MARS tokens are your ticket to the stars! #MARSProject #InterstellarTravel #FSPStaking #STPXStaking #MarsMission
0:45

🌌 From Earth to Mars, just a step away—each step carries endless possibilities and dreams! The future is right in front of us. 🚀 MARS Project makes interstellar travel not just a sci-fi dream, but a reality within reach! ✨ By staking FSP & STP-X, we pave the way for you to journey to Mars and step into a new era of interstellar civilization. 💫 Every MARS token carries the power to cross galaxies and bring you closer to the future. Let’s embark on this journey together, from Earth to Mars, creating our own interstellar legacy! The future is here, and MARS tokens are your ticket to the stars! #MARSProject #InterstellarTravel #FSPStaking #STPXStaking #MarsMission

Mars Project

13,490 görüntüleme • 1 yıl önce

Many of us know this already but how do we get the sheep to understand and see what they are actually doing in their efforts of trying to take control? This video points out what they are doing and is straight to the point. Step one: train the people only to consume. Step two: infiltrate the adults on the news. Step three: indoctrinate the children through the schools and the music and the apps on the phones that they use. Step four: separate the right from the left. Step five: separate the white from the black. Step six: separate the rich from the poor. Use religion and equality to separate them more. Step seven: fabricate a problem made of lies. Step eight: put it on the news every night. Step nine: when people start to fight and divide take control, this is called situational design.
0:30

Many of us know this already but how do we get the sheep to understand and see what they are actually doing in their efforts of trying to take control? This video points out what they are doing and is straight to the point. Step one: train the people only to consume. Step two: infiltrate the adults on the news. Step three: indoctrinate the children through the schools and the music and the apps on the phones that they use. Step four: separate the right from the left. Step five: separate the white from the black. Step six: separate the rich from the poor. Use religion and equality to separate them more. Step seven: fabricate a problem made of lies. Step eight: put it on the news every night. Step nine: when people start to fight and divide take control, this is called situational design.

🌸𝓐𝓾𝓭𝓻𝓮𝔂🌸

10,822 görüntüleme • 2 ay önce

Anthropic and Andrew Ng built an agent that uses 90% fewer tokens from scratch: they dropped the entire book of Frankenstein into a prompt - 108,000 tokens asked one question the input dropped from 108,000 tokens to 11 here's how: step 1 → put everything that never changes at the top - tools, then system, then docs step 2 → mark where the static part ends. everything above it gets cached step 3 → one stray space breaks it - and you pay full price again step 4 → the cache dies in 5 min - every read resets the clock step 5 → cached tokens don't count against your rate limits. free headroom most people never touch this - it pays for itself on day one watch & bookmark - this 1-hour brilliant course ↓
1:04:05

Anthropic and Andrew Ng built an agent that uses 90% fewer tokens from scratch: they dropped the entire book of Frankenstein into a prompt - 108,000 tokens asked one question the input dropped from 108,000 tokens to 11 here's how: step 1 → put everything that never changes at the top - tools, then system, then docs step 2 → mark where the static part ends. everything above it gets cached step 3 → one stray space breaks it - and you pay full price again step 4 → the cache dies in 5 min - every read resets the clock step 5 → cached tokens don't count against your rate limits. free headroom most people never touch this - it pays for itself on day one watch & bookmark - this 1-hour brilliant course ↓

codila

524,336 görüntüleme • 3 gün önce