CLIP by hand ✍️ ~ 13 steps walkthrough below
CLIP, Contrastive Language-Image Pre-training, is OpenAI's answer to a question that sounds impossible: how do you put a sentence and a picture in the same space?
CLIP shipped when OpenAI was still open, and those embeddings were shared far and wide. Almost every multimodal model you use today descends from them.
How does it work?
Goal: learn one shared embedding space for text and images.
= 1. Given =
A mini batch of three text-image pairs. OpenAI trained the original on 400 million.
= 2. Text to vectors =
Let us look up each word with word2vec.
= 3. Image to vectors =
We cut each image into two patches and flatten them. Now text and pixels are both just numbers.
= 4. The other pairs =
Repeat steps 2 and 3 for the rest of the batch.
= 5. Encode =
Let us push both sides through their encoders, a linear layer and a ReLU. In practice these are transformers, but the shape of the operation is the same.
= 6. Mean pooling =
We average across the columns, so each image and each sentence collapses to a single vector.
= 7. Projection =
The text vectors are 3D and the image vectors are 4D, so they cannot be compared at all. A linear layer projects both to 2D. That 2D space is the shared embedding space, and getting here is the whole point of the model.
= 8. Prepare for matmul =
Let us copy the text vectors down and the transposed image vectors across.
= 9. MatMul =
We multiply, which takes the dot product of every text vector with every image vector. Each cell is one estimate of how well a sentence matches a picture.
= 10. Softmax, e to the power =
Raise e to each cell. To keep it hand sized we approximate e with 3.
= 11. Softmax, sum =
Sum each row for image to text, each column for text to image.
= 12. Softmax, normalize =
Divide, and out come two similarity matrices, one per direction.
= 13. Loss gradients =
The targets are identity matrices: a pair that belongs together should score 1, every other cell 0. Subtract the target from the similarity and you have the gradients, in both directions.
The takeaway: pairing a picture with a sentence comes down to a single dot product. Everything before step 9 is the work of getting them into one shared space, so that the dot product finally means something.
💾 Save this post!
Kommentare aus dem Originalbeitrag auf X (Twitter)
@ajs6888 •
这类手绘拆解比论文图好懂太多了
11 Like
@rtheoryxyz •
CLIP was the moment embeddings stopped feeling like a backend trick and started acting like a shared coordinate system for culture
11 Like
@aster_dev_ •
Im really inspire by your work to build my own sheets like these for models you havent covered yet.
11 Like
@ProfTomYeh •
I would love to see. 😊
@RaoulDukeDegen •
clip zero shot skills blew past supervised models on tons of tasks back in 2021
11 Like
@Sudhanva2424 •
Your pace of uploading vedio is amazing
@ProfTomYeh •
Each one in the original series took me a long time to design. This round is my revision with better notes. I am glad you finally discovered this series.
@Sudhanva2424 •
You are amazing sir , thank you these kind of vedios really helps us remember things well