Navigation überspringen
CLIP by hand ✍️ ~ 13 steps walkthrough below
0:00 / 0:36

CLIP by hand ✍️ ~ 13 steps walkthrough below

Tom Yeh

@ProfTomYeh65.764 Follower

Auf X besuchen

20.929 Aufrufe

CLIP by hand ✍️ ~ 13 steps walkthrough below CLIP, Contrastive Language-Image Pre-training, is OpenAI's answer to a question that sounds impossible: how do you put a sentence and a picture in the same space? CLIP shipped when OpenAI was still open, and those embeddings were shared far and wide. Almost every multimodal model you use today descends from them. How does it work? Goal: learn one shared embedding space for text and images. = 1. Given = A mini batch of three text-image pairs. OpenAI trained the original on 400 million. = 2. Text to vectors = Let us look up each word with word2vec. = 3. Image to vectors = We cut each image into two patches and flatten them. Now text and pixels are both just numbers. = 4. The other pairs = Repeat steps 2 and 3 for the rest of the batch. = 5. Encode = Let us push both sides through their encoders, a linear layer and a ReLU. In practice these are transformers, but the shape of the operation is the same. = 6. Mean pooling = We average across the columns, so each image and each sentence collapses to a single vector. = 7. Projection = The text vectors are 3D and the image vectors are 4D, so they cannot be compared at all. A linear layer projects both to 2D. That 2D space is the shared embedding space, and getting here is the whole point of the model. = 8. Prepare for matmul = Let us copy the text vectors down and the transposed image vectors across. = 9. MatMul = We multiply, which takes the dot product of every text vector with every image vector. Each cell is one estimate of how well a sentence matches a picture. = 10. Softmax, e to the power = Raise e to each cell. To keep it hand sized we approximate e with 3. = 11. Softmax, sum = Sum each row for image to text, each column for text to image. = 12. Softmax, normalize = Divide, and out come two similarity matrices, one per direction. = 13. Loss gradients = The targets are identity matrices: a pair that belongs together should score 1, every other cell 0. Subtract the target from the similarity and you have the gradients, in both directions. The takeaway: pairing a picture with a sentence comes down to a single dot product. Everything before step 9 is the work of getting them into one shared space, so that the dot product finally means something. 💾 Save this post!

8 Kommentare

Kommentare aus dem Originalbeitrag auf X (Twitter)

  • Profilbild von 安叫兽|Bird🕊️ 🔶 BNB

    @ajs6888

    这类手绘拆解比论文图好懂太多了

    1 Like

  • Profilbild von robertus

    @rtheoryxyz

    CLIP was the moment embeddings stopped feeling like a backend trick and started acting like a shared coordinate system for culture

    1 Like

  • Profilbild von Aster Dev

    @aster_dev_

    Im really inspire by your work to build my own sheets like these for models you havent covered yet.

    1 Like

  • Profilbild von Tom Yeh

    @ProfTomYeh

    I would love to see. 😊

  • Profilbild von RaoulDuke

    @RaoulDukeDegen

    clip zero shot skills blew past supervised models on tons of tasks back in 2021

    1 Like

  • Profilbild von Sudhanva Savyasachi S

    @Sudhanva2424

    Your pace of uploading vedio is amazing

  • Profilbild von Tom Yeh

    @ProfTomYeh

    Each one in the original series took me a long time to design. This round is my revision with better notes. I am glad you finally discovered this series.

  • Profilbild von Sudhanva Savyasachi S

    @Sudhanva2424

    You are amazing sir , thank you these kind of vedios really helps us remember things well