Loading video...
Video Failed to Load
Next mlx-vlm release will ship with continuous batching support on the server ๐ What's coming: โ Continuous batching โ new requests join the active batch immediately, no waiting. Mixed image + text batches supported โ OpenAI-compatible API โ field-for-field match with mlx-lm, reasoning/content split for thinking models, tag-aware streaming... show more
82,349 views โข 4 months ago โขvia X (Twitter)
0 Comments
No comments available
Comments from the original post will appear here
![[CLIP] by Hand โ๏ธ The CLIP (Contrastive LanguageโImage Pre-training) model, a groundbreaking work by OpenAI, redefines the intersection of computer vision and natural language processing. It is the basis of all the multi-modal foundation models we see today. How does CLIP work? Goal: ๐จ Learn a shared embedding space for text and image [1] Given โณ A mini batch of 3 text-image pairs โณ OpenAI used 400 million text-image pairs to train its original CLIP model. Process 1st pair: "big table" [2] ๐ช Text โ 2 Vectors (3D) โณ Look up word embedding vectors using word2vec. [3] ๐ฉ Image โ 2 Vectors (4D) โณ Divide the image into two patches. โณ Flatten each patch [4] Process other pairs โณ Repeat [2]-[3] [5] ๐ช Text Encoder & ๐ฉ Image Encoder โณ Encode input vectors into feature vectors โณ Here, both encoders are simple one layer perceptron (linear + ReLU) โณ In practice, the encoders are usually transformer models. [6] ๐ช ๐ฉ Mean Pooling: 2 โ 1 vector โณ Average 2 feature vectors into a single vector by averaging across the columns โณ The goal is to have one vector to represent each image or text [7] ๐ช ๐ฉ -> ๐จ Projection โณ Note that the text and image feature vectors from the encoders have different dimensions (3D vs. 4D). โณ Use a linear layer to project image and text vectors to a 2D shared embedding space. ๐๏ธ Contrastive Pre-training ๐๏ธ [8] Prepare for MatMul โณ Copy text vectors (T1,T2,T3) โณ Copy the transpose of image vectors (I1,I2,I3) โณ They are all in the 2D shared embedding space. [9] ๐ฆ MatMul โณ Multiply T and I matrices. โณ This is equivalent to taking dot product between every pair of image and text vectors. โณ The purpose is to use dot product to estimate the similarity between a pair of image-text. [10] ๐ฆ Softmax: e^x โณ Raise e to the power of the number in each cell โณ To simplify hand calculation, we approximate e^โก with 3^โก. [11] ๐ฆ Softmax: โ โณ Sum each row for ๐ฉ imageโ๐ช text โณ Sum each column for ๐ช textโ ๐ฉ image [12] ๐ฆ Softmax: 1 / sum โณ Divide each element by the column sum to obtain a similarity matrix for ๐ช textโ๐ฉ image โณ Divide each element by the row sum to obtain a similarity matrix for ๐ฉ imageโ๐ช text [13] ๐ฅ Loss Gradients โณ The "Targets" for the similarity matrices are Identity Matrices. โณ Why? If I and T come from the same pair (i=j), we want the highest value, which is 1, and 0 otherwise. โณ Apply the simple equation of [Similarity - Target] to compute gradients of for both directions. โณ Why so simple? Because when Softmax and Cross-Entropy Loss are used together, the math magically works out that way. โณ These gradients kick off the backpropagation process to update weights and biases of the encoders and projection layers (red borders).](https://image.24vids.com/tw-1801370086152016034/ext_tw_video_thumb/1801368762937131008/pu/img/GuRh5lcJ6M7HsrkY.jpg)
