Загрузка видео...

Не удалось загрузить видео

На главную

ComfyUI Tutorial for Beginners: Text-to-Image & Image-to-Image Workflows One graph, two workflows and Module 2 is where you build it node by node instead of loading someone else's. We start from a blank canvas and wire every connection, then change a single value and turn the same graph into...

29,658 просмотров • 3 дней назад •via X (Twitter)

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

CLIP by hand ✍️ ~ 13 steps walkthrough below CLIP, Contrastive Language-Image Pre-training, is OpenAI's answer to a question that sounds impossible: how do you put a sentence and a picture in the same space? CLIP shipped when OpenAI was still open, and those embeddings were shared far and wide. Almost every multimodal model you use today descends from them. How does it work? Goal: learn one shared embedding space for text and images. = 1. Given = A mini batch of three text-image pairs. OpenAI trained the original on 400 million. = 2. Text to vectors = Let us look up each word with word2vec. = 3. Image to vectors = We cut each image into two patches and flatten them. Now text and pixels are both just numbers. = 4. The other pairs = Repeat steps 2 and 3 for the rest of the batch. = 5. Encode = Let us push both sides through their encoders, a linear layer and a ReLU. In practice these are transformers, but the shape of the operation is the same. = 6. Mean pooling = We average across the columns, so each image and each sentence collapses to a single vector. = 7. Projection = The text vectors are 3D and the image vectors are 4D, so they cannot be compared at all. A linear layer projects both to 2D. That 2D space is the shared embedding space, and getting here is the whole point of the model. = 8. Prepare for matmul = Let us copy the text vectors down and the transposed image vectors across. = 9. MatMul = We multiply, which takes the dot product of every text vector with every image vector. Each cell is one estimate of how well a sentence matches a picture. = 10. Softmax, e to the power = Raise e to each cell. To keep it hand sized we approximate e with 3. = 11. Softmax, sum = Sum each row for image to text, each column for text to image. = 12. Softmax, normalize = Divide, and out come two similarity matrices, one per direction. = 13. Loss gradients = The targets are identity matrices: a pair that belongs together should score 1, every other cell 0. Subtract the target from the similarity and you have the gradients, in both directions. The takeaway: pairing a picture with a sentence comes down to a single dot product. Everything before step 9 is the work of getting them into one shared space, so that the dot product finally means something. 💾 Save this post!

Tom Yeh

20,750 просмотров • 1 месяц назад

context engineering vs graph engineering. every few months the list gets a new word and everyone treats it as a replacement for the last one. these two are not on the same list. one decides what the model sees this turn, the other decides what exists at all. the cleanest way to tell them apart is to ask what a single unit of work looks like. > context engineering is the window the window opens empty, every single time. you assemble what goes in it. the prompt, the docs, the history, the tool results. the assembling is the work. the window only grows. it never shrinks on its own, so eventually something gets dropped. usually from the middle. usually without telling you. then the turn ends and the window is thrown away. not archived, thrown away. the next turn opens empty again and you re-explain what you already explained. good context engineering is knowing what to leave out, not what to pack in. the unit of work is one window. > graph engineering is the structure the same material arrives from the same sources. instead of packing it into a window, you pull entities out of it, resolve the duplicates into one node, and write typed edges between them. nothing here is stored as text you hope to find again. it is stored as a thing with a name and its connections to other things. when the turn ends, the graph is still there. the next turn does not start from zero. it starts by querying what already exists, and the query walks edges instead of guessing at similarity. good graph engineering is deciding what counts as the same thing twice. the unit of work is one relationship. > they are not alternatives the graph is what refills the window. context engineering decides what fits. graph engineering decides what there is to choose from. remove the graph and every session starts blind. remove the context work and the best structure in the world arrives as an unreadable dump. that also tells you which one broke. the answer drifted from what you actually said, or forgot something from this same session. that is the window. the answer is coherent but invents a connection that does not exist, or cannot join two facts it has clearly seen. that is the structure. people debug the prompt because the prompt is the easiest thing to edit. it keeps taking the blame for failures that live a layer down. save this - then read the full breakdown below

Hanako

19,160 просмотров • 1 месяц назад

[CLIP] by Hand ✍️ The CLIP (Contrastive Language–Image Pre-training) model, a groundbreaking work by OpenAI, redefines the intersection of computer vision and natural language processing. It is the basis of all the multi-modal foundation models we see today. How does CLIP work? Goal: 🟨 Learn a shared embedding space for text and image [1] Given ↳ A mini batch of 3 text-image pairs ↳ OpenAI used 400 million text-image pairs to train its original CLIP model. Process 1st pair: "big table" [2] 🟪 Text → 2 Vectors (3D) ↳ Look up word embedding vectors using word2vec. [3] 🟩 Image → 2 Vectors (4D) ↳ Divide the image into two patches. ↳ Flatten each patch [4] Process other pairs ↳ Repeat [2]-[3] [5] 🟪 Text Encoder & 🟩 Image Encoder ↳ Encode input vectors into feature vectors ↳ Here, both encoders are simple one layer perceptron (linear + ReLU) ↳ In practice, the encoders are usually transformer models. [6] 🟪 🟩 Mean Pooling: 2 → 1 vector ↳ Average 2 feature vectors into a single vector by averaging across the columns ↳ The goal is to have one vector to represent each image or text [7] 🟪 🟩 -> 🟨 Projection ↳ Note that the text and image feature vectors from the encoders have different dimensions (3D vs. 4D). ↳ Use a linear layer to project image and text vectors to a 2D shared embedding space. 🏋️ Contrastive Pre-training 🏋️ [8] Prepare for MatMul ↳ Copy text vectors (T1,T2,T3) ↳ Copy the transpose of image vectors (I1,I2,I3) ↳ They are all in the 2D shared embedding space. [9] 🟦 MatMul ↳ Multiply T and I matrices. ↳ This is equivalent to taking dot product between every pair of image and text vectors. ↳ The purpose is to use dot product to estimate the similarity between a pair of image-text. [10] 🟦 Softmax: e^x ↳ Raise e to the power of the number in each cell ↳ To simplify hand calculation, we approximate e^□ with 3^□. [11] 🟦 Softmax: ∑ ↳ Sum each row for 🟩 image→🟪 text ↳ Sum each column for 🟪 text→ 🟩 image [12] 🟦 Softmax: 1 / sum ↳ Divide each element by the column sum to obtain a similarity matrix for 🟪 text→🟩 image ↳ Divide each element by the row sum to obtain a similarity matrix for 🟩 image→🟪 text [13] 🟥 Loss Gradients ↳ The "Targets" for the similarity matrices are Identity Matrices. ↳ Why? If I and T come from the same pair (i=j), we want the highest value, which is 1, and 0 otherwise. ↳ Apply the simple equation of [Similarity - Target] to compute gradients of for both directions. ↳ Why so simple? Because when Softmax and Cross-Entropy Loss are used together, the math magically works out that way. ↳ These gradients kick off the backpropagation process to update weights and biases of the encoders and projection layers (red borders).

Tom Yeh

67,883 просмотров • 2 лет назад

Anthropic ex-engineer runs an internal graph that costs $6 a month and catches what a $300,000 eval suite misses. No retrieval layer. Seven nodes and one rule about who's allowed to change their mind. He published the whole schema. His version starts from the opposite idea. A graph is not an execution order. It's a memory of why. Seven nodes. Every edge carries the reason it exists: > INTENT - states what the task is for. Never how > DECOMPOSE - splits it into steps, each with a stated assumption > WORKER - executes one step. Sees nothing else > AUDIT - checks the output against the assumption, not the goal > DRIFT - compares the current step to INTENT and flags divergence > LEDGER - stores every decision with the assumption that justified it > ROOT - holds the graph, and when an assumption breaks, re-runs every step built on it Six nodes act. One node remembers why they acted. Every step carries the assumption that made it correct, so a false one only reruns what stood on it. That's the entire design. A pipeline that forgets its reasons has to redo all of it or trust all of it. He replayed a month of agent runs. 4,100 steps, 380 built on an assumption that was wrong by day three. The old pipeline shipped all 380 and linked none of them. Everyone else builds graphs where output moves forward and the reasoning evaporates. He built one where the reason travels with the result. The blast radius is the output nobody else produces. The article below is the full build - node prompts, the assumption format, the invalidation rule that finds every step downstream of a broken one. Save it. You'll want it open in the other

wast3

304,987 просмотров • 10 дней назад