Загрузка видео...

Не удалось загрузить видео

На главную

➡️Machine Learning from Scratch, part 52: tokenization. Before a language model reads your text, the text is cut into small pieces called tokens: whole words, parts of words, single letters or punctuation marks. Every token has a fixed ID in a vocabulary, each ID gets its own vector (the...

18,000 просмотров • 4 дней назад •via TikTok

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

➡️ Machine Learning from Scratch, part 31: the transformer. A transformer turns every token into a list of numbers, adds its position, and then lets every token look at the others. That step is called attention: each token has a query (what am I looking for), a key (what do I contain) and a value (what I pass on). The better a query matches a key, the more of that value flows in. Real transformers repeat this in many layers with several attention heads each. The design comes from the 2017 Google paper "Attention Is All You Need" by Ashish Vaswani and colleagues, first used for translation. A large language model is a transformer trained to predict the next token, again and again, on a huge amount of text. Our own run, nothing downloaded. attention.py is one self-attention layer in 16 lines of numpy. Every word gets a query, a key and a value of 8 numbers, starting random, trained from scratch with hand-written backpropagation on 192 made-up sentences like "the dog ate the carrot because it was hungry". The task: find out which pet "it" means. On 64 new test sentences "it" puts 99.95 percent of its attention on the right word on average (the lowest single sentence gets 99.9 percent; the program prints this as 100%). Before training it was 10 percent, spread evenly. The cat sentence from the video is one of the test sentences. Honest notes: our task is easy on purpose. Every sentence has exactly one pet, so "it" only has to learn to look for pets, and it learns that within the first pass. Real sentences with two candidates need more context, which is what many layers are for. Our small version has no positions, no scaling, no mask, one head, and only "it" asks a question. Every word of the test sentences also appears in training; the combinations are new. The 3D space in the video shows the real keys of our model, projected to three axes (two of them stretched so you can see them). The stacked layers are a sketch. #machinelearning #ai #transformer #python #llm

MachineLearningToGo

81,400 просмотров • 1 месяц назад

➡️ Machine Learning from Scratch, part 47: the Hopfield network. In 1982 the physicist John Hopfield described a network that works as a memory, and 42 years later this work was part of the Nobel Prize in Physics. Each neuron is one pixel that is either on or off, and every neuron is connected to every other neuron, so 100 pixels need 4,950 connections. There are no layers: signals go back and forth until the network settles. Each neuron takes a weighted sum of its inputs and switches on if the sum is positive, off if it is negative. Storing a picture is simple: for every pair of pixels with the same colour the connection gets stronger, otherwise weaker. Every stored picture becomes a valley in the energy landscape, and no switch can ever raise the energy, so a noisy picture rolls downhill into the nearest stored memory. The code (hopfield.py, 16 lines of numpy, pictures in pics.txt) stores three 10 x 10 pictures, a heart, a house and a smiley, and flips 30 of the 100 pixels in each. After a single pass over all neurons all three come back exactly; the energy of the heart falls from -6.90 to -48.90, the energy of the stored heart. With 100 neurons and random patterns (10 flipped pixels, 20 runs) it recalls 5 patterns every time, 10 patterns in 94.5 % of the cases, 14 in 68 %, 20 in about one in four (24 %) and 30 in 1 %. Honest notes: the weights are kept as whole numbers (100 times the Hebb weight), so a sum of exactly zero switches the neuron on, as the rule says. The landscape in the video is a picture of the idea; the valley floors and the height of the rolling ball are the real energies from the run. #machinelearning #ai #coding #neuralnetworks #nobelprize

MachineLearningToGo

12,000 просмотров • 17 дней назад

How good is GPT-4-Vision at extracting text from images? I wanted to find the limit - but I found weirdness instead Most surprising: GPT-4V performance varies depending on the *structure* of text it sees Let me explain A set of images with progressively more text was presented to GPT-4-Vision. GPT-4V was asked what text it saw in the image. The response from the model was compared against the image’s original text and scored for similarity. The model was tested with 4 types of text: essay, random words, random tokens, and random characters. Findings: * Performance degrades - Yes, the models are good at basic OCR, but as you get more text and words then performance drops (this is expected) * Type of context matters - You should expect different recall on your texts based on your context types * Hallucination Errors - I thought that the model would make errors of omission (it wouldn’t return all the words). But instead the model mostly made hallucination errors - it replaced words with made up words. * Evals Matter - This test in isolation doesn’t mean that your data will have the same results, but it should motivate you to create eval tests for your data and anticipate errors which are hard to spot Notes: * Next step would be to add additional image types like tables or PDFs * GPT-4V would routinely get stuck in repeat-token-loops when trying to extract random tokens * GPT-4V would refuse to answer most random character images

Greg Kamradt

49,111 просмотров • 2 лет назад

CLIP by hand ✍️ ~ 13 steps walkthrough below CLIP, Contrastive Language-Image Pre-training, is OpenAI's answer to a question that sounds impossible: how do you put a sentence and a picture in the same space? CLIP shipped when OpenAI was still open, and those embeddings were shared far and wide. Almost every multimodal model you use today descends from them. How does it work? Goal: learn one shared embedding space for text and images. = 1. Given = A mini batch of three text-image pairs. OpenAI trained the original on 400 million. = 2. Text to vectors = Let us look up each word with word2vec. = 3. Image to vectors = We cut each image into two patches and flatten them. Now text and pixels are both just numbers. = 4. The other pairs = Repeat steps 2 and 3 for the rest of the batch. = 5. Encode = Let us push both sides through their encoders, a linear layer and a ReLU. In practice these are transformers, but the shape of the operation is the same. = 6. Mean pooling = We average across the columns, so each image and each sentence collapses to a single vector. = 7. Projection = The text vectors are 3D and the image vectors are 4D, so they cannot be compared at all. A linear layer projects both to 2D. That 2D space is the shared embedding space, and getting here is the whole point of the model. = 8. Prepare for matmul = Let us copy the text vectors down and the transposed image vectors across. = 9. MatMul = We multiply, which takes the dot product of every text vector with every image vector. Each cell is one estimate of how well a sentence matches a picture. = 10. Softmax, e to the power = Raise e to each cell. To keep it hand sized we approximate e with 3. = 11. Softmax, sum = Sum each row for image to text, each column for text to image. = 12. Softmax, normalize = Divide, and out come two similarity matrices, one per direction. = 13. Loss gradients = The targets are identity matrices: a pair that belongs together should score 1, every other cell 0. Subtract the target from the similarity and you have the gradients, in both directions. The takeaway: pairing a picture with a sentence comes down to a single dot product. Everything before step 9 is the work of getting them into one shared space, so that the dot product finally means something. 💾 Save this post!

Tom Yeh

20,929 просмотров • 3 месяцев назад

Tokenization -- turning text into a sequence of integers -- is a key part of generative AI, and most API providers charge per million tokens. How does tokenization work? Learn the details of tokenization and RAG optimization in Retrieval Optimization: From Tokenization to Vector Quantization, created in collaboration with Qdrant and taught by its Developer Relations Lead, Kacper Łukawski. This course focuses on Retrieval augmented generation (RAG), which has two steps: First, a retriever finds relevant information; then, the generator uses what’s retrieved as context to produce a response. You’ll learn to optimize the first step (the retriever) by understanding how tokenization works and how it impacts the relevance of your search. In addition, you will also learn to measure and improve retrieval quality, speed, and memory. In detail, you’ll: - Learn about the internal workings of the embedding models and how your text turns into vectors. - Understand how several tokenizers, such as Byte-Pair Encoding, WordPiece, Unigram, and SentencePiece work. - Explore common challenges with tokenizers, such as unknown tokens, domain-specific identifiers, and numerical values, that can negatively affect your vector search. - Understand how to measure the quality of your search across relevance, ranking, and score-related metrics. - Understand how the main parameters in "HNSW", a graph-based algorithm, affect the relevance and speed of vector search, and how to tune its parameters. - Experiment with the three major quantization methods – product, scalar, and binary – and learn how they impact memory requirements, search quality, and speed. By the end of this course, you’ll have a solid understanding of how tokenization functions and how to optimize vector search in your RAG systems. Please sign up here!

Andrew Ng

146,313 просмотров • 2 лет назад

[CLIP] by Hand ✍️ The CLIP (Contrastive Language–Image Pre-training) model, a groundbreaking work by OpenAI, redefines the intersection of computer vision and natural language processing. It is the basis of all the multi-modal foundation models we see today. How does CLIP work? Goal: 🟨 Learn a shared embedding space for text and image [1] Given ↳ A mini batch of 3 text-image pairs ↳ OpenAI used 400 million text-image pairs to train its original CLIP model. Process 1st pair: "big table" [2] 🟪 Text → 2 Vectors (3D) ↳ Look up word embedding vectors using word2vec. [3] 🟩 Image → 2 Vectors (4D) ↳ Divide the image into two patches. ↳ Flatten each patch [4] Process other pairs ↳ Repeat [2]-[3] [5] 🟪 Text Encoder & 🟩 Image Encoder ↳ Encode input vectors into feature vectors ↳ Here, both encoders are simple one layer perceptron (linear + ReLU) ↳ In practice, the encoders are usually transformer models. [6] 🟪 🟩 Mean Pooling: 2 → 1 vector ↳ Average 2 feature vectors into a single vector by averaging across the columns ↳ The goal is to have one vector to represent each image or text [7] 🟪 🟩 -> 🟨 Projection ↳ Note that the text and image feature vectors from the encoders have different dimensions (3D vs. 4D). ↳ Use a linear layer to project image and text vectors to a 2D shared embedding space. 🏋️ Contrastive Pre-training 🏋️ [8] Prepare for MatMul ↳ Copy text vectors (T1,T2,T3) ↳ Copy the transpose of image vectors (I1,I2,I3) ↳ They are all in the 2D shared embedding space. [9] 🟦 MatMul ↳ Multiply T and I matrices. ↳ This is equivalent to taking dot product between every pair of image and text vectors. ↳ The purpose is to use dot product to estimate the similarity between a pair of image-text. [10] 🟦 Softmax: e^x ↳ Raise e to the power of the number in each cell ↳ To simplify hand calculation, we approximate e^□ with 3^□. [11] 🟦 Softmax: ∑ ↳ Sum each row for 🟩 image→🟪 text ↳ Sum each column for 🟪 text→ 🟩 image [12] 🟦 Softmax: 1 / sum ↳ Divide each element by the column sum to obtain a similarity matrix for 🟪 text→🟩 image ↳ Divide each element by the row sum to obtain a similarity matrix for 🟩 image→🟪 text [13] 🟥 Loss Gradients ↳ The "Targets" for the similarity matrices are Identity Matrices. ↳ Why? If I and T come from the same pair (i=j), we want the highest value, which is 1, and 0 otherwise. ↳ Apply the simple equation of [Similarity - Target] to compute gradients of for both directions. ↳ Why so simple? Because when Softmax and Cross-Entropy Loss are used together, the math magically works out that way. ↳ These gradients kick off the backpropagation process to update weights and biases of the encoders and projection layers (red borders).

Tom Yeh

67,924 просмотров • 2 лет назад

➡️ Machine Learning from Scratch, part 35: supervised vs unsupervised learning. In supervised learning every training example comes with the right answer, a label. The model guesses, a loss function measures how wrong the guess was, and training makes that error smaller step by step. In unsupervised learning there are no labels. The model still has a goal, but it is measured on the data itself. k-means is the classic example: place a few group centers, give every point to its nearest center, add up the squared distances, move each center to the middle of its points and repeat. That total never gets larger, and after a few rounds the groups stop changing. Our own run, nothing downloaded. learn.py (16 lines of numpy) makes 300 points in 2D from 3 true groups and learns them both ways. With labels (a nearest-middle classifier trained on 200 points) it gets 100 of 100 new points right. Without labels, k-means brings the total squared distance down from 2927 to 741, and its groups match the true ones for 298 of 300 points. Honest notes: our three groups are made up and well separated on purpose, so both methods do very well; real data is messier. The 298 counts each found group as the true group most of its points belong to (here that is a clean one-to-one match). k-means starts from far-apart points, because a random start can get stuck in a worse grouping. The algorithm cannot know what a group means, so a person still has to name it. #machinelearning #ai #supervisedlearning #coding #python

MachineLearningToGo

26,000 просмотров • 1 месяц назад

Former Meta Chief AI Scientist Yann LeCun on the three paradigms of machine learning — and why the third is what made ChatGPT possible: Here's each one, and where it breaks. First, supervised learning. You tell the machine the answer. "You show it a picture, let's say of a table, and you tell it this is a table. So it's supervised because you tell it what the correct answer is." Get it wrong, and the machine rewrites itself: "The system computes its output, and if it says something else than table, then it's going to adjust its parameters, its internal structure, so that the output it produces gets closer to the output you want." Repeat at scale and something more than memorisation appears: "Eventually the system will find a way to recognize every image you trained it on, but also images it's never seen that are similar to the one you train it on. This is called a generalization ability." The limit: a human has to supply every single answer. That doesn't scale to the size of the internet. Second, reinforcement learning. You don't give the answer, only a verdict. "You don't tell the system what the correct answer is. You only tell it whether the answer it produced was good or bad." Learning to ride a bike, essentially: "You try to ride a bike and you don't know how to ride the bike and after a while you fall. So you know you did something bad and so you change your strategy a little bit. And eventually you learn how to ride a bike." For years the field assumed this was the closest thing to how animals actually learn. Yann LeCun's verdict: "Now it turns out reinforcement learning is extremely inefficient." It dominates wherever failure is free: "It works really well if you want to train a system to play chess or play go or poker, because you can have the system play millions and millions of games against itself and basically fine-tune itself. But it doesn't really work in the real world." The limit, in one image: "If you want to train a car to drive itself, you're not going to do it with reinforcement learning. It's going to crash thousands of times." On robotics he's careful rather than dismissive: "Reinforcement learning can be part of the solution, but it's not the complete answer. It's not sufficient." Third, self-supervised learning. You tell the machine nothing at all. "And this is what has enabled the recent progress in natural language understanding and chatbots." The strange part is that you stop asking for a task: "You don't train the system to accomplish any particular task. You just train it to basically capture the structure..." The method is deliberate sabotage: "You take a piece of text, you corrupt it in some way, by for example removing some words, and then you train a big neural net to predict the words that are missing." And one narrow version of that trick runs every chatbot on Earth: "A special case of this is that you take a piece of text and the last word in that text is not visible, and so you train the system to predict the last word in that text — and this is the way large language models are trained on." So why did the third one win? Supervised learning needs a human. Reinforcement learning needs a crash. Self-supervised learning needs neither — because the missing word and the correct answer are the same thing. The data grades itself.

Big Brain AI

49,293 просмотров • 3 месяцев назад

Big win for local LLM inference! This new inference engine runs LLMs 4x faster than llama.cpp: I ran Qwen3.5 9B on a base M5 MacBook Pro with three engines: - llama.cpp: 22.0 tokens/s - MLX: 25.1 tokens/s - uzu: 92.1 tokens/s The video below shows all three runs side by side. The first two numbers don't mean llama.cpp or MLX are badly built. But both are sitting right under a hard limit of local decoding. When an LLM generates text, every new token needs one full pass through the model, and that pass reads all of the model's weights from memory. A base M5 reads memory at about 153 GB/s. A 4-bit 9B model weighs about 5.2 GB. So at one token per pass, the ceiling is roughly 153 ÷ 5.2 ≈ 29 tokens/s, however fast the GPU computes. uzu gets past that ceiling by producing several tokens per pass instead of one: ↳ A small drafter (DFlash) proposes candidates for the next 16 positions at once. ↳ A tiny adapter (Weaver, 56.7M params) picks among them left to right, so the guesses form coherent text, and keeps a few alternative branches as a tree. ↳ The 9B model checks the whole tree in a single pass and keeps the longest path it agrees with. The main model still decides every token that ends up in the output. GitHub repo: (don't forget to star it 🌟) In my run, each pass of the 9B model produced 7.5 tokens on average. Ordinary decoding produces exactly 1. The weights and kernels are designed around this too. uzu ships its own 4-bit checkpoints, and on M5 chips it runs the verification step on the GPU's int8 matrix path. uzu is an open-source inference engine by Mirai Labs, written in Rust with Metal kernels. It has Swift, Python, TypeScript and Rust bindings, so the same engine runs inside a Mac or iOS app. I shared the GitHub repo above. If you want to dive deeper, my co-founder wrote a detailed breakdown of why local inference is memory-bound and how uzu's speculative decoding gets past it. Read it below.

Avi Chawla

128,879 просмотров • 21 дней назад

➡️ Machine Learning from Scratch, part 37: softmax. Every neuron first adds up its inputs times their weights, then an activation function reshapes the result. Without activation functions, any number of layers would act like one single linear layer. Softmax is the special activation at the very end of a network that sorts things into classes. The last layer has one output neuron per class, and each one gives a raw score that can be any number, even negative. Softmax looks at all of them at once: it raises e to the power of each score, then divides by the sum. Now every output lies between 0 and 1, all outputs add up to exactly 1, and the class with the highest one is the prediction. Unlike an S curve on each neuron, softmax couples the whole layer: if one score rises, the other probabilities fall. Our own run, nothing downloaded. softmax.py (16 lines of numpy) makes 300 points in three classes and trains a small network with 2 inputs, 4 hidden neurons and 3 output neurons on 200 of them. On the 100 new points it gets 90 right. For the point (-1, 0) it says 95% class 1, 1% class 2 and 4% class 3. Honest notes: these percentages are the model's own confidence. On the test points our model is 94.9% sure on average but right only 90% of the time, so it is a bit too confident, which is common for neural networks. Our learning rate sits close to the edge of stability, so a tiny change in the start can move the last digit (we show our exact run). The 3D hills are the real probabilities of our trained network over the input plane. #machinelearning #coding #softmax #neuralnetwork #deeplearning

MachineLearningToGo

16,500 просмотров • 1 месяц назад

➡️ Machine Learning from Scratch, part 15: a recommender, the kind of algorithm that picks what your feed shows you. Our app is simulated: 300 people and 60 videos. Every person has a hidden taste, every video has a hidden style, and each person has watched only about 12 of the 60 videos. All the algorithm ever sees is who liked which of the videos they watched. How it learns: every person gets two numbers, and every video gets two numbers. Think of them as arrows on a map. For every video a person has seen, the two arrows turn into a chance of a like with the sigmoid from part 2. The algorithm compares that chance with what really happened and nudges both arrows a little. That is gradient descent from part 8, and the whole method is called matrix factorisation. The results, all measured on the videos nobody showed it: After 25 rounds it guesses 85.0% of the hidden likes right. After 3000 rounds it gets 87.8% right. The best any method could do here is 91.1%, because some likes are just random. For every person it recommends the three unseen videos with the strongest match, and 98% of them are videos the person really likes. A random unseen video gets a like 51% of the time. The first person on the list watched only 5 videos. From those five it guessed the other 55 and got 51 of them right. We ran the same program on 20 different simulated apps, and it always landed between 84 and 88%. Real apps use hundreds of numbers for every person and every video, plus signals like watch time and skips. The core idea stays the same: learn numbers for people and for videos, and match them. #machinelearning #ai #recommendersystem #python #coding

MachineLearningToGo

89,300 просмотров • 1 месяц назад