Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

Stop reading attention is all you need. Transformers (LLMs) clearly explained with visuals: - Real GPT-2 visualizer - tokens to embeddings - 12 attention heads (Q, K, V, scale, mask, softmax) - multi-head Q/K/V - residual + MLP - next-token probabilities That’s the difference between reading about transformers and...

0x0SojalSec's profile picture

Md Ismail Šojal 🕷️

54,335 subscribers

22,537 görüntüleme • 12 gün önce •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

New short course: Attention in Transformers: Concepts and Code in PyTorch. Last week we released a course on how LLM transformers work. This week, go deeper and learn about the technical ideas behind the attention mechanism, and see how to code it in PyTorch. This course is built with Joshua Starmer, Founder and CEO of StatQuest. The attention mechanism was a breakthrough that led to transformers, the architecture powering large language models like ChatGPT. Transformers, introduced in the 2017 paper: "Attention is All You Need" by Viswani and others, took off because of its highly scalable design. In this course, you’ll learn how the attention mechanism, a key element of transformer-based LLMs, works and implement it in PyTorch. You'll develop deep intuition about building reliable, functional, and scalable AI applications. What you will do: - Understand the evolution of the attention mechanism, a key breakthrough that led to transformers. - Learn the relationships between word embeddings, positional embeddings, and attention. - Learn about the Query, Key, and Value matrices, and how to produce and use them in attention. - Walk through the math required to calculate self-attention and masked self-attention to learn why and how they work. - Understand the difference between self-attention and masked self-attention and how one is used in the encoder to build context-aware embeddings and the other is used in the decoder for generative outputs. - Learn the details of the encoder-decoder architecture, cross-attention, and multi-head attention and how they are all incorporated into a transformer. - Use PyTorch to code a class that implements self-attention, masked self-attention, and multi-head attention. There're lots of exciting technical details in this course. Please sign up here:
2:41

New short course: Attention in Transformers: Concepts and Code in PyTorch. Last week we released a course on how LLM transformers work. This week, go deeper and learn about the technical ideas behind the attention mechanism, and see how to code it in PyTorch. This course is built with Joshua Starmer, Founder and CEO of StatQuest. The attention mechanism was a breakthrough that led to transformers, the architecture powering large language models like ChatGPT. Transformers, introduced in the 2017 paper: "Attention is All You Need" by Viswani and others, took off because of its highly scalable design. In this course, you’ll learn how the attention mechanism, a key element of transformer-based LLMs, works and implement it in PyTorch. You'll develop deep intuition about building reliable, functional, and scalable AI applications. What you will do: - Understand the evolution of the attention mechanism, a key breakthrough that led to transformers. - Learn the relationships between word embeddings, positional embeddings, and attention. - Learn about the Query, Key, and Value matrices, and how to produce and use them in attention. - Walk through the math required to calculate self-attention and masked self-attention to learn why and how they work. - Understand the difference between self-attention and masked self-attention and how one is used in the encoder to build context-aware embeddings and the other is used in the decoder for generative outputs. - Learn the details of the encoder-decoder architecture, cross-attention, and multi-head attention and how they are all incorporated into a transformer. - Use PyTorch to code a class that implements self-attention, masked self-attention, and multi-head attention. There're lots of exciting technical details in this course. Please sign up here:

Andrew Ng

132,400 görüntüleme • 1 yıl önce

New course: Transformers in Practice. You'll get a practical view of how transformer-based LLMs work, so you can reason about their behavior, diagnose problems like slow inference, and make smarter decisions about deployment. This course is built in partnership with AMD and taught by Sharon Zhou. You'll see how transformers generate text one token at a time, how the model decides which earlier words matter most when predicting the next one, and how techniques like quantization speed up inference on GPUs. This is not a video-only course; interactive visualizations throughout let you play with these concepts and build intuition that sticks. Skills you'll gain: - Understand why LLMs hallucinate, and RAG and chain-of-thought shape what they generate - Look inside the model to see how attention and layers combine to predict the next token - Diagnose inference bottlenecks and learn the techniques that speed up transformers on GPUs Join and understand what's really happening inside your LLMs:
1:00

New course: Transformers in Practice. You'll get a practical view of how transformer-based LLMs work, so you can reason about their behavior, diagnose problems like slow inference, and make smarter decisions about deployment. This course is built in partnership with AMD and taught by Sharon Zhou. You'll see how transformers generate text one token at a time, how the model decides which earlier words matter most when predicting the next one, and how techniques like quantization speed up inference on GPUs. This is not a video-only course; interactive visualizations throughout let you play with these concepts and build intuition that sticks. Skills you'll gain: - Understand why LLMs hallucinate, and RAG and chain-of-thought shape what they generate - Look inside the model to see how attention and layers combine to predict the next token - Diagnose inference bottlenecks and learn the techniques that speed up transformers on GPUs Join and understand what's really happening inside your LLMs:

Andrew Ng

124,677 görüntüleme • 4 ay önce

[Self-Attention] by Hand ✍️ Self-attention is what enables LLMs to understand context. How does it work? This exercise demonstrates how to calculate a 6-3 attention head by hand. Note that if we have two instances of this, we get 6-6 attention (i.e., multi-head attention, n=2). -- 𝗚𝗼𝗮𝗹 -- Transform [6D Features 🟧] to [3D Attention Weighted Features 🟦] -- 𝗪𝗮𝗹𝗸𝘁𝗵𝗿𝗼𝘂𝗴𝗵 -- [1] Given ↳ A set of 4 feature vectors (6-D): x1,x2,x3,x4 [2] Query, Key, Value ↳ Multiply features x's with linear transformation matrices WQ, WK, and WV, to obtain query vectors (q1,q2,q3,q4), key vectors (k1,k2,k3,k4), and value vectors (v1,v2,v3,v4). ↳ "Self" refers to the fact that both queries and keys are derived from the same set of features. [3] 🟪 Prepare for MatMul ↳ Copy query vectors ↳ Copy the transpose of key vectors [4] 🟪 MatMul ↳ Multiply K^T and Q ↳ This is equivalent to taking dot product between every pair of query and key vectors. ↳ The purpose is to use dot product as an estimate of the "matching score" between every key-value pair. ↳ This estimate makes sense because dot product is the numerator of Cosine Similarity between two vectors. [5] 🟨 Scale ↳ Scale each element by the square root of dk, which is the dimension of key vectors (dk=3). ↳ The purpose is to normalize the impact of the dk on matching scores, even if we scale dk to 32, 64, or 128. ↳ To simplify hand calculation, we approximate [ □/sqrt(3) ] with [ floor(□/2) ]. [6] 🟩 Softmax: e^x ↳ Raise e to the power of the number in each cell ↳ To simplify hand calculation, we approximate e^□ with 3^□. [7] 🟩 Softmax: ∑ ↳ Sum across each column [8] 🟩 Softmax: 1 / sum ↳ For each column, divide each element by the column sum ↳ The purpose is normalize each column so that the numbers sum to 1. In other words, each column is a probability distribution of attention, and we have four of them. ↳ The result is the Attention Weight Matrix (A) (yellow) [9] 🟦 MatMul ↳ Multiply the value vectors (Vs) with the Attention Weight Matrix (A) ↳ The results are the attention weighted features Zs. ↳ They are fed to the position-wise feed forward network in the next layer.
0:32

[Self-Attention] by Hand ✍️ Self-attention is what enables LLMs to understand context. How does it work? This exercise demonstrates how to calculate a 6-3 attention head by hand. Note that if we have two instances of this, we get 6-6 attention (i.e., multi-head attention, n=2). -- 𝗚𝗼𝗮𝗹 -- Transform [6D Features 🟧] to [3D Attention Weighted Features 🟦] -- 𝗪𝗮𝗹𝗸𝘁𝗵𝗿𝗼𝘂𝗴𝗵 -- [1] Given ↳ A set of 4 feature vectors (6-D): x1,x2,x3,x4 [2] Query, Key, Value ↳ Multiply features x's with linear transformation matrices WQ, WK, and WV, to obtain query vectors (q1,q2,q3,q4), key vectors (k1,k2,k3,k4), and value vectors (v1,v2,v3,v4). ↳ "Self" refers to the fact that both queries and keys are derived from the same set of features. [3] 🟪 Prepare for MatMul ↳ Copy query vectors ↳ Copy the transpose of key vectors [4] 🟪 MatMul ↳ Multiply K^T and Q ↳ This is equivalent to taking dot product between every pair of query and key vectors. ↳ The purpose is to use dot product as an estimate of the "matching score" between every key-value pair. ↳ This estimate makes sense because dot product is the numerator of Cosine Similarity between two vectors. [5] 🟨 Scale ↳ Scale each element by the square root of dk, which is the dimension of key vectors (dk=3). ↳ The purpose is to normalize the impact of the dk on matching scores, even if we scale dk to 32, 64, or 128. ↳ To simplify hand calculation, we approximate [ □/sqrt(3) ] with [ floor(□/2) ]. [6] 🟩 Softmax: e^x ↳ Raise e to the power of the number in each cell ↳ To simplify hand calculation, we approximate e^□ with 3^□. [7] 🟩 Softmax: ∑ ↳ Sum across each column [8] 🟩 Softmax: 1 / sum ↳ For each column, divide each element by the column sum ↳ The purpose is normalize each column so that the numbers sum to 1. In other words, each column is a probability distribution of attention, and we have four of them. ↳ The result is the Attention Weight Matrix (A) (yellow) [9] 🟦 MatMul ↳ Multiply the value vectors (Vs) with the Attention Weight Matrix (A) ↳ The results are the attention weighted features Zs. ↳ They are fed to the position-wise feed forward network in the next layer.

Tom Yeh

101,213 görüntüleme • 2 yıl önce

Thanksgiving-week treat: an epic conversation on Frontier AI with Lukasz Kaiser -co-author of “Attention Is All You Need” (Transformers) and leading research scientist at OpenAI working on GPT-5.1-era reasoning models. 00:00 – Cold open and intro 01:29 – “AI slowdown” vs a wild week of new frontier models 08:03 – Low-hanging fruit, infra, RL training and better data 11:39 – What is a reasoning model, in plain language 17:02 – Chain-of-thought and training the thinking process with RL 21:39 – Łukasz’s path: from logic and France to Google and Kurzweil 24:20 – Inside the Transformer story and what “attention” really means 28:42 – From Google Brain to OpenAI: culture, scale and GPUs 32:49 – What’s next for pre-training, GPUs and distillation 37:29 – Can we still understand these models? Circuits, sparsity and black boxes 39:42 – GPT-4 → GPT-5 → GPT-5.1: what actually changed 42:40 – Post-training, safety and teaching GPT-5.1 different tones 46:16 – How long should GPT-5.1 think? Reasoning tokens and jagged abilities 47:43 – The five-year-old’s dot puzzle that still breaks frontier models 52:22 – Generalization, child-like learning and whether reasoning is enough 53:48 – Beyond Transformers: ARC, LeCun’s ideas and multimodal bottlenecks 56:10 – GPT-5.1 Codex Max, long-running agents and compaction 1:00:06 – Will foundation models eat most apps? The translation analogy and trust 1:02:34 – What still needs to be solved, and where AI might go next
1:05:25

Thanksgiving-week treat: an epic conversation on Frontier AI with Lukasz Kaiser -co-author of “Attention Is All You Need” (Transformers) and leading research scientist at OpenAI working on GPT-5.1-era reasoning models. 00:00 – Cold open and intro 01:29 – “AI slowdown” vs a wild week of new frontier models 08:03 – Low-hanging fruit, infra, RL training and better data 11:39 – What is a reasoning model, in plain language 17:02 – Chain-of-thought and training the thinking process with RL 21:39 – Łukasz’s path: from logic and France to Google and Kurzweil 24:20 – Inside the Transformer story and what “attention” really means 28:42 – From Google Brain to OpenAI: culture, scale and GPUs 32:49 – What’s next for pre-training, GPUs and distillation 37:29 – Can we still understand these models? Circuits, sparsity and black boxes 39:42 – GPT-4 → GPT-5 → GPT-5.1: what actually changed 42:40 – Post-training, safety and teaching GPT-5.1 different tones 46:16 – How long should GPT-5.1 think? Reasoning tokens and jagged abilities 47:43 – The five-year-old’s dot puzzle that still breaks frontier models 52:22 – Generalization, child-like learning and whether reasoning is enough 53:48 – Beyond Transformers: ARC, LeCun’s ideas and multimodal bottlenecks 56:10 – GPT-5.1 Codex Max, long-running agents and compaction 1:00:06 – Will foundation models eat most apps? The translation analogy and trust 1:02:34 – What still needs to be solved, and where AI might go next

Matt Turck

168,007 görüntüleme • 9 ay önce

This is probably the most entertaining way to understand one of AI’s hardest AI debates. Transformer vs Post-Transformer, argued by leading researchers, inside a real physical boxing ring. Both technically deep and genuinely entertaining. I was glued for the entire 1 hour 20 minutes. So many super cool points to learn. 🥊 Transformers - Transformers still own the present because they work at scale. They are simple, trainable, hardware-friendly, and already power the strongest AI systems we use today. - The Transformer is basically a memory machine. It stores information as keys and values, then uses attention to pull back the most useful parts when answering. - The real Transformer advantage is not just “attention.” The bigger advantage is that it fits modern hardware extremely well, so it can process huge batches of tokens fast. - Scaling is still the brutal rule. If you give Transformers more compute, more data, and more parameters, they usually keep getting better. Any Post-Transformer architecture has to scale just as well, or better. - It is not enough to look clever on small tests, because the real question is whether it improves faster than Transformers when scaled up. - A replacement cannot be slightly better. Because the whole AI stack is already built around Transformers, the next architecture may need to be around 10x better to force everyone to switch. - Transformers are powerful, but they may be brute force. A human does not need to read the entire internet many times to become smart, but current LLMs need enormous data and compute. 🥊 Post-Transformer - Post-Transformer people are not saying Transformers are bad. They are saying Transformers may be the best current tool, not the final form of machine intelligence. - The biggest Post-Transformer target is native reasoning and continual learning. Today’s LLM reasoning often feels like text-based step-by-step work added on top, instead of thinking happening naturally inside the model. - Latent reasoning is one possible next step. That means the model reasons inside its own hidden internal space, instead of writing every thought out as words. - Continual learning is still a major weakness. Humans keep learning from experience, but most Transformer-based models are trained, frozen, and then only adapt inside the prompt. - Long context is not the same as real memory. A model can read a huge prompt, but that is different from building a life history, learning from mistakes, and updating beliefs over time. - The future may be hybrid, not a clean replacement. Transformers may stay as 1 building block while newer systems add better memory, better reasoning, and better learning loops. - The most interesting possibility is that Transformers may help discover their own successor. AI agents are already getting better at research and coding, so the next architecture may come from AI-assisted architecture search. ------- - Benchmarks are a problem. Many public benchmarks are easy to game, so they may show leaderboard strength without proving deeper intelligence. - Perplexity is still probably a great metric to evaluate frontier models,, because it tests prediction quality. --- Overall, Transformers continue to dominate, but the frontier is clearly widening. Pathway’s BDH (Dragon Hatchling — brain-inspired reasoning architecture), Sakana AI’s CTMs (Continuous Thought Machines — models that think over time), and Liquid AI’s LFMs (Liquid Foundation Models — efficient multimodal foundation models) - all of these show how the frontier is expanding. --- From “Pathway (pathway[.]com)” Youtube channel (link in comment) Zuzanna Stamirowska
2:35

This is probably the most entertaining way to understand one of AI’s hardest AI debates. Transformer vs Post-Transformer, argued by leading researchers, inside a real physical boxing ring. Both technically deep and genuinely entertaining. I was glued for the entire 1 hour 20 minutes. So many super cool points to learn. 🥊 Transformers - Transformers still own the present because they work at scale. They are simple, trainable, hardware-friendly, and already power the strongest AI systems we use today. - The Transformer is basically a memory machine. It stores information as keys and values, then uses attention to pull back the most useful parts when answering. - The real Transformer advantage is not just “attention.” The bigger advantage is that it fits modern hardware extremely well, so it can process huge batches of tokens fast. - Scaling is still the brutal rule. If you give Transformers more compute, more data, and more parameters, they usually keep getting better. Any Post-Transformer architecture has to scale just as well, or better. - It is not enough to look clever on small tests, because the real question is whether it improves faster than Transformers when scaled up. - A replacement cannot be slightly better. Because the whole AI stack is already built around Transformers, the next architecture may need to be around 10x better to force everyone to switch. - Transformers are powerful, but they may be brute force. A human does not need to read the entire internet many times to become smart, but current LLMs need enormous data and compute. 🥊 Post-Transformer - Post-Transformer people are not saying Transformers are bad. They are saying Transformers may be the best current tool, not the final form of machine intelligence. - The biggest Post-Transformer target is native reasoning and continual learning. Today’s LLM reasoning often feels like text-based step-by-step work added on top, instead of thinking happening naturally inside the model. - Latent reasoning is one possible next step. That means the model reasons inside its own hidden internal space, instead of writing every thought out as words. - Continual learning is still a major weakness. Humans keep learning from experience, but most Transformer-based models are trained, frozen, and then only adapt inside the prompt. - Long context is not the same as real memory. A model can read a huge prompt, but that is different from building a life history, learning from mistakes, and updating beliefs over time. - The future may be hybrid, not a clean replacement. Transformers may stay as 1 building block while newer systems add better memory, better reasoning, and better learning loops. - The most interesting possibility is that Transformers may help discover their own successor. AI agents are already getting better at research and coding, so the next architecture may come from AI-assisted architecture search. ------- - Benchmarks are a problem. Many public benchmarks are easy to game, so they may show leaderboard strength without proving deeper intelligence. - Perplexity is still probably a great metric to evaluate frontier models,, because it tests prediction quality. --- Overall, Transformers continue to dominate, but the frontier is clearly widening. Pathway’s BDH (Dragon Hatchling — brain-inspired reasoning architecture), Sakana AI’s CTMs (Continuous Thought Machines — models that think over time), and Liquid AI’s LFMs (Liquid Foundation Models — efficient multimodal foundation models) - all of these show how the frontier is expanding. --- From “Pathway (pathway[.]com)” Youtube channel (link in comment) Zuzanna Stamirowska

Rohan Paul

89,110 görüntüleme • 3 ay önce

Self Attention by hand ✍️ ~ 9 steps walkthrough below Self-attention is what enables LLMs to understand context. How does it work? So I drew and calculated one entirely by hand. Goal: turn four 6D features into four 3D attention weighted features, filling in every cell yourself. = 1. Given = Four feature vectors, six dimensions each, one per position. = 2. Query, key, value = Let us multiply the features by WQ, WK and WV. Queries, keys and values all come out of the same four features, and that is what the word "self" is doing in self-attention. = 3. Prepare for MatMul = We copy the queries across the top and the transposed keys down the side. Lining the two up is half the work. = 4. MatMul = Let us multiply K transpose by Q. Every cell is the dot product of one key with one query, which we use as a matching score. That works because the dot product is the numerator of cosine similarity: it is how alike two vectors are, before anyone divides by their lengths. = 5. Scale = We divide by the square root of dk, the dimension of a key vector, here 3. Without it the scores grow with the dimension and a 64-wide head would swamp the softmax. To keep the page doable in pen, the drawing approximates dividing by root 3 with halving. = 6. e to the power = Let us raise e to the power of each score. This is the first half of softmax, and the drawing uses 3 in place of e, which is close enough to do in your head. = 7. Sum = We add up each column: 16, 6, 7 and 12. = 8. Normalize = Let us divide every cell by its column sum. That gives the attention weight matrix in yellow, and each of its four columns is now a probability distribution over the four positions. The decimals are nudged as they are rounded, so every column still sums to exactly 1. = 9. MatMul = We multiply the value vectors by those weights. Each output is a blend of all four values, mixed in the proportion the attention matrix just decided, and it goes to the position-wise feed forward network in the next layer: the FFN box at the bottom of the page. The outputs: Attention weights (A), by column = [.2, .6, 0, .2], [.2, .4, .2, .2], [.4, .2, 0, .4], [.1, .7, .1, .1] Attention weighted features (Z) = [8, 2, 6], [8, 4, 4], [16, 4, 2], [4, 2, 7] The takeaway: attention is a weighted average, and everything before step 9 exists to decide the weights. Compare every position with every other, turn the scores into one distribution per position, then blend. 💾 Save this post!
0:22

Self Attention by hand ✍️ ~ 9 steps walkthrough below Self-attention is what enables LLMs to understand context. How does it work? So I drew and calculated one entirely by hand. Goal: turn four 6D features into four 3D attention weighted features, filling in every cell yourself. = 1. Given = Four feature vectors, six dimensions each, one per position. = 2. Query, key, value = Let us multiply the features by WQ, WK and WV. Queries, keys and values all come out of the same four features, and that is what the word "self" is doing in self-attention. = 3. Prepare for MatMul = We copy the queries across the top and the transposed keys down the side. Lining the two up is half the work. = 4. MatMul = Let us multiply K transpose by Q. Every cell is the dot product of one key with one query, which we use as a matching score. That works because the dot product is the numerator of cosine similarity: it is how alike two vectors are, before anyone divides by their lengths. = 5. Scale = We divide by the square root of dk, the dimension of a key vector, here 3. Without it the scores grow with the dimension and a 64-wide head would swamp the softmax. To keep the page doable in pen, the drawing approximates dividing by root 3 with halving. = 6. e to the power = Let us raise e to the power of each score. This is the first half of softmax, and the drawing uses 3 in place of e, which is close enough to do in your head. = 7. Sum = We add up each column: 16, 6, 7 and 12. = 8. Normalize = Let us divide every cell by its column sum. That gives the attention weight matrix in yellow, and each of its four columns is now a probability distribution over the four positions. The decimals are nudged as they are rounded, so every column still sums to exactly 1. = 9. MatMul = We multiply the value vectors by those weights. Each output is a blend of all four values, mixed in the proportion the attention matrix just decided, and it goes to the position-wise feed forward network in the next layer: the FFN box at the bottom of the page. The outputs: Attention weights (A), by column = [.2, .6, 0, .2], [.2, .4, .2, .2], [.4, .2, 0, .4], [.1, .7, .1, .1] Attention weighted features (Z) = [8, 2, 6], [8, 4, 4], [16, 4, 2], [4, 2, 7] The takeaway: attention is a weighted average, and everything before step 9 exists to decide the weights. Compare every position with every other, turn the scores into one distribution per position, then blend. 💾 Save this post!

Tom Yeh

27,292 görüntüleme • 1 ay önce

Parents often ask me about the best way to engage infants in lap reading - noting, at the very earliest stages, that their babies aren’t always real attentive to books. It’s important, first and foremost, to note that infant attention spans are very short. So know that it’s okay to start small… reading for just a minute or two at a time. But there are also ways you can optimize your chances of a successful experience. One trick is to literally turn things around, as illustrated by this great video from justjessfood on IG, who (with the assistance of a supportive pillow) has turned lap reading into a face-to-face social experience. There’s nothing your baby loves more than your face… as evidenced by this sweet little guy who arguably takes more delight in mom’s expressive reading than in a view of the book itself. And that’s just fine! He’s still accruing all the language benefits as they bond. And talk about an appreciative audience! Next time you read with your baby try configuring the situation so that they can see both you and the book. You may find it makes a big difference.
0:50

Parents often ask me about the best way to engage infants in lap reading - noting, at the very earliest stages, that their babies aren’t always real attentive to books. It’s important, first and foremost, to note that infant attention spans are very short. So know that it’s okay to start small… reading for just a minute or two at a time. But there are also ways you can optimize your chances of a successful experience. One trick is to literally turn things around, as illustrated by this great video from justjessfood on IG, who (with the assistance of a supportive pillow) has turned lap reading into a face-to-face social experience. There’s nothing your baby loves more than your face… as evidenced by this sweet little guy who arguably takes more delight in mom’s expressive reading than in a view of the book itself. And that’s just fine! He’s still accruing all the language benefits as they bond. And talk about an appreciative audience! Next time you read with your baby try configuring the situation so that they can see both you and the book. You may find it makes a big difference.

Dan Wuori

40,597 görüntüleme • 1 yıl önce

this is free f*cking gold for AI engineers 10 repos i'd actually keep, from python fundamentals to LLMs, agents and production AI. skip the random tutorials 1. Python-100-Days - a 100 day path through fundamentals, data analysis, web dev ↳ 2. Generative AI for Beginners - LLM basics, prompting, RAG, agents, fine-tuning ↳ 3. LLMs from Scratch - build one yourself: tokenization, attention, transformers, training ↳ 4. ML for Beginners - 12 weeks, 26 lessons of classical ML ↳ 5. OpenAI Cookbook - working examples to go from learning to building ↳ 6. Stable Diffusion - the original implementation and research code ↳ 7. AI Agents for Beginners - tool use, RAG, agent frameworks, multi-agent systems ↳ 8. AI for Beginners - 12 weeks, 24 lessons: neural nets, vision, NLP ↳ 9. LLM App - RAG pipelines, enterprise search, real-time data, vector search ↳ 10. Segment Anything - Meta's foundation model for image segmentation ↳ pick by where you are: > python -> Python-100-Days > ML -> ML-For-Beginners > LLMs -> LLMs-from-scratch > agents -> AI-Agents-for-Beginners > building -> OpenAI Cookbook, LLM App > vision -> Segment Anything pick one, build something, then move to the next
0:24

this is free f*cking gold for AI engineers 10 repos i'd actually keep, from python fundamentals to LLMs, agents and production AI. skip the random tutorials 1. Python-100-Days - a 100 day path through fundamentals, data analysis, web dev ↳ 2. Generative AI for Beginners - LLM basics, prompting, RAG, agents, fine-tuning ↳ 3. LLMs from Scratch - build one yourself: tokenization, attention, transformers, training ↳ 4. ML for Beginners - 12 weeks, 26 lessons of classical ML ↳ 5. OpenAI Cookbook - working examples to go from learning to building ↳ 6. Stable Diffusion - the original implementation and research code ↳ 7. AI Agents for Beginners - tool use, RAG, agent frameworks, multi-agent systems ↳ 8. AI for Beginners - 12 weeks, 24 lessons: neural nets, vision, NLP ↳ 9. LLM App - RAG pipelines, enterprise search, real-time data, vector search ↳ 10. Segment Anything - Meta's foundation model for image segmentation ↳ pick by where you are: > python -> Python-100-Days > ML -> ML-For-Beginners > LLMs -> LLMs-from-scratch > agents -> AI-Agents-for-Beginners > building -> OpenAI Cookbook, LLM App > vision -> Segment Anything pick one, build something, then move to the next

Mr. Buzzoni

32,332 görüntüleme • 5 gün önce

Announcing How Transformer LLMs Work, created with Jay Alammar and Maarten Grootendorst, co-authors of the beautifully illustrated book, “Hands-On Large Language Models.” This course offers a deep dive into the inner workings of the transformer architecture that powers large language models (LLMs). The transformer architecture revolutionized generative AI; in fact, the "GPT" in ChatGPT stands for "Generative Pre-Trained Transformer." Originally introduced in the Google Brain team's groundbreaking 2017 paper "Attention Is All You Need," by Vaswani and others, transformers were a highly scalable model for machine translation tasks. Variants of this architecture now power today’s LLMs such as those from OpenAI, Google, Meta, Cohere, Anthropic and DeepSeek. In this course, you’ll learn in detail how LLMs process text. You'll also work through code examples that illustrate that transformer's individual components. In details, you’ll learn: - How the representation of language has evolved, from Bag-of-Words to Word2Vec embeddings to the transformer architecture that captures a word's meanings taking into account the context of other words in the input. - How inputs are broken down into tokens before they are sent to the language model. - The details of a transformer's main stages: Tokenization and embedding, the stack of transformer blocks, and the language model head. - The inner workings of the transformer block, including attention, which calculates relevance scores, and the feedforward layer, which incorporates stored information learned in training. - How cached calculations make transformers faster. - Some of the most recent ideas in the latest models such as Mixture-of-Experts (MoE) which uses multiple sub-models and a router on each layer to improve the quality of LLMs. By the end of this course, you’ll have a deep understanding of how LLMs actually process text and be able to read through papers describing the latest models and understand the details. Gaining this intuition will improve your approach to building LLM applications. Please sign up here:
3:36

Announcing How Transformer LLMs Work, created with Jay Alammar and Maarten Grootendorst, co-authors of the beautifully illustrated book, “Hands-On Large Language Models.” This course offers a deep dive into the inner workings of the transformer architecture that powers large language models (LLMs). The transformer architecture revolutionized generative AI; in fact, the "GPT" in ChatGPT stands for "Generative Pre-Trained Transformer." Originally introduced in the Google Brain team's groundbreaking 2017 paper "Attention Is All You Need," by Vaswani and others, transformers were a highly scalable model for machine translation tasks. Variants of this architecture now power today’s LLMs such as those from OpenAI, Google, Meta, Cohere, Anthropic and DeepSeek. In this course, you’ll learn in detail how LLMs process text. You'll also work through code examples that illustrate that transformer's individual components. In details, you’ll learn: - How the representation of language has evolved, from Bag-of-Words to Word2Vec embeddings to the transformer architecture that captures a word's meanings taking into account the context of other words in the input. - How inputs are broken down into tokens before they are sent to the language model. - The details of a transformer's main stages: Tokenization and embedding, the stack of transformer blocks, and the language model head. - The inner workings of the transformer block, including attention, which calculates relevance scores, and the feedforward layer, which incorporates stored information learned in training. - How cached calculations make transformers faster. - Some of the most recent ideas in the latest models such as Mixture-of-Experts (MoE) which uses multiple sub-models and a router on each layer to improve the quality of LLMs. By the end of this course, you’ll have a deep understanding of how LLMs actually process text and be able to read through papers describing the latest models and understand the details. Gaining this intuition will improve your approach to building LLM applications. Please sign up here:

Andrew Ng

259,920 görüntüleme • 1 yıl önce

Elon Musk was asked: “What’s one invention that’s made us worse, not better?” His answer: short-form video. He called it straight-up “brain rot.” And he’s not wrong. A local news report highlighted how kids are getting flooded with dopamine hits every 15–30 seconds from YouTube Shorts and TikTok-style content. Brain scans show overactivation in the reward centers, which over time trains the brain to crave instant gratification, shortens attention spans, and contributes to attention problems, behavioral issues, and even emotional dysregulation. Doctors are now seeing cases where it’s hard to tell the difference between true ADHD and what they’re calling “environmental ADHD” caused by excessive screen use. 78–84% of kids aged 2–12 are on YouTube, often for 2+ hours a day. This one feels especially urgent for parents. How much short-form video are your kids (or you) consuming daily — and have you noticed any real impact on attention span or mood?
1:57

Elon Musk was asked: “What’s one invention that’s made us worse, not better?” His answer: short-form video. He called it straight-up “brain rot.” And he’s not wrong. A local news report highlighted how kids are getting flooded with dopamine hits every 15–30 seconds from YouTube Shorts and TikTok-style content. Brain scans show overactivation in the reward centers, which over time trains the brain to crave instant gratification, shortens attention spans, and contributes to attention problems, behavioral issues, and even emotional dysregulation. Doctors are now seeing cases where it’s hard to tell the difference between true ADHD and what they’re calling “environmental ADHD” caused by excessive screen use. 78–84% of kids aged 2–12 are on YouTube, often for 2+ hours a day. This one feels especially urgent for parents. How much short-form video are your kids (or you) consuming daily — and have you noticed any real impact on attention span or mood?

Camus

579,417 görüntüleme • 4 ay önce

3Blue1Brown (grant sanderson) asked a jane street researcher whether people modelling markets think about time as continuous rather than discrete. "when you're modeling real world time series, they're not always so cleanly discrete, or as regular." "i do also think when you're modeling real world physical phenomena it is actually quite nice to say time is continuous. we just sample it at some points that we care for." in language the index is the whole object. token 1, token 2, token 3. there is no duration between tokens unless you add one. market data comes with a clock attached. quotes and trades arrive when they arrive, a 1m or daily bar is a grid you laid over them afterwards, and the waiting time between two events carries information: intensity, liquidity, whether anything is happening at all.
1:08

3Blue1Brown (grant sanderson) asked a jane street researcher whether people modelling markets think about time as continuous rather than discrete. "when you're modeling real world time series, they're not always so cleanly discrete, or as regular." "i do also think when you're modeling real world physical phenomena it is actually quite nice to say time is continuous. we just sample it at some points that we care for." in language the index is the whole object. token 1, token 2, token 3. there is no duration between tokens unless you add one. market data comes with a clock attached. quotes and trades arrive when they arrive, a 1m or daily bar is a grid you laid over them afterwards, and the waiting time between two events carries information: intensity, liquidity, whether anything is happening at all.

Goshawk Trades

174,634 görüntüleme • 20 gün önce

Researchers found a way to make LLMs 8.5x faster! (without compromising accuracy) Speculative decoding is quite an effective way to address the single-token bottleneck in traditional LLM inference. A small "draft" model first generates the next several tokens, then the large model verifies all of them at once in a single forward pass. If a token at any position is wrong, you keep everything before it and restart from there. This never does worse than normal decoding. But current drafters in Speculative decoding still guess one token at a time. That makes the drafting step itself a bottleneck, capping real-world speedups at 2-3x. DFlash is a new technique that swaps the autoregressive drafter with a lightweight block diffusion model that guesses all tokens in one parallel shot. Drafting cost stays flat no matter how many tokens you speculate. On top of that, the drafter is conditioned on hidden features pulled from multiple layers of the target model and injected into every draft layer, so it makes significantly better guesses than a drafter working from scratch. In the side-by-side demo below, vanilla decoding runs at 48.5 tokens/sec. DFlash hits 415 tokens/sec on the same model, with zero quality loss. It's already integrated with vLLM, SGLang, and Transformers, with draft models on HuggingFace for several models like Qwen3, Qwen3.5, Llama 3.1, Kimi-K2.5, gpt-oss, and many more. I have shared the GitHub repo in the replies! KV caching is another must-know technique to boost LLM inference. I recently wrote an article about it. Read it below. 👉 Over to you: What use case are you working on that can benefit from this new technique?
0:26

Researchers found a way to make LLMs 8.5x faster! (without compromising accuracy) Speculative decoding is quite an effective way to address the single-token bottleneck in traditional LLM inference. A small "draft" model first generates the next several tokens, then the large model verifies all of them at once in a single forward pass. If a token at any position is wrong, you keep everything before it and restart from there. This never does worse than normal decoding. But current drafters in Speculative decoding still guess one token at a time. That makes the drafting step itself a bottleneck, capping real-world speedups at 2-3x. DFlash is a new technique that swaps the autoregressive drafter with a lightweight block diffusion model that guesses all tokens in one parallel shot. Drafting cost stays flat no matter how many tokens you speculate. On top of that, the drafter is conditioned on hidden features pulled from multiple layers of the target model and injected into every draft layer, so it makes significantly better guesses than a drafter working from scratch. In the side-by-side demo below, vanilla decoding runs at 48.5 tokens/sec. DFlash hits 415 tokens/sec on the same model, with zero quality loss. It's already integrated with vLLM, SGLang, and Transformers, with draft models on HuggingFace for several models like Qwen3, Qwen3.5, Llama 3.1, Kimi-K2.5, gpt-oss, and many more. I have shared the GitHub repo in the replies! KV caching is another must-know technique to boost LLM inference. I recently wrote an article about it. Read it below. 👉 Over to you: What use case are you working on that can benefit from this new technique?

Avi Chawla

157,390 görüntüleme • 4 ay önce

Director Lee Hangyeol speaks with DFN about Taehyung Q: Your connection with V? 🧑🏻 With Taehyung, since working on solo music video projects, we had many opportunities to work together, so our connection became quite deep. His visuals are extremely special, but in real life he has this really “friend-next-door” kind of charm, and that’s actually the kind of charm I like the most. Honestly, all the BTS members are like that. Because I wanted to bring out that kind of charm, I mentioned the opinion that it would be nice if the intro were structured in a fun way. As soon as Taehyung heard that, he said, “Hyung, the thing I’m most confident in is being funny.” Because he said that, I think that’s how an iconic scene could be born. (
0:34

Director Lee Hangyeol speaks with DFN about Taehyung Q: Your connection with V? 🧑🏻 With Taehyung, since working on solo music video projects, we had many opportunities to work together, so our connection became quite deep. His visuals are extremely special, but in real life he has this really “friend-next-door” kind of charm, and that’s actually the kind of charm I like the most. Honestly, all the BTS members are like that. Because I wanted to bring out that kind of charm, I mentioned the opinion that it would be nice if the intro were structured in a fun way. As soon as Taehyung heard that, he said, “Hyung, the thing I’m most confident in is being funny.” Because he said that, I think that’s how an iconic scene could be born. (

Taehyung Naver

32,421 görüntüleme • 4 ay önce

tylercowen is bullish on AI education — here's why. 00:00 -- Preview 00:24 -- President Carlos Carvalho's AI-generated intro 03:21 -- Cowen reacts to UATX's campus 04:38 -- The AI revolution is here. Who will lose the most? 06:05 -- AI lawyers 07:17 -- Don't underestimate this 10:41 -- Changes to the "upper upper middle class" 12:38 -- How to be successful 13:43 -- The rise of managerial empires 14:02 -- When will we have the first billion dollar company with one employee? 16:05 -- 10-20 year forecast 16:19 -- Why education is so behind 17:01 -- Should you be bullish on UATX? 18:36 -- Should you still read Homer? 21:50 -- Write to think 25:01 -- Meet more people 25:42 -- How to get hired 26:54 -- Is AI your best mentor? 38:17 -- How to curb cheating 39:02 -- The new life of the mind 42:34 -- Q&A: Will there be more status associated with real education or AI education? 45:50 -- Q&A: Why do tech-savvy students need to practice using AI? 47:56 -- Q&A: Do LLMs atrophy your mind? 49:29 -- Q&A: How do you avoid AI-dependency? 51:05 -- Q&A: Isn't this vision lonely and isolating? 53:06 -- Q&A: Do students need teachers? 55:36 -- Q&A: What are the four most important courses for undergrads? 57:49 -- Q&A: Which AI company will win the AI race in the next five years and why? 59:22 -- Q&A: Can AI teach religion? 01:01:32 -- Q&A: Will AI narrow or widen our world? 01:04:37 -- Q&A: What makes us human? 01:05:42 -- Q&A: What is art? 01:08:33 -- Q&A: It's easy to catch cheaters
1:10:31

tylercowen is bullish on AI education — here's why. 00:00 -- Preview 00:24 -- President Carlos Carvalho's AI-generated intro 03:21 -- Cowen reacts to UATX's campus 04:38 -- The AI revolution is here. Who will lose the most? 06:05 -- AI lawyers 07:17 -- Don't underestimate this 10:41 -- Changes to the "upper upper middle class" 12:38 -- How to be successful 13:43 -- The rise of managerial empires 14:02 -- When will we have the first billion dollar company with one employee? 16:05 -- 10-20 year forecast 16:19 -- Why education is so behind 17:01 -- Should you be bullish on UATX? 18:36 -- Should you still read Homer? 21:50 -- Write to think 25:01 -- Meet more people 25:42 -- How to get hired 26:54 -- Is AI your best mentor? 38:17 -- How to curb cheating 39:02 -- The new life of the mind 42:34 -- Q&A: Will there be more status associated with real education or AI education? 45:50 -- Q&A: Why do tech-savvy students need to practice using AI? 47:56 -- Q&A: Do LLMs atrophy your mind? 49:29 -- Q&A: How do you avoid AI-dependency? 51:05 -- Q&A: Isn't this vision lonely and isolating? 53:06 -- Q&A: Do students need teachers? 55:36 -- Q&A: What are the four most important courses for undergrads? 57:49 -- Q&A: Which AI company will win the AI race in the next five years and why? 59:22 -- Q&A: Can AI teach religion? 01:01:32 -- Q&A: Will AI narrow or widen our world? 01:04:37 -- Q&A: What makes us human? 01:05:42 -- Q&A: What is art? 01:08:33 -- Q&A: It's easy to catch cheaters

University of Austin (UATX)

27,770 görüntüleme • 7 ay önce

You can now watch any AI paper instead of reading it There's a new site called arXivisual that turns ANY arXiv paper into a fully animated breakdown for free. Just add "isual" to the link becomes Paste it and an agent pipeline breaks the paper into sections, finds the concepts worth animating, writes Manim code for each one, and narrates it with voice You scroll, the narration plays, and each animation sits right next to the paragraph it explains Tried it on the original Transformers paper. Attention finally made sense to me in a way 10 blog posts never did, and it's open source. Four college kids built it at a hackathon Here's what I'd do if I were you: 1. Pick the one paper you've been avoiding 2. Change the link and paste it in 3. Scroll through it on your phone with headphones in 4. Ask your AI to explain the one section you still don't get You'll learn more in 20 minutes than a whole weekend of pretending to read PDFs This is how you read research in 2026
0:34

You can now watch any AI paper instead of reading it There's a new site called arXivisual that turns ANY arXiv paper into a fully animated breakdown for free. Just add "isual" to the link becomes Paste it and an agent pipeline breaks the paper into sections, finds the concepts worth animating, writes Manim code for each one, and narrates it with voice You scroll, the narration plays, and each animation sits right next to the paragraph it explains Tried it on the original Transformers paper. Attention finally made sense to me in a way 10 blog posts never did, and it's open source. Four college kids built it at a hackathon Here's what I'd do if I were you: 1. Pick the one paper you've been avoiding 2. Change the link and paste it in 3. Scroll through it on your phone with headphones in 4. Ask your AI to explain the one section you still don't get You'll learn more in 20 minutes than a whole weekend of pretending to read PDFs This is how you read research in 2026

Hasan Toor

121,111 görüntüleme • 8 gün önce

Qwen 3.8 Flash Next (125B-A6B) now runs natively in mlx-serve on Apple Silicon. No Python, Zig + Metal. (Preview Release) M4 Max, 4-bit pack (~75 GB resident): - ~70 tok/s serial, ~98 tok/s with MTP when tested using `npx llmprobe` - prefill ~700 tok/s, sparse attention past 2k tokens with fused kernels - images and video input supported Hyper-connections, n-gram embeddings gathered straight from disk, QSA block selection, 5 new Metal kernels. All open source, parity-tested against the HF reference. Kernel work: - hyper-connection read: 3 Metal kernels, split-K, write folded into the next read - GDN decode: 2 fused dispatches per layer - hd-256 attention: mask arm, sparse prefill + verify stay fused - all greedy-equivalent to the unfused path If you port any of this to your engine, a mention would be nice. MLX-Serve Preview 26.8.11 build: 4bit Speed Pack: 4-8bit Quality Pack: As for the model, i'm not sure yet, it feels verbose, and Opus like. Thinks a lot. Goes on long self testing loops/talking with itself, but need more time with it. It seems like a good model for agentic things.
0:25

Qwen 3.8 Flash Next (125B-A6B) now runs natively in mlx-serve on Apple Silicon. No Python, Zig + Metal. (Preview Release) M4 Max, 4-bit pack (~75 GB resident): - ~70 tok/s serial, ~98 tok/s with MTP when tested using `npx llmprobe` - prefill ~700 tok/s, sparse attention past 2k tokens with fused kernels - images and video input supported Hyper-connections, n-gram embeddings gathered straight from disk, QSA block selection, 5 new Metal kernels. All open source, parity-tested against the HF reference. Kernel work: - hyper-connection read: 3 Metal kernels, split-K, write folded into the next read - GDN decode: 2 fused dispatches per layer - hd-256 attention: mask arm, sparse prefill + verify stay fused - all greedy-equivalent to the unfused path If you port any of this to your engine, a mention would be nice. MLX-Serve Preview 26.8.11 build: 4bit Speed Pack: 4-8bit Quality Pack: As for the model, i'm not sure yet, it feels verbose, and Opus like. Thinks a lot. Goes on long self testing loops/talking with itself, but need more time with it. It seems like a good model for agentic things.

David Dalcu

122,199 görüntüleme • 20 gün önce

Brendan Schaub responds to Nate Diaz’s tweet in which Diaz called him “a big ol’ pus*y.”: “I will snap his fuc*ing neck.” “I guess he needs to do this to stay current. I retired 15 years ago, man. That being said, that’s social media, this isn’t real. Cause here’s what’s real: I will fu*k you up! That’s real. I’m not asking for this. I don’t wanna fu*k him up. I’m not trying to fu*k him up. End of the day if he walked in right now and I said ‘Chin, Case, can you guys leave?’—only one of us is going to walk out…unscathed, and it’s me! Every fu*king day of the week, 365 days a year—I will snap his fu*king neck. I’m not this fake gangster. I don’t go looking for fights. I know I’m tough, I don’t have to get on Twitter and blast it out and prove it every day because I’m insecure cause whatever sh*t he has going on with his childhood trauma. At the end of the day, I will twist your fu*king neck off. Ask your boys. And the difference between you and I, Nate—a million things don’t get me started—but the difference between you and I is that if you start crying because your brother, who has CTE, the difference between you and I is that if your brother—and Shane Carwin is a brother to me—is if you came out starting to get emotional about your brother’s issues with CTE and fighting and all this sh*t, cause we know he has his issues, I would support you. (🎥: The Schaub Show / BrendanSchaub on X)
5:32

Brendan Schaub responds to Nate Diaz’s tweet in which Diaz called him “a big ol’ pus*y.”: “I will snap his fuc*ing neck.” “I guess he needs to do this to stay current. I retired 15 years ago, man. That being said, that’s social media, this isn’t real. Cause here’s what’s real: I will fu*k you up! That’s real. I’m not asking for this. I don’t wanna fu*k him up. I’m not trying to fu*k him up. End of the day if he walked in right now and I said ‘Chin, Case, can you guys leave?’—only one of us is going to walk out…unscathed, and it’s me! Every fu*king day of the week, 365 days a year—I will snap his fu*king neck. I’m not this fake gangster. I don’t go looking for fights. I know I’m tough, I don’t have to get on Twitter and blast it out and prove it every day because I’m insecure cause whatever sh*t he has going on with his childhood trauma. At the end of the day, I will twist your fu*king neck off. Ask your boys. And the difference between you and I, Nate—a million things don’t get me started—but the difference between you and I is that if you start crying because your brother, who has CTE, the difference between you and I is that if your brother—and Shane Carwin is a brother to me—is if you came out starting to get emotional about your brother’s issues with CTE and fighting and all this sh*t, cause we know he has his issues, I would support you. (🎥: The Schaub Show / BrendanSchaub on X)

AFeldmanMMA

1,150,358 görüntüleme • 1 yıl önce

Another Chinese student made $400K for last 30 days He used Markov Chains and Claude Even without little knowledge you can make money on it How it works : > Markov Property : future depends only on the current state > State Space : the set of all possible states in the system > Transition Probability : the chance of moving from one state to another > Transition Matrix : a table of all transition probabilities between states > Steady State : the long-run equilibrium distribution as steps → infinity Just using these 5 rules you will be able to trade with 55%-65% wr Chinese trader has 71% wr and $418k PNL Account : Top formulas for trading so : 1) The Markov Property RSI = 100 - (100 / (1 + RS)) RS = Avg Gain / Avg Loss (N=14) RSI > 70 → Sell | RSI < 30 → Buy 2) State Distribution SMA = (P1+P2+...+PN) / N EMA = P×k + EMA_prev×(1-k), k = 2/(N+1) Fast MA crosses UP → Buy (Golden Cross) Fast MA crosses DOWN → Sell (Death Cross) 3) Expected Value EV = (p × W) − (q × L) p = win prob q = 1 − p W = profit if win L = loss if lose My result is 66,3% and $780 profit for last week I keep on learning trading every day and going to make my first $10k Trade with me :
0:21

Another Chinese student made $400K for last 30 days He used Markov Chains and Claude Even without little knowledge you can make money on it How it works : > Markov Property : future depends only on the current state > State Space : the set of all possible states in the system > Transition Probability : the chance of moving from one state to another > Transition Matrix : a table of all transition probabilities between states > Steady State : the long-run equilibrium distribution as steps → infinity Just using these 5 rules you will be able to trade with 55%-65% wr Chinese trader has 71% wr and $418k PNL Account : Top formulas for trading so : 1) The Markov Property RSI = 100 - (100 / (1 + RS)) RS = Avg Gain / Avg Loss (N=14) RSI > 70 → Sell | RSI < 30 → Buy 2) State Distribution SMA = (P1+P2+...+PN) / N EMA = P×k + EMA_prev×(1-k), k = 2/(N+1) Fast MA crosses UP → Buy (Golden Cross) Fast MA crosses DOWN → Sell (Death Cross) 3) Expected Value EV = (p × W) − (q × L) p = win prob q = 1 − p W = profit if win L = loss if lose My result is 66,3% and $780 profit for last week I keep on learning trading every day and going to make my first $10k Trade with me :

Lummox

11,242 görüntüleme • 5 ay önce

Want to create an avatar from a single image? FlexAvatar is a transformer model that creates full 360°, high-quality, and expressive 3D head avatar from just a single portrait image in minutes. Real-time Demo: FlexAvatar's lightweight architecture allows both animation and rendering in real-time, enabling interactive user experiences. To create a new 3D head avatar, only one image is required, e.g., from a webcam. The final avatar is ready after 2 minutes. Architecture: Under the hood, FlexAvatar adopts a transformer-based encoder-decoder design. The encoder maps the input image onto a latent avatar space, while the decoder produces 3D Gaussian attribute maps by incorporating the animation signal via cross-attention. The model learns all facial animations directly from the data without relying on pre-built 3D face models. This equips the avatars with realistic facial expressions. The internal avatar latent space can be conveniently used to integrate additional observations of a person via fitting. This enables use-cases where more than one image of a person is available, e.g., from a phone scan of the person. We train jointly on 2D monocular videos and multi-view data. However, in monocular videos, the animation signal leaks the target viewpoint, causing the model to produce incomplete 3D heads. We call this phenomenon entanglement of driving signal and target viewpoint. To prevent entanglement, we introduce bias sinks. These are learnable tokens that indicate whether a training sample stems from a monocular or a multi-view dataset. During training, the model learns to produce incomplete 3D heads only when the monocular token is present. During inference, FlexAvatar then always uses the multi-view token for which the model has learned to produce complete 3D heads. This simple design allows to combine the generalizability from monocular data with the quality of multi-view data. FlexAvatar summary: - Input: Single-image, phone scan, or monocular video - Output: Full 360° head avatar - Expressive animations - Real-time rendering and animation - Generalization to any portrait - Create a new avatar in 2 minutes - Use bias sinks to combine 2D and 3D data 🏠 🌍 🎥 Great work by Tobias Kirschstein and Simon Giebenhain!
0:30

Want to create an avatar from a single image? FlexAvatar is a transformer model that creates full 360°, high-quality, and expressive 3D head avatar from just a single portrait image in minutes. Real-time Demo: FlexAvatar's lightweight architecture allows both animation and rendering in real-time, enabling interactive user experiences. To create a new 3D head avatar, only one image is required, e.g., from a webcam. The final avatar is ready after 2 minutes. Architecture: Under the hood, FlexAvatar adopts a transformer-based encoder-decoder design. The encoder maps the input image onto a latent avatar space, while the decoder produces 3D Gaussian attribute maps by incorporating the animation signal via cross-attention. The model learns all facial animations directly from the data without relying on pre-built 3D face models. This equips the avatars with realistic facial expressions. The internal avatar latent space can be conveniently used to integrate additional observations of a person via fitting. This enables use-cases where more than one image of a person is available, e.g., from a phone scan of the person. We train jointly on 2D monocular videos and multi-view data. However, in monocular videos, the animation signal leaks the target viewpoint, causing the model to produce incomplete 3D heads. We call this phenomenon entanglement of driving signal and target viewpoint. To prevent entanglement, we introduce bias sinks. These are learnable tokens that indicate whether a training sample stems from a monocular or a multi-view dataset. During training, the model learns to produce incomplete 3D heads only when the monocular token is present. During inference, FlexAvatar then always uses the multi-view token for which the model has learned to produce complete 3D heads. This simple design allows to combine the generalizability from monocular data with the quality of multi-view data. FlexAvatar summary: - Input: Single-image, phone scan, or monocular video - Output: Full 360° head avatar - Expressive animations - Real-time rendering and animation - Generalization to any portrait - Create a new avatar in 2 minutes - Use bias sinks to combine 2D and 3D data 🏠 🌍 🎥 Great work by Tobias Kirschstein and Simon Giebenhain!

Matthias Niessner

96,334 görüntüleme • 9 ay önce