Loading video...

Video Failed to Load

Go Home

building large language models from scratch by Sebastian Raschka was a great chance for me to sit down and study again all the LLM basics > token and positional embeddings > self-attention and what QKV is about > causal & multi-head attention studying llms and how they work can...

Hesamation's profile picture

ℏεsam

38,524 subscribers

49,292 views • 1 year ago •via X (Twitter)

EducationHealth & WellnessScience & Technology
LIVE

Anya Rossi• Live Now

Private livecam show

3 Comments

ℏεsam's profile picture
ℏεsam1 year ago

thanks professor @ProfTomYeh for the video, he advised me to share the notes to spread the word and I'm happy I did. here's the link to the repo if you're curios.

Power Homeschool's profile picture
Power Homeschool2 years ago

The Acellus® Learning System automates much of the busy-work associated with grading & keeping records so that parents can focus on what matters most–helping your child succeed. Enroll now! ⬇️

Callum's profile picture
Callum1 year ago

Great insights I came to the AI space late, so I was a little confused when I cracked open Tensorflow. I expected training to provide visibility to the underlying data, but was disappointed when I found out this wasn’t the case. I started structuring data myself, and found I was able to extract meaningful results and Create, Read, Update and Delete nodes and edges on the fly within my own Cube4D tensors. It made me come to the breakthrough that model training is just finding patterns within the data, and found that I was able to bypass the need for brute force model training by simply mapping the relationships in the schema itself. I essentially created a tesseract of data, which is a cornerstone of my Relational Intelligence Framework. I’d be keen to hear others thoughts on my approach to see if I’m on to something here. I’ve built, tested, validated, formalized it and licensed it behind CC-BY-NC-SA 4.0, but no one seems to understand it. Would really be keen to chat to others in the space.

Related Videos

New short course: Attention in Transformers: Concepts and Code in PyTorch. Last week we released a course on how LLM transformers work. This week, go deeper and learn about the technical ideas behind the attention mechanism, and see how to code it in PyTorch. This course is built with Joshua Starmer, Founder and CEO of StatQuest. The attention mechanism was a breakthrough that led to transformers, the architecture powering large language models like ChatGPT. Transformers, introduced in the 2017 paper: "Attention is All You Need" by Viswani and others, took off because of its highly scalable design. In this course, you’ll learn how the attention mechanism, a key element of transformer-based LLMs, works and implement it in PyTorch. You'll develop deep intuition about building reliable, functional, and scalable AI applications. What you will do: - Understand the evolution of the attention mechanism, a key breakthrough that led to transformers. - Learn the relationships between word embeddings, positional embeddings, and attention. - Learn about the Query, Key, and Value matrices, and how to produce and use them in attention. - Walk through the math required to calculate self-attention and masked self-attention to learn why and how they work. - Understand the difference between self-attention and masked self-attention and how one is used in the encoder to build context-aware embeddings and the other is used in the decoder for generative outputs. - Learn the details of the encoder-decoder architecture, cross-attention, and multi-head attention and how they are all incorporated into a transformer. - Use PyTorch to code a class that implements self-attention, masked self-attention, and multi-head attention. There're lots of exciting technical details in this course. Please sign up here:
2:41

New short course: Attention in Transformers: Concepts and Code in PyTorch. Last week we released a course on how LLM transformers work. This week, go deeper and learn about the technical ideas behind the attention mechanism, and see how to code it in PyTorch. This course is built with Joshua Starmer, Founder and CEO of StatQuest. The attention mechanism was a breakthrough that led to transformers, the architecture powering large language models like ChatGPT. Transformers, introduced in the 2017 paper: "Attention is All You Need" by Viswani and others, took off because of its highly scalable design. In this course, you’ll learn how the attention mechanism, a key element of transformer-based LLMs, works and implement it in PyTorch. You'll develop deep intuition about building reliable, functional, and scalable AI applications. What you will do: - Understand the evolution of the attention mechanism, a key breakthrough that led to transformers. - Learn the relationships between word embeddings, positional embeddings, and attention. - Learn about the Query, Key, and Value matrices, and how to produce and use them in attention. - Walk through the math required to calculate self-attention and masked self-attention to learn why and how they work. - Understand the difference between self-attention and masked self-attention and how one is used in the encoder to build context-aware embeddings and the other is used in the decoder for generative outputs. - Learn the details of the encoder-decoder architecture, cross-attention, and multi-head attention and how they are all incorporated into a transformer. - Use PyTorch to code a class that implements self-attention, masked self-attention, and multi-head attention. There're lots of exciting technical details in this course. Please sign up here:

Andrew Ng

132,285 views • 1 year ago

Met a guy making $1.6 million/year as an LLM engineer. I asked him how he learned LLMs from scratch. He sent me the exact video that got him in. A 1 hour course on how LLMs actually work. He shows how transformers inside LLMs like ChatGPT & Claude are actually built. I watched it last night. Halfway through, I realized LLM architecture is way simpler than they make it look. Bookmark this and read the article below. • 00:00 - LLM foundations • 04:21 - LLM tokenization • 05:43 - LLMs vector embeddings • 22:16 - attention mechanism of LLM • 43:42 - LLM multi head attention
57:39

Met a guy making $1.6 million/year as an LLM engineer. I asked him how he learned LLMs from scratch. He sent me the exact video that got him in. A 1 hour course on how LLMs actually work. He shows how transformers inside LLMs like ChatGPT & Claude are actually built. I watched it last night. Halfway through, I realized LLM architecture is way simpler than they make it look. Bookmark this and read the article below. • 00:00 - LLM foundations • 04:21 - LLM tokenization • 05:43 - LLMs vector embeddings • 22:16 - attention mechanism of LLM • 43:42 - LLM multi head attention

Roan

80,126 views • 16 days ago

A quick video about how Come-from-Beyond discovered that you can actually break any Large Language Model by trolling it with complex questions. It is called a "Zero Delta" exploit and all LLM models are susceptible to it. I managed to recreate this on Grok and the video shows the result. With regards to all LLMs out there from $QUBIC - building the real AGI. Qubic My YouTube Channel:
2:21

A quick video about how Come-from-Beyond discovered that you can actually break any Large Language Model by trolling it with complex questions. It is called a "Zero Delta" exploit and all LLM models are susceptible to it. I managed to recreate this on Grok and the video shows the result. With regards to all LLMs out there from $QUBIC - building the real AGI. Qubic My YouTube Channel:

retrodrive ⛏

13,625 views • 1 year ago

Embeddings (and how to create them) are, perhaps, the most interesting idea behind Large Language Models. I built a simple model to help you understand embeddings from scratch. Here is a step-by-step video explanation:
51:43

Embeddings (and how to create them) are, perhaps, the most interesting idea behind Large Language Models. I built a simple model to help you understand embeddings from scratch. Here is a step-by-step video explanation:

Santiago

44,107 views • 1 year ago

LLMs can make sense of retrieved context because of how transformers work. In one of the lessons from the Retrieval Augmented Generation (RAG) course, we unpack how LLMs process augmented prompts using token embeddings, positional vectors, and multi-head attention. Understanding these internals helps you design more reliable and efficient RAG systems. Watch the breakdown and keep learning how to build production-ready RAG systems in this course, taught by Zain:
2:09

LLMs can make sense of retrieved context because of how transformers work. In one of the lessons from the Retrieval Augmented Generation (RAG) course, we unpack how LLMs process augmented prompts using token embeddings, positional vectors, and multi-head attention. Understanding these internals helps you design more reliable and efficient RAG systems. Watch the breakdown and keep learning how to build production-ready RAG systems in this course, taught by Zain:

DeepLearning.AI

11,500 views • 1 year ago

Cam Boozer was asked about the noise and how he’s let the work do the talking: “It’s why you haven’t seen a lot from me on social media because I want to put my head down and work and try to be the best player I can be.” And he’s shown how good he is!〽️
0:48

Cam Boozer was asked about the noise and how he’s let the work do the talking: “It’s why you haven’t seen a lot from me on social media because I want to put my head down and work and try to be the best player I can be.” And he’s shown how good he is!〽️

CamBoozer Muse

34,626 views • 1 month ago

I hear so often from the Dommes I work with that they struggle with people online fetichizing them and simply seeing them for how sexy and beautiful they are. They project their fantasies and their desires onto you. That stops immediately once you move the attention from you to them. From 'look at me' to 'I see you'. What does that look like? When you create content, think of them and what this scene or that narrative is evoking. What will they learn from you? What they want is not to passively watch how sexy you are, but for you to train them, to give them instructions, to teach them, to guide them, to be in charge, to command them. This is not being an object but the main subject. The Authority figure. How is your content already doing that. The sexy photos can still be there, they are important to already capture des attention. But what you do with that attention once you have it, is where the power dynamic is established. Positioning yourself as more than a stunning Goddess, but actually a woman who has a voice, opinions, perspective, a philosophy, a way to doing things, teaching them what you like, how you like it, why you like it, already makes them want to be that for you. You hold the attention, you hold the power, so you direct it. And for that, you want them to know you get them and you know what lives within them... that creates the desire for you to be the one exposing it. You instantly build trust. Not because you demanded it, but because you earned it: you showed them you know what you are doing. You have experience, you understand them. They are not told to come see you, they are seduced into it. They desire it. And they will work for it. This will attract better clients (real subs) and instead of you trying to get their attention, they will work to earn yours. If you want to learn more about power dynamics, building a brand as a Pro or the psychology behind BDSM, you can now access all my trainings and classes in one place for a fraction of the cost of The Dominatrix Academy. And you can reinvest the total amount towards the Program. Message me [SECRET] for the details. This offer is not available on my website.
0:51

I hear so often from the Dommes I work with that they struggle with people online fetichizing them and simply seeing them for how sexy and beautiful they are. They project their fantasies and their desires onto you. That stops immediately once you move the attention from you to them. From 'look at me' to 'I see you'. What does that look like? When you create content, think of them and what this scene or that narrative is evoking. What will they learn from you? What they want is not to passively watch how sexy you are, but for you to train them, to give them instructions, to teach them, to guide them, to be in charge, to command them. This is not being an object but the main subject. The Authority figure. How is your content already doing that. The sexy photos can still be there, they are important to already capture des attention. But what you do with that attention once you have it, is where the power dynamic is established. Positioning yourself as more than a stunning Goddess, but actually a woman who has a voice, opinions, perspective, a philosophy, a way to doing things, teaching them what you like, how you like it, why you like it, already makes them want to be that for you. You hold the attention, you hold the power, so you direct it. And for that, you want them to know you get them and you know what lives within them... that creates the desire for you to be the one exposing it. You instantly build trust. Not because you demanded it, but because you earned it: you showed them you know what you are doing. You have experience, you understand them. They are not told to come see you, they are seduced into it. They desire it. And they will work for it. This will attract better clients (real subs) and instead of you trying to get their attention, they will work to earn yours. If you want to learn more about power dynamics, building a brand as a Pro or the psychology behind BDSM, you can now access all my trainings and classes in one place for a fraction of the cost of The Dominatrix Academy. And you can reinvest the total amount towards the Program. Message me [SECRET] for the details. This offer is not available on my website.

Ms. Malissia

16,120 views • 3 months ago

"Elon Musk is a genius by all accounts. His commitment and his work, like I asked him one time about how does he learn. And the amount of research, the amount of study that he does is unheard of, but he'll always say the most important thing is imagination. You can learn anything that you want to learn. You can study all these things that you have in a book. If you don't have the imagination to take it to another level, it doesn't mean anything. I sat down and spoke with Elon for a couple hours. It was really, really a joy." — Kobe Bryant
0:28

"Elon Musk is a genius by all accounts. His commitment and his work, like I asked him one time about how does he learn. And the amount of research, the amount of study that he does is unheard of, but he'll always say the most important thing is imagination. You can learn anything that you want to learn. You can study all these things that you have in a book. If you don't have the imagination to take it to another level, it doesn't mean anything. I sat down and spoke with Elon for a couple hours. It was really, really a joy." — Kobe Bryant

DogeDesigner

241,985 views • 1 month ago

I’m at Davos and it’s clear that the global budget deficits require the most urgent attention from policy makers. I have conducted a study and have a solution. If you want a clear picture of what’s happening, and what we can do about it, I urge you to read my new free study “How Countries Go Broke.” You can find Part One here: #WEF25 #RayDalio
2:08

I’m at Davos and it’s clear that the global budget deficits require the most urgent attention from policy makers. I have conducted a study and have a solution. If you want a clear picture of what’s happening, and what we can do about it, I urge you to read my new free study “How Countries Go Broke.” You can find Part One here: #WEF25 #RayDalio

Ray Dalio

76,294 views • 1 year ago

The goal isn't despair, it's indifference. It's enlightenment. It's having the ability to step in and out of the muck as you see fit without a handler or a sponsor coming down on you for it. I get fired up about things but this isn't my entire identity. I can put my phone down and walk away whenever I want and what a blessing that is.
4:00

The goal isn't despair, it's indifference. It's enlightenment. It's having the ability to step in and out of the muck as you see fit without a handler or a sponsor coming down on you for it. I get fired up about things but this isn't my entire identity. I can put my phone down and walk away whenever I want and what a blessing that is.

Shipwreck

20,812 views • 6 months ago

Just saw this video by Prankstar on Tunde Ednut’s page on Instagram and I think I have seen it before. Now you will notice somethings in this video and one of them are how easy it was to talk to the Christian ladies maybe because of the way he dressed or his approach when going to meet them. But that’s not what I want you to see….. loolz 😀 What I want you to see is the fact that many of this Christian ladies are single and are all looking for or wanting to get into a relationship and maybe get married in the future. Where am I going with this? If you sit down and craft a well written title that can help Christian sisters meet good guys and partners who are interested in relationships, what to say, what not to say and how they can get to know if a guy is serious and wants them, you will sell like wild fire. Especially if your title is well crafted and appeal to that niche. Now on the other side, if you can help men to learn how to dress well and source for quality but cheap outfits, things to say to a woman and things not to say, how to not look desperate when having a conversation with a girl they like, this will also sell like wild fire. Each of the demograph approach this guy used can be crafted to a product that will sell. There’s money sitting somewhere inside this video for you if you sit down, watch it and get your onions and slice it. I hope to see you at the other side of success… cheers 🥂
8:02

Just saw this video by Prankstar on Tunde Ednut’s page on Instagram and I think I have seen it before. Now you will notice somethings in this video and one of them are how easy it was to talk to the Christian ladies maybe because of the way he dressed or his approach when going to meet them. But that’s not what I want you to see….. loolz 😀 What I want you to see is the fact that many of this Christian ladies are single and are all looking for or wanting to get into a relationship and maybe get married in the future. Where am I going with this? If you sit down and craft a well written title that can help Christian sisters meet good guys and partners who are interested in relationships, what to say, what not to say and how they can get to know if a guy is serious and wants them, you will sell like wild fire. Especially if your title is well crafted and appeal to that niche. Now on the other side, if you can help men to learn how to dress well and source for quality but cheap outfits, things to say to a woman and things not to say, how to not look desperate when having a conversation with a girl they like, this will also sell like wild fire. Each of the demograph approach this guy used can be crafted to a product that will sell. There’s money sitting somewhere inside this video for you if you sit down, watch it and get your onions and slice it. I hope to see you at the other side of success… cheers 🥂

Kingjames 𝐗 ✯

192,518 views • 28 days ago

.tobi lutke removed all the Norman doors from Shopify’s offices because he couldn’t have excellent people surrounded by bad design. Ramp Cofounder Eric Glyman says he learned the same lesson by studying Breville toasters: “ If you care about design, you should buy a Breville toaster.” “ One of the big problems in a lot of B2B software is: it starts out clean and elegant. There's a bunch of businesses who love the product and they start serving more customers and have more requirements and more ideas.” “Before you know it, you end up with these disasters of B2B softwares which require a PhD to learn how to use.” “ If you were to go ask people, ‘What do you want in a toaster or in a microwave?’ They might say I like toast but sometimes I want to reheat pizza. I should have a ‘reheat frozen pizza’ mode. Or maybe you're cooking stuff and want it to be more like an oven, or you want to make popcorn and it's a toaster/microwave and all these things.” “Before you know it, you end up with these toasters filled with buttons and buttons and knobs and dials.” “That is literally asking customers what they want and building exactly that.” “ if you watch people toast things, what they actually do falls into two categories: you hit start and either you've undercooked the toast and need to put it back in, or you've overdone it and have to start over.” “If you look at the [Breville] there are only a few buttons. One of the four buttons is called ‘A Bit More.’” “ No one would ask you for ‘A Bit More.’ They just want the perfect level of toast and it's hard to know that, so they built that in.”
3:43

.tobi lutke removed all the Norman doors from Shopify’s offices because he couldn’t have excellent people surrounded by bad design. Ramp Cofounder Eric Glyman says he learned the same lesson by studying Breville toasters: “ If you care about design, you should buy a Breville toaster.” “ One of the big problems in a lot of B2B software is: it starts out clean and elegant. There's a bunch of businesses who love the product and they start serving more customers and have more requirements and more ideas.” “Before you know it, you end up with these disasters of B2B softwares which require a PhD to learn how to use.” “ If you were to go ask people, ‘What do you want in a toaster or in a microwave?’ They might say I like toast but sometimes I want to reheat pizza. I should have a ‘reheat frozen pizza’ mode. Or maybe you're cooking stuff and want it to be more like an oven, or you want to make popcorn and it's a toaster/microwave and all these things.” “Before you know it, you end up with these toasters filled with buttons and buttons and knobs and dials.” “That is literally asking customers what they want and building exactly that.” “ if you watch people toast things, what they actually do falls into two categories: you hit start and either you've undercooked the toast and need to put it back in, or you've overdone it and have to start over.” “If you look at the [Breville] there are only a few buttons. One of the four buttons is called ‘A Bit More.’” “ No one would ask you for ‘A Bit More.’ They just want the perfect level of toast and it's hard to know that, so they built that in.”

David Senra

172,242 views • 1 month ago

The future of AI is open-source. And ollama is the easiest way to build AI applications with open-source LLMs. Here's how to build a free, private RAG app using open-source tools. We'll use: - Ollama for LLMs and embedding models - PostgreSQL for data storage and retrieval - pgai Vectorizer for embedding creation and sync (I use Nomic for embeddings and tinnyllama as my LLM but you can substitute them for any models on Ollama)
15:16

The future of AI is open-source. And ollama is the easiest way to build AI applications with open-source LLMs. Here's how to build a free, private RAG app using open-source tools. We'll use: - Ollama for LLMs and embedding models - PostgreSQL for data storage and retrieval - pgai Vectorizer for embedding creation and sync (I use Nomic for embeddings and tinnyllama as my LLM but you can substitute them for any models on Ollama)

Avthar

34,261 views • 1 year ago

🩷 : I bought a self-improvement book about how to get better at talking to people. I’m really not good at making conversation, especially with people I’m not close to. The version of me around my friends and my fans is exactly the same, but when I’m around other people, I’m actually not very good at making conversation. I was talking with my friends about a conversation I had with a celebrity I follow on social media, even though we’re not close. My friend was like, “Why didn’t you keep the conversation going?” and I replied, “Wait, really? I supposed to do that right?” It probably makes me seem stuck-up, but honestly, I’m just scared. I keep worrying, “What if they don’t want to talk to me?” So I don’t dare to start a conversation or say too much. But my friend told me, 'Just go for it! You never know if they want to talk to you or not. Sure, it might feel a little awkward or funny if it turns out they don't want to talk with you, but that's how you get closer to people.' That’s why I bought the book to read. I don’t want to be like this, but I just overthink and get scared beforehand. ꒰ #junewanwimol 𓈒 ꒱
1:35

🩷 : I bought a self-improvement book about how to get better at talking to people. I’m really not good at making conversation, especially with people I’m not close to. The version of me around my friends and my fans is exactly the same, but when I’m around other people, I’m actually not very good at making conversation. I was talking with my friends about a conversation I had with a celebrity I follow on social media, even though we’re not close. My friend was like, “Why didn’t you keep the conversation going?” and I replied, “Wait, really? I supposed to do that right?” It probably makes me seem stuck-up, but honestly, I’m just scared. I keep worrying, “What if they don’t want to talk to me?” So I don’t dare to start a conversation or say too much. But my friend told me, 'Just go for it! You never know if they want to talk to you or not. Sure, it might feel a little awkward or funny if it turns out they don't want to talk with you, but that's how you get closer to people.' That’s why I bought the book to read. I don’t want to be like this, but I just overthink and get scared beforehand. ꒰ #junewanwimol 𓈒 ꒱

𝗷𝗲𝗹𝗹𝘆𝗺𝗼𝗼𝗻𝗶𝗲 🕊️

72,870 views • 19 days ago

Rafa Nadal & Roger Federer were asked how they want to be remembered in a new campaign for Louis Vuitton Rafa: “At the end what will make me feel proud and happy is when I leave this world, if the tournament directors, the people who work at the tournaments, staff, ATP, say good things about who I was on the tour, or who I am as a person more than a tennis player. Because as a tennis player, at the end, we have the titles, we have the achievements and that's how it is. I achieved more than what I ever dreamed of. So for me at the end, the legacy in terms of a human being, after achieving all the things that we achieved, for me is the most important thing.” ❤️ Roger: “I hope if I'm remembered not only as a tennis player, but as a person behind what I gave to the game and what I represented to the game, and it was less about all my Wimbledon victories or whatever it was, that would mean a lot to me. If I can be remembered more as the personality rather than just a player, that would be great. And if I was a good role model for kids, that would make me happy too… And it would be nice, like Rafa said, if the people are happy to see us again. Not like ‘oh no, that guy again,’” 😂 These two. 🥹
1:27

Rafa Nadal & Roger Federer were asked how they want to be remembered in a new campaign for Louis Vuitton Rafa: “At the end what will make me feel proud and happy is when I leave this world, if the tournament directors, the people who work at the tournaments, staff, ATP, say good things about who I was on the tour, or who I am as a person more than a tennis player. Because as a tennis player, at the end, we have the titles, we have the achievements and that's how it is. I achieved more than what I ever dreamed of. So for me at the end, the legacy in terms of a human being, after achieving all the things that we achieved, for me is the most important thing.” ❤️ Roger: “I hope if I'm remembered not only as a tennis player, but as a person behind what I gave to the game and what I represented to the game, and it was less about all my Wimbledon victories or whatever it was, that would mean a lot to me. If I can be remembered more as the personality rather than just a player, that would be great. And if I was a good role model for kids, that would make me happy too… And it would be nice, like Rafa said, if the people are happy to see us again. Not like ‘oh no, that guy again,’” 😂 These two. 🥹

The Tennis Letter

525,786 views • 2 years ago

I dont think you can use marriage to get my attention in this big 2026 because ee no dey sharkk me again. Before I want to get married but now it feels like a thre@t to me. All I'm thinking about now is how to get rich and go on vacation" — Lady reveals
1:00

I dont think you can use marriage to get my attention in this big 2026 because ee no dey sharkk me again. Before I want to get married but now it feels like a thre@t to me. All I'm thinking about now is how to get rich and go on vacation" — Lady reveals

TENIOLA

46,720 views • 4 months ago