正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

building large language models from scratch by Sebastian Raschka was a great chance for me to sit down and study again all the LLM basics > token and positional embeddings > self-attention and what QKV is about > causal & multi-head attention studying llms and how they work can... show more

ℏεsam

38,524 subscribers

49,292 次观看 • 1 年前 •via X (Twitter)

教育健康养生科学技术

Anya Rossi• Live Now

Private livecam show

3 条评论

ℏεsam 的头像

ℏεsam1 年前

thanks professor @ProfTomYeh for the video, he advised me to share the notes to spread the word and I'm happy I did. here's the link to the repo if you're curios.

Power Homeschool 的头像

Power Homeschool2 年前

The Acellus® Learning System automates much of the busy-work associated with grading & keeping records so that parents can focus on what matters most–helping your child succeed. Enroll now! ⬇️

Callum 的头像

Callum1 年前

Great insights I came to the AI space late, so I was a little confused when I cracked open Tensorflow. I expected training to provide visibility to the underlying data, but was disappointed when I found out this wasn’t the case. I started structuring data myself, and found I was able to extract meaningful results and Create, Read, Update and Delete nodes and edges on the fly within my own Cube4D tensors. It made me come to the breakthrough that model training is just finding patterns within the data, and found that I was able to bypass the need for brute force model training by simply mapping the relationships in the schema itself. I essentially created a tesseract of data, which is a cornerstone of my Relational Intelligence Framework. I’d be keen to hear others thoughts on my approach to see if I’m on to something here. I’ve built, tested, validated, formalized it and licensed it behind CC-BY-NC-SA 4.0, but no one seems to understand it. Would really be keen to chat to others in the space.

相关视频

New short course: Attention in Transformers: Concepts and Code in PyTorch. Last week we released a course on how LLM transformers work. This week, go deeper and learn about the technical ideas behind the attention mechanism, and see how to code it in PyTorch. This course is built with Joshua Starmer, Founder and CEO of StatQuest. The attention mechanism was a breakthrough that led to transformers, the architecture powering large language models like ChatGPT. Transformers, introduced in the 2017 paper: "Attention is All You Need" by Viswani and others, took off because of its highly scalable design. In this course, you’ll learn how the attention mechanism, a key element of transformer-based LLMs, works and implement it in PyTorch. You'll develop deep intuition about building reliable, functional, and scalable AI applications. What you will do: - Understand the evolution of the attention mechanism, a key breakthrough that led to transformers. - Learn the relationships between word embeddings, positional embeddings, and attention. - Learn about the Query, Key, and Value matrices, and how to produce and use them in attention. - Walk through the math required to calculate self-attention and masked self-attention to learn why and how they work. - Understand the difference between self-attention and masked self-attention and how one is used in the encoder to build context-aware embeddings and the other is used in the decoder for generative outputs. - Learn the details of the encoder-decoder architecture, cross-attention, and multi-head attention and how they are all incorporated into a transformer. - Use PyTorch to code a class that implements self-attention, masked self-attention, and multi-head attention. There're lots of exciting technical details in this course. Please sign up here:

New short course: Attention in Transformers: Concepts and Code in PyTorch. Last week we released a course on how LLM transformers work. This week, go deeper and learn about the technical ideas behind the attention mechanism, and see how to code it in PyTorch. This course is built with Joshua Starmer, Founder and CEO of StatQuest. The attention mechanism was a breakthrough that led to transformers, the architecture powering large language models like ChatGPT. Transformers, introduced in the 2017 paper: "Attention is All You Need" by Viswani and others, took off because of its highly scalable design. In this course, you’ll learn how the attention mechanism, a key element of transformer-based LLMs, works and implement it in PyTorch. You'll develop deep intuition about building reliable, functional, and scalable AI applications. What you will do: - Understand the evolution of the attention mechanism, a key breakthrough that led to transformers. - Learn the relationships between word embeddings, positional embeddings, and attention. - Learn about the Query, Key, and Value matrices, and how to produce and use them in attention. - Walk through the math required to calculate self-attention and masked self-attention to learn why and how they work. - Understand the difference between self-attention and masked self-attention and how one is used in the encoder to build context-aware embeddings and the other is used in the decoder for generative outputs. - Learn the details of the encoder-decoder architecture, cross-attention, and multi-head attention and how they are all incorporated into a transformer. - Use PyTorch to code a class that implements self-attention, masked self-attention, and multi-head attention. There're lots of exciting technical details in this course. Please sign up here:

Andrew Ng

132,285 次观看 • 1 年前

Met a guy making $1.6 million/year as an LLM engineer. I asked him how he learned LLMs from scratch. He sent me the exact video that got him in. A 1 hour course on how LLMs actually work. He shows how transformers inside LLMs like ChatGPT & Claude are actually built. I watched it last night. Halfway through, I realized LLM architecture is way simpler than they make it look. Bookmark this and read the article below. • 00:00 - LLM foundations • 04:21 - LLM tokenization • 05:43 - LLMs vector embeddings • 22:16 - attention mechanism of LLM • 43:42 - LLM multi head attention

Met a guy making $1.6 million/year as an LLM engineer. I asked him how he learned LLMs from scratch. He sent me the exact video that got him in. A 1 hour course on how LLMs actually work. He shows how transformers inside LLMs like ChatGPT & Claude are actually built. I watched it last night. Halfway through, I realized LLM architecture is way simpler than they make it look. Bookmark this and read the article below. • 00:00 - LLM foundations • 04:21 - LLM tokenization • 05:43 - LLMs vector embeddings • 22:16 - attention mechanism of LLM • 43:42 - LLM multi head attention

Roan

80,650 次观看 • 22 天前

Embeddings (and how to create them) are, perhaps, the most interesting idea behind Large Language Models. I built a simple model to help you understand embeddings from scratch. Here is a step-by-step video explanation:

Embeddings (and how to create them) are, perhaps, the most interesting idea behind Large Language Models. I built a simple model to help you understand embeddings from scratch. Here is a step-by-step video explanation:

Santiago

44,107 次观看 • 1 年前

LLMs can make sense of retrieved context because of how transformers work. In one of the lessons from the Retrieval Augmented Generation (RAG) course, we unpack how LLMs process augmented prompts using token embeddings, positional vectors, and multi-head attention. Understanding these internals helps you design more reliable and efficient RAG systems. Watch the breakdown and keep learning how to build production-ready RAG systems in this course, taught by Zain:

LLMs can make sense of retrieved context because of how transformers work. In one of the lessons from the Retrieval Augmented Generation (RAG) course, we unpack how LLMs process augmented prompts using token embeddings, positional vectors, and multi-head attention. Understanding these internals helps you design more reliable and efficient RAG systems. Watch the breakdown and keep learning how to build production-ready RAG systems in this course, taught by Zain:

DeepLearning.AI

11,500 次观看 • 1 年前

Cam Boozer was asked about the noise and how he’s let the work do the talking: “It’s why you haven’t seen a lot from me on social media because I want to put my head down and work and try to be the best player I can be.” And he’s shown how good he is!〽️

Cam Boozer was asked about the noise and how he’s let the work do the talking: “It’s why you haven’t seen a lot from me on social media because I want to put my head down and work and try to be the best player I can be.” And he’s shown how good he is!〽️

CamBoozer Muse

34,626 次观看 • 1 个月前

$I hear so often from the Dommes I work with that they struggle with people online fetichizing them and simply seeing them for how sexy and beautiful they are. They project their fantasies and their desires onto you. That stops immediately once you move the attention from you to them. From 'look at me' to 'I see you'. What does that look like? When you create content, think of them and what this scene or that narrative is evoking. What will they learn from you? What they want is not to passively watch how sexy you are, but for you to train them, to give them instructions, to teach them, to guide them, to be in charge, to command them. This is not being an object but the main subject. The Authority figure. How is your content already doing that. The sexy photos can still be there, they are important to already capture des attention. But what you do with that attention once you have it, is where the power dynamic is established. Positioning yourself as more than a stunning Goddess, but actually a woman who has a voice, opinions, perspective, a philosophy, a way to doing things, teaching them what you like, how you like it, why you like it, already makes them want to be that for you. You hold the attention, you hold the power, so you direct it. And for that, you want them to know you get them and you know what lives within them... that creates the desire for you to be the one exposing it. You instantly build trust. Not because you demanded it, but because you earned it: you showed them you know what you are doing. You have experience, you understand them. They are not told to come see you, they are seduced into it. They desire it. And they will work for it. This will attract better clients (real subs) and instead of you trying to get their attention, they will work to earn yours. If you want to learn more about power dynamics, building a brand as a Pro or the psychology behind BDSM, you can now access all my trainings and classes in one place for a fraction of the cost of The Dominatrix Academy. And you can reinvest the total amount towards the Program. Message me [SECRET] for the details. This offer is not available on my website.$

I hear so often from the Dommes I work with that they struggle with people online fetichizing them and simply seeing them for how sexy and beautiful they are. They project their fantasies and their desires onto you. That stops immediately once you move the attention from you to them. From 'look at me' to 'I see you'. What does that look like? When you create content, think of them and what this scene or that narrative is evoking. What will they learn from you? What they want is not to passively watch how sexy you are, but for you to train them, to give them instructions, to teach them, to guide them, to be in charge, to command them. This is not being an object but the main subject. The Authority figure. How is your content already doing that. The sexy photos can still be there, they are important to already capture des attention. But what you do with that attention once you have it, is where the power dynamic is established. Positioning yourself as more than a stunning Goddess, but actually a woman who has a voice, opinions, perspective, a philosophy, a way to doing things, teaching them what you like, how you like it, why you like it, already makes them want to be that for you. You hold the attention, you hold the power, so you direct it. And for that, you want them to know you get them and you know what lives within them... that creates the desire for you to be the one exposing it. You instantly build trust. Not because you demanded it, but because you earned it: you showed them you know what you are doing. You have experience, you understand them. They are not told to come see you, they are seduced into it. They desire it. And they will work for it. This will attract better clients (real subs) and instead of you trying to get their attention, they will work to earn yours. If you want to learn more about power dynamics, building a brand as a Pro or the psychology behind BDSM, you can now access all my trainings and classes in one place for a fraction of the cost of The Dominatrix Academy. And you can reinvest the total amount towards the Program. Message me [SECRET] for the details. This offer is not available on my website.

Ms. Malissia

16,120 次观看 • 3 个月前

"Elon Musk is a genius by all accounts. His commitment and his work, like I asked him one time about how does he learn. And the amount of research, the amount of study that he does is unheard of, but he'll always say the most important thing is imagination. You can learn anything that you want to learn. You can study all these things that you have in a book. If you don't have the imagination to take it to another level, it doesn't mean anything. I sat down and spoke with Elon for a couple hours. It was really, really a joy." — Kobe Bryant

"Elon Musk is a genius by all accounts. His commitment and his work, like I asked him one time about how does he learn. And the amount of research, the amount of study that he does is unheard of, but he'll always say the most important thing is imagination. You can learn anything that you want to learn. You can study all these things that you have in a book. If you don't have the imagination to take it to another level, it doesn't mean anything. I sat down and spoke with Elon for a couple hours. It was really, really a joy." — Kobe Bryant

DogeDesigner

241,985 次观看 • 1 个月前

The goal isn't despair, it's indifference. It's enlightenment. It's having the ability to step in and out of the muck as you see fit without a handler or a sponsor coming down on you for it. I get fired up about things but this isn't my entire identity. I can put my phone down and walk away whenever I want and what a blessing that is.

The goal isn't despair, it's indifference. It's enlightenment. It's having the ability to step in and out of the muck as you see fit without a handler or a sponsor coming down on you for it. I get fired up about things but this isn't my entire identity. I can put my phone down and walk away whenever I want and what a blessing that is.

Shipwreck

20,812 次观看 • 6 个月前

Just saw this video by Prankstar on Tunde Ednut’s page on Instagram and I think I have seen it before. Now you will notice somethings in this video and one of them are how easy it was to talk to the Christian ladies maybe because of the way he dressed or his approach when going to meet them. But that’s not what I want you to see….. loolz 😀 What I want you to see is the fact that many of this Christian ladies are single and are all looking for or wanting to get into a relationship and maybe get married in the future. Where am I going with this? If you sit down and craft a well written title that can help Christian sisters meet good guys and partners who are interested in relationships, what to say, what not to say and how they can get to know if a guy is serious and wants them, you will sell like wild fire. Especially if your title is well crafted and appeal to that niche. Now on the other side, if you can help men to learn how to dress well and source for quality but cheap outfits, things to say to a woman and things not to say, how to not look desperate when having a conversation with a girl they like, this will also sell like wild fire. Each of the demograph approach this guy used can be crafted to a product that will sell. There’s money sitting somewhere inside this video for you if you sit down, watch it and get your onions and slice it. I hope to see you at the other side of success… cheers 🥂

Just saw this video by Prankstar on Tunde Ednut’s page on Instagram and I think I have seen it before. Now you will notice somethings in this video and one of them are how easy it was to talk to the Christian ladies maybe because of the way he dressed or his approach when going to meet them. But that’s not what I want you to see….. loolz 😀 What I want you to see is the fact that many of this Christian ladies are single and are all looking for or wanting to get into a relationship and maybe get married in the future. Where am I going with this? If you sit down and craft a well written title that can help Christian sisters meet good guys and partners who are interested in relationships, what to say, what not to say and how they can get to know if a guy is serious and wants them, you will sell like wild fire. Especially if your title is well crafted and appeal to that niche. Now on the other side, if you can help men to learn how to dress well and source for quality but cheap outfits, things to say to a woman and things not to say, how to not look desperate when having a conversation with a girl they like, this will also sell like wild fire. Each of the demograph approach this guy used can be crafted to a product that will sell. There’s money sitting somewhere inside this video for you if you sit down, watch it and get your onions and slice it. I hope to see you at the other side of success… cheers 🥂

Kingjames 𝐗 ✯

192,518 次观看 • 1 个月前

.tobi lutke removed all the Norman doors from Shopify’s offices because he couldn’t have excellent people surrounded by bad design. Ramp Cofounder Eric Glyman says he learned the same lesson by studying Breville toasters: “ If you care about design, you should buy a Breville toaster.” “ One of the big problems in a lot of B2B software is: it starts out clean and elegant. There's a bunch of businesses who love the product and they start serving more customers and have more requirements and more ideas.” “Before you know it, you end up with these disasters of B2B softwares which require a PhD to learn how to use.” “ If you were to go ask people, ‘What do you want in a toaster or in a microwave?’ They might say I like toast but sometimes I want to reheat pizza. I should have a ‘reheat frozen pizza’ mode. Or maybe you're cooking stuff and want it to be more like an oven, or you want to make popcorn and it's a toaster/microwave and all these things.” “Before you know it, you end up with these toasters filled with buttons and buttons and knobs and dials.” “That is literally asking customers what they want and building exactly that.” “ if you watch people toast things, what they actually do falls into two categories: you hit start and either you've undercooked the toast and need to put it back in, or you've overdone it and have to start over.” “If you look at the [Breville] there are only a few buttons. One of the four buttons is called ‘A Bit More.’” “ No one would ask you for ‘A Bit More.’ They just want the perfect level of toast and it's hard to know that, so they built that in.”

.tobi lutke removed all the Norman doors from Shopify’s offices because he couldn’t have excellent people surrounded by bad design. Ramp Cofounder Eric Glyman says he learned the same lesson by studying Breville toasters: “ If you care about design, you should buy a Breville toaster.” “ One of the big problems in a lot of B2B software is: it starts out clean and elegant. There's a bunch of businesses who love the product and they start serving more customers and have more requirements and more ideas.” “Before you know it, you end up with these disasters of B2B softwares which require a PhD to learn how to use.” “ If you were to go ask people, ‘What do you want in a toaster or in a microwave?’ They might say I like toast but sometimes I want to reheat pizza. I should have a ‘reheat frozen pizza’ mode. Or maybe you're cooking stuff and want it to be more like an oven, or you want to make popcorn and it's a toaster/microwave and all these things.” “Before you know it, you end up with these toasters filled with buttons and buttons and knobs and dials.” “That is literally asking customers what they want and building exactly that.” “ if you watch people toast things, what they actually do falls into two categories: you hit start and either you've undercooked the toast and need to put it back in, or you've overdone it and have to start over.” “If you look at the [Breville] there are only a few buttons. One of the four buttons is called ‘A Bit More.’” “ No one would ask you for ‘A Bit More.’ They just want the perfect level of toast and it's hard to know that, so they built that in.”

David Senra

172,242 次观看 • 1 个月前

The future of AI is open-source. And ollama is the easiest way to build AI applications with open-source LLMs. Here's how to build a free, private RAG app using open-source tools. We'll use: - Ollama for LLMs and embedding models - PostgreSQL for data storage and retrieval - pgai Vectorizer for embedding creation and sync (I use Nomic for embeddings and tinnyllama as my LLM but you can substitute them for any models on Ollama)

The future of AI is open-source. And ollama is the easiest way to build AI applications with open-source LLMs. Here's how to build a free, private RAG app using open-source tools. We'll use: - Ollama for LLMs and embedding models - PostgreSQL for data storage and retrieval - pgai Vectorizer for embedding creation and sync (I use Nomic for embeddings and tinnyllama as my LLM but you can substitute them for any models on Ollama)

Avthar

34,261 次观看 • 1 年前

🩷 : I bought a self-improvement book about how to get better at talking to people. I’m really not good at making conversation, especially with people I’m not close to. The version of me around my friends and my fans is exactly the same, but when I’m around other people, I’m actually not very good at making conversation. I was talking with my friends about a conversation I had with a celebrity I follow on social media, even though we’re not close. My friend was like, “Why didn’t you keep the conversation going?” and I replied, “Wait, really? I supposed to do that right?” It probably makes me seem stuck-up, but honestly, I’m just scared. I keep worrying, “What if they don’t want to talk to me?” So I don’t dare to start a conversation or say too much. But my friend told me, 'Just go for it! You never know if they want to talk to you or not. Sure, it might feel a little awkward or funny if it turns out they don't want to talk with you, but that's how you get closer to people.' That’s why I bought the book to read. I don’t want to be like this, but I just overthink and get scared beforehand. ꒰ #junewanwimol 𓈒 ꒱

🩷 : I bought a self-improvement book about how to get better at talking to people. I’m really not good at making conversation, especially with people I’m not close to. The version of me around my friends and my fans is exactly the same, but when I’m around other people, I’m actually not very good at making conversation. I was talking with my friends about a conversation I had with a celebrity I follow on social media, even though we’re not close. My friend was like, “Why didn’t you keep the conversation going?” and I replied, “Wait, really? I supposed to do that right?” It probably makes me seem stuck-up, but honestly, I’m just scared. I keep worrying, “What if they don’t want to talk to me?” So I don’t dare to start a conversation or say too much. But my friend told me, 'Just go for it! You never know if they want to talk to you or not. Sure, it might feel a little awkward or funny if it turns out they don't want to talk with you, but that's how you get closer to people.' That’s why I bought the book to read. I don’t want to be like this, but I just overthink and get scared beforehand. ꒰ #junewanwimol 𓈒 ꒱

𝗷𝗲𝗹𝗹𝘆𝗺𝗼𝗼𝗻𝗶𝗲 🕊️

72,870 次观看 • 25 天前

Rafa Nadal & Roger Federer were asked how they want to be remembered in a new campaign for Louis Vuitton Rafa: “At the end what will make me feel proud and happy is when I leave this world, if the tournament directors, the people who work at the tournaments, staff, ATP, say good things about who I was on the tour, or who I am as a person more than a tennis player. Because as a tennis player, at the end, we have the titles, we have the achievements and that's how it is. I achieved more than what I ever dreamed of. So for me at the end, the legacy in terms of a human being, after achieving all the things that we achieved, for me is the most important thing.” ❤️ Roger: “I hope if I'm remembered not only as a tennis player, but as a person behind what I gave to the game and what I represented to the game, and it was less about all my Wimbledon victories or whatever it was, that would mean a lot to me. If I can be remembered more as the personality rather than just a player, that would be great. And if I was a good role model for kids, that would make me happy too… And it would be nice, like Rafa said, if the people are happy to see us again. Not like ‘oh no, that guy again,’” 😂 These two. 🥹

Rafa Nadal & Roger Federer were asked how they want to be remembered in a new campaign for Louis Vuitton Rafa: “At the end what will make me feel proud and happy is when I leave this world, if the tournament directors, the people who work at the tournaments, staff, ATP, say good things about who I was on the tour, or who I am as a person more than a tennis player. Because as a tennis player, at the end, we have the titles, we have the achievements and that's how it is. I achieved more than what I ever dreamed of. So for me at the end, the legacy in terms of a human being, after achieving all the things that we achieved, for me is the most important thing.” ❤️ Roger: “I hope if I'm remembered not only as a tennis player, but as a person behind what I gave to the game and what I represented to the game, and it was less about all my Wimbledon victories or whatever it was, that would mean a lot to me. If I can be remembered more as the personality rather than just a player, that would be great. And if I was a good role model for kids, that would make me happy too… And it would be nice, like Rafa said, if the people are happy to see us again. Not like ‘oh no, that guy again,’” 😂 These two. 🥹

The Tennis Letter

525,823 次观看 • 2 年前

Jack Dorsey on becoming a better storyteller: "I found myself very early on thinking about something like thinking about this early idea for Twitter and saying to myself, I could build this awesome. You have those shower-like moments, or you're walking at midnight in some town in New York City, and you've got these amazing brand ideas. And then you start thinking, well, I could really start doing this if only X and if I had this person or if this technology existed or if this happened or this happened. And what I realized was that I was constantly making excuses for not working on it. And then the window had passed, and then I couldn't do anything. So I think it's really, really important to write it out or to draw it out or to code it. But you need to get it out of your head. And the reason you have to get it out of your head is that you need to be able to see it on a surface that is not in your mind. And once you can see it, and once you can step back from it, then you can also decide this passes my filter, my constraints, so maybe I can show it and share it with some other people. And then they will be like that's the stupidest idea ever and or that's somewhat interesting, but maybe this and this and this. So the sooner you can do that, then you have a lot of momentum around it, and you can really decide if you want to commit to it and work on it more or put it on the shelf for a later date. And the realization that I think everyone needs to have about that latter option, putting it on the shelf, is that you can come back to it and it will surface back up in another piece of work or another idea at some point in your life. So having that ability to close off a chapter and move on is really, really important. You can't have all these open threads, and that's what I realized I was doing. And that also encouraged me to really write more and to really think about what's the story? How are people coming to this? And like when I show my friends this, how are they going to react and I would write it down. I would actually treat it like a play. And when I realized that I was writing plays, I read a lot more plays for style and for substance and for technique and I think it's really good. I think there is another company that I have always looked towards for inspiration and I know a number of people in this room probably have a similar company in mind, which is Apple. Apple, I think, is run like a theater company. It has a great sense of pacing, has a great sense of story and has a great sense of execution and it's all about event-driven, it's all stage-driven, the stage being a billboard or the stage being a keynote or the stage being a product launch. All of it has a very, very cohesive end-to-end story. I mean you think about what happened when Steve Jobs came back to the company. The first thing he did was kill every product line the company was working on. And for two years,rs they had no product on the market whatsoever. All they had were a bunch of posters all around the world with Steve Jobs' heroes, and it said, think different. And it was just focused on bringing up the brand and making people aware of the brand again and how the brand is aligning to this particular feeling and story. And then they came out with the iMac and then built iTunes and then the iPod, and they realized that, wait a minute, people are carrying music on their phones now, so we better build a phone, an iPhone. And so this unfolding of the plot and the epic story has been very, very interesting to watch, especially if you look back to that time when he came back to the company. So I've learned a lot from that company and other companies that operate in a similar fashion."

Founder Mode

107,213 次观看 • 7 个月前