Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Zhilin at GTC: Introducing Attention Residuals Learning selective memory, rather than mechanically accumulating everything, is the beauty of attention. Many of you have probably read Attention Is All You Need, the 2017 Transformer paper that brought “human-like” attention into the model’s field of view. From that point on, models... no longer simply read everything in a mechanical way. Instead, they began to develop a sense of what matters more and what matters less across the text, choosing to retain the more important information. Recently, Kimi applied this idea of attention to the temporal dimension, then rotated it 90 degrees into the model’s depth dimension. This allows the model to have attention not only over time, but also throughout the process of information transmission across layers—giving it a more intelligent way to understand and process information.show more

Kimi.ai

182,122 subscribers

115,162 views • 5 months ago •via X (Twitter)

Science & Technology

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

Steve Nash engaged in self-regulated learning early! His father, canny as a fox, pointed out what he did well and encouraged him to bottle the details. Remember this, remember that! Remember this, remember that! Memory - but it was never about remembering the performance outcomes, it was always performance process - the actions and decisions that helped the young Nash to score, not the scoring itself. All detail! All detail! This has been called attentional density (by neuroscientist David Schwartz). When you pay attention to something, your brain fires in patterns akin to that something. The more you think about it, the more the brain fires that way. The more the brain fires that way, the more chance you have of remembering what you’re paying attention to. Pay attention to the details! Pay attention to the details!

Steve Nash engaged in self-regulated learning early! His father, canny as a fox, pointed out what he did well and encouraged him to bottle the details. Remember this, remember that! Remember this, remember that! Memory - but it was never about remembering the performance outcomes, it was always performance process - the actions and decisions that helped the young Nash to score, not the scoring itself. All detail! All detail! This has been called attentional density (by neuroscientist David Schwartz). When you pay attention to something, your brain fires in patterns akin to that something. The more you think about it, the more the brain fires that way. The more the brain fires that way, the more chance you have of remembering what you’re paying attention to. Pay attention to the details! Pay attention to the details!

Daniel Abrahams

50,933 views • 7 months ago

New short course: Attention in Transformers: Concepts and Code in PyTorch. Last week we released a course on how LLM transformers work. This week, go deeper and learn about the technical ideas behind the attention mechanism, and see how to code it in PyTorch. This course is built with Joshua Starmer, Founder and CEO of StatQuest. The attention mechanism was a breakthrough that led to transformers, the architecture powering large language models like ChatGPT. Transformers, introduced in the 2017 paper: "Attention is All You Need" by Viswani and others, took off because of its highly scalable design. In this course, you’ll learn how the attention mechanism, a key element of transformer-based LLMs, works and implement it in PyTorch. You'll develop deep intuition about building reliable, functional, and scalable AI applications. What you will do: - Understand the evolution of the attention mechanism, a key breakthrough that led to transformers. - Learn the relationships between word embeddings, positional embeddings, and attention. - Learn about the Query, Key, and Value matrices, and how to produce and use them in attention. - Walk through the math required to calculate self-attention and masked self-attention to learn why and how they work. - Understand the difference between self-attention and masked self-attention and how one is used in the encoder to build context-aware embeddings and the other is used in the decoder for generative outputs. - Learn the details of the encoder-decoder architecture, cross-attention, and multi-head attention and how they are all incorporated into a transformer. - Use PyTorch to code a class that implements self-attention, masked self-attention, and multi-head attention. There're lots of exciting technical details in this course. Please sign up here:

New short course: Attention in Transformers: Concepts and Code in PyTorch. Last week we released a course on how LLM transformers work. This week, go deeper and learn about the technical ideas behind the attention mechanism, and see how to code it in PyTorch. This course is built with Joshua Starmer, Founder and CEO of StatQuest. The attention mechanism was a breakthrough that led to transformers, the architecture powering large language models like ChatGPT. Transformers, introduced in the 2017 paper: "Attention is All You Need" by Viswani and others, took off because of its highly scalable design. In this course, you’ll learn how the attention mechanism, a key element of transformer-based LLMs, works and implement it in PyTorch. You'll develop deep intuition about building reliable, functional, and scalable AI applications. What you will do: - Understand the evolution of the attention mechanism, a key breakthrough that led to transformers. - Learn the relationships between word embeddings, positional embeddings, and attention. - Learn about the Query, Key, and Value matrices, and how to produce and use them in attention. - Walk through the math required to calculate self-attention and masked self-attention to learn why and how they work. - Understand the difference between self-attention and masked self-attention and how one is used in the encoder to build context-aware embeddings and the other is used in the decoder for generative outputs. - Learn the details of the encoder-decoder architecture, cross-attention, and multi-head attention and how they are all incorporated into a transformer. - Use PyTorch to code a class that implements self-attention, masked self-attention, and multi-head attention. There're lots of exciting technical details in this course. Please sign up here:

Andrew Ng

132,400 views • 1 year ago

6⃣ Main Different Modes of Attention + explanations Can you notice when you are spontaneously in each of these types of attention throughout the week? Most think attention mode 2. is the only way to be concentrated = sustaining engagement with desired phenomena for long periods of time. But actually you can learn to maintain each mode of attention and concentrate in different ways - knowing this radically opens up one’s potential in meditation practice and access to many more states of mind and insights. 🤯 ¡This animation visually illustrates different ways in which attention can behave, but note that these different modes are relevant across the senses (not just with seeing)!

6⃣ Main Different Modes of Attention + explanations Can you notice when you are spontaneously in each of these types of attention throughout the week? Most think attention mode 2. is the only way to be concentrated = sustaining engagement with desired phenomena for long periods of time. But actually you can learn to maintain each mode of attention and concentrate in different ways - knowing this radically opens up one’s potential in meditation practice and access to many more states of mind and insights. 🤯 ¡This animation visually illustrates different ways in which attention can behave, but note that these different modes are relevant across the senses (not just with seeing)!

Roger This

12,801 views • 1 year ago

POSOBIEC: People won’t read a Supreme Court ruling, but they’ll watch a Netflix miniseries on it If you're not paying attention to who is in control of this, then you're not paying attention to who's in control of the minds of the masses

POSOBIEC: People won’t read a Supreme Court ruling, but they’ll watch a Netflix miniseries on it If you're not paying attention to who is in control of this, then you're not paying attention to who's in control of the minds of the masses

Jack Posobiec

310,143 views • 9 months ago

Alex Pereira shares what has frustrated him most following his loss to Ciryl Gane: "Herb Dean came in, and I said, 'Listen, I understand that sometimes in a fight, eye pokes can happen and low blows can happen... but this guy has a history of shots to the back of the head, so please pay attention to those during the fight.' After we spoke with him, he brought Mike Beltran over and said, 'Just so you understand, the back of the head is here.' So it seemed like he absorbed the information. Then, the next day, seeing everything that happened and knowing that we brought it to his attention, he did not do his job."

Alex Pereira shares what has frustrated him most following his loss to Ciryl Gane: "Herb Dean came in, and I said, 'Listen, I understand that sometimes in a fight, eye pokes can happen and low blows can happen... but this guy has a history of shots to the back of the head, so please pay attention to those during the fight.' After we spoke with him, he brought Mike Beltran over and said, 'Just so you understand, the back of the head is here.' So it seemed like he absorbed the information. Then, the next day, seeing everything that happened and knowing that we brought it to his attention, he did not do his job."

Ariel Helwani

758,943 views • 3 months ago

I would pay close attention to this clip about USOs (Unidentified, Submerged Objects) from our recent episode of WEAPONIZED. You are going to be hearing a lot more about this soon. The information is right in front of you - if you read between the lines - and soon enough, you won’t need to read between the lines 😉🪼 WATCH : LISTEN : LEARN :

I would pay close attention to this clip about USOs (Unidentified, Submerged Objects) from our recent episode of WEAPONIZED. You are going to be hearing a lot more about this soon. The information is right in front of you - if you read between the lines - and soon enough, you won’t need to read between the lines 😉🪼 WATCH : LISTEN : LEARN :

Jeremy Kenyon Lockyer Corbell

132,000 views • 1 year ago

🇺🇸 Chamath just revealed the truth on Rogan… attention is what runs it all. Google ranks the web by attention. Facebook and Instagram are built to capture attention. The most important AI paper ever written is literally called "Attention Is All You Need." Every system that now runs the world reduces to the same single currency. And at some point, that stops sounding like branding and starts sounding like a warning. Joe Rogan, Chamath Palihapitiya

🇺🇸 Chamath just revealed the truth on Rogan… attention is what runs it all. Google ranks the web by attention. Facebook and Instagram are built to capture attention. The most important AI paper ever written is literally called "Attention Is All You Need." Every system that now runs the world reduces to the same single currency. And at some point, that stops sounding like branding and starts sounding like a warning. Joe Rogan, Chamath Palihapitiya

Mario Nawfal

314,947 views • 4 months ago

my favorite bit of attention to detail in this scene is that the Pilots and operator speak English, rather than Japanese Extra bit of realism that makes me appreciate it all the more

my favorite bit of attention to detail in this scene is that the Pilots and operator speak English, rather than Japanese Extra bit of realism that makes me appreciate it all the more

MA5K CEO

188,571 views • 1 year ago

xQc explains the difference between Twitch fame, real fame, and TikTok fame "In the market of attention..nobody's famous on Twitch, like none of us are famous..but if u were to put it on a scale, definitely more famous than anybody on TikTok" "They'll literally scroll through and see oh TikTok, very cool move on..Twitch is more like they have to click on the page, they have get on there, ppl go in the chat..ppl spend at least a small amount of time watching this sh*t and giving it attention"

xQc explains the difference between Twitch fame, real fame, and TikTok fame "In the market of attention..nobody's famous on Twitch, like none of us are famous..but if u were to put it on a scale, definitely more famous than anybody on TikTok" "They'll literally scroll through and see oh TikTok, very cool move on..Twitch is more like they have to click on the page, they have get on there, ppl go in the chat..ppl spend at least a small amount of time watching this sh*t and giving it attention"

yeet

41,094 views • 2 months ago

“I Started Losing My Sense of Self. I’ve actually never even talked about this. I’ve never told anybody this because it sounds probably stupid, but I really started paying attention to depth perception and trying to view the world in more of a three-dimensional experience. It’s also like I lose a sense of self - not completely, but maybe 75% of the way there.” Sadhguru

“I Started Losing My Sense of Self. I’ve actually never even talked about this. I’ve never told anybody this because it sounds probably stupid, but I really started paying attention to depth perception and trying to view the world in more of a three-dimensional experience. It’s also like I lose a sense of self - not completely, but maybe 75% of the way there.” Sadhguru

Shawn Ryan

290,328 views • 2 months ago

I have investigated the situation. EUW is doomed and we are only getting worse, give soloq 5% more of your attention and euw is saved, players arent even able to process basic information anymore

I have investigated the situation. EUW is doomed and we are only getting worse, give soloq 5% more of your attention and euw is saved, players arent even able to process basic information anymore

Veigarv2

199,363 views • 2 years ago

Why is it so challenging for Vice President Kamala Harris to construct a straightforward sentence? READ: "I feel very strongly about the importance as a general matter of engaging in U.S. policy as it relates to foreign affairs in a way that we pay attention, of course, to the immediate concerns and threats if they exist, but that we also pay attention to 10, 20, 30 years down the line and what we are developing now that will be to the benefit of our country then."

Why is it so challenging for Vice President Kamala Harris to construct a straightforward sentence? READ: "I feel very strongly about the importance as a general matter of engaging in U.S. policy as it relates to foreign affairs in a way that we pay attention, of course, to the immediate concerns and threats if they exist, but that we also pay attention to 10, 20, 30 years down the line and what we are developing now that will be to the benefit of our country then."

Simon Ateba

1,758,113 views • 3 years ago

#Colts HC Shane Steichen on QB Anthony Richardson: “I’ve said ‘the process, the process’ for a long time, and what I mean by that is the attention to detail in everything he does. From the classroom to the walkthroughs to practice …” “He’s gonna start the rest of the season.”

#Colts HC Shane Steichen on QB Anthony Richardson: “I’ve said ‘the process, the process’ for a long time, and what I mean by that is the attention to detail in everything he does. From the classroom to the walkthroughs to practice …” “He’s gonna start the rest of the season.”

James Boyd

140,105 views • 1 year ago

$I hear so often from the Dommes I work with that they struggle with people online fetichizing them and simply seeing them for how sexy and beautiful they are. They project their fantasies and their desires onto you. That stops immediately once you move the attention from you to them. From 'look at me' to 'I see you'. What does that look like? When you create content, think of them and what this scene or that narrative is evoking. What will they learn from you? What they want is not to passively watch how sexy you are, but for you to train them, to give them instructions, to teach them, to guide them, to be in charge, to command them. This is not being an object but the main subject. The Authority figure. How is your content already doing that. The sexy photos can still be there, they are important to already capture des attention. But what you do with that attention once you have it, is where the power dynamic is established. Positioning yourself as more than a stunning Goddess, but actually a woman who has a voice, opinions, perspective, a philosophy, a way to doing things, teaching them what you like, how you like it, why you like it, already makes them want to be that for you. You hold the attention, you hold the power, so you direct it. And for that, you want them to know you get them and you know what lives within them... that creates the desire for you to be the one exposing it. You instantly build trust. Not because you demanded it, but because you earned it: you showed them you know what you are doing. You have experience, you understand them. They are not told to come see you, they are seduced into it. They desire it. And they will work for it. This will attract better clients (real subs) and instead of you trying to get their attention, they will work to earn yours. If you want to learn more about power dynamics, building a brand as a Pro or the psychology behind BDSM, you can now access all my trainings and classes in one place for a fraction of the cost of The Dominatrix Academy. And you can reinvest the total amount towards the Program. Message me [SECRET] for the details. This offer is not available on my website.$

I hear so often from the Dommes I work with that they struggle with people online fetichizing them and simply seeing them for how sexy and beautiful they are. They project their fantasies and their desires onto you. That stops immediately once you move the attention from you to them. From 'look at me' to 'I see you'. What does that look like? When you create content, think of them and what this scene or that narrative is evoking. What will they learn from you? What they want is not to passively watch how sexy you are, but for you to train them, to give them instructions, to teach them, to guide them, to be in charge, to command them. This is not being an object but the main subject. The Authority figure. How is your content already doing that. The sexy photos can still be there, they are important to already capture des attention. But what you do with that attention once you have it, is where the power dynamic is established. Positioning yourself as more than a stunning Goddess, but actually a woman who has a voice, opinions, perspective, a philosophy, a way to doing things, teaching them what you like, how you like it, why you like it, already makes them want to be that for you. You hold the attention, you hold the power, so you direct it. And for that, you want them to know you get them and you know what lives within them... that creates the desire for you to be the one exposing it. You instantly build trust. Not because you demanded it, but because you earned it: you showed them you know what you are doing. You have experience, you understand them. They are not told to come see you, they are seduced into it. They desire it. And they will work for it. This will attract better clients (real subs) and instead of you trying to get their attention, they will work to earn yours. If you want to learn more about power dynamics, building a brand as a Pro or the psychology behind BDSM, you can now access all my trainings and classes in one place for a fraction of the cost of The Dominatrix Academy. And you can reinvest the total amount towards the Program. Message me [SECRET] for the details. This offer is not available on my website.

Ms. Malissia

17,098 views • 4 months ago

I am a physicist and I'm profoundly opposed to any idea of non-physical explanations that contradict physics. So that's a no-no and really doesn't make sense. However, there are ways in which both emergent properties such as minds and life and so on have an effect. And as you said, also abstractions. Now the fact that the theory of good explanations led to the idea that abstractions are real things was slightly surprising to me. I wasn't expecting the link, at least wasn't expecting it to be so strong as it is. But the thing is, if you think about how to explain events, physical events like a footprint on the moon, how do you explain how that happened? Well, it happened because of human ideas, of science. And human ideas, you could say in this reductionist sense that as you rightly say is the prevailing mode of explanation and the prevailing idea is to look down on other modes of explanation, that those ideas are nothing more than configurations of atoms. So some physicists, some rocket scientists put their brain into certain configurations of atoms and those atoms then acted on other atoms which then ended up making a footprint on the moon. Now what that misses is the explanation of why certain configurations of atoms put footprints on the moon while others, the overwhelming majority of configurations that human brains, even human brains have been put into in history, do not have that effect. And it's because there's a certain type of information. And this information can't in my view be reduced to statements about atoms because if you think about what that information does, it is in brains but the same information then gets transferred into, let's say, sound waves in air and then it gets transferred into ink on paper and then it gets transferred into magnetic domains inside a computer which then control a machine that instantiates those ideas in bits of steel and silicon and so on and so on. There's an immense chain of instantiations of the same information. And it's only special kinds of information that have this property that they are preserved and instantiated in successive physical modes. So what is being transmitted, what is having the causal effect is not the atoms but the fact that the atoms instantiate certain kinds of information and not other kinds. So therefore it is the information that is having the causal effect. If a particular instantiation of that information were damaged, then processes would come along to fix it, whether or not they could fix the physical instantiation. For example, if the computer goes wrong, then we don't use the corrupted information. We go back and rescue the information from a different computer and we throw away the atoms that at one point instantiated it. So the information causes itself to remain in existence. Now I think there's no way out of that mode of explanation. And if explanation is going to be the fundamental thing about our criterion, for example, about what is or isn't real, then we have to say that information and this particular kind which we call knowledge is real and really does cause things. David Deutsch

I am a physicist and I'm profoundly opposed to any idea of non-physical explanations that contradict physics. So that's a no-no and really doesn't make sense. However, there are ways in which both emergent properties such as minds and life and so on have an effect. And as you said, also abstractions. Now the fact that the theory of good explanations led to the idea that abstractions are real things was slightly surprising to me. I wasn't expecting the link, at least wasn't expecting it to be so strong as it is. But the thing is, if you think about how to explain events, physical events like a footprint on the moon, how do you explain how that happened? Well, it happened because of human ideas, of science. And human ideas, you could say in this reductionist sense that as you rightly say is the prevailing mode of explanation and the prevailing idea is to look down on other modes of explanation, that those ideas are nothing more than configurations of atoms. So some physicists, some rocket scientists put their brain into certain configurations of atoms and those atoms then acted on other atoms which then ended up making a footprint on the moon. Now what that misses is the explanation of why certain configurations of atoms put footprints on the moon while others, the overwhelming majority of configurations that human brains, even human brains have been put into in history, do not have that effect. And it's because there's a certain type of information. And this information can't in my view be reduced to statements about atoms because if you think about what that information does, it is in brains but the same information then gets transferred into, let's say, sound waves in air and then it gets transferred into ink on paper and then it gets transferred into magnetic domains inside a computer which then control a machine that instantiates those ideas in bits of steel and silicon and so on and so on. There's an immense chain of instantiations of the same information. And it's only special kinds of information that have this property that they are preserved and instantiated in successive physical modes. So what is being transmitted, what is having the causal effect is not the atoms but the fact that the atoms instantiate certain kinds of information and not other kinds. So therefore it is the information that is having the causal effect. If a particular instantiation of that information were damaged, then processes would come along to fix it, whether or not they could fix the physical instantiation. For example, if the computer goes wrong, then we don't use the corrupted information. We go back and rescue the information from a different computer and we throw away the atoms that at one point instantiated it. So the information causes itself to remain in existence. Now I think there's no way out of that mode of explanation. And if explanation is going to be the fundamental thing about our criterion, for example, about what is or isn't real, then we have to say that information and this particular kind which we call knowledge is real and really does cause things. David Deutsch

Deutsch Explains

45,735 views • 2 years ago

Elon Musk on learning physics: What matters are not the formulas, but the thinking process that led to the discovery of those formulas. “I just recommend studying physics. What matters is not remembering a bunch of formulas. What matters is the thinking process that led to the discovery of those formulas. It's sort of critical thinking, first principles analysis, trying to understand what is true at a most fundamental level and then reasoning up from there and testing your conclusions against the most fundamental truths in any given arena. This is how you can figure out whether something is more likely to be true or not. It's good to think in terms of probabilities. If you receive information about a subject, that should change the probability of your conclusion, but not the certainty of your conclusion. Physics teaches you that you should not be 100% certain about any given prediction. Now, there's some things that are highly likely, but physics teaches you you got to assign a probability to something being true. And then as you learn more information, your original conclusion may be wrong, then you should then change your mind based on the new information.” Town Hall in Lancaster, Pennsylvania, October 24, 2024

Elon Musk on learning physics: What matters are not the formulas, but the thinking process that led to the discovery of those formulas. “I just recommend studying physics. What matters is not remembering a bunch of formulas. What matters is the thinking process that led to the discovery of those formulas. It's sort of critical thinking, first principles analysis, trying to understand what is true at a most fundamental level and then reasoning up from there and testing your conclusions against the most fundamental truths in any given arena. This is how you can figure out whether something is more likely to be true or not. It's good to think in terms of probabilities. If you receive information about a subject, that should change the probability of your conclusion, but not the certainty of your conclusion. Physics teaches you that you should not be 100% certain about any given prediction. Now, there's some things that are highly likely, but physics teaches you you got to assign a probability to something being true. And then as you learn more information, your original conclusion may be wrong, then you should then change your mind based on the new information.” Town Hall in Lancaster, Pennsylvania, October 24, 2024

ELON CLIPS

67,501 views • 7 months ago

Elon Musk exposes how the deception is going on with the media: “The biggest deception is the choice of narrative. The media can write a story about this or that, but only a few stories can go on the front page So you’re deciding what people should pay attention to. Instead of it being what people actually care about, it’s what a handful of editors care about. They’re telling the people what to pay attention to” This is exactly why Elon built 𝕏 the way he did…..so the truth and important stories bubble up organically from the people, not dictated by a small group of influential editors

Elon Musk exposes how the deception is going on with the media: “The biggest deception is the choice of narrative. The media can write a story about this or that, but only a few stories can go on the front page So you’re deciding what people should pay attention to. Instead of it being what people actually care about, it’s what a handful of editors care about. They’re telling the people what to pay attention to” This is exactly why Elon built 𝕏 the way he did…..so the truth and important stories bubble up organically from the people, not dictated by a small group of influential editors

X Freeze

13,721 views • 6 months ago