Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Zhilin at GTC: Introducing Attention Residuals Learning selective memory, rather than mechanically accumulating everything, is the beauty of attention. Many of you have probably read Attention Is All You Need, the 2017 Transformer paper that brought “human-like” attention into the model’s field of view. From that point on, models... no longer simply read everything in a mechanical way. Instead, they began to develop a sense of what matters more and what matters less across the text, choosing to retain the more important information. Recently, Kimi applied this idea of attention to the temporal dimension, then rotated it 90 degrees into the model’s depth dimension. This allows the model to have attention not only over time, but also throughout the process of information transmission across layers—giving it a more intelligent way to understand and process information.show more

Kimi.ai

170,918 subscribers

114,765 views • 2 months ago •via X (Twitter)

Science & Technology

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

Steve Nash engaged in self-regulated learning early! His father, canny as a fox, pointed out what he did well and encouraged him to bottle the details. Remember this, remember that! Remember this, remember that! Memory - but it was never about remembering the performance outcomes, it was always performance process - the actions and decisions that helped the young Nash to score, not the scoring itself. All detail! All detail! This has been called attentional density (by neuroscientist David Schwartz). When you pay attention to something, your brain fires in patterns akin to that something. The more you think about it, the more the brain fires that way. The more the brain fires that way, the more chance you have of remembering what you’re paying attention to. Pay attention to the details! Pay attention to the details!

Steve Nash engaged in self-regulated learning early! His father, canny as a fox, pointed out what he did well and encouraged him to bottle the details. Remember this, remember that! Remember this, remember that! Memory - but it was never about remembering the performance outcomes, it was always performance process - the actions and decisions that helped the young Nash to score, not the scoring itself. All detail! All detail! This has been called attentional density (by neuroscientist David Schwartz). When you pay attention to something, your brain fires in patterns akin to that something. The more you think about it, the more the brain fires that way. The more the brain fires that way, the more chance you have of remembering what you’re paying attention to. Pay attention to the details! Pay attention to the details!

Daniel Abrahams

50,912 views • 4 months ago

New short course: Attention in Transformers: Concepts and Code in PyTorch. Last week we released a course on how LLM transformers work. This week, go deeper and learn about the technical ideas behind the attention mechanism, and see how to code it in PyTorch. This course is built with Joshua Starmer, Founder and CEO of StatQuest. The attention mechanism was a breakthrough that led to transformers, the architecture powering large language models like ChatGPT. Transformers, introduced in the 2017 paper: "Attention is All You Need" by Viswani and others, took off because of its highly scalable design. In this course, you’ll learn how the attention mechanism, a key element of transformer-based LLMs, works and implement it in PyTorch. You'll develop deep intuition about building reliable, functional, and scalable AI applications. What you will do: - Understand the evolution of the attention mechanism, a key breakthrough that led to transformers. - Learn the relationships between word embeddings, positional embeddings, and attention. - Learn about the Query, Key, and Value matrices, and how to produce and use them in attention. - Walk through the math required to calculate self-attention and masked self-attention to learn why and how they work. - Understand the difference between self-attention and masked self-attention and how one is used in the encoder to build context-aware embeddings and the other is used in the decoder for generative outputs. - Learn the details of the encoder-decoder architecture, cross-attention, and multi-head attention and how they are all incorporated into a transformer. - Use PyTorch to code a class that implements self-attention, masked self-attention, and multi-head attention. There're lots of exciting technical details in this course. Please sign up here:

New short course: Attention in Transformers: Concepts and Code in PyTorch. Last week we released a course on how LLM transformers work. This week, go deeper and learn about the technical ideas behind the attention mechanism, and see how to code it in PyTorch. This course is built with Joshua Starmer, Founder and CEO of StatQuest. The attention mechanism was a breakthrough that led to transformers, the architecture powering large language models like ChatGPT. Transformers, introduced in the 2017 paper: "Attention is All You Need" by Viswani and others, took off because of its highly scalable design. In this course, you’ll learn how the attention mechanism, a key element of transformer-based LLMs, works and implement it in PyTorch. You'll develop deep intuition about building reliable, functional, and scalable AI applications. What you will do: - Understand the evolution of the attention mechanism, a key breakthrough that led to transformers. - Learn the relationships between word embeddings, positional embeddings, and attention. - Learn about the Query, Key, and Value matrices, and how to produce and use them in attention. - Walk through the math required to calculate self-attention and masked self-attention to learn why and how they work. - Understand the difference between self-attention and masked self-attention and how one is used in the encoder to build context-aware embeddings and the other is used in the decoder for generative outputs. - Learn the details of the encoder-decoder architecture, cross-attention, and multi-head attention and how they are all incorporated into a transformer. - Use PyTorch to code a class that implements self-attention, masked self-attention, and multi-head attention. There're lots of exciting technical details in this course. Please sign up here:

Andrew Ng

132,135 views • 1 year ago

6⃣ Main Different Modes of Attention + explanations Can you notice when you are spontaneously in each of these types of attention throughout the week? Most think attention mode 2. is the only way to be concentrated = sustaining engagement with desired phenomena for long periods of time. But actually you can learn to maintain each mode of attention and concentrate in different ways - knowing this radically opens up one’s potential in meditation practice and access to many more states of mind and insights. 🤯 ¡This animation visually illustrates different ways in which attention can behave, but note that these different modes are relevant across the senses (not just with seeing)!

6⃣ Main Different Modes of Attention + explanations Can you notice when you are spontaneously in each of these types of attention throughout the week? Most think attention mode 2. is the only way to be concentrated = sustaining engagement with desired phenomena for long periods of time. But actually you can learn to maintain each mode of attention and concentrate in different ways - knowing this radically opens up one’s potential in meditation practice and access to many more states of mind and insights. 🤯 ¡This animation visually illustrates different ways in which attention can behave, but note that these different modes are relevant across the senses (not just with seeing)!

Roger This

12,742 views • 1 year ago

POSOBIEC: People won’t read a Supreme Court ruling, but they’ll watch a Netflix miniseries on it If you're not paying attention to who is in control of this, then you're not paying attention to who's in control of the minds of the masses

POSOBIEC: People won’t read a Supreme Court ruling, but they’ll watch a Netflix miniseries on it If you're not paying attention to who is in control of this, then you're not paying attention to who's in control of the minds of the masses

Jack Posobiec

310,114 views • 6 months ago

Alex Pereira shares what has frustrated him most following his loss to Ciryl Gane: "Herb Dean came in, and I said, 'Listen, I understand that sometimes in a fight, eye pokes can happen and low blows can happen... but this guy has a history of shots to the back of the head, so please pay attention to those during the fight.' After we spoke with him, he brought Mike Beltran over and said, 'Just so you understand, the back of the head is here.' So it seemed like he absorbed the information. Then, the next day, seeing everything that happened and knowing that we brought it to his attention, he did not do his job."

Alex Pereira shares what has frustrated him most following his loss to Ciryl Gane: "Herb Dean came in, and I said, 'Listen, I understand that sometimes in a fight, eye pokes can happen and low blows can happen... but this guy has a history of shots to the back of the head, so please pay attention to those during the fight.' After we spoke with him, he brought Mike Beltran over and said, 'Just so you understand, the back of the head is here.' So it seemed like he absorbed the information. Then, the next day, seeing everything that happened and knowing that we brought it to his attention, he did not do his job."

Ariel Helwani

469,603 views • 7 hours ago

🇺🇸 Chamath just revealed the truth on Rogan… attention is what runs it all. Google ranks the web by attention. Facebook and Instagram are built to capture attention. The most important AI paper ever written is literally called "Attention Is All You Need." Every system that now runs the world reduces to the same single currency. And at some point, that stops sounding like branding and starts sounding like a warning. Joe Rogan, Chamath Palihapitiya

🇺🇸 Chamath just revealed the truth on Rogan… attention is what runs it all. Google ranks the web by attention. Facebook and Instagram are built to capture attention. The most important AI paper ever written is literally called "Attention Is All You Need." Every system that now runs the world reduces to the same single currency. And at some point, that stops sounding like branding and starts sounding like a warning. Joe Rogan, Chamath Palihapitiya

Mario Nawfal

314,295 views • 1 month ago

$Raskin: This seems to me to be one more pathetic effort to distract and divert attention from the fact that the president's name appeared more than 1,000 times already in the small fraction of material released from Epstein and from the failure of the majority to move to disclose this information.$

Raskin: This seems to me to be one more pathetic effort to distract and divert attention from the fact that the president's name appeared more than 1,000 times already in the small fraction of material released from Epstein and from the failure of the majority to move to disclose this information.

Acyn

26,735 views • 7 months ago

You may change your mental vibrations by an effort of WILL in the direction of deliberately fixing the Attention upon a more desirable state. Will directs the ATTENTION, and Attention changes the Vibration. Cultivate the Art of Attention by means of the Will, and you have solved the secret of the MASTERY of Moods and Mental States ~ Kybalion

You may change your mental vibrations by an effort of WILL in the direction of deliberately fixing the Attention upon a more desirable state. Will directs the ATTENTION, and Attention changes the Vibration. Cultivate the Art of Attention by means of the Will, and you have solved the secret of the MASTERY of Moods and Mental States ~ Kybalion

WhatdoIknow

17,146 views • 9 months ago

Fr. Sam Sawyer: Trump wants to be the center of attention all the time. Pope Leo is pulling our attention away from the president and putting our attention back on God, on Jesus, on the call to peace and the message of the Gospel. Trump seems to have trouble with that.

Fr. Sam Sawyer: Trump wants to be the center of attention all the time. Pope Leo is pulling our attention away from the president and putting our attention back on God, on Jesus, on the call to peace and the message of the Gospel. Trump seems to have trouble with that.

FactPost

35,952 views • 2 months ago

my favorite bit of attention to detail in this scene is that the Pilots and operator speak English, rather than Japanese Extra bit of realism that makes me appreciate it all the more

my favorite bit of attention to detail in this scene is that the Pilots and operator speak English, rather than Japanese Extra bit of realism that makes me appreciate it all the more

MA5K CEO

186,778 views • 10 months ago

I have investigated the situation. EUW is doomed and we are only getting worse, give soloq 5% more of your attention and euw is saved, players arent even able to process basic information anymore

I have investigated the situation. EUW is doomed and we are only getting worse, give soloq 5% more of your attention and euw is saved, players arent even able to process basic information anymore

Veigarv2

199,334 views • 2 years ago

Why is it so challenging for Vice President Kamala Harris to construct a straightforward sentence? READ: "I feel very strongly about the importance as a general matter of engaging in U.S. policy as it relates to foreign affairs in a way that we pay attention, of course, to the immediate concerns and threats if they exist, but that we also pay attention to 10, 20, 30 years down the line and what we are developing now that will be to the benefit of our country then."

Why is it so challenging for Vice President Kamala Harris to construct a straightforward sentence? READ: "I feel very strongly about the importance as a general matter of engaging in U.S. policy as it relates to foreign affairs in a way that we pay attention, of course, to the immediate concerns and threats if they exist, but that we also pay attention to 10, 20, 30 years down the line and what we are developing now that will be to the benefit of our country then."

Simon Ateba

1,757,147 views • 2 years ago

Attention: Members of the clergy listen to Charlie Kirk explain what is happening to the young people across the nation. They no longer want the Christian music with guitars but are seeking traditions. They are tired of the wokeness and liberalism and are seeking feelings of holiness. This is a learning lesson…..

Attention: Members of the clergy listen to Charlie Kirk explain what is happening to the young people across the nation. They no longer want the Christian music with guitars but are seeking traditions. They are tired of the wokeness and liberalism and are seeking feelings of holiness. This is a learning lesson…..

Erica 🇺🇸🇺🇸🇺🇸

32,001 views • 1 year ago

#Colts HC Shane Steichen on QB Anthony Richardson: “I’ve said ‘the process, the process’ for a long time, and what I mean by that is the attention to detail in everything he does. From the classroom to the walkthroughs to practice …” “He’s gonna start the rest of the season.”

#Colts HC Shane Steichen on QB Anthony Richardson: “I’ve said ‘the process, the process’ for a long time, and what I mean by that is the attention to detail in everything he does. From the classroom to the walkthroughs to practice …” “He’s gonna start the rest of the season.”

James Boyd

140,105 views • 1 year ago

$I hear so often from the Dommes I work with that they struggle with people online fetichizing them and simply seeing them for how sexy and beautiful they are. They project their fantasies and their desires onto you. That stops immediately once you move the attention from you to them. From 'look at me' to 'I see you'. What does that look like? When you create content, think of them and what this scene or that narrative is evoking. What will they learn from you? What they want is not to passively watch how sexy you are, but for you to train them, to give them instructions, to teach them, to guide them, to be in charge, to command them. This is not being an object but the main subject. The Authority figure. How is your content already doing that. The sexy photos can still be there, they are important to already capture des attention. But what you do with that attention once you have it, is where the power dynamic is established. Positioning yourself as more than a stunning Goddess, but actually a woman who has a voice, opinions, perspective, a philosophy, a way to doing things, teaching them what you like, how you like it, why you like it, already makes them want to be that for you. You hold the attention, you hold the power, so you direct it. And for that, you want them to know you get them and you know what lives within them... that creates the desire for you to be the one exposing it. You instantly build trust. Not because you demanded it, but because you earned it: you showed them you know what you are doing. You have experience, you understand them. They are not told to come see you, they are seduced into it. They desire it. And they will work for it. This will attract better clients (real subs) and instead of you trying to get their attention, they will work to earn yours. If you want to learn more about power dynamics, building a brand as a Pro or the psychology behind BDSM, you can now access all my trainings and classes in one place for a fraction of the cost of The Dominatrix Academy. And you can reinvest the total amount towards the Program. Message me [SECRET] for the details. This offer is not available on my website.$

I hear so often from the Dommes I work with that they struggle with people online fetichizing them and simply seeing them for how sexy and beautiful they are. They project their fantasies and their desires onto you. That stops immediately once you move the attention from you to them. From 'look at me' to 'I see you'. What does that look like? When you create content, think of them and what this scene or that narrative is evoking. What will they learn from you? What they want is not to passively watch how sexy you are, but for you to train them, to give them instructions, to teach them, to guide them, to be in charge, to command them. This is not being an object but the main subject. The Authority figure. How is your content already doing that. The sexy photos can still be there, they are important to already capture des attention. But what you do with that attention once you have it, is where the power dynamic is established. Positioning yourself as more than a stunning Goddess, but actually a woman who has a voice, opinions, perspective, a philosophy, a way to doing things, teaching them what you like, how you like it, why you like it, already makes them want to be that for you. You hold the attention, you hold the power, so you direct it. And for that, you want them to know you get them and you know what lives within them... that creates the desire for you to be the one exposing it. You instantly build trust. Not because you demanded it, but because you earned it: you showed them you know what you are doing. You have experience, you understand them. They are not told to come see you, they are seduced into it. They desire it. And they will work for it. This will attract better clients (real subs) and instead of you trying to get their attention, they will work to earn yours. If you want to learn more about power dynamics, building a brand as a Pro or the psychology behind BDSM, you can now access all my trainings and classes in one place for a fraction of the cost of The Dominatrix Academy. And you can reinvest the total amount towards the Program. Message me [SECRET] for the details. This offer is not available on my website.

Ms. Malissia

14,297 views • 1 month ago

The inventor of Disagg Prefill, Hao Zhang , drops a new technique called Core Attention Disagg (CAD) which optimizes long context training by disaggregating only the core attention softmax(QK^T)V from the rest of the layers. This is compared to Attention-FFN Disagg (AFD) that attention (including QKV proj & Attention Output Proj) from FFN layers. CAD decouples the layer with quadratic compute growth with respect to seq length from the rest of the layers which only has near linear growth.

The inventor of Disagg Prefill, Hao Zhang , drops a new technique called Core Attention Disagg (CAD) which optimizes long context training by disaggregating only the core attention softmax(QK^T)V from the rest of the layers. This is compared to Attention-FFN Disagg (AFD) that attention (including QKV proj & Attention Output Proj) from FFN layers. CAD decouples the layer with quadratic compute growth with respect to seq length from the rest of the layers which only has near linear growth.

SemiAnalysis

21,896 views • 5 months ago

This isn’t empowerment, it is attention monetized… the more you show, the more you get rewarded… and the more you rely on it, the less you build anything real. Y’all are being taught to use your body more and less of brain… brainwashed into believing that’s the power. Tragic!

This isn’t empowerment, it is attention monetized… the more you show, the more you get rewarded… and the more you rely on it, the less you build anything real. Y’all are being taught to use your body more and less of brain… brainwashed into believing that’s the power. Tragic!

Dr. God Abeg ooo

10,225 views • 1 month ago

🚨RAOUL PAL: MEMECOINS ARE A GAME OF ATTENTION "This entire industry trades on attention, and there's only so many of us in the industry. It's this game of attention, and memes have got it now because everyone believes that they can make the next 100x." Source: Raoul Pal

🚨RAOUL PAL: MEMECOINS ARE A GAME OF ATTENTION "This entire industry trades on attention, and there's only so many of us in the industry. It's this game of attention, and memes have got it now because everyone believes that they can make the next 100x." Source: Raoul Pal

Mario Nawfal

219,942 views • 1 year ago

Elon Musk exposes how the deception is going on with the media: “The biggest deception is the choice of narrative. The media can write a story about this or that, but only a few stories can go on the front page So you’re deciding what people should pay attention to. Instead of it being what people actually care about, it’s what a handful of editors care about. They’re telling the people what to pay attention to” This is exactly why Elon built 𝕏 the way he did…..so the truth and important stories bubble up organically from the people, not dictated by a small group of influential editors

Elon Musk exposes how the deception is going on with the media: “The biggest deception is the choice of narrative. The media can write a story about this or that, but only a few stories can go on the front page So you’re deciding what people should pay attention to. Instead of it being what people actually care about, it’s what a handful of editors care about. They’re telling the people what to pay attention to” This is exactly why Elon built 𝕏 the way he did…..so the truth and important stories bubble up organically from the people, not dictated by a small group of influential editors

X Freeze

13,721 views • 3 months ago