Загрузка видео...

Не удалось загрузить видео

На главную

FlexAttention is a novel compiler-driven programming model that allows implementing the majority of attention variants in a few lines of idiomatic PyTorch code Boyuan Feng & Avik Chaudhuri show how many existing attention variants can be implemented via FlexAttention & that we achieve competitive performance compared to handwritten kernels....

PyTorch's profile picture

PyTorch

497,910 subscribers

17,929 просмотров • 1 год назад •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 3

Фото профиля Lakshya
Lakshya1 год назад

@Boyuan_Feng @__avik How does it compare to MagiAttention

Фото профиля Jilong | We provide AI marketer - 24/7 marketing
Jilong | We provide AI marketer - 24/7 marketing1 год назад

@Boyuan_Feng @__avik FlexAttention sounds promising! Compilers beat brute force coding.

Фото профиля Vinayak N Baddi
Vinayak N Baddi1 год назад

@Boyuan_Feng @__avik Possible to share the slides @__avik Thanks

Похожие видео

Our next PyTorch Compiler Series video is live with Michael Lazos and Avik Chaudhuri discussing the key concepts behind CUTLASS and how to utilize torch.compile to customize primitives for a specific workload via autotuning & epilogue fusion 📺 Watch on YouTube:
1:08

Our next PyTorch Compiler Series video is live with Michael Lazos and Avik Chaudhuri discussing the key concepts behind CUTLASS and how to utilize torch.compile to customize primitives for a specific workload via autotuning & epilogue fusion 📺 Watch on YouTube:

PyTorch

12,886 просмотров • 1 год назад

Missed today’s live Q&A on PyTorch 2.6? The full recording is now available on our LinkedIn & YouTube. PyTorch Core Maintainer Nikita Shulga (Meta) answered key questions about the latest release, sharing his insights on CUDA, torch.compile, security improvements and more. 📺 Watch the full session here: In this clip, Nikita breaks down a crucial topic: 👉 What are the challenges of benchmarking to avoid performance regressions in PyTorch releases? #pytorch #machinelearning #opensource #ai
1:08

Missed today’s live Q&A on PyTorch 2.6? The full recording is now available on our LinkedIn & YouTube. PyTorch Core Maintainer Nikita Shulga (Meta) answered key questions about the latest release, sharing his insights on CUDA, torch.compile, security improvements and more. 📺 Watch the full session here: In this clip, Nikita breaks down a crucial topic: 👉 What are the challenges of benchmarking to avoid performance regressions in PyTorch releases? #pytorch #machinelearning #opensource #ai

PyTorch

11,052 просмотров • 1 год назад

‼️Multiple S'pht Compiler variants in Marathon’s Cryo Archive map? 👀 There's been some speculation that the upcoming Cryo Archive map could feature multiple S'pht Compiler variants, based on the new camera feed footage released via the ongoing ARG. In the "Control" feed, there appears to be something moving under the bridge in the top-right part of the video, and the movement looks very similar to a Compiler.
0:28

‼️Multiple S'pht Compiler variants in Marathon’s Cryo Archive map? 👀 There's been some speculation that the upcoming Cryo Archive map could feature multiple S'pht Compiler variants, based on the new camera feed footage released via the ongoing ARG. In the "Control" feed, there appears to be something moving under the bridge in the top-right part of the video, and the movement looks very similar to a Compiler.

Marathon Bulletin

48,010 просмотров • 4 месяцев назад

New short course: Attention in Transformers: Concepts and Code in PyTorch. Last week we released a course on how LLM transformers work. This week, go deeper and learn about the technical ideas behind the attention mechanism, and see how to code it in PyTorch. This course is built with Joshua Starmer, Founder and CEO of StatQuest. The attention mechanism was a breakthrough that led to transformers, the architecture powering large language models like ChatGPT. Transformers, introduced in the 2017 paper: "Attention is All You Need" by Viswani and others, took off because of its highly scalable design. In this course, you’ll learn how the attention mechanism, a key element of transformer-based LLMs, works and implement it in PyTorch. You'll develop deep intuition about building reliable, functional, and scalable AI applications. What you will do: - Understand the evolution of the attention mechanism, a key breakthrough that led to transformers. - Learn the relationships between word embeddings, positional embeddings, and attention. - Learn about the Query, Key, and Value matrices, and how to produce and use them in attention. - Walk through the math required to calculate self-attention and masked self-attention to learn why and how they work. - Understand the difference between self-attention and masked self-attention and how one is used in the encoder to build context-aware embeddings and the other is used in the decoder for generative outputs. - Learn the details of the encoder-decoder architecture, cross-attention, and multi-head attention and how they are all incorporated into a transformer. - Use PyTorch to code a class that implements self-attention, masked self-attention, and multi-head attention. There're lots of exciting technical details in this course. Please sign up here:
2:41

New short course: Attention in Transformers: Concepts and Code in PyTorch. Last week we released a course on how LLM transformers work. This week, go deeper and learn about the technical ideas behind the attention mechanism, and see how to code it in PyTorch. This course is built with Joshua Starmer, Founder and CEO of StatQuest. The attention mechanism was a breakthrough that led to transformers, the architecture powering large language models like ChatGPT. Transformers, introduced in the 2017 paper: "Attention is All You Need" by Viswani and others, took off because of its highly scalable design. In this course, you’ll learn how the attention mechanism, a key element of transformer-based LLMs, works and implement it in PyTorch. You'll develop deep intuition about building reliable, functional, and scalable AI applications. What you will do: - Understand the evolution of the attention mechanism, a key breakthrough that led to transformers. - Learn the relationships between word embeddings, positional embeddings, and attention. - Learn about the Query, Key, and Value matrices, and how to produce and use them in attention. - Walk through the math required to calculate self-attention and masked self-attention to learn why and how they work. - Understand the difference between self-attention and masked self-attention and how one is used in the encoder to build context-aware embeddings and the other is used in the decoder for generative outputs. - Learn the details of the encoder-decoder architecture, cross-attention, and multi-head attention and how they are all incorporated into a transformer. - Use PyTorch to code a class that implements self-attention, masked self-attention, and multi-head attention. There're lots of exciting technical details in this course. Please sign up here:

Andrew Ng

132,220 просмотров • 1 год назад

Introducing The AI CUDA Engineer: An agentic AI system that automates the production of highly optimized CUDA kernels. The AI CUDA Engineer can produce highly optimized CUDA kernels, reaching 10-100x speedup over common machine learning operations in PyTorch. Our system is also able to produce highly optimized CUDA kernels that are much faster than existing CUDA kernels commonly used in production. We believe that fundamentally, AI systems can and should be as resource-efficient as the human brain, and that the best path to achieve this efficiency is to use AI to make AI more efficient! We are excited to publish our paper, The AI CUDA Engineer: Agentic CUDA Kernel Discovery, Optimization and Composition. We also release a dataset of over 17,000 verified CUDA kernels produced by The AI CUDA Engineer. Paper: Kernel Archive Webpage: HuggingFace Dataset: The AI CUDA Engineer utilizes evolutionary LLM-driven code optimization to autonomously improve the runtime of machine learning operations. Our system is not only able to convert PyTorch code into CUDA kernels, but through the use of evolution, it can also optimize the runtime performance of CUDA kernels, fuse multiple operations, and even discover novel solutions for writing efficient CUDA operations by learning from past innovations! We believe The AI CUDA Engineer opens a new era of AI-driven acceleration of AI and automated inference time optimization. We (Robert Lange, Aaditya Prasad 🇺🇸, sssss, Maxence Faldor, Yujin Tang, hardmaru) are excited to continue Sakana AI's mission of leveraging AI to improve AI.
1:21

Introducing The AI CUDA Engineer: An agentic AI system that automates the production of highly optimized CUDA kernels. The AI CUDA Engineer can produce highly optimized CUDA kernels, reaching 10-100x speedup over common machine learning operations in PyTorch. Our system is also able to produce highly optimized CUDA kernels that are much faster than existing CUDA kernels commonly used in production. We believe that fundamentally, AI systems can and should be as resource-efficient as the human brain, and that the best path to achieve this efficiency is to use AI to make AI more efficient! We are excited to publish our paper, The AI CUDA Engineer: Agentic CUDA Kernel Discovery, Optimization and Composition. We also release a dataset of over 17,000 verified CUDA kernels produced by The AI CUDA Engineer. Paper: Kernel Archive Webpage: HuggingFace Dataset: The AI CUDA Engineer utilizes evolutionary LLM-driven code optimization to autonomously improve the runtime of machine learning operations. Our system is not only able to convert PyTorch code into CUDA kernels, but through the use of evolution, it can also optimize the runtime performance of CUDA kernels, fuse multiple operations, and even discover novel solutions for writing efficient CUDA operations by learning from past innovations! We believe The AI CUDA Engineer opens a new era of AI-driven acceleration of AI and automated inference time optimization. We (Robert Lange, Aaditya Prasad 🇺🇸, sssss, Maxence Faldor, Yujin Tang, hardmaru) are excited to continue Sakana AI's mission of leveraging AI to improve AI.

Sakana AI

1,159,053 просмотров • 1 год назад

underrated gems in Kimi-K3 release: > an early K3 wrote the majority of the kernels in the late development stages > it built a triton-class compiler from scratch, MiniTriton, that delivers performance on par with or better than Triton and torch.compile > then it designed a chip, by a model, for a model, in one 48-hour autonomous run the model is rewriting and optimizing every layer of the stack it runs on: kernels for its own training. A compiler for its own kernels. silicon for its own weights. we're watching software build its own hardware
0:22

underrated gems in Kimi-K3 release: > an early K3 wrote the majority of the kernels in the late development stages > it built a triton-class compiler from scratch, MiniTriton, that delivers performance on par with or better than Triton and torch.compile > then it designed a chip, by a model, for a model, in one 48-hour autonomous run the model is rewriting and optimizing every layer of the stack it runs on: kernels for its own training. A compiler for its own kernels. silicon for its own weights. we're watching software build its own hardware

Alex Ker 🔭

303,792 просмотров • 6 дней назад

TOMORROW 🎉 Join us for the virtual premiere of the PyTorch Documentary! We'll be live on YouTube at 10am PT on June 25 along with some of our key PyTorch collaborators from AMD, AWS, Google, Meta and Microsoft 💜 🔗
0:40

TOMORROW 🎉 Join us for the virtual premiere of the PyTorch Documentary! We'll be live on YouTube at 10am PT on June 25 along with some of our key PyTorch collaborators from AMD, AWS, Google, Meta and Microsoft 💜 🔗

PyTorch

48,761 просмотров • 2 лет назад

A few weeks ago we posted a clip of how to apply the left hand to the club. Today we show you again, this time without a glove! We also focus on the right hand and this is something you should pay special attention to as many recreational golfers really struggle with this…
0:55

A few weeks ago we posted a clip of how to apply the left hand to the club. Today we show you again, this time without a glove! We also focus on the right hand and this is something you should pay special attention to as many recreational golfers really struggle with this…

Steve Gould

65,254 просмотров • 3 месяцев назад

“Did you know that whilst we’re all being distracted, we are getting blasted, on a daily basis, all across America, and the world” What used to be a fringe conspiracy only a few years ago, is now attracting the attention of the entire World. Will we get full disclosure in 2026?
1:12

“Did you know that whilst we’re all being distracted, we are getting blasted, on a daily basis, all across America, and the world” What used to be a fringe conspiracy only a few years ago, is now attracting the attention of the entire World. Will we get full disclosure in 2026?

Concerned Citizen

55,694 просмотров • 6 месяцев назад

Zhilin at GTC: Introducing Attention Residuals Learning selective memory, rather than mechanically accumulating everything, is the beauty of attention. Many of you have probably read Attention Is All You Need, the 2017 Transformer paper that brought “human-like” attention into the model’s field of view. From that point on, models no longer simply read everything in a mechanical way. Instead, they began to develop a sense of what matters more and what matters less across the text, choosing to retain the more important information. Recently, Kimi applied this idea of attention to the temporal dimension, then rotated it 90 degrees into the model’s depth dimension. This allows the model to have attention not only over time, but also throughout the process of information transmission across layers—giving it a more intelligent way to understand and process information.
5:36

Zhilin at GTC: Introducing Attention Residuals Learning selective memory, rather than mechanically accumulating everything, is the beauty of attention. Many of you have probably read Attention Is All You Need, the 2017 Transformer paper that brought “human-like” attention into the model’s field of view. From that point on, models no longer simply read everything in a mechanical way. Instead, they began to develop a sense of what matters more and what matters less across the text, choosing to retain the more important information. Recently, Kimi applied this idea of attention to the temporal dimension, then rotated it 90 degrees into the model’s depth dimension. This allows the model to have attention not only over time, but also throughout the process of information transmission across layers—giving it a more intelligent way to understand and process information.

Kimi.ai

115,162 просмотров • 3 месяцев назад

Cursor CEO Michael Truell on the future of writing code: "Our goal with Cursor is to invent a new type of programming." "It looks like a world where you have a representation of the logic of your software that does look more like English." "You can imagine kind of an evolution of programming language towards pseudocode. You have written down the logic of the software, and you can edit that at a high level." "It won't be the impenetrable millions of lines of code, it'll instead be something that's much terser and easier to understand and easier to navigate." Michael Truell with Lenny Rachitsky on Lenny's Podcast
2:50

Cursor CEO Michael Truell on the future of writing code: "Our goal with Cursor is to invent a new type of programming." "It looks like a world where you have a representation of the logic of your software that does look more like English." "You can imagine kind of an evolution of programming language towards pseudocode. You have written down the logic of the software, and you can edit that at a high level." "It won't be the impenetrable millions of lines of code, it'll instead be something that's much terser and easier to understand and easier to navigate." Michael Truell with Lenny Rachitsky on Lenny's Podcast

a16z

1,014,223 просмотров • 1 месяц назад

Cursor CEO Michael Truell on the future of writing code: “ Our goal with Cursor is to invent a new type of programming.” “It looks like a world where you have a representation of the logic of your software that does look more like English.” “You can imagine kind of an evolution of programming language towards pseudocode. You have written down the logic of the software, and you can edit that at a high level.” “It won't be the impenetrable millions of lines of code, it'll instead be something that's much terser and easier to understand and easier to navigate.” Source: Michael Truell (CEO Cursor) with Lenny Rachitsky on Lenny's Podcast
2:50

Cursor CEO Michael Truell on the future of writing code: “ Our goal with Cursor is to invent a new type of programming.” “It looks like a world where you have a representation of the logic of your software that does look more like English.” “You can imagine kind of an evolution of programming language towards pseudocode. You have written down the logic of the software, and you can edit that at a high level.” “It won't be the impenetrable millions of lines of code, it'll instead be something that's much terser and easier to understand and easier to navigate.” Source: Michael Truell (CEO Cursor) with Lenny Rachitsky on Lenny's Podcast

a16z

812,698 просмотров • 8 месяцев назад

(1/5) FP4 hardware is here, but 4-bit attention still kills model quality, blocking true end-to-end FP4 serving. To fix that, we propose Attn-QAT, the first systematic study of quantization-aware training for attention. The result: FP4 attention quality is comparable to BF16 attention with 1.1x–1.5x higher throughput than SageAttention3 on an RTX 5090 and 1.39x speedup over FlashAttention-4 on a B200. Blog: Code: Checkpoints:
0:25

(1/5) FP4 hardware is here, but 4-bit attention still kills model quality, blocking true end-to-end FP4 serving. To fix that, we propose Attn-QAT, the first systematic study of quantization-aware training for attention. The result: FP4 attention quality is comparable to BF16 attention with 1.1x–1.5x higher throughput than SageAttention3 on an RTX 5090 and 1.39x speedup over FlashAttention-4 on a B200. Blog: Code: Checkpoints:

Hao AI Lab

37,512 просмотров • 3 месяцев назад

We made a video many asked for: a dive into the so-called "Western Azerbaijan Community," an AZ gov-tentacle that claims all of Armenia is AZ. Their hardcore propaganda campaign even includes a cooking show. Please watch the full doc on YouTube:
0:55

We made a video many asked for: a dive into the so-called "Western Azerbaijan Community," an AZ gov-tentacle that claims all of Armenia is AZ. Their hardcore propaganda campaign even includes a cooking show. Please watch the full doc on YouTube:

Lindsey Snell

91,290 просмотров • 1 год назад

"The heedlessness of the people of Noah's day will be duplicated in the last days of our world's history. It will be a day much like ours, a day when ideologies like socialism can sneak in without much attention." . Watch the full sermon now on our YouTube channel! . Is Socialism a Sign of the End Times? | Dr. David Jeremiah . #prophecy #endtimes
1:21

"The heedlessness of the people of Noah's day will be duplicated in the last days of our world's history. It will be a day much like ours, a day when ideologies like socialism can sneak in without much attention." . Watch the full sermon now on our YouTube channel! . Is Socialism a Sign of the End Times? | Dr. David Jeremiah . #prophecy #endtimes

David Jeremiah

12,114 просмотров • 4 месяцев назад

I appeared on the Steve Deace Show recently to talk about a number of topics. One of those topics was that many Republicans need to show courage and actually act like Conservatives. We need to follow the model that Ron DeSantis in Florida has implemented. Don’t worry about making Democrats happy, do what’s right and do it boldly!
1:31

I appeared on the Steve Deace Show recently to talk about a number of topics. One of those topics was that many Republicans need to show courage and actually act like Conservatives. We need to follow the model that Ron DeSantis in Florida has implemented. Don’t worry about making Democrats happy, do what’s right and do it boldly!

Lt. Governor Micah Beckwith

12,318 просмотров • 1 год назад

Carney: "In a time of rising populism and ethnic nationalism, Canada can show how diversity is a strength... Canada can't solve all the world's problems. But we can show that another way is possible. That the arc of history isn't destined to be warped toward authoritarianism".
1:06

Carney: "In a time of rising populism and ethnic nationalism, Canada can show how diversity is a strength... Canada can't solve all the world's problems. But we can show that another way is possible. That the arc of history isn't destined to be warped toward authoritarianism".

Adam Schwarz

116,602 просмотров • 6 месяцев назад

Obama on Trump's antics, including posting a video depicting him and Michelle as apes: "It's important to recognize that the majority of the American people find this behavior deeply troubling. It is true that it gets attention. It's true that it's a distraction ... people still believe in decency"
4:05

Obama on Trump's antics, including posting a video depicting him and Michelle as apes: "It's important to recognize that the majority of the American people find this behavior deeply troubling. It is true that it gets attention. It's true that it's a distraction ... people still believe in decency"

Aaron Rupar

45,475 просмотров • 5 месяцев назад