Loading video...

Video Failed to Load

Go Home

Introducing The AI CUDA Engineer: An agentic AI system that automates the production of highly optimized CUDA kernels. The AI CUDA Engineer can produce highly optimized CUDA kernels, reaching 10-100x speedup over common machine learning operations in PyTorch. Our system is also able to produce highly optimized CUDA kernels...

SakanaAILabs's profile picture

Sakana AI

140,570 subscribers

1,160,948 views • 1 year ago •via X (Twitter)

Science & TechnologyEducation
LIVE

Anya Rossi• Live Now

Private livecam show

10 Comments

vittorio's profile picture
vittorio1 year ago

japan bros are back again

Globant's profile picture
Globant1 year ago

🚀 Agentic AI Systems are changing what AI can do by having the power to act independently. Unlike traditional AI, which needs constant human supervision, these systems can operate more autonomously. Learn how this shift is leading to smarter solutions that can transform industries.➡️ #TechTrends2025

ludwig's profile picture
ludwig1 year ago

I’m going to sleep if I wake up to this having 1M+ views I will read the paper tomorrow morning else pls give me a vibe check chat

Bing Xu's profile picture
Bing Xu1 year ago

I quickly take a look of their report on phone, there are a few misleading parts: 1. Torch C++ code is not CUDA kernel, it is calling CUDNN under hood. 2. The highlighted example Conv3D GroupNorm, conv code is not generated at all. The speedup doesn’t make sense if numerical is wrong. 3. It claims wmma can be faster than PyTorch (CUBLAS), is definitely wrong. Probably benchmark error.

main's profile picture
main1 year ago

isn't there clearly something wrong with level_1->15_Matmul_for_lower_triangular_matrices? claimed 152.9x speedup for the kernel on the left over the code on the right. really?

Viraat's profile picture
Viraat1 year ago

Hey - wondering if you all are only working with large enterprises right now. If not, we’d love to chat! This would be extremely useful to us - we’re building low-bit models to run efficiently on Jetsons. Generating optimized CUDA code for these would be a game-changer!

aizk ✡️'s profile picture
aizk ✡️1 year ago

A slow 14 seconds in AI developments

Dan Mac's profile picture
Dan Mac1 year ago

guys seriously I can't take it anymore need to slow down

Kristof's profile picture
Kristof1 year ago

Japan is back

Dr Futuro - e/acc's profile picture
Dr Futuro - e/acc1 year ago

Wow, Japan is back! 🇯🇵

Related Videos

🎉 Stoked to share The AI CUDA Engineer 👷 - our end-to-end approach for automating the design and optimization of CUDA Kernels using agentic systems. Blog 📰: Paper 📜: WebUI 📈: Dataset 💽: Awesome team work done with Aaditya Prasad 🇺🇸, Suuun, Maxence Faldor, Yujin Tang, hardmaru 🤗
3:40

🎉 Stoked to share The AI CUDA Engineer 👷 - our end-to-end approach for automating the design and optimization of CUDA Kernels using agentic systems. Blog 📰: Paper 📜: WebUI 📈: Dataset 💽: Awesome team work done with Aaditya Prasad 🇺🇸, Suuun, Maxence Faldor, Yujin Tang, hardmaru 🤗

Robert Lange

42,174 views • 1 year ago

Part 1: CUDA Graphs - Intro & Basics - how normal CPU ←→ GPU execution works - what a kernel launch is and where CPU launch overhead comes from - why launching many small kernels individually can become a bottleneck - what CUDA Graphs are and how they reduce CPU launch overhead - common benefits of using CUDA Graphs - the two ways to create CUDA Graphs: a. Explicit Graph Construction b. Stream Capture - where these graph creation methods are used - the CUDA Graph lifecycle: capture → instantiate/prepare → replay this is the first ever content that ive recorded and taught, its genuinely so much harder to record than i thought lmao, feedback's are appreciated and hopefully it just gets better from here youtube link shared below
6:26

Part 1: CUDA Graphs - Intro & Basics - how normal CPU ←→ GPU execution works - what a kernel launch is and where CPU launch overhead comes from - why launching many small kernels individually can become a bottleneck - what CUDA Graphs are and how they reduce CPU launch overhead - common benefits of using CUDA Graphs - the two ways to create CUDA Graphs: a. Explicit Graph Construction b. Stream Capture - where these graph creation methods are used - the CUDA Graph lifecycle: capture → instantiate/prepare → replay this is the first ever content that ive recorded and taught, its genuinely so much harder to record than i thought lmao, feedback's are appreciated and hopefully it just gets better from here youtube link shared below

mohit

29,135 views • 2 months ago

AI that responds in under 100ms doesn't happen by accident. Together AI VP of Kernels Dan Fu shares how they use NVIDIA CUDA, TensorRT-LLM, Dynamo and Together ATLAS on NVIDIA Blackwell to power ultra-low-latency inference and long context code generation for Cursor. From building megakernels that fuse an entire model's forward pass into a single launch to enabling real-time voice agents, Together AI is optimizing latency at every layer of the stack.
1:51

AI that responds in under 100ms doesn't happen by accident. Together AI VP of Kernels Dan Fu shares how they use NVIDIA CUDA, TensorRT-LLM, Dynamo and Together ATLAS on NVIDIA Blackwell to power ultra-low-latency inference and long context code generation for Cursor. From building megakernels that fuse an entire model's forward pass into a single launch to enabling real-time voice agents, Together AI is optimizing latency at every layer of the stack.

NVIDIA AI Infrastructure

20,738 views • 3 months ago

Day 14/90 of Inference Engineering I didn't finish the MNIST in CUDA project today, which made me hesitant to post since I didn't want to put out some slop. But in the spirit of being consistent and showing up, here's a concept I've revisited today now that I've started writing CUDA again. This is an animation on the CUDA memory life cycle. Generally speaking, when writing CUDA, memory management goes through the following stages: 1. Allocate memory on the host 2.Initialize / populate the memory 3.Allocate device memory 4.Copy host data to the device 5. Launch the kernel 6. Copy device data back to the host 7. Free the memory - On a brief reflection, I'm starting to write the matmul kernels in CUDA instead of in C. So this in itself is another paradigm shift in thinking. Excited to hone my technique over the coming week. I'll update you guys when I've completed the project; otherwise I'll just make a post like usual on what I've learned during the day!
0:22

Day 14/90 of Inference Engineering I didn't finish the MNIST in CUDA project today, which made me hesitant to post since I didn't want to put out some slop. But in the spirit of being consistent and showing up, here's a concept I've revisited today now that I've started writing CUDA again. This is an animation on the CUDA memory life cycle. Generally speaking, when writing CUDA, memory management goes through the following stages: 1. Allocate memory on the host 2.Initialize / populate the memory 3.Allocate device memory 4.Copy host data to the device 5. Launch the kernel 6. Copy device data back to the host 7. Free the memory - On a brief reflection, I'm starting to write the matmul kernels in CUDA instead of in C. So this in itself is another paradigm shift in thinking. Excited to hone my technique over the coming week. I'll update you guys when I've completed the project; otherwise I'll just make a post like usual on what I've learned during the day!

max fu

42,808 views • 2 months ago

Cerebras just had the biggest IPO of the year. Founder Andrew Feldman says the 3 most important things he had to convince investors of while doing the roadshow were that demand for inference is going to 1,000,000x, the GPU isn't the only way to do compute, and that the CUDA moat is overstated. What he said: "Jensen said some time ago on Brad Gerstner's podcast that the demand for inference will grow by a 1,000,000x, and nobody believed him. And at the same time, you saw Sam Altman displaying real vision and going out and trying to lock up huge amounts of compute, memory, data centers, and power, because he saw it too." "[We tried] to share what that means — what exponential demand means. And that we're still so early, and yet the demand for AI compute is overwhelming." "The other thing is that there are lots of ways to do this. The GPU isn't the only way. You've got TPUs, Trainium, and us. There are lots of different ways to build a solution here." "And finally — the notion that CUDA is this grand lock-in is overplayed. Gemini 3, which is an excellent model, was trained on TPUs with no CUDA. The Anthropic models were trained on Trainium with no CUDA. Some of the best models, some of the most interesting things are being done without CUDA. And that lock-in might be overplayed." $CBRS
1:46

Cerebras just had the biggest IPO of the year. Founder Andrew Feldman says the 3 most important things he had to convince investors of while doing the roadshow were that demand for inference is going to 1,000,000x, the GPU isn't the only way to do compute, and that the CUDA moat is overstated. What he said: "Jensen said some time ago on Brad Gerstner's podcast that the demand for inference will grow by a 1,000,000x, and nobody believed him. And at the same time, you saw Sam Altman displaying real vision and going out and trying to lock up huge amounts of compute, memory, data centers, and power, because he saw it too." "[We tried] to share what that means — what exponential demand means. And that we're still so early, and yet the demand for AI compute is overwhelming." "The other thing is that there are lots of ways to do this. The GPU isn't the only way. You've got TPUs, Trainium, and us. There are lots of different ways to build a solution here." "And finally — the notion that CUDA is this grand lock-in is overplayed. Gemini 3, which is an excellent model, was trained on TPUs with no CUDA. The Anthropic models were trained on Trainium with no CUDA. Some of the best models, some of the most interesting things are being done without CUDA. And that lock-in might be overplayed." $CBRS

TBPN

36,474 views • 4 months ago

We're excited to back Sail Research, the inference platform built for long-horizon, asynchronous agents. The next wave of AI isn't a person typing into a box. It's agents running for minutes or hours, burning orders of magnitude more tokens per task. At that scale, cost and capacity beat raw latency, and teams are already hitting budget caps. Neil Movva and Samir Menon are solving this full-stack, from CUDA kernels to the API. Hear Neil and Aditya Naganath on why inference is being rebuilt for the age of agents.
3:35

We're excited to back Sail Research, the inference platform built for long-horizon, asynchronous agents. The next wave of AI isn't a person typing into a box. It's agents running for minutes or hours, burning orders of magnitude more tokens per task. At that scale, cost and capacity beat raw latency, and teams are already hitting budget caps. Neil Movva and Samir Menon are solving this full-stack, from CUDA kernels to the API. Hear Neil and Aditya Naganath on why inference is being rebuilt for the age of agents.

Kleiner Perkins

17,495 views • 3 months ago

Everyone says learn inference, learn CUDA,learn ML But almost nobody tells you exactly what to learn, how to learn it, or how to know whether you’ve actually learned it. Introducing a free, open-source platform to learn and practice Machine Learning, CUDA, and inference through hands-on coding. Instead of just watching lectures or reading theory, you’ll: • Learn the theory behind each concept • Follow step-by-step implementation examples • Solve coding challenges based on what you learned • Implement algorithms from scratch, from foundational ML to Inference and CUDA kernel • Practice problems in a Codeforces-style environment • Get new Problem of the Day challenges with ratings The goal isn't just to teach you how to write the code. It’s to help you understand what your code is actually doing underneath. We’re building Trentorch because advanced education shouldn’t be locked behind expensive subscriptions. Money shouldn’t be the barrier to learning ML, inference, or CUDA. A free alternative to platforms that charge students heavily every month. signup and start solving problems now
0:35

Everyone says learn inference, learn CUDA,learn ML But almost nobody tells you exactly what to learn, how to learn it, or how to know whether you’ve actually learned it. Introducing a free, open-source platform to learn and practice Machine Learning, CUDA, and inference through hands-on coding. Instead of just watching lectures or reading theory, you’ll: • Learn the theory behind each concept • Follow step-by-step implementation examples • Solve coding challenges based on what you learned • Implement algorithms from scratch, from foundational ML to Inference and CUDA kernel • Practice problems in a Codeforces-style environment • Get new Problem of the Day challenges with ratings The goal isn't just to teach you how to write the code. It’s to help you understand what your code is actually doing underneath. We’re building Trentorch because advanced education shouldn’t be locked behind expensive subscriptions. Money shouldn’t be the barrier to learning ML, inference, or CUDA. A free alternative to platforms that charge students heavily every month. signup and start solving problems now

AADITYANSHA

100,367 views • 25 days ago

Nvidia announces the new RTX Spark, a new platform powered by the NX1 CPU, and shows off Spark laptops running 007 First Light and Forza 6. The CPU has 20 ARM based cores and a Blackwell RTX GPU with 6144 CUDA Cores. This is the same core count as a 5070, but with 128GB of unified LPDDR5X RAM memory sitting in the same package as the CPU and GPU. The entire Nvidia software stack is available, particularly CUDA, vital for AI. Nvidia's new laptops will likely be ideal for running local LLMs be cause the unified memory means you can load models up to 120-180B parameters (quantized). These laptops are expected to ship later this year and could become strong competitors to high-end MacBooks and even Mac Studios for local AI workloads, thanks to CUDA support and unified memory. Price is unannounced.
2:24

Nvidia announces the new RTX Spark, a new platform powered by the NX1 CPU, and shows off Spark laptops running 007 First Light and Forza 6. The CPU has 20 ARM based cores and a Blackwell RTX GPU with 6144 CUDA Cores. This is the same core count as a 5070, but with 128GB of unified LPDDR5X RAM memory sitting in the same package as the CPU and GPU. The entire Nvidia software stack is available, particularly CUDA, vital for AI. Nvidia's new laptops will likely be ideal for running local LLMs be cause the unified memory means you can load models up to 120-180B parameters (quantized). These laptops are expected to ship later this year and could become strong competitors to high-end MacBooks and even Mac Studios for local AI workloads, thanks to CUDA support and unified memory. Price is unannounced.

Grummz

30,573 views • 4 months ago

Chamath: US AI Startups Can Learn A Lot from DeepSeek "This is a case where necessity was the mother of invention." On E213, Chamath Palihapitiya explained how, even if the $6M number is inaccurate, DeepSeek still had some impressive breakthroughs: GRPO > PPO for reinforcement learning "These guys were like, 'Well, how am I going to do this reinforcement learning thing?' They invented a totally different algorithm." "It uses a lot less computer memory and it's highly performant." PTX > CUDA to build the model "And then the second thing that was crazy is everybody is used to building models and compiling through CUDA, which is NVIDIA's proprietary language." "And these guys worked totally around CUDA and they did something called PTX, which goes right to the bare metal." What US AI startups can learn from this: "We, meaning the West, with all the money that we've had, didn't come up with these ideas." "And I think part of why we didn't come up is not that we're not smart enough to do it, but we weren't forced to because the constraints didn't exist." "And so I just wonder how we make sure we learn this principle, meaning, when the AI company wakes up and rolls out of bed and some VC gives them $200M, maybe that's not the right answer for a Series A or a Seed." "And maybe the right answer is $2M so that they do these DeepSeek-like innovations."
0:55

Chamath: US AI Startups Can Learn A Lot from DeepSeek "This is a case where necessity was the mother of invention." On E213, Chamath Palihapitiya explained how, even if the $6M number is inaccurate, DeepSeek still had some impressive breakthroughs: GRPO > PPO for reinforcement learning "These guys were like, 'Well, how am I going to do this reinforcement learning thing?' They invented a totally different algorithm." "It uses a lot less computer memory and it's highly performant." PTX > CUDA to build the model "And then the second thing that was crazy is everybody is used to building models and compiling through CUDA, which is NVIDIA's proprietary language." "And these guys worked totally around CUDA and they did something called PTX, which goes right to the bare metal." What US AI startups can learn from this: "We, meaning the West, with all the money that we've had, didn't come up with these ideas." "And I think part of why we didn't come up is not that we're not smart enough to do it, but we weren't forced to because the constraints didn't exist." "And so I just wonder how we make sure we learn this principle, meaning, when the AI company wakes up and rolls out of bed and some VC gives them $200M, maybe that's not the right answer for a Series A or a Seed." "And maybe the right answer is $2M so that they do these DeepSeek-like innovations."

The All-In Podcast

108,815 views • 1 year ago

The CUDA moat is real, but probably not for long, says CEO of AI infrastructure platform Modal Erik Bernhardsson. He says he's bullish on alternative accelerators over the 2-3 year timeline, even though there's currently zero demand from his customers for TPUs, etc. "The cost today of rewriting your software to run on those stacks is very high... But the cost is going to go down." "You're going to have software that basically lets you take CUDA-compatible stuff and run it on alternative accelerators."
1:33

The CUDA moat is real, but probably not for long, says CEO of AI infrastructure platform Modal Erik Bernhardsson. He says he's bullish on alternative accelerators over the 2-3 year timeline, even though there's currently zero demand from his customers for TPUs, etc. "The cost today of rewriting your software to run on those stacks is very high... But the cost is going to go down." "You're going to have software that basically lets you take CUDA-compatible stuff and run it on alternative accelerators."

TBPN

35,189 views • 4 months ago

PyTorch 2.13 brings FlexAttention to Apple Silicon, cuts peak memory by up to 4× for large-vocabulary models with nn.LinearCrossEntropyLoss, and updates distributed training, compilation, profiling, and on-device inference. Our live Q&A examined CUDA version support, CuTeDSL in TorchInductor, Python 3.15, ExecuTorch, torch.compile, and plans for PyTorch 2.14. Andrey Talman (Meta), albanD (Meta), and Piotr Bialecki (NVIDIA) joined moderator Chris Gottbrath (Gottbrath Technologies). This clip explains how FlexAttention compiles kernels specialized for the requested masking pattern, allowing them to skip unnecessary computation. It also covers speedups of up to approximately 12× over SDPA in tested sparse configurations and a deterministic backward path on CUDA that removes one source of nondeterminism in gradient computation. 🔗 Watch the full PyTorch 2.13 Release Live Q&A:
1:30

PyTorch 2.13 brings FlexAttention to Apple Silicon, cuts peak memory by up to 4× for large-vocabulary models with nn.LinearCrossEntropyLoss, and updates distributed training, compilation, profiling, and on-device inference. Our live Q&A examined CUDA version support, CuTeDSL in TorchInductor, Python 3.15, ExecuTorch, torch.compile, and plans for PyTorch 2.14. Andrey Talman (Meta), albanD (Meta), and Piotr Bialecki (NVIDIA) joined moderator Chris Gottbrath (Gottbrath Technologies). This clip explains how FlexAttention compiles kernels specialized for the requested masking pattern, allowing them to skip unnecessary computation. It also covers speedups of up to approximately 12× over SDPA in tested sparse configurations and a deterministic backward path on CUDA that removes one source of nondeterminism in gradient computation. 🔗 Watch the full PyTorch 2.13 Release Live Q&A:

PyTorch

18,591 views • 2 months ago

Can LLMs invent better ways to train LLMs? At Sakana AI, we’re pioneering AI-driven methods to automate AI research and discovery. We’re excited to release DiscoPOP: a new SOTA preference optimization algorithm that was discovered and written by an LLM! Our method leverages LLMs to propose and implement new preference optimization algorithms. We then train models with those algorithms and evaluate their performance, providing feedback to the LLM. By repeating this process for multiple generations in an evolutionary loop, the LLM discovers many highly-performant and novel preference optimization objectives! Paper: GitHub: Model: We proudly collaborated with the University of Oxford (Foerster Lab for AI Research (now part of BOLD)) and Cambridge University (Mihaela van der Schaar) on this groundbreaking project. Looking ahead, we envision a future where AI-driven research reduces the need for extensive human intervention and computational resources. This will accelerate scientific discoveries and innovation, pushing the boundaries of what AI can achieve.
0:22

Can LLMs invent better ways to train LLMs? At Sakana AI, we’re pioneering AI-driven methods to automate AI research and discovery. We’re excited to release DiscoPOP: a new SOTA preference optimization algorithm that was discovered and written by an LLM! Our method leverages LLMs to propose and implement new preference optimization algorithms. We then train models with those algorithms and evaluate their performance, providing feedback to the LLM. By repeating this process for multiple generations in an evolutionary loop, the LLM discovers many highly-performant and novel preference optimization objectives! Paper: GitHub: Model: We proudly collaborated with the University of Oxford (Foerster Lab for AI Research (now part of BOLD)) and Cambridge University (Mihaela van der Schaar) on this groundbreaking project. Looking ahead, we envision a future where AI-driven research reduces the need for extensive human intervention and computational resources. This will accelerate scientific discoveries and innovation, pushing the boundaries of what AI can achieve.

Sakana AI

556,037 views • 2 years ago

World Labs CEO Dr. Fei-Fei Li says AI must change how we teach and evaluate students: "AI must change learning. AI must change K-16 learning." "The most precious resource of our entire world is human capital." "When we have gotten the technology that can answer standardized tests... when AI can do better than an average human, it's not about humans are bad. It's about we need to change the education system." "We need to change how we evaluate. We need to change the way we empower teachers to educate the next generation of students where they can use these tools, be empowered, and do things that we can never imagine." "All of the kids today should not be scared of AI. They should feel the human agency to lead AI, to use AI in the right way, and to use AI to make the impact that they want to make for the world." Fei-Fei Li at Bloomberg Tech live with Emily Chang
1:35

World Labs CEO Dr. Fei-Fei Li says AI must change how we teach and evaluate students: "AI must change learning. AI must change K-16 learning." "The most precious resource of our entire world is human capital." "When we have gotten the technology that can answer standardized tests... when AI can do better than an average human, it's not about humans are bad. It's about we need to change the education system." "We need to change how we evaluate. We need to change the way we empower teachers to educate the next generation of students where they can use these tools, be empowered, and do things that we can never imagine." "All of the kids today should not be scared of AI. They should feel the human agency to lead AI, to use AI in the right way, and to use AI to make the impact that they want to make for the world." Fei-Fei Li at Bloomberg Tech live with Emily Chang

a16z

82,018 views • 4 months ago

We gave our AI system a spec, and in under 2 weeks, it designed, verified, and deployed a chip that beats NVIDIA. It’s built for low-power physical AI workloads. We’re running live inference on >B+ parameter models like Llama, Qwen, and Kimi, serving at 3.4x better perf/watt than NVIDIA Jetson. From just a specification, our AI system autonomously generated all of the RTL Design, UVM verification, formal proofs, firmware, drivers, and kernels, co-designing the model, software and silicon as one optimization loop. Better AI can now design better chips to run AI, leading to a loop of recursive-self improvement towards our path to abundant intelligence.
1:04

We gave our AI system a spec, and in under 2 weeks, it designed, verified, and deployed a chip that beats NVIDIA. It’s built for low-power physical AI workloads. We’re running live inference on >B+ parameter models like Llama, Qwen, and Kimi, serving at 3.4x better perf/watt than NVIDIA Jetson. From just a specification, our AI system autonomously generated all of the RTL Design, UVM verification, formal proofs, firmware, drivers, and kernels, co-designing the model, software and silicon as one optimization loop. Better AI can now design better chips to run AI, leading to a loop of recursive-self improvement towards our path to abundant intelligence.

Architect Labs

674,470 views • 1 month ago

Ben Thompson explains how LLMs greatly diminished Nvidia's CUDA moat even as they sent the stock to the moon "So the weird thing about large language models is they were obviously incredible for Nvidia. That's why their stock went to the moon." "They have been on and off the most valuable company in the world." "It was also very bad for Nvidia. And the reason it was bad for Nvidia is that the play with CUDA is to build a developer ecosystem on top of CUDA." "But CUDA only works on Nvidia GPUs. So you get CUDA for free. It's easier to use, and it's a tremendous investment. Nvidia almost went under trying to build CUDA at a time when no one understood what they were doing or why they were wasting money on it." "And that's why Jensen Huang will get bristly, particularly when people question their rent-seeking or profit, whatever. It's like, no, they earned their spot fair and square." "Absolutely. It shouldn't be forgotten. They have earned every dollar they've gotten through 25 years of taking massive risks." "It bottomed out in October 2022. I wrote an article like three weeks before ChatGPT came out, tracing their bottoming-out history and their search for what was next." "'Nvidia in the Valley.' So, go back to this GTC. So I wrote an article at the time called 'Nvidia Waves and Moats'." "And what was interesting about that GTC was, number one, it was very boring. All the cool stuff kind of got scrubbed out." "Now, Jensen Huang has brought that stuff back, so the last few GTCs he's more talking about other things. Now it comes across as, oh, you're still looking for something beyond the LLM." "Because the problem with the LLM is it shifts the developer platform far above where Nvidia sits. All the activity is happening on top of LLMs. And so no one who's writing an AI application today is using CUDA." "Now, some people are, if you're training your own model and you're doing some low-level things or non-LLM things." "But the vast majority of the energy and all the money and the ecosystem is far removed from CUDA." "They have no idea and don't need to know or care what chips their application is running on. They're just on the OpenAI API, or the Anthropic API, or using Bedrock on Amazon, and it's sitting on Trainium, and they're using a Chinese open-source model. It's totally abstracted away, and this is why LLMs were bad for Nvidia." "Now, again, all the money they made along the way is worth it, but their moat has been tremendously diminished." "CUDA is still a moat if you need to do stuff that requires CUDA. But the vast majority of stuff, in energy, doesn't require CUDA, like in a post-LLM world."
3:04

Ben Thompson explains how LLMs greatly diminished Nvidia's CUDA moat even as they sent the stock to the moon "So the weird thing about large language models is they were obviously incredible for Nvidia. That's why their stock went to the moon." "They have been on and off the most valuable company in the world." "It was also very bad for Nvidia. And the reason it was bad for Nvidia is that the play with CUDA is to build a developer ecosystem on top of CUDA." "But CUDA only works on Nvidia GPUs. So you get CUDA for free. It's easier to use, and it's a tremendous investment. Nvidia almost went under trying to build CUDA at a time when no one understood what they were doing or why they were wasting money on it." "And that's why Jensen Huang will get bristly, particularly when people question their rent-seeking or profit, whatever. It's like, no, they earned their spot fair and square." "Absolutely. It shouldn't be forgotten. They have earned every dollar they've gotten through 25 years of taking massive risks." "It bottomed out in October 2022. I wrote an article like three weeks before ChatGPT came out, tracing their bottoming-out history and their search for what was next." "'Nvidia in the Valley.' So, go back to this GTC. So I wrote an article at the time called 'Nvidia Waves and Moats'." "And what was interesting about that GTC was, number one, it was very boring. All the cool stuff kind of got scrubbed out." "Now, Jensen Huang has brought that stuff back, so the last few GTCs he's more talking about other things. Now it comes across as, oh, you're still looking for something beyond the LLM." "Because the problem with the LLM is it shifts the developer platform far above where Nvidia sits. All the activity is happening on top of LLMs. And so no one who's writing an AI application today is using CUDA." "Now, some people are, if you're training your own model and you're doing some low-level things or non-LLM things." "But the vast majority of the energy and all the money and the ecosystem is far removed from CUDA." "They have no idea and don't need to know or care what chips their application is running on. They're just on the OpenAI API, or the Anthropic API, or using Bedrock on Amazon, and it's sitting on Trainium, and they're using a Chinese open-source model. It's totally abstracted away, and this is why LLMs were bad for Nvidia." "Now, again, all the money they made along the way is worth it, but their moat has been tremendously diminished." "CUDA is still a moat if you need to do stuff that requires CUDA. But the vast majority of stuff, in energy, doesn't require CUDA, like in a post-LLM world."

Fireside Alpha

12,980 views • 1 month ago

Modal engineer who teaches Full-Stack Deep Learning explained everything an AI engineer needs to know about GPUs in 20 minutes - better than $800 CUDA workshops. memory bandwidth -> SM occupancy -> kernel fusion -> arithmetic intensity -> know when you're compute-bound vs memory-bound. That loop is how you stop overpaying for GPUs you're barely using. VRAM math + roofline model + Modal serverless GPUs + kernel profiling - that's the stack. Watch and save it, then check your own GPU utilization tonight.
19:25

Modal engineer who teaches Full-Stack Deep Learning explained everything an AI engineer needs to know about GPUs in 20 minutes - better than $800 CUDA workshops. memory bandwidth -> SM occupancy -> kernel fusion -> arithmetic intensity -> know when you're compute-bound vs memory-bound. That loop is how you stop overpaying for GPUs you're barely using. VRAM math + roofline model + Modal serverless GPUs + kernel profiling - that's the stack. Watch and save it, then check your own GPU utilization tonight.

h100envy

18,442 views • 3 months ago

Ex-vLLM core contributor explained how to make LLM inference 10x cheaper in 34 minutes - better than $3000 inference optimization bootcamps. request comes in -> check LMCache -> hit? load KV cache from CPU/SSD/remote -> skip prefill -> serve. That loop is why Bloomberg and other production stacks now push 300 terabytes of KV cache per week. LMCache + vLLM + CPU/SSD/remote storage + zero-copy CUDA kernels - that's the stack. Watch and save it, then wire the KV-offload into your inference stack.
34:46

Ex-vLLM core contributor explained how to make LLM inference 10x cheaper in 34 minutes - better than $3000 inference optimization bootcamps. request comes in -> check LMCache -> hit? load KV cache from CPU/SSD/remote -> skip prefill -> serve. That loop is why Bloomberg and other production stacks now push 300 terabytes of KV cache per week. LMCache + vLLM + CPU/SSD/remote storage + zero-copy CUDA kernels - that's the stack. Watch and save it, then wire the KV-offload into your inference stack.

h100envy

61,470 views • 2 months ago