Загрузка видео...

Не удалось загрузить видео

На главную

Understanding GPUs will change your AI Engineering journey beyond agents and LLM. Rise of LLMs and deep learning introduced dedicated GPU hardware which serves your models for training and inference. You don't need to start with C++, CUDA, and Maths, but: > Components of a GPU and roles >...

kmeanskaran's profile picture

Karan🧋

15,037 subscribers

33,453 просмотров • 1 месяц назад •via X (Twitter)

Наука и технологииОбразование
LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 2

Фото профиля Anderson
Anderson1 месяц назад

everyone will skip to the benchmarks anyway

Фото профиля Lewu
Lewu1 месяц назад

Would you include a worked example that budgets weights, KV cache, and runtime overhead separately? Comparing the same model at different context lengths and concurrency levels would make the memory discussion much more concrete.

Похожие видео

🚨 YOUR GPU IS PROBABLY WASTING MORE THAN YOU THINK. vLLM is built to squeeze far more useful work out of your GPU when serving LLMs. Running an LLM at scale isn’t just about having a powerful GPU. The real problem is how efficiently you use its memory and compute. That’s where vLLM comes in. → High-throughput LLM inference and serving → PagedAttention for smarter KV-cache memory management → Continuous batching to keep GPUs busy → Prefix caching + chunked prefill → OpenAI-compatible API out of the box → Supports a huge range of modern LLM architectures → Quantization support for running models more efficiently And the crazy part? You can start an OpenAI-compatible inference server with: `vllm serve ` So your application can talk to your own model almost like it’s talking to OpenAI. The bigger idea: Don’t just buy more GPUs. Make the GPUs you already have work harder. That’s why vLLM has become such a major project in LLM inference. 🔥 #vLLM #AI #LLM #Inference #GPU #MachineLearning #AIInfrastructure #OpenSource #AIAgents #Developers
0:20

🚨 YOUR GPU IS PROBABLY WASTING MORE THAN YOU THINK. vLLM is built to squeeze far more useful work out of your GPU when serving LLMs. Running an LLM at scale isn’t just about having a powerful GPU. The real problem is how efficiently you use its memory and compute. That’s where vLLM comes in. → High-throughput LLM inference and serving → PagedAttention for smarter KV-cache memory management → Continuous batching to keep GPUs busy → Prefix caching + chunked prefill → OpenAI-compatible API out of the box → Supports a huge range of modern LLM architectures → Quantization support for running models more efficiently And the crazy part? You can start an OpenAI-compatible inference server with: `vllm serve ` So your application can talk to your own model almost like it’s talking to OpenAI. The bigger idea: Don’t just buy more GPUs. Make the GPUs you already have work harder. That’s why vLLM has become such a major project in LLM inference. 🔥 #vLLM #AI #LLM #Inference #GPU #MachineLearning #AIInfrastructure #OpenSource #AIAgents #Developers

Vikas gupta

14,138 просмотров • 1 месяц назад

AI inference is already live on UOMI Router. Access frontier open-source models through a decentralized inference network with: • Low-cost inference • OpenAI-compatible APIs • Verifiable compute • On-chain settlement Start building: And if you have idle GPUs, there's now a second way to participate. The UOMI Provider whitelist is officially open. Instead of letting your hardware sit unused, you can contribute GPU power to the UOMI Inference Network and earn from real AI demand. Join the Provider whitelist:
0:34

AI inference is already live on UOMI Router. Access frontier open-source models through a decentralized inference network with: • Low-cost inference • OpenAI-compatible APIs • Verifiable compute • On-chain settlement Start building: And if you have idle GPUs, there's now a second way to participate. The UOMI Provider whitelist is officially open. Instead of letting your hardware sit unused, you can contribute GPU power to the UOMI Inference Network and earn from real AI demand. Join the Provider whitelist:

Uomi

19,009 просмотров • 4 месяцев назад

Fast matrix multiplication on GPUs has traditionally meant wrestling with threads, shared memory, and low-level hardware details. This webinar explores how NVIDIA’s CUDA Tile model—and its Julia port, cuTile.jl—makes high-performance GPU programming more accessible. Join Dr. Andy Terrel of NVIDIA and Dr. Tim Besard of JuliaHub to see real examples across linear algebra, AI inference, and HPC. Register here - #JuliaLang #GPUProgramming #CUDA #HPC #AIInfrastructure
1:36

Fast matrix multiplication on GPUs has traditionally meant wrestling with threads, shared memory, and low-level hardware details. This webinar explores how NVIDIA’s CUDA Tile model—and its Julia port, cuTile.jl—makes high-performance GPU programming more accessible. Join Dr. Andy Terrel of NVIDIA and Dr. Tim Besard of JuliaHub to see real examples across linear algebra, AI inference, and HPC. Register here - #JuliaLang #GPUProgramming #CUDA #HPC #AIInfrastructure

JuliaHub

10,461 просмотров • 6 месяцев назад

What is AI inference engineering, why is it such an in-demand skill, and how do you break into the field? With author of Inference Engineering Philip Kiely and head of training at Baseten Charlie O'Neill 0:00: What is inference? 2:47: History of inference 4:59: Downstream effects of AI research on inference 13:54: What you'll learn from Inference Engineering 16:14: Advice for engineers transitioning into AI 19:00: Open source models driving inference growth 20:55: Specialization vs. frontier closed models 23:51: "Big Token" and the importance of open source AI 27:18: Where to get Inference Engineering
27:58

What is AI inference engineering, why is it such an in-demand skill, and how do you break into the field? With author of Inference Engineering Philip Kiely and head of training at Baseten Charlie O'Neill 0:00: What is inference? 2:47: History of inference 4:59: Downstream effects of AI research on inference 13:54: What you'll learn from Inference Engineering 16:14: Advice for engineers transitioning into AI 19:00: Open source models driving inference growth 20:55: Specialization vs. frontier closed models 23:51: "Big Token" and the importance of open source AI 27:18: Where to get Inference Engineering

Madison Kanna

119,268 просмотров • 7 месяцев назад

SITUATION EXPLAINED: Cerebras raised $5.55 billion in their IPO and closing their first day of trading valued at $66 billion, making it the biggest US tech IPO since Snowflake in 2020. Cerebras makes Wafer-Scale Engine chips built for AI inference. We asked Sarah Fong the main difference between wafer-scale chips and traditional GPUs: - GPUs are great at parallel work (graphics, training) - AI inference is sequential, AKA one token at a time This causes the "memory wall" problem: - Every GPU core needs model weights, KV cache, and activations to do its math - On a GPU, that data lives in off-chip memory (HBM) - Cores constantly load and offload from off-chip memory, which is a huge bottleneck; hardware accounts for ~70% of inference latency Cerebras' chips: -Dinner-plate sized (vs. GPUs which are palm-sized) with tens of thousands of cores -Memory sits directly on top of the cores as distributed SRAM -Weights and KV cache can be accessed at on-chip speeds in the PB/s range, compared with off-chip speeds in the TB/s range achieved by GPUs with HBM.
3:14

SITUATION EXPLAINED: Cerebras raised $5.55 billion in their IPO and closing their first day of trading valued at $66 billion, making it the biggest US tech IPO since Snowflake in 2020. Cerebras makes Wafer-Scale Engine chips built for AI inference. We asked Sarah Fong the main difference between wafer-scale chips and traditional GPUs: - GPUs are great at parallel work (graphics, training) - AI inference is sequential, AKA one token at a time This causes the "memory wall" problem: - Every GPU core needs model weights, KV cache, and activations to do its math - On a GPU, that data lives in off-chip memory (HBM) - Cores constantly load and offload from off-chip memory, which is a huge bottleneck; hardware accounts for ~70% of inference latency Cerebras' chips: -Dinner-plate sized (vs. GPUs which are palm-sized) with tens of thousands of cores -Memory sits directly on top of the cores as distributed SRAM -Weights and KV cache can be accessed at on-chip speeds in the PB/s range, compared with off-chip speeds in the TB/s range achieved by GPUs with HBM.

MTS

44,057 просмотров • 5 месяцев назад

📢 Inference is ON 😎 Run any AI model on dedicated GPU hardware, starting today. The GPU is yours while you're on it, the full price is displayed before you commit, and the model is whatever you bring from Hugging Face. Wild concept, we know! 👇
0:29

📢 Inference is ON 😎 Run any AI model on dedicated GPU hardware, starting today. The GPU is yours while you're on it, the full price is displayed before you commit, and the model is whatever you bring from Hugging Face. Wild concept, we know! 👇

Ocean Network

459,461 просмотров • 1 месяц назад

AI agents are shaping the future of automation and intelligence, but they need powerful infrastructure to thrive. ⚡ NodeAI delivers high-performance GPUs to scale, deploy, and optimize your AI agents with unmatched efficiency. #AI #GPU #NODEAI
0:36

AI agents are shaping the future of automation and intelligence, but they need powerful infrastructure to thrive. ⚡ NodeAI delivers high-performance GPUs to scale, deploy, and optimize your AI agents with unmatched efficiency. #AI #GPU #NODEAI

Node AI | $GPU

114,550 просмотров • 1 год назад

Modal engineer who teaches Full-Stack Deep Learning explained everything an AI engineer needs to know about GPUs in 20 minutes - better than $800 CUDA workshops. memory bandwidth -> SM occupancy -> kernel fusion -> arithmetic intensity -> know when you're compute-bound vs memory-bound. That loop is how you stop overpaying for GPUs you're barely using. VRAM math + roofline model + Modal serverless GPUs + kernel profiling - that's the stack. Watch and save it, then check your own GPU utilization tonight.
19:25

Modal engineer who teaches Full-Stack Deep Learning explained everything an AI engineer needs to know about GPUs in 20 minutes - better than $800 CUDA workshops. memory bandwidth -> SM occupancy -> kernel fusion -> arithmetic intensity -> know when you're compute-bound vs memory-bound. That loop is how you stop overpaying for GPUs you're barely using. VRAM math + roofline model + Modal serverless GPUs + kernel profiling - that's the stack. Watch and save it, then check your own GPU utilization tonight.

h100envy

18,442 просмотров • 3 месяцев назад

Gavin Baker (Gavin Baker) says the disaggregation of inference can extend GPU useful lives from 3-4 years to 10-15. That may single-handedly save private credit and reduce the financing rates for GPUs, which will drive demand and help finance the build-out. "The disaggregation of prefill and inference is going to be amazing for the useful lives of GPU and may single-handedly save private credit. Private credit is in pain from these SaaS loans. But there's a lot of private credit in GPUs too. They were underwriting that to 3-4. The disaggregation of inference means that these GPUs are going to have 10 or 15-year lives. The AI skeptics are like, "Oh, these companies are all cooking their books. The useful life of a GPU is only a year or two. The useful life of a CPU is only four years because the rapid technological change." No. What rapid technological change has done with the disaggregation of prefill and inference is you can put a Cerebras system or Groq LPUs effectively in front of a Hopper or even an Ampere, use that Hopper and Ampere for prefill, and extend the useful life of that GPU until it melts. This is going to be really good for the whole private credit industry. It's gonna help finance the AI build-out. Because if you can start to finance GPUs at 5% or 6% instead of – I think CoreWeave's lowest financing was low sevens – that actually mathematically changes the cost to finance this build-out."
1:51

Gavin Baker (Gavin Baker) says the disaggregation of inference can extend GPU useful lives from 3-4 years to 10-15. That may single-handedly save private credit and reduce the financing rates for GPUs, which will drive demand and help finance the build-out. "The disaggregation of prefill and inference is going to be amazing for the useful lives of GPU and may single-handedly save private credit. Private credit is in pain from these SaaS loans. But there's a lot of private credit in GPUs too. They were underwriting that to 3-4. The disaggregation of inference means that these GPUs are going to have 10 or 15-year lives. The AI skeptics are like, "Oh, these companies are all cooking their books. The useful life of a GPU is only a year or two. The useful life of a CPU is only four years because the rapid technological change." No. What rapid technological change has done with the disaggregation of prefill and inference is you can put a Cerebras system or Groq LPUs effectively in front of a Hopper or even an Ampere, use that Hopper and Ampere for prefill, and extend the useful life of that GPU until it melts. This is going to be really good for the whole private credit industry. It's gonna help finance the AI build-out. Because if you can start to finance GPUs at 5% or 6% instead of – I think CoreWeave's lowest financing was low sevens – that actually mathematically changes the cost to finance this build-out."

Invest Like the Best

211,336 просмотров • 5 месяцев назад

$15,000 for a Mac sounds insane. Until you realize what Apple is actually selling. 256GB of unified memory. Not a huge GPU farm. Not an NVIDIA rack. Not 4 different machines. Just one box that can keep multiple large LLMs in memory and run them locally. That changes the equation for local AI. NVIDIA wins when you need raw GPU compute. Apple gets interesting when the bottleneck is memory capacity. Because once your model doesn't fit into 24GB, 32GB or 48GB of VRAM, buying a faster GPU doesn't solve the problem. You need more memory. And that's where machines like the Mac Studio start looking less crazy. The interesting question isn't: “Is a $15K Mac faster than an NVIDIA GPU?” It isn't. The real question is: “How much are you willing to pay to make huge local models actually fit?” That’s where unified memory gets interesting.
0:44

$15,000 for a Mac sounds insane. Until you realize what Apple is actually selling. 256GB of unified memory. Not a huge GPU farm. Not an NVIDIA rack. Not 4 different machines. Just one box that can keep multiple large LLMs in memory and run them locally. That changes the equation for local AI. NVIDIA wins when you need raw GPU compute. Apple gets interesting when the bottleneck is memory capacity. Because once your model doesn't fit into 24GB, 32GB or 48GB of VRAM, buying a faster GPU doesn't solve the problem. You need more memory. And that's where machines like the Mac Studio start looking less crazy. The interesting question isn't: “Is a $15K Mac faster than an NVIDIA GPU?” It isn't. The real question is: “How much are you willing to pay to make huge local models actually fit?” That’s where unified memory gets interesting.

KyzoroX

19,536 просмотров • 2 месяцев назад

Inference providers hate this man... Your LLM deployment can deliver 30 to 50% better performance after initial deployment. If your current inference provider isn't learning your workload and tuning the stack continuously after deployment, you need another provider. Wafer's continual inference agents do that work. Wafer's continual inference agents profile your production workload to identify where scheduling, decoding, or GPU memory pressure limits performance. The agents benchmark changes to batching, serving engines, and GPU kernels against your workload. Each candidate must preserve correctness and meet your reliability targets before it ships. The agents keep profiling as traffic changes, identifying new bottlenecks and further opportunities to improve performance per dollar. Watch emilio andere explain how that work continues after deployment in his conversation with theCUBE.
1:06

Inference providers hate this man... Your LLM deployment can deliver 30 to 50% better performance after initial deployment. If your current inference provider isn't learning your workload and tuning the stack continuously after deployment, you need another provider. Wafer's continual inference agents do that work. Wafer's continual inference agents profile your production workload to identify where scheduling, decoding, or GPU memory pressure limits performance. The agents benchmark changes to batching, serving engines, and GPU kernels against your workload. Each candidate must preserve correctness and meet your reliability targets before it ships. The agents keep profiling as traffic changes, identifying new bottlenecks and further opportunities to improve performance per dollar. Watch emilio andere explain how that work continues after deployment in his conversation with theCUBE.

wafer

42,840 просмотров • 1 месяц назад

Groq's LPU is faster than GPUs, handling requests and responding more quickly. Groq's LPUs don't need speedy data delivery like Nvidia GPUs do because they don't have HBM in their system. They use SRAM, which is about 20 times faster than what GPUs use. Since inference runs use way less data than model training, Groq's LPU is more energy-efficient. It reads less from external memory and uses less power than a Nvidia GPU for inference tasks. The LPU works differently from GPUs. It uses a Temporal Instruction Set Computer architecture, so it doesn't have to reload data from memory as often as GPUs do with High Bandwidth Memory (HBM). This helps avoid issues with HBM shortages and keeps costs down. If Groq's LPU is used in places that do AI processing, you might not need special storage for Nvidia GPUs. The LPU doesn't demand super-fast storage like GPUs do. Groq is claiming that its technology could replace GPUs in AI tasks with its powerful chip and software.
7:57

Groq's LPU is faster than GPUs, handling requests and responding more quickly. Groq's LPUs don't need speedy data delivery like Nvidia GPUs do because they don't have HBM in their system. They use SRAM, which is about 20 times faster than what GPUs use. Since inference runs use way less data than model training, Groq's LPU is more energy-efficient. It reads less from external memory and uses less power than a Nvidia GPU for inference tasks. The LPU works differently from GPUs. It uses a Temporal Instruction Set Computer architecture, so it doesn't have to reload data from memory as often as GPUs do with High Bandwidth Memory (HBM). This helps avoid issues with HBM shortages and keeps costs down. If Groq's LPU is used in places that do AI processing, you might not need special storage for Nvidia GPUs. The LPU doesn't demand super-fast storage like GPUs do. Groq is claiming that its technology could replace GPUs in AI tasks with its powerful chip and software.

Brian Roemmele

87,244 просмотров • 2 лет назад

.Logan broke down Nockchain's GTM strategy Here's the lowdown: - Nvidia's licensing model blocks data centers from using consumer GPUs (4090, 5090, 3090) - If you use them, you lose your enterprise GPU license - But on a dollars-per-compute basis, consumer GPUs are insanely efficient - Nockchain coordinates the world's idle consumer compute into a single base layer - Top layer: any compute product (training, inference, agents) can plug in - The hardware base is already there $NOCK
2:24

.Logan broke down Nockchain's GTM strategy Here's the lowdown: - Nvidia's licensing model blocks data centers from using consumer GPUs (4090, 5090, 3090) - If you use them, you lose your enterprise GPU license - But on a dollars-per-compute basis, consumer GPUs are insanely efficient - Nockchain coordinates the world's idle consumer compute into a single base layer - Top layer: any compute product (training, inference, agents) can plug in - The hardware base is already there $NOCK

MCG

13,675 просмотров • 4 месяцев назад

$CRWV CEO says “older generations of GPUs are going to have a longer useful life than anyone anticipated” as CoreWeave signs a fully priced A100 contract through 2029 for hardware first launched in 2020. That means a 2020-era GPU can keep monetizing already-powered facilities that newer systems may not physically fit into while frontier workloads cascade to newer architectures and inference shifts down the stack. $NVDA CUDA strengthens that model by continuously optimizing Ampere, Hopper and Blackwell so the same workloads require less compute over time without forcing a hardware replacement.
0:25

$CRWV CEO says “older generations of GPUs are going to have a longer useful life than anyone anticipated” as CoreWeave signs a fully priced A100 contract through 2029 for hardware first launched in 2020. That means a 2020-era GPU can keep monetizing already-powered facilities that newer systems may not physically fit into while frontier workloads cascade to newer architectures and inference shifts down the stack. $NVDA CUDA strengthens that model by continuously optimizing Ampere, Hopper and Blackwell so the same workloads require less compute over time without forcing a hardware replacement.

Shay Boloor

140,947 просмотров • 2 месяцев назад

The Way Forward & A New Chapter For AI powered by GamerHash AI & $GHX. Welcome to the future of bleeding edge connectivity and cloud infrastructure ⚡ GamerHash AI is the answer to the billion dollar question on how to offset the GPU scarcity, enable exponential growth of AI through utilizing consumer-level GPUs and establishing a powerful sharing economy by sourcing a global network of gamers. The synergy of our GPU #DePIN and blockchain for #AI #inference is the technology of the future that already works today. See you in the network!
1:01

The Way Forward & A New Chapter For AI powered by GamerHash AI & $GHX. Welcome to the future of bleeding edge connectivity and cloud infrastructure ⚡ GamerHash AI is the answer to the billion dollar question on how to offset the GPU scarcity, enable exponential growth of AI through utilizing consumer-level GPUs and establishing a powerful sharing economy by sourcing a global network of gamers. The synergy of our GPU #DePIN and blockchain for #AI #inference is the technology of the future that already works today. See you in the network!

GamerHash AI

1,866,164 просмотров • 2 лет назад

I wrote GPU kernels for GPT2 124M inference engine from scratch in ~1500 lines of CUDA. Here I explain it simply.
3:45

I wrote GPU kernels for GPT2 124M inference engine from scratch in ~1500 lines of CUDA. Here I explain it simply.

ninjaHattori1

160,092 просмотров • 1 месяц назад

GPU tradeoff series: A100 is not much more powerful than 4090 🫠 GPU Perf and Price: - 4090: 330 fp16 TFLOPs, $1,749 - A100 (80GB): 312 fp16 TFLOPs, $20,000 > A100 is 11.4X more pricy Training speed for GPT-2(124M) with llm.c: - 4090: 153K tokens/s - A100 (80GB): 195K tokens/s > A100 is only 1.3X faster (both trained using a single card, A100 llm.c training is shown in the video, 4090 video is in the quoted tweet) Conclusion: 4090 has a much better cost vs performance ratio Why: As in the H100 vs. 4090 comparison, the biggest difference between A100 and 4090 is their GPU memory size/bandwidth and cross-GPU communication bandwidth, which does not matter too much if your model can fit into a single 4090. Specs: 4090: - GPU memory size: 24GB - memory bandwidth: 1 TB/s - communication bandwidth: 64 GB/s A100: - GPU memory size: 80GB - memory bandwidth: 2 TB/s - communication bandwidth: 900 GB/s Nvidia killed off NVLink (a high-speed communication link that connects GPUs) on 4090. (Jensen Huang smiling face) If multiple 4090s could be interconnected via NVLink, their performance would be closer to datacenter-grade A100 GPUs, even for training larger models. Additionally, 4090 isn't allowed in datacenters, that's how Nvidia makes 💰💰💰
0:32

GPU tradeoff series: A100 is not much more powerful than 4090 🫠 GPU Perf and Price: - 4090: 330 fp16 TFLOPs, $1,749 - A100 (80GB): 312 fp16 TFLOPs, $20,000 > A100 is 11.4X more pricy Training speed for GPT-2(124M) with llm.c: - 4090: 153K tokens/s - A100 (80GB): 195K tokens/s > A100 is only 1.3X faster (both trained using a single card, A100 llm.c training is shown in the video, 4090 video is in the quoted tweet) Conclusion: 4090 has a much better cost vs performance ratio Why: As in the H100 vs. 4090 comparison, the biggest difference between A100 and 4090 is their GPU memory size/bandwidth and cross-GPU communication bandwidth, which does not matter too much if your model can fit into a single 4090. Specs: 4090: - GPU memory size: 24GB - memory bandwidth: 1 TB/s - communication bandwidth: 64 GB/s A100: - GPU memory size: 80GB - memory bandwidth: 2 TB/s - communication bandwidth: 900 GB/s Nvidia killed off NVLink (a high-speed communication link that connects GPUs) on 4090. (Jensen Huang smiling face) If multiple 4090s could be interconnected via NVLink, their performance would be closer to datacenter-grade A100 GPUs, even for training larger models. Additionally, 4090 isn't allowed in datacenters, that's how Nvidia makes 💰💰💰

Yuchen Jin

234,951 просмотров • 2 лет назад