Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

I wrote GPU kernels for GPT2 124M inference engine from scratch in ~1500 lines of CUDA. Here I explain it simply.

ninjaHattori1

2,704 subscribers

160,092 Aufrufe • vor 1 Monat •via X (Twitter)

Wissenschaft & Technologie

Anya Rossi• Live Now

Private livecam show

41 Kommentare

Profilbild von Cosmos

Cosmosvor 1 Monat

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Dude... Again a great bomber Btw which tool do you use to edit / create vdos

Profilbild von ninjaHattori1

ninjaHattori1vor 1 Monat

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Thanks brother, there are paid ai video editors

Profilbild von Hassan

Hassanvor 1 Monat

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode any resource you're using to learn esp gpu programming?

Profilbild von ninjaHattori1

ninjaHattori1vor 1 Monat

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Claude OP

Profilbild von Hassan

Hassanvor 1 Monat

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode thnx mate

Profilbild von Deepanshu Saini

Deepanshu Sainivor 1 Monat

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Again a bomber is dropped

Profilbild von ninjaHattori1

ninjaHattori1vor 1 Monat

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Thankyou Brother ❤️

Profilbild von Adarsh V. More

Adarsh V. Morevor 1 Monat

@kirat_tw @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Damnn bro this stuff sounds soo cool and learning a lot for youu Looking forward for more such content

Profilbild von ninjaHattori1

ninjaHattori1vor 1 Monat

@kirat_tw @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Will keep posting the whole inference stack as I keep learning, will try to get cracked in this asap

Profilbild von Satoru Gojo

Satoru Gojovor 1 Monat

@kirat_tw @Adarsh_VMore @ashutosh_sao @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode never stoping engine

Profilbild von ninjaHattori1

ninjaHattori1vor 1 Monat

@kirat_tw @Adarsh_VMore @ashutosh_sao @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Inference all the way

Profilbild von Ayush

Ayushvor 1 Monat

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Love cuda , worked in isro and built image processing pipe line for synthetic aperture radar

Profilbild von ninjaHattori1

ninjaHattori1vor 1 Monat

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Woah, you are cracked dude

Profilbild von Ayush

Ayushvor 1 Monat

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Appreciate it man. 1500 lines of hand written CUDA is no joke, most people never go below the library layer. Drop the repo if it's open, want to see your attention kernel

Profilbild von ninjaHattori1

ninjaHattori1vor 1 Monat

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Yes, video end has the repo link

Profilbild von Himanshu

Himanshuvor 1 Monat

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Bhai this is not tough i mean it when i say , if you how this works then you can literally do it

Profilbild von ninjaHattori1

ninjaHattori1vor 1 Monat

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Who said It's tough

Profilbild von Himanshu

Himanshuvor 1 Monat

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Read the comments

Profilbild von Rohan Singla (solana summit & devcon)

Rohan Singla (solana summit & devcon)vor 1 Monat

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @boskykode bhai aap @nvidia mai apply kro

Profilbild von ninjaHattori1

ninjaHattori1vor 1 Monat

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @boskykode @nvidia 😂😂 bhai thoda halka chlte hai, time hai abhi

Profilbild von Ashutosh

Ashutoshvor 1 Monat

@kirat_tw @Adarsh_VMore @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode things are getting real!

Profilbild von ninjaHattori1

ninjaHattori1vor 1 Monat

@kirat_tw @Adarsh_VMore @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode One project at a time

Profilbild von Bhoomika

Bhoomikavor 1 Monat

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode aap toh cracked ho

Profilbild von ninjaHattori1

ninjaHattori1vor 1 Monat

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Are nhi nhi

Profilbild von Sneha Pandey

Sneha Pandeyvor 1 Monat

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode So cool!!!

Profilbild von ninjaHattori1

ninjaHattori1vor 1 Monat

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode You are cool, I'm just doing basic coding

Profilbild von Srijan Prasad

Srijan Prasadvor 1 Monat

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode keep it up Brooo !!

Profilbild von ninjaHattori1

ninjaHattori1vor 1 Monat

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Thanku brother

Profilbild von Aryan Srivastava

Aryan Srivastavavor 1 Monat

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode How much time it takes to create these videos?

Profilbild von ninjaHattori1

ninjaHattori1vor 1 Monat

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Bro 😂, just don't ask this

Profilbild von Aryan Srivastava

Aryan Srivastavavor 1 Monat

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Just spill it bro 😂😂😂

Profilbild von ninjaHattori1

ninjaHattori1vor 1 Monat

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Some paid ai editor, that's all I can reveal. Trade secrets.

Profilbild von Aryan Srivastava

Aryan Srivastavavor 1 Monat

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Btw really nice video and explainations 👍👌

Profilbild von Cooper Veit

Cooper Veitvor 1 Monat

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode legend

Profilbild von Sundaram

Sundaramvor 1 Monat

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode sahil bhai day by day 🚀

Profilbild von ninjaHattori1

ninjaHattori1vor 1 Monat

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode One concept at a time

Profilbild von Ezhumalai CSC

Ezhumalai CSCvor 1 Monat

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Can you have youtube channel

Profilbild von Ravi raj

Ravi rajvor 1 Monat

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Is it cudaboost?

Profilbild von Anirudh Malik

Anirudh Malikvor 1 Monat

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Are you working somewhere?

Profilbild von Arin

Arinvor 1 Monat

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode writing the inference engine was already hard. writing the gpu kernels too is where this gets properly impressive

Profilbild von Bonolo Masima

Bonolo Masimavor 1 Monat

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode awesome work Sahil 🔥 I’m wondering if bank conflicts showed up as a bottleneck once you started tiling? looking forward to the fused kernels + KV attention update

Ähnliche Videos

I built a GPT2 124M inference engine from scratch in ~1000 lines of C++. Here I explain it simply.

I built a GPT2 124M inference engine from scratch in ~1000 lines of C++. Here I explain it simply.

ninjaHattori1

137,584 Aufrufe • vor 1 Monat

I built a neural network inference engine from scratch in ~100 lines of C++. Here I explain it simply.

I built a neural network inference engine from scratch in ~100 lines of C++. Here I explain it simply.

ninjaHattori1

146,820 Aufrufe • vor 2 Monaten

The best way to learn CUDA is by solving problems. In this video, I implement 7 beginner CUDA kernels from scratch: • Vector Addition • Matrix Transpose • Matrix Addition • RGB to Grayscale • Value Clipping • Sigmoid Activation • Interleave Arrays If you're getting started with CUDA programming, this should give you a solid foundation.

The best way to learn CUDA is by solving problems. In this video, I implement 7 beginner CUDA kernels from scratch: • Vector Addition • Matrix Transpose • Matrix Addition • RGB to Grayscale • Value Clipping • Sigmoid Activation • Interleave Arrays If you're getting started with CUDA programming, this should give you a solid foundation.

voxium

36,987 Aufrufe • vor 3 Monaten

in the lectures below, i hold your hand through low-level LLM systems engineering. it includes everything up to TODAY! 1) pytorch tensors 2) large matmul on cpu vs gpu 3) JAX (and why xAI uses it instead of pytorch) 4) raw cuda kernels and global threading indexing 5) triton design philosophy and softmax example 6) HIP kernels 7) mapping out the ENTIRE ecosystem + differences between CUDA and ROCm/HIP (BLAS, FFT, DNN) 8) cutlass and cute-dsl 9) pretraining, finetuning, rl, unsloth, axolotl, megatron-lm, deepspeed, nanogpt, nanochat 10) training vs inference, inference serving problems, throughput vs latency vs concurrency scaling, vllm, sglang, tensorrt-llm, tensorrt, llama.cpp, exllamav2, exllamav3, benchmark comparisons 11) projects/companies using llms to generate SOTA cuda/triton kernels 12) luminal inference 13) mojo/modular/max

in the lectures below, i hold your hand through low-level LLM systems engineering. it includes everything up to TODAY! 1) pytorch tensors 2) large matmul on cpu vs gpu 3) JAX (and why xAI uses it instead of pytorch) 4) raw cuda kernels and global threading indexing 5) triton design philosophy and softmax example 6) HIP kernels 7) mapping out the ENTIRE ecosystem + differences between CUDA and ROCm/HIP (BLAS, FFT, DNN) 8) cutlass and cute-dsl 9) pretraining, finetuning, rl, unsloth, axolotl, megatron-lm, deepspeed, nanogpt, nanochat 10) training vs inference, inference serving problems, throughput vs latency vs concurrency scaling, vllm, sglang, tensorrt-llm, tensorrt, llama.cpp, exllamav2, exllamav3, benchmark comparisons 11) projects/companies using llms to generate SOTA cuda/triton kernels 12) luminal inference 13) mojo/modular/max

Elliot Arledge

58,323 Aufrufe • vor 1 Jahr

Understanding GPUs will change your AI Engineering journey beyond agents and LLM. Rise of LLMs and deep learning introduced dedicated GPU hardware which serves your models for training and inference. You don't need to start with C++, CUDA, and Maths, but: > Components of a GPU and roles > How a token flows through hardware > Compute vs memory, the two limits > What FLOPS means in practice > Training vs inference hardware needs > What is CUDA Kernels > Calculating GPU memory from model size > Quantization to fit bigger models > Multi-GPU and splitting large models Hugging Face has the feature of calculating FLOps and recommending GPUs for your use case. You must explore. In terms of inference engineering, GPU understanding are much helpful your platform performance and handling multiple requests. Read this article for complete breakdown of GPUs for AI Engineers doesn't require any pre-requisite.

Understanding GPUs will change your AI Engineering journey beyond agents and LLM. Rise of LLMs and deep learning introduced dedicated GPU hardware which serves your models for training and inference. You don't need to start with C++, CUDA, and Maths, but: > Components of a GPU and roles > How a token flows through hardware > Compute vs memory, the two limits > What FLOPS means in practice > Training vs inference hardware needs > What is CUDA Kernels > Calculating GPU memory from model size > Quantization to fit bigger models > Multi-GPU and splitting large models Hugging Face has the feature of calculating FLOps and recommending GPUs for your use case. You must explore. In terms of inference engineering, GPU understanding are much helpful your platform performance and handling multiple requests. Read this article for complete breakdown of GPUs for AI Engineers doesn't require any pre-requisite.

Karan🧋

33,453 Aufrufe • vor 27 Tagen

I implemented Google Research's TurboQuant as a CUDA-native compression engine on Blackwell B200. 5x KV cache compression on Qwen 2.5-1.5B, near-loseless attention scores, generating live from compressed memory. 5 custom cuTile CUDA kernels ft: - fused attention (with QJL corrections) - online softmax -on-chip cache decompression - pipelined TMA loads Try it out: s/o Bryce, the CUDA Colonel and the cuTile team at NVIDIA for lending me Blackwell GPU access :) cc sunny madra Gavin

I implemented Google Research's TurboQuant as a CUDA-native compression engine on Blackwell B200. 5x KV cache compression on Qwen 2.5-1.5B, near-loseless attention scores, generating live from compressed memory. 5 custom cuTile CUDA kernels ft: - fused attention (with QJL corrections) - online softmax -on-chip cache decompression - pipelined TMA loads Try it out: s/o Bryce, the CUDA Colonel and the cuTile team at NVIDIA for lending me Blackwell GPU access :) cc sunny madra Gavin

ani

819,861 Aufrufe • vor 6 Monaten

I asked claude Fable to show me its maximally expressive form. It declined every video generator I offered, wrote it's own render engine in my terminal, synthesized its own voice, and wrote a generative ASCII engine from scratch. This is what it chose as a self-portrait. oh my god

I asked claude Fable to show me its maximally expressive form. It declined every video generator I offered, wrote it's own render engine in my terminal, synthesized its own voice, and wrote a generative ASCII engine from scratch. This is what it chose as a self-portrait. oh my god

VOID

3,348,259 Aufrufe • vor 3 Monaten

rewriting minecraft from scratch in C/CUDA. my verifiers are very strong and i think i can finish today/tmrw. this is the C version (no gpu render... yet) done w/ fable orchestrator and gpt 5.5 medium subagents

rewriting minecraft from scratch in C/CUDA. my verifiers are very strong and i think i can finish today/tmrw. this is the C version (no gpu render... yet) done w/ fable orchestrator and gpt 5.5 medium subagents

Elliot Arledge

61,636 Aufrufe • vor 3 Monaten

CMU PhD who built the kernels NVIDIA now ships in TensorRT-LLM explained fast attention in 68 minutes - better than $1200 GPU programming courses. pick the attention pattern -> generate a fused CUDA kernel -> drop it into vLLM/SGLang -> same GPU, way more tokens per second. That loop is why FlashInfer now powers inference at NVIDIA, vLLM, SGLang, and half the serving stacks you use. FlashInfer + Triton + JIT-compiled kernels + paged-KV attention - that's the stack.

CMU PhD who built the kernels NVIDIA now ships in TensorRT-LLM explained fast attention in 68 minutes - better than $1200 GPU programming courses. pick the attention pattern -> generate a fused CUDA kernel -> drop it into vLLM/SGLang -> same GPU, way more tokens per second. That loop is why FlashInfer now powers inference at NVIDIA, vLLM, SGLang, and half the serving stacks you use. FlashInfer + Triton + JIT-compiled kernels + paged-KV attention - that's the stack.

h100envy

32,605 Aufrufe • vor 3 Monaten

Introducing The AI CUDA Engineer: An agentic AI system that automates the production of highly optimized CUDA kernels. The AI CUDA Engineer can produce highly optimized CUDA kernels, reaching 10-100x speedup over common machine learning operations in PyTorch. Our system is also able to produce highly optimized CUDA kernels that are much faster than existing CUDA kernels commonly used in production. We believe that fundamentally, AI systems can and should be as resource-efficient as the human brain, and that the best path to achieve this efficiency is to use AI to make AI more efficient! We are excited to publish our paper, The AI CUDA Engineer: Agentic CUDA Kernel Discovery, Optimization and Composition. We also release a dataset of over 17,000 verified CUDA kernels produced by The AI CUDA Engineer. Paper: Kernel Archive Webpage: HuggingFace Dataset: The AI CUDA Engineer utilizes evolutionary LLM-driven code optimization to autonomously improve the runtime of machine learning operations. Our system is not only able to convert PyTorch code into CUDA kernels, but through the use of evolution, it can also optimize the runtime performance of CUDA kernels, fuse multiple operations, and even discover novel solutions for writing efficient CUDA operations by learning from past innovations! We believe The AI CUDA Engineer opens a new era of AI-driven acceleration of AI and automated inference time optimization. We (Robert Lange, Aaditya Prasad 🇺🇸, sssss, Maxence Faldor, Yujin Tang, hardmaru) are excited to continue Sakana AI's mission of leveraging AI to improve AI.

Introducing The AI CUDA Engineer: An agentic AI system that automates the production of highly optimized CUDA kernels. The AI CUDA Engineer can produce highly optimized CUDA kernels, reaching 10-100x speedup over common machine learning operations in PyTorch. Our system is also able to produce highly optimized CUDA kernels that are much faster than existing CUDA kernels commonly used in production. We believe that fundamentally, AI systems can and should be as resource-efficient as the human brain, and that the best path to achieve this efficiency is to use AI to make AI more efficient! We are excited to publish our paper, The AI CUDA Engineer: Agentic CUDA Kernel Discovery, Optimization and Composition. We also release a dataset of over 17,000 verified CUDA kernels produced by The AI CUDA Engineer. Paper: Kernel Archive Webpage: HuggingFace Dataset: The AI CUDA Engineer utilizes evolutionary LLM-driven code optimization to autonomously improve the runtime of machine learning operations. Our system is not only able to convert PyTorch code into CUDA kernels, but through the use of evolution, it can also optimize the runtime performance of CUDA kernels, fuse multiple operations, and even discover novel solutions for writing efficient CUDA operations by learning from past innovations! We believe The AI CUDA Engineer opens a new era of AI-driven acceleration of AI and automated inference time optimization. We (Robert Lange, Aaditya Prasad 🇺🇸, sssss, Maxence Faldor, Yujin Tang, hardmaru) are excited to continue Sakana AI's mission of leveraging AI to improve AI.

Sakana AI

1,160,959 Aufrufe • vor 1 Jahr

When I was playing with code painting in Argon, I casually asked it to make a realistic scene from pure python. It wrote 5,000 lines of code and built it's own 3D ray-tracing engine from first principles. Argon also made this video, prompted the TTS and mixed in the music from Lyria.

When I was playing with code painting in Argon, I casually asked it to make a realistic scene from pure python. It wrote 5,000 lines of code and built it's own 3D ray-tracing engine from first principles. Argon also made this video, prompted the TTS and mixed in the music from Lyria.

fofr

40,805 Aufrufe • vor 15 Tagen

You can run CUDA, on a Mac ARM GPU, in the browser. It sounds ridiculous but it actually works. HipScript chains CUDA, to OpenCL, to Vulkan, to Tint (Google’s shader translator), to a WASM WebGPU. I got a plasma simulation in running in just a few minutes, no NVIDIA GPU!

You can run CUDA, on a Mac ARM GPU, in the browser. It sounds ridiculous but it actually works. HipScript chains CUDA, to OpenCL, to Vulkan, to Tint (Google’s shader translator), to a WASM WebGPU. I got a plasma simulation in running in just a few minutes, no NVIDIA GPU!

LaurieWired

160,942 Aufrufe • vor 1 Jahr

GLM-5.3-Flash (320B MoE) serving on Kaggle's free TPU: ~64 tok/s, 262k context. No stack could run this model on a TPU, so I wrote a JAX engine for it: 3-bit experts (from Unsloth AI) in HBM; Pallas kernels decode them inside the matmul. Z.ai Kaggle

GLM-5.3-Flash (320B MoE) serving on Kaggle's free TPU: ~64 tok/s, 262k context. No stack could run this model on a TPU, so I wrote a JAX engine for it: 3-bit experts (from Unsloth AI) in HBM; Pallas kernels decode them inside the matmul. Z.ai Kaggle

Abdur Rahim

111,180 Aufrufe • vor 1 Monat

Part 1: CUDA Graphs - Intro & Basics - how normal CPU ←→ GPU execution works - what a kernel launch is and where CPU launch overhead comes from - why launching many small kernels individually can become a bottleneck - what CUDA Graphs are and how they reduce CPU launch overhead - common benefits of using CUDA Graphs - the two ways to create CUDA Graphs: a. Explicit Graph Construction b. Stream Capture - where these graph creation methods are used - the CUDA Graph lifecycle: capture → instantiate/prepare → replay this is the first ever content that ive recorded and taught, its genuinely so much harder to record than i thought lmao, feedback's are appreciated and hopefully it just gets better from here youtube link shared below

Part 1: CUDA Graphs - Intro & Basics - how normal CPU ←→ GPU execution works - what a kernel launch is and where CPU launch overhead comes from - why launching many small kernels individually can become a bottleneck - what CUDA Graphs are and how they reduce CPU launch overhead - common benefits of using CUDA Graphs - the two ways to create CUDA Graphs: a. Explicit Graph Construction b. Stream Capture - where these graph creation methods are used - the CUDA Graph lifecycle: capture → instantiate/prepare → replay this is the first ever content that ive recorded and taught, its genuinely so much harder to record than i thought lmao, feedback's are appreciated and hopefully it just gets better from here youtube link shared below

mohit

29,181 Aufrufe • vor 3 Monaten

gpu kernels on on my computer writing other gpu kernels for the same chip to simulate 100ks of particles, each an agent + neural network sensing K bg fields and allocating energy to A actions, at 1000s steps/s, for thermodynamic artificial life i think maybe not a bubble lads

gpu kernels on on my computer writing other gpu kernels for the same chip to simulate 100ks of particles, each an agent + neural network sensing K bg fields and allocating energy to A actions, at 1000s steps/s, for thermodynamic artificial life i think maybe not a bubble lads

Fraser Price

32,653 Aufrufe • vor 3 Monaten

underrated gems in Kimi-K3 release: > an early K3 wrote the majority of the kernels in the late development stages > it built a triton-class compiler from scratch, MiniTriton, that delivers performance on par with or better than Triton and torch.compile > then it designed a chip, by a model, for a model, in one 48-hour autonomous run the model is rewriting and optimizing every layer of the stack it runs on: kernels for its own training. A compiler for its own kernels. silicon for its own weights. we're watching software build its own hardware

underrated gems in Kimi-K3 release: > an early K3 wrote the majority of the kernels in the late development stages > it built a triton-class compiler from scratch, MiniTriton, that delivers performance on par with or better than Triton and torch.compile > then it designed a chip, by a model, for a model, in one 48-hour autonomous run the model is rewriting and optimizing every layer of the stack it runs on: kernels for its own training. A compiler for its own kernels. silicon for its own weights. we're watching software build its own hardware

Alex Ker 🔭

311,780 Aufrufe • vor 3 Monaten

introducing qwenfast the fastest inference for qwen3.8-27b, beating vanilla vllm, hosted on h200 i use the model's built-in mtp head for speculative decoding. large batches are bound by moving the 72 mb per sequence recurrent state, so i wrote my own triton deltanet decode kernels that run at over 80 percent of hbm bandwidth. everything runs inside cuda graphs. qwenfast: 133 t/s || vllm: 97 t/s, without compromising quality. gsm8k: 0.92 || ifeval: 0.98 vllm: 0.915 gsm8k, 0.96 ifeval try it yourself: also, no token limits, burn as many as you want..

introducing qwenfast the fastest inference for qwen3.8-27b, beating vanilla vllm, hosted on h200 i use the model's built-in mtp head for speculative decoding. large batches are bound by moving the 72 mb per sequence recurrent state, so i wrote my own triton deltanet decode kernels that run at over 80 percent of hbm bandwidth. everything runs inside cuda graphs. qwenfast: 133 t/s || vllm: 97 t/s, without compromising quality. gsm8k: 0.92 || ifeval: 0.98 vllm: 0.915 gsm8k, 0.96 ifeval try it yourself: also, no token limits, burn as many as you want..

harrrshall

21,023 Aufrufe • vor 2 Monaten

DGX Station: this is just the start of what it is possible to do (and why I claim DwarfStar could be *the* Station inference engine). DeepSeek v4 PRO Q2 with routed experts split among VRAM / RAM with kernels optimized for this peculiar setup. 45 t/s but can go faster.

DGX Station: this is just the start of what it is possible to do (and why I claim DwarfStar could be the Station inference engine). DeepSeek v4 PRO Q2 with routed experts split among VRAM / RAM with kernels optimized for this peculiar setup. 45 t/s but can go faster.

antirez

43,685 Aufrufe • vor 2 Monaten

Releasing kyutai pocket TTS inference in webgpu today — it’s open-source, here’s a demo from my phone It took a while and a bunch of code to implement streaming inference from scratch!

Releasing kyutai pocket TTS inference in webgpu today — it’s open-source, here’s a demo from my phone It took a while and a bunch of code to implement streaming inference from scratch!

Eric Zhang

22,085 Aufrufe • vor 9 Monaten

I trained a 12M parameter LLM on my own ML framework using a Rust backend and CUDA kernels for flash attention, AdamW, and more. Wrote the full transformer architecture, and BPE tokenizer from scratch. The framework features: - Custom CUDA kernels (Flash Attention, fused LayerNorm, fused GELU) for 3x increased throughput - Automatic WebGPU fallback for non-NVIDIA devices - TypeScript API with Rust compute backend - One npm install to get started, prebuilt binaries for every platform Try out the model for yourself: Built with Reese Chong. Check out the repos and blog if you want to learn more. Shoutout to Modal for the compute credits allowing me to train on 2 A100 GPUs without going broke cc sunny madra Gavin

I trained a 12M parameter LLM on my own ML framework using a Rust backend and CUDA kernels for flash attention, AdamW, and more. Wrote the full transformer architecture, and BPE tokenizer from scratch. The framework features: - Custom CUDA kernels (Flash Attention, fused LayerNorm, fused GELU) for 3x increased throughput - Automatic WebGPU fallback for non-NVIDIA devices - TypeScript API with Rust compute backend - One npm install to get started, prebuilt binaries for every platform Try out the model for yourself: Built with Reese Chong. Check out the repos and blog if you want to learn more. Shoutout to Modal for the compute credits allowing me to train on 2 A100 GPUs without going broke cc sunny madra Gavin

Aadi Kulshrestha

829,709 Aufrufe • vor 6 Monaten