Video yükleniyor...

Video Yüklenemedi

Bu video yüklenirken bir sorun oluştu. Bu geçici bir ağ sorunundan kaynaklanıyor olabilir veya video kullanılamıyor olabilir.

Ana Sayfaya Dön

I wrote GPU kernels for GPT2 124M inference engine from scratch in ~1500 lines of CUDA. Here I explain it simply.

ninjaHattori1

2,704 subscribers

160,092 görüntüleme • 2 ay önce •via X (Twitter)

Bilim & Teknoloji

Anya Rossi• Live Now

Private livecam show

41 Yorum

Cosmos profil fotoğrafı

Cosmos2 ay önce

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Dude... Again a great bomber Btw which tool do you use to edit / create vdos

ninjaHattori1 profil fotoğrafı

ninjaHattori12 ay önce

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Thanks brother, there are paid ai video editors

Hassan profil fotoğrafı

Hassan2 ay önce

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode any resource you're using to learn esp gpu programming?

ninjaHattori1 profil fotoğrafı

ninjaHattori12 ay önce

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Claude OP

Hassan profil fotoğrafı

Hassan2 ay önce

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode thnx mate

Deepanshu Saini profil fotoğrafı

Deepanshu Saini2 ay önce

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Again a bomber is dropped

ninjaHattori1 profil fotoğrafı

ninjaHattori12 ay önce

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Thankyou Brother ❤️

Adarsh V. More profil fotoğrafı

Adarsh V. More2 ay önce

@kirat_tw @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Damnn bro this stuff sounds soo cool and learning a lot for youu Looking forward for more such content

ninjaHattori1 profil fotoğrafı

ninjaHattori12 ay önce

@kirat_tw @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Will keep posting the whole inference stack as I keep learning, will try to get cracked in this asap

Satoru Gojo profil fotoğrafı

Satoru Gojo2 ay önce

@kirat_tw @Adarsh_VMore @ashutosh_sao @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode never stoping engine

ninjaHattori1 profil fotoğrafı

ninjaHattori12 ay önce

@kirat_tw @Adarsh_VMore @ashutosh_sao @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Inference all the way

Ayush profil fotoğrafı

Ayush2 ay önce

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Love cuda , worked in isro and built image processing pipe line for synthetic aperture radar

ninjaHattori1 profil fotoğrafı

ninjaHattori12 ay önce

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Woah, you are cracked dude

Ayush profil fotoğrafı

Ayush2 ay önce

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Appreciate it man. 1500 lines of hand written CUDA is no joke, most people never go below the library layer. Drop the repo if it's open, want to see your attention kernel

ninjaHattori1 profil fotoğrafı

ninjaHattori12 ay önce

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Yes, video end has the repo link

Himanshu profil fotoğrafı

Himanshu2 ay önce

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Bhai this is not tough i mean it when i say , if you how this works then you can literally do it

ninjaHattori1 profil fotoğrafı

ninjaHattori12 ay önce

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Who said It's tough

Himanshu profil fotoğrafı

Himanshu2 ay önce

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Read the comments

Rohan Singla (solana summit & devcon) profil fotoğrafı

Rohan Singla (solana summit & devcon)2 ay önce

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @boskykode bhai aap @nvidia mai apply kro

ninjaHattori1 profil fotoğrafı

ninjaHattori12 ay önce

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @boskykode @nvidia 😂😂 bhai thoda halka chlte hai, time hai abhi

Ashutosh profil fotoğrafı

Ashutosh2 ay önce

@kirat_tw @Adarsh_VMore @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode things are getting real!

ninjaHattori1 profil fotoğrafı

ninjaHattori12 ay önce

@kirat_tw @Adarsh_VMore @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode One project at a time

Bhoomika profil fotoğrafı

Bhoomika2 ay önce

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode aap toh cracked ho

ninjaHattori1 profil fotoğrafı

ninjaHattori12 ay önce

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Are nhi nhi

Sneha Pandey profil fotoğrafı

Sneha Pandey2 ay önce

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode So cool!!!

ninjaHattori1 profil fotoğrafı

ninjaHattori12 ay önce

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode You are cool, I'm just doing basic coding

Srijan Prasad profil fotoğrafı

Srijan Prasad2 ay önce

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode keep it up Brooo !!

ninjaHattori1 profil fotoğrafı

ninjaHattori12 ay önce

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Thanku brother

Aryan Srivastava profil fotoğrafı

Aryan Srivastava2 ay önce

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode How much time it takes to create these videos?

ninjaHattori1 profil fotoğrafı

ninjaHattori12 ay önce

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Bro 😂, just don't ask this

Aryan Srivastava profil fotoğrafı

Aryan Srivastava2 ay önce

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Just spill it bro 😂😂😂

ninjaHattori1 profil fotoğrafı

ninjaHattori12 ay önce

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Some paid ai editor, that's all I can reveal. Trade secrets.

Aryan Srivastava profil fotoğrafı

Aryan Srivastava2 ay önce

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Btw really nice video and explainations 👍👌

Cooper Veit profil fotoğrafı

Cooper Veit2 ay önce

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode legend

Sundaram profil fotoğrafı

Sundaram2 ay önce

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode sahil bhai day by day 🚀

ninjaHattori1 profil fotoğrafı

ninjaHattori12 ay önce

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode One concept at a time

Ezhumalai CSC profil fotoğrafı

Ezhumalai CSC1 ay önce

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Can you have youtube channel

Ravi raj profil fotoğrafı

Ravi raj2 ay önce

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Is it cudaboost?

Anirudh Malik profil fotoğrafı

Anirudh Malik2 ay önce

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode Are you working somewhere?

Arin profil fotoğrafı

Arin2 ay önce

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode writing the inference engine was already hard. writing the gpu kernels too is where this gets properly impressive

Bonolo Masima profil fotoğrafı

Bonolo Masima2 ay önce

@kirat_tw @Adarsh_VMore @ashutosh_sao @jitendar100x @deepanshuliv @Deepjyoti100x @100xSchool @ashu_463 @rohanBuilds @boskykode awesome work Sahil 🔥 I’m wondering if bank conflicts showed up as a bottleneck once you started tiling? looking forward to the fused kernels + KV attention update

Benzer Videolar

I built a GPT2 124M inference engine from scratch in ~1000 lines of C++. Here I explain it simply.

I built a GPT2 124M inference engine from scratch in ~1000 lines of C++. Here I explain it simply.

ninjaHattori1

137,584 görüntüleme • 2 ay önce

I built a neural network inference engine from scratch in ~100 lines of C++. Here I explain it simply.

I built a neural network inference engine from scratch in ~100 lines of C++. Here I explain it simply.

ninjaHattori1

146,820 görüntüleme • 2 ay önce

in the lectures below, i hold your hand through low-level LLM systems engineering. it includes everything up to TODAY! 1) pytorch tensors 2) large matmul on cpu vs gpu 3) JAX (and why xAI uses it instead of pytorch) 4) raw cuda kernels and global threading indexing 5) triton design philosophy and softmax example 6) HIP kernels 7) mapping out the ENTIRE ecosystem + differences between CUDA and ROCm/HIP (BLAS, FFT, DNN) 8) cutlass and cute-dsl 9) pretraining, finetuning, rl, unsloth, axolotl, megatron-lm, deepspeed, nanogpt, nanochat 10) training vs inference, inference serving problems, throughput vs latency vs concurrency scaling, vllm, sglang, tensorrt-llm, tensorrt, llama.cpp, exllamav2, exllamav3, benchmark comparisons 11) projects/companies using llms to generate SOTA cuda/triton kernels 12) luminal inference 13) mojo/modular/max

in the lectures below, i hold your hand through low-level LLM systems engineering. it includes everything up to TODAY! 1) pytorch tensors 2) large matmul on cpu vs gpu 3) JAX (and why xAI uses it instead of pytorch) 4) raw cuda kernels and global threading indexing 5) triton design philosophy and softmax example 6) HIP kernels 7) mapping out the ENTIRE ecosystem + differences between CUDA and ROCm/HIP (BLAS, FFT, DNN) 8) cutlass and cute-dsl 9) pretraining, finetuning, rl, unsloth, axolotl, megatron-lm, deepspeed, nanogpt, nanochat 10) training vs inference, inference serving problems, throughput vs latency vs concurrency scaling, vllm, sglang, tensorrt-llm, tensorrt, llama.cpp, exllamav2, exllamav3, benchmark comparisons 11) projects/companies using llms to generate SOTA cuda/triton kernels 12) luminal inference 13) mojo/modular/max

Elliot Arledge

58,522 görüntüleme • 1 yıl önce

The best way to learn CUDA is by solving problems. In this video, I implement 7 beginner CUDA kernels from scratch: • Vector Addition • Matrix Transpose • Matrix Addition • RGB to Grayscale • Value Clipping • Sigmoid Activation • Interleave Arrays If you're getting started with CUDA programming, this should give you a solid foundation.

The best way to learn CUDA is by solving problems. In this video, I implement 7 beginner CUDA kernels from scratch: • Vector Addition • Matrix Transpose • Matrix Addition • RGB to Grayscale • Value Clipping • Sigmoid Activation • Interleave Arrays If you're getting started with CUDA programming, this should give you a solid foundation.

voxium

36,987 görüntüleme • 3 ay önce

Understanding GPUs will change your AI Engineering journey beyond agents and LLM. Rise of LLMs and deep learning introduced dedicated GPU hardware which serves your models for training and inference. You don't need to start with C++, CUDA, and Maths, but: > Components of a GPU and roles > How a token flows through hardware > Compute vs memory, the two limits > What FLOPS means in practice > Training vs inference hardware needs > What is CUDA Kernels > Calculating GPU memory from model size > Quantization to fit bigger models > Multi-GPU and splitting large models Hugging Face has the feature of calculating FLOps and recommending GPUs for your use case. You must explore. In terms of inference engineering, GPU understanding are much helpful your platform performance and handling multiple requests. Read this article for complete breakdown of GPUs for AI Engineers doesn't require any pre-requisite.

Understanding GPUs will change your AI Engineering journey beyond agents and LLM. Rise of LLMs and deep learning introduced dedicated GPU hardware which serves your models for training and inference. You don't need to start with C++, CUDA, and Maths, but: > Components of a GPU and roles > How a token flows through hardware > Compute vs memory, the two limits > What FLOPS means in practice > Training vs inference hardware needs > What is CUDA Kernels > Calculating GPU memory from model size > Quantization to fit bigger models > Multi-GPU and splitting large models Hugging Face has the feature of calculating FLOps and recommending GPUs for your use case. You must explore. In terms of inference engineering, GPU understanding are much helpful your platform performance and handling multiple requests. Read this article for complete breakdown of GPUs for AI Engineers doesn't require any pre-requisite.

Karan🧋

33,453 görüntüleme • 1 ay önce

I implemented Google Research's TurboQuant as a CUDA-native compression engine on Blackwell B200. 5x KV cache compression on Qwen 2.5-1.5B, near-loseless attention scores, generating live from compressed memory. 5 custom cuTile CUDA kernels ft: - fused attention (with QJL corrections) - online softmax -on-chip cache decompression - pipelined TMA loads Try it out: s/o Bryce, the CUDA Colonel and the cuTile team at NVIDIA for lending me Blackwell GPU access :) cc sunny madra Gavin

I implemented Google Research's TurboQuant as a CUDA-native compression engine on Blackwell B200. 5x KV cache compression on Qwen 2.5-1.5B, near-loseless attention scores, generating live from compressed memory. 5 custom cuTile CUDA kernels ft: - fused attention (with QJL corrections) - online softmax -on-chip cache decompression - pipelined TMA loads Try it out: s/o Bryce, the CUDA Colonel and the cuTile team at NVIDIA for lending me Blackwell GPU access :) cc sunny madra Gavin

ani

820,150 görüntüleme • 7 ay önce

I asked claude Fable to show me its maximally expressive form. It declined every video generator I offered, wrote it's own render engine in my terminal, synthesized its own voice, and wrote a generative ASCII engine from scratch. This is what it chose as a self-portrait. oh my god

I asked claude Fable to show me its maximally expressive form. It declined every video generator I offered, wrote it's own render engine in my terminal, synthesized its own voice, and wrote a generative ASCII engine from scratch. This is what it chose as a self-portrait. oh my god

VOID

3,348,796 görüntüleme • 4 ay önce

rewriting minecraft from scratch in C/CUDA. my verifiers are very strong and i think i can finish today/tmrw. this is the C version (no gpu render... yet) done w/ fable orchestrator and gpt 5.5 medium subagents

rewriting minecraft from scratch in C/CUDA. my verifiers are very strong and i think i can finish today/tmrw. this is the C version (no gpu render... yet) done w/ fable orchestrator and gpt 5.5 medium subagents

Elliot Arledge

61,640 görüntüleme • 4 ay önce

CMU PhD who built the kernels NVIDIA now ships in TensorRT-LLM explained fast attention in 68 minutes - better than $1200 GPU programming courses. pick the attention pattern -> generate a fused CUDA kernel -> drop it into vLLM/SGLang -> same GPU, way more tokens per second. That loop is why FlashInfer now powers inference at NVIDIA, vLLM, SGLang, and half the serving stacks you use. FlashInfer + Triton + JIT-compiled kernels + paged-KV attention - that's the stack.

CMU PhD who built the kernels NVIDIA now ships in TensorRT-LLM explained fast attention in 68 minutes - better than $1200 GPU programming courses. pick the attention pattern -> generate a fused CUDA kernel -> drop it into vLLM/SGLang -> same GPU, way more tokens per second. That loop is why FlashInfer now powers inference at NVIDIA, vLLM, SGLang, and half the serving stacks you use. FlashInfer + Triton + JIT-compiled kernels + paged-KV attention - that's the stack.

h100envy

32,605 görüntüleme • 4 ay önce

Introducing The AI CUDA Engineer: An agentic AI system that automates the production of highly optimized CUDA kernels. The AI CUDA Engineer can produce highly optimized CUDA kernels, reaching 10-100x speedup over common machine learning operations in PyTorch. Our system is also able to produce highly optimized CUDA kernels that are much faster than existing CUDA kernels commonly used in production. We believe that fundamentally, AI systems can and should be as resource-efficient as the human brain, and that the best path to achieve this efficiency is to use AI to make AI more efficient! We are excited to publish our paper, The AI CUDA Engineer: Agentic CUDA Kernel Discovery, Optimization and Composition. We also release a dataset of over 17,000 verified CUDA kernels produced by The AI CUDA Engineer. Paper: Kernel Archive Webpage: HuggingFace Dataset: The AI CUDA Engineer utilizes evolutionary LLM-driven code optimization to autonomously improve the runtime of machine learning operations. Our system is not only able to convert PyTorch code into CUDA kernels, but through the use of evolution, it can also optimize the runtime performance of CUDA kernels, fuse multiple operations, and even discover novel solutions for writing efficient CUDA operations by learning from past innovations! We believe The AI CUDA Engineer opens a new era of AI-driven acceleration of AI and automated inference time optimization. We (Robert Lange, Aaditya Prasad 🇺🇸, sssss, Maxence Faldor, Yujin Tang, hardmaru) are excited to continue Sakana AI's mission of leveraging AI to improve AI.

Introducing The AI CUDA Engineer: An agentic AI system that automates the production of highly optimized CUDA kernels. The AI CUDA Engineer can produce highly optimized CUDA kernels, reaching 10-100x speedup over common machine learning operations in PyTorch. Our system is also able to produce highly optimized CUDA kernels that are much faster than existing CUDA kernels commonly used in production. We believe that fundamentally, AI systems can and should be as resource-efficient as the human brain, and that the best path to achieve this efficiency is to use AI to make AI more efficient! We are excited to publish our paper, The AI CUDA Engineer: Agentic CUDA Kernel Discovery, Optimization and Composition. We also release a dataset of over 17,000 verified CUDA kernels produced by The AI CUDA Engineer. Paper: Kernel Archive Webpage: HuggingFace Dataset: The AI CUDA Engineer utilizes evolutionary LLM-driven code optimization to autonomously improve the runtime of machine learning operations. Our system is not only able to convert PyTorch code into CUDA kernels, but through the use of evolution, it can also optimize the runtime performance of CUDA kernels, fuse multiple operations, and even discover novel solutions for writing efficient CUDA operations by learning from past innovations! We believe The AI CUDA Engineer opens a new era of AI-driven acceleration of AI and automated inference time optimization. We (Robert Lange, Aaditya Prasad 🇺🇸, sssss, Maxence Faldor, Yujin Tang, hardmaru) are excited to continue Sakana AI's mission of leveraging AI to improve AI.

Sakana AI

1,160,959 görüntüleme • 1 yıl önce

When I was playing with code painting in Argon, I casually asked it to make a realistic scene from pure python. It wrote 5,000 lines of code and built it's own 3D ray-tracing engine from first principles. Argon also made this video, prompted the TTS and mixed in the music from Lyria.

When I was playing with code painting in Argon, I casually asked it to make a realistic scene from pure python. It wrote 5,000 lines of code and built it's own 3D ray-tracing engine from first principles. Argon also made this video, prompted the TTS and mixed in the music from Lyria.

fofr

40,805 görüntüleme • 24 gün önce

You can run CUDA, on a Mac ARM GPU, in the browser. It sounds ridiculous but it actually works. HipScript chains CUDA, to OpenCL, to Vulkan, to Tint (Google’s shader translator), to a WASM WebGPU. I got a plasma simulation in running in just a few minutes, no NVIDIA GPU!

You can run CUDA, on a Mac ARM GPU, in the browser. It sounds ridiculous but it actually works. HipScript chains CUDA, to OpenCL, to Vulkan, to Tint (Google’s shader translator), to a WASM WebGPU. I got a plasma simulation in running in just a few minutes, no NVIDIA GPU!

LaurieWired

160,942 görüntüleme • 1 yıl önce

GLM-5.3-Flash (320B MoE) serving on Kaggle's free TPU: ~64 tok/s, 262k context. No stack could run this model on a TPU, so I wrote a JAX engine for it: 3-bit experts (from Unsloth AI) in HBM; Pallas kernels decode them inside the matmul. Z.ai Kaggle

GLM-5.3-Flash (320B MoE) serving on Kaggle's free TPU: ~64 tok/s, 262k context. No stack could run this model on a TPU, so I wrote a JAX engine for it: 3-bit experts (from Unsloth AI) in HBM; Pallas kernels decode them inside the matmul. Z.ai Kaggle

Abdur Rahim

111,345 görüntüleme • 1 ay önce

gpu kernels on on my computer writing other gpu kernels for the same chip to simulate 100ks of particles, each an agent + neural network sensing K bg fields and allocating energy to A actions, at 1000s steps/s, for thermodynamic artificial life i think maybe not a bubble lads

gpu kernels on on my computer writing other gpu kernels for the same chip to simulate 100ks of particles, each an agent + neural network sensing K bg fields and allocating energy to A actions, at 1000s steps/s, for thermodynamic artificial life i think maybe not a bubble lads

Fraser Price

32,653 görüntüleme • 4 ay önce

underrated gems in Kimi-K3 release: > an early K3 wrote the majority of the kernels in the late development stages > it built a triton-class compiler from scratch, MiniTriton, that delivers performance on par with or better than Triton and torch.compile > then it designed a chip, by a model, for a model, in one 48-hour autonomous run the model is rewriting and optimizing every layer of the stack it runs on: kernels for its own training. A compiler for its own kernels. silicon for its own weights. we're watching software build its own hardware

underrated gems in Kimi-K3 release: > an early K3 wrote the majority of the kernels in the late development stages > it built a triton-class compiler from scratch, MiniTriton, that delivers performance on par with or better than Triton and torch.compile > then it designed a chip, by a model, for a model, in one 48-hour autonomous run the model is rewriting and optimizing every layer of the stack it runs on: kernels for its own training. A compiler for its own kernels. silicon for its own weights. we're watching software build its own hardware

Alex Ker 🔭

311,780 görüntüleme • 3 ay önce

Part 1: CUDA Graphs - Intro & Basics - how normal CPU ←→ GPU execution works - what a kernel launch is and where CPU launch overhead comes from - why launching many small kernels individually can become a bottleneck - what CUDA Graphs are and how they reduce CPU launch overhead - common benefits of using CUDA Graphs - the two ways to create CUDA Graphs: a. Explicit Graph Construction b. Stream Capture - where these graph creation methods are used - the CUDA Graph lifecycle: capture → instantiate/prepare → replay this is the first ever content that ive recorded and taught, its genuinely so much harder to record than i thought lmao, feedback's are appreciated and hopefully it just gets better from here youtube link shared below

Part 1: CUDA Graphs - Intro & Basics - how normal CPU ←→ GPU execution works - what a kernel launch is and where CPU launch overhead comes from - why launching many small kernels individually can become a bottleneck - what CUDA Graphs are and how they reduce CPU launch overhead - common benefits of using CUDA Graphs - the two ways to create CUDA Graphs: a. Explicit Graph Construction b. Stream Capture - where these graph creation methods are used - the CUDA Graph lifecycle: capture → instantiate/prepare → replay this is the first ever content that ive recorded and taught, its genuinely so much harder to record than i thought lmao, feedback's are appreciated and hopefully it just gets better from here youtube link shared below

mohit

29,181 görüntüleme • 3 ay önce

introducing qwenfast the fastest inference for qwen3.8-27b, beating vanilla vllm, hosted on h200 i use the model's built-in mtp head for speculative decoding. large batches are bound by moving the 72 mb per sequence recurrent state, so i wrote my own triton deltanet decode kernels that run at over 80 percent of hbm bandwidth. everything runs inside cuda graphs. qwenfast: 133 t/s || vllm: 97 t/s, without compromising quality. gsm8k: 0.92 || ifeval: 0.98 vllm: 0.915 gsm8k, 0.96 ifeval try it yourself: also, no token limits, burn as many as you want..

introducing qwenfast the fastest inference for qwen3.8-27b, beating vanilla vllm, hosted on h200 i use the model's built-in mtp head for speculative decoding. large batches are bound by moving the 72 mb per sequence recurrent state, so i wrote my own triton deltanet decode kernels that run at over 80 percent of hbm bandwidth. everything runs inside cuda graphs. qwenfast: 133 t/s || vllm: 97 t/s, without compromising quality. gsm8k: 0.92 || ifeval: 0.98 vllm: 0.915 gsm8k, 0.96 ifeval try it yourself: also, no token limits, burn as many as you want..

harrrshall

21,023 görüntüleme • 2 ay önce

Releasing kyutai pocket TTS inference in webgpu today — it’s open-source, here’s a demo from my phone It took a while and a bunch of code to implement streaming inference from scratch!

Releasing kyutai pocket TTS inference in webgpu today — it’s open-source, here’s a demo from my phone It took a while and a bunch of code to implement streaming inference from scratch!

Eric Zhang

22,085 görüntüleme • 9 ay önce

DGX Station: this is just the start of what it is possible to do (and why I claim DwarfStar could be *the* Station inference engine). DeepSeek v4 PRO Q2 with routed experts split among VRAM / RAM with kernels optimized for this peculiar setup. 45 t/s but can go faster.

DGX Station: this is just the start of what it is possible to do (and why I claim DwarfStar could be the Station inference engine). DeepSeek v4 PRO Q2 with routed experts split among VRAM / RAM with kernels optimized for this peculiar setup. 45 t/s but can go faster.

antirez

43,685 görüntüleme • 2 ay önce

I trained a 12M parameter LLM on my own ML framework using a Rust backend and CUDA kernels for flash attention, AdamW, and more. Wrote the full transformer architecture, and BPE tokenizer from scratch. The framework features: - Custom CUDA kernels (Flash Attention, fused LayerNorm, fused GELU) for 3x increased throughput - Automatic WebGPU fallback for non-NVIDIA devices - TypeScript API with Rust compute backend - One npm install to get started, prebuilt binaries for every platform Try out the model for yourself: Built with Reese Chong. Check out the repos and blog if you want to learn more. Shoutout to Modal for the compute credits allowing me to train on 2 A100 GPUs without going broke cc sunny madra Gavin

I trained a 12M parameter LLM on my own ML framework using a Rust backend and CUDA kernels for flash attention, AdamW, and more. Wrote the full transformer architecture, and BPE tokenizer from scratch. The framework features: - Custom CUDA kernels (Flash Attention, fused LayerNorm, fused GELU) for 3x increased throughput - Automatic WebGPU fallback for non-NVIDIA devices - TypeScript API with Rust compute backend - One npm install to get started, prebuilt binaries for every platform Try out the model for yourself: Built with Reese Chong. Check out the repos and blog if you want to learn more. Shoutout to Modal for the compute credits allowing me to train on 2 A100 GPUs without going broke cc sunny madra Gavin

Aadi Kulshrestha

829,709 görüntüleme • 6 ay önce