Loading video...

Video Failed to Load

Go Home

🧵1/2 Andrej Karpathy at GPU MODE workshop on llm.c! 👇Full Video

7etsuo's profile picture

tetsuo.ai 💹🧲

83,939 subscribers

16,177 views • 1 year ago •via X (Twitter)

News & PoliticsScience & TechnologyEducation
LIVE

Anya Rossi• Live Now

Private livecam show

8 Comments

tetsuo.ai 💹🧲's profile picture
tetsuo.ai 💹🧲1 year ago

LLMs in simple, pure C/CUDA: Neural Networks: Zero to Hero: Full Video:

UserInterface's profile picture
UserInterface4 years ago

Need Professional Video Production, Music Videos, Commercials, Graphic Design, or Photo Retouching? We will take your project from concept to completion. #services #creative #DMV

Daniel O'Leary's profile picture
Daniel O'Leary1 year ago

$TETSUO 💪

Bui Dinh Ngoc's profile picture
Bui Dinh Ngoc1 year ago

This is so cool!

tetsuo.ai 💹🧲's profile picture
tetsuo.ai 💹🧲1 year ago

yeah it is!

maxwellsdemon⏳'s profile picture
maxwellsdemon⏳1 year ago

he mentions at the end of the talk i think using LLMs as intermediate compilers which generate application specific llm.c files to accelerate workloads instead of relying on cuda/ptx (or high level apis like triton)

Jared / eacc's profile picture
Jared / eacc1 year ago

this is awesome

Jared / eacc's profile picture
Jared / eacc1 year ago

how can i explain these to simple people?.....

Related Videos

CUDA MODE hackathon today! Here's Andrej Karpathy on the 🏖️ origin story of llm.c, and what it hints at for the fast, simple, llm-compiled future of custom software.
23:28

CUDA MODE hackathon today! Here's Andrej Karpathy on the 🏖️ origin story of llm.c, and what it hints at for the fast, simple, llm-compiled future of custom software.

swyx

97,440 views • 1 year ago

Andrej Karpathy on how to fight entropy at startups:
1:48

Andrej Karpathy on how to fight entropy at startups:

Z Fellows

26,368 views • 1 year ago

Andrej Karpathy's (Andrej Karpathy) keynote yesterday at AI Startup School in San Francisco.
39:31

Andrej Karpathy's (Andrej Karpathy) keynote yesterday at AI Startup School in San Francisco.

Y Combinator

2,133,221 views • 1 year ago

Andrej Karpathy on Tensor Cores and TF32 precision
4:31

Andrej Karpathy on Tensor Cores and TF32 precision

ℏεsam

112,979 views • 1 year ago

flash attention explained by Andrej Andrej Karpathy
5:37

flash attention explained by Andrej Andrej Karpathy

ℏεsam

168,185 views • 1 year ago

56,000+ tokens/sec at just 80 MHz. 🤯 I burned a full Transformer with KV cache into a custom chip. Designed gate by gate as a 100% digital integrated circuit. Prototyped on a FPGA. (No GPU. No CPU) Just pure digital silicon running Andrej Karpathy microGPT, spelling out names on a tiny LCD. This is GateGPT 👇
0:24

56,000+ tokens/sec at just 80 MHz. 🤯 I burned a full Transformer with KV cache into a custom chip. Designed gate by gate as a 100% digital integrated circuit. Prototyped on a FPGA. (No GPU. No CPU) Just pure digital silicon running Andrej Karpathy microGPT, spelling out names on a tiny LCD. This is GateGPT 👇

Fabio Guzman

519,514 views • 2 days ago

Andrej @Karpathy explains on the Dwarkesh Patel podcast why he's not impressed by demos anymore Full episode:
4:28

Andrej @Karpathy explains on the Dwarkesh Patel podcast why he's not impressed by demos anymore Full episode:

Whole Mars Catalog

37,615 views • 3 months ago

Greg and Andrej Karpathy attemping to use grill at OpenAI 2017 offsite
0:28

Greg and Andrej Karpathy attemping to use grill at OpenAI 2017 offsite

Yaroslav Bulatov

817,989 views • 3 months ago

Flash Attention explained by Andrej Karpathy
5:35

Flash Attention explained by Andrej Karpathy

ℏεsam

105,390 views • 1 year ago

Andrej Karpathy (Andrej Karpathy) — co-founded OpenAI, led AI at Tesla, coined "vibe coding." In 4 minutes he explains why software is changing - and why Claude Skills, MCP servers, and AI agents aren't hype anymore. They're the foundation of how software gets built from now on. Imo, worth every second (i've added subtitles)👇
4:22

Andrej Karpathy (Andrej Karpathy) — co-founded OpenAI, led AI at Tesla, coined "vibe coding." In 4 minutes he explains why software is changing - and why Claude Skills, MCP servers, and AI agents aren't hype anymore. They're the foundation of how software gets built from now on. Imo, worth every second (i've added subtitles)👇

darkzodchi

453,653 views • 2 months ago

Andrej Karpathy admits he’s struggling with AI
8:02

Andrej Karpathy admits he’s struggling with AI

Mo

331,101 views • 28 days ago

Andrej Karpathy Professor Tolkien on what The Lord of The Rings is all about:
1:03

Andrej Karpathy Professor Tolkien on what The Lord of The Rings is all about:

𝐑𝐮𝐠𝐠𝐚

13,821 views • 10 months ago

I gave a talk at GPU MODE workshop last week on llm.c - the origin story of llm.c - being naked in the world without PyTorch and having to re-invent Array, Autograd, Device, Dtype, Compile, Distributed - how to port a PyTorch layer to 1) explicit PyTorch - and then to 2) write the backward pass - 3) port forward & backward pass to C - 4) string all the layers together - achieving one file of C with no dependencies that compiles and runs ~instantly, where all memory is pre-planned and allocated a single time, fully deterministic, portable code that can run on a potato or a von Neumann probe - how most of llm.c was built at 1am-7am in a water villa porch in Maldives and why this is the recommended way to develop software - convert all of it to run in CUDA on GPU in fp32 - port matmul to cuBLAS - port attention to cuDNN flash-attention - introduce bfloat16 mixed precision - introduce many more optimizations and features like kernel fusions, Packed128, stochastic rounding, full determinism - add multi-GPU training, NCCL, sharded optimizer - add multi-node with MPI or file system or socket - reproduce GPT-2 (1.6B) on one 8XH100 node in 24 hours for $672 in llm.c, achieving (at the time) 29% less memory, 19% faster training that PyTorch nightly, and much faster compile & run - how open source development attracts Avengers from the internet - port to training Llama 3 imminent (branch exists) - many other notable forks - last thought: how software abstractions like Python/PyTorch and everything else really exist only because humans are finite in knowledge, IQ and attention, and how with increasing AI capability LLMs may export custom binaries like llm.c for any application directly, tearing apart and refactoring all abstractions as needed. More links in reply
23:16

I gave a talk at GPU MODE workshop last week on llm.c - the origin story of llm.c - being naked in the world without PyTorch and having to re-invent Array, Autograd, Device, Dtype, Compile, Distributed - how to port a PyTorch layer to 1) explicit PyTorch - and then to 2) write the backward pass - 3) port forward & backward pass to C - 4) string all the layers together - achieving one file of C with no dependencies that compiles and runs ~instantly, where all memory is pre-planned and allocated a single time, fully deterministic, portable code that can run on a potato or a von Neumann probe - how most of llm.c was built at 1am-7am in a water villa porch in Maldives and why this is the recommended way to develop software - convert all of it to run in CUDA on GPU in fp32 - port matmul to cuBLAS - port attention to cuDNN flash-attention - introduce bfloat16 mixed precision - introduce many more optimizations and features like kernel fusions, Packed128, stochastic rounding, full determinism - add multi-GPU training, NCCL, sharded optimizer - add multi-node with MPI or file system or socket - reproduce GPT-2 (1.6B) on one 8XH100 node in 24 hours for $672 in llm.c, achieving (at the time) 29% less memory, 19% faster training that PyTorch nightly, and much faster compile & run - how open source development attracts Avengers from the internet - port to training Llama 3 imminent (branch exists) - many other notable forks - last thought: how software abstractions like Python/PyTorch and everything else really exist only because humans are finite in knowledge, IQ and attention, and how with increasing AI capability LLMs may export custom binaries like llm.c for any application directly, tearing apart and refactoring all abstractions as needed. More links in reply

Andrej Karpathy

335,861 views • 1 year ago

Just posted video of my talk on "DeFi in the MEV Era" at the recent @Paradigm research workshop. 🧵
35:46

Just posted video of my talk on "DeFi in the MEV Era" at the recent @Paradigm research workshop. 🧵

ciamac moallemi

19,634 views • 1 year ago

Recent from No Priors with Andrej Karpathy on Tesla and Waymo in self-driving🤔
1:48

Recent from No Priors with Andrej Karpathy on Tesla and Waymo in self-driving🤔

Elad Gil

32,448 views • 1 year ago

Andrej Karpathy (OpenAI co-founder) on AI's weirdest blind spot: jokes haven't improved in 4 years.
0:29

Andrej Karpathy (OpenAI co-founder) on AI's weirdest blind spot: jokes haven't improved in 4 years.

Big Brain AI

39,304 views • 1 month ago

we got Andrej Karpathy to drive our self-driving golf cart!
3:13

we got Andrej Karpathy to drive our self-driving golf cart!

Georg von Manstein

132,476 views • 1 month ago

Inspired by Andrej Karpathy :) A local markdown editor + terminal agent workspace.
4:38

Inspired by Andrej Karpathy :) A local markdown editor + terminal agent workspace.

killian

95,424 views • 1 month ago

Andrej featuring Hyperbolic in his latest video was not on my bingo card! 🎉 "My favorite place to interact with the base models is the company called Hyperbolic." Thank you Andrej Karpathy for using and mentioning us! Other exciting news: - We surpassed 100k users - Since our GPU market launch in late December, nearly all H100s have been rented out, driving strong ARR growth.
0:20

Andrej featuring Hyperbolic in his latest video was not on my bingo card! 🎉 "My favorite place to interact with the base models is the company called Hyperbolic." Thank you Andrej Karpathy for using and mentioning us! Other exciting news: - We surpassed 100k users - Since our GPU market launch in late December, nearly all H100s have been rented out, driving strong ARR growth.

Yuchen Jin

104,768 views • 1 year ago