Reasoning from scratch round 2: In this video, I cover the text generation process in LLMs and KV caching (to prepare the base model before adding reasoning techniques in the upcoming ones). 00:00 Introduction and reasoning model demo 01:55 How to work through the book 05:00 Chapter 2 overview... 08:25 Checking PyTorch and hardware support 10:26 Apple silicon and MPS caveats 15:00 Cloud GPU options 16:08 Tokens and tokenization 18:20 Qwen3 and the Reasoning From Scratch package 23:05 Encoding and decoding text 26:24 Downloading weights and selecting a device 31:01 Loading the pretrained Qwen3 model 34:32 How LLMs generate text 36:47 Input tensors and batch dimensions 41:48 Running the model in inference mode 44:11 Logits and next-token predictions 49:21 Greedy decoding with argmax 52:28 Building a streaming text generator 01:01:28 Generating text and handling end-of-sequence tokens 01:06:00 Benchmarking text generation 01:14:34 How KV caching works 01:17:22 Adding KV caching and measuring the speedup 01:24:31 Model compilation with torch.compile 01:30:33 Combining compilation with KV caching 01:32:53 Comparing CPU and GPU performance 01:35:32 Recap and next stepsshow more

And here's the YouTube version:

Best public walkthrough of the week. This is how you turn GPUs into tokens people can actually use.

compile plus kv cache at 1:30 is where most reimplementations lose the win. a cache that grows one token per step changes tensor shapes every step, so torch compile recompiles or goes dynamic and the speedup evaporates. preallocate to max length, write by index.

Yes, I have another version with pre-allocated tensors in the bonus materials. Downside is it uses more memory on short queries, or users would have to commit to a max-length manually when instantiating.

qwen3 showing up in local demos keeps catching me off guard

compact and resource efficient

so good, thanks for sharing 🔥

Thankss for sharing, this seems so useful

the torch compile plus kv cache combo is exactly what i needed to see

Worth stating early for anyone following along: the KV cache is the memory budget, not the weights. 7B in 4-bit is 3.9 GB. Whether 8 GB then holds 7k tokens or 28k comes down to KV head count, and there's no parameter count in that equation.

On that note, I have an interactive memory calculator here:

That's the calculator I'd point people to. What it makes obvious once you move the sliders: the weight term is fixed and the cache term isn't. It scales with context and batch together, so the config that fits at batch 1 is exactly the one that OOMs in production.

KV caching setup is the less glamorous but essential part. Good to see that covered.

The chapter timestamps are such a thoughtful touch. It gives people learning around work or school a way to return to one idea without starting over. Thank you for making space for that kind of learning, Sebastian.

Yaaaaay , Raschka again🥳👏🏻

benchmarking KV caching next_

benchmarks are included :)

yeah_

KV caching first so later reasoning videos build on a working generator

Amazing work! Would love to see the next series explore reasoning with looped Transformers. I’m experimenting with this too, but with limited compute I’m looping only the middle layers. Interestingly, cos θ barely moves there, may hint that much of the reasoning happens there.

Thanks for suggesting. I think this wouldn't really change the reasoning methods (RLVR, inference scaling) in any way though; it's just an architecture tweak.

KV caching before reasoning is the right order, a reasoning model is mostly just a much longer decode loop

This was super helpful, thanks for putting it together. The KV caching part finally clicked for me. Looking forward to the next videos.

Nice, glad you found it helpful. In addition, I also have a from-scratch KV Cache article here if useful:

Hi Sebastian, thanks for sharing the KV cache article and the videos. They were really clear and helpful. I was wondering if you’re currently looking for anyone to help with from-scratch LLM or inference work, whether that’s implementation, tutorials, or related projects. I’d be happy to contribute if there’s an opportunity.

This is going to be quite busy.🤣

Round 2 of explaining KV caching while the actual reasoning techniques stay in 'upcoming.' Can't wait for Round 3 where we finally show the reasoning.

Lots of groundwork to be laid when doing things from scratch 😅

KV caching before reasoning is the bit I want to see. does the speedup hold once the reasoning loop gets long?

Thank you for all you do to spread LLM education. We love you 😘

KV caching + compile combo is the quiet win. Cuts my agent loops in half when I test locally.

Thank you for getting me hooked on reading and learning about the technical details of LLMs. I discovered your writing through threads in which you explained BERT models.

Oh BERT, good old times 😊

does the KV section cover what breaks over a long context?

KV caching shouldn't break over long context, the results are indistinguishable with and without KV caching; it's just a (big) performance improvement

right, I was putting KV caching in the wrong pile

🙌

KV caching is the part most from-scratch series skip, so this is the useful half. One check worth putting on camera: run the same prompt cached and uncached, then diff the token ids. When they disagree, the cache is where the numerics moved. Do yours agree?

KV cache is the part people skip until generation feels broken. Right order: make the base fast, then add reasoning.

Will this Qwen3 base model handle the KV cache memory growth better than the older versions? The speedup from KV caching is great but the memory pressure on consumer GPUs is the real headache for long context.

先把生成和 KV cache 讲透再上推理技巧，顺序很对。很多人一上来堆技巧，底座其实还没稳住。

Maybe speech recognition for the next video?

Thank you so much

KV cache before the reasoning tricks the unsexy chapter that actually decides if your “reasoner” ships or OOMs.

KV cache before the reasoning tricks — the unsexy chapter that decides if your “reasoner” ships or OOMs.

when the reasoning techniques land in the next videos, does the kv caching setup still hold or does it need rework

The exact same KV cache will be used in all future contents in this series. However, note that KV caching is not used for training portions of course (due to the nature of autoregressive training) but then again for inferencing with the trained model