Sebastian Raschka
@rasbt • 537,351 subscribers
ML/AI research engineer. Ex stats professor. Author of "Build a Large Language Model From Scratch" (https://t.co/O8LAAMRzzW) & reasoning (https://t.co/5TueQKx2Fk)
Videos
Reasoning from scratch, round number 6! An introduction (and implementation) of Reinforcement Learning with Verifiable Rewards (RLVR) and Group Relative Policy Optimization (GRPO). 00:00 Introduction 01:54 What makes a reasoning model different? 04:25 Reasoning traces and model capability 08:29 Accuracy and format rewards 11:34 Aha moments and DeepSeek-R1 training 14:41 Reasoning effort and answer length 18:38 RLHF and RLVR 23:04 GRPO vs. PPO 26:40 GRPO explained with a cooking analogy 31:43 The KL term and simplified GRPO 35:04 Loading the pretrained model 36:07 Loading the MATH training data 39:26 Sampling model responses 46:30 Computing verifiable rewards 49:55 Computing advantages 51:54 Token and sequence log probabilities 55:29 Implementing sequence log probabilities 57:37 Fixing the inference-mode error 1:02:24 Computing the GRPO loss 1:04:37 Putting the GRPO step together 1:09:19 The GRPO training loop 1:12:57 Training settings, logging, and checkpoints 1:17:24 Running training and inspecting outputs 1:19:28 Loading and evaluating checkpoints 1:22:33 MATH-500 results and training stability 1:24:05 Memory requirements and next steps
Sebastian Raschka76,474 просмотров • 1 месяц назад
Reasoning From Scratch: Reinforcement Learning with Verifiable Rewards (RLVR) round 2. Covering clipped policy ratios, KL loss term, format rewards, and other GRPO tips & tricks. 00:00 Introduction and recap 01:52 Interpreting basic GRPO training metrics 06:34 Planned improvements to GRPO 08:58 Running longer training jobs with Python scripts 13:39 Running the baseline GRPO training script 17:29 Loading and plotting training logs 19:29 Diagnosing unstable training 23:55 Evaluating checkpoints on MATH-500 26:26 Downloading existing checkpoints 30:09 Tracking advantage statistics 34:53 Understanding entropy 40:32 Computing entropy in PyTorch 44:17 Interpreting entropy values 48:58 Adding entropy tracking to GRPO 53:36 Analyzing advantage and entropy metrics 56:18 Stabilizing GRPO with clipped policy ratios 1:03:27 Implementing the clipped policy loss 1:09:39 Analyzing clipped policy training results 1:11:25 KL divergence and reward hacking 1:15:12 Adding a KL loss term 1:20:34 Limitations of the simplified KL loss 1:23:04 Format rewards and think tags 1:25:47 Adding special tokens to the tokenizer 1:30:29 Implementing the format reward 1:35:56 Analyzing format reward training 1:38:25 Rewarding format only for correct answers 1:40:48 Further GRPO improvements from research 1:45:43 Next steps and distillation
Sebastian Raschka60,891 просмотров • 1 месяц назад
Reasoning From Scratch round 8, the finale! This last lecture in the series covers distillation as an efficient alternative to the previous reinforcement learning training runs. (Planning to take a brief break, but I will be back!) 00:00 Introduction and series recap 01:07 Why distill a reasoning model? 02:51 Soft vs. hard distillation 11:06 On-policy vs. off-policy distillation 18:09 Preparing data for reasoning distillation 22:35 Loading the DeepSeek R1 MATH dataset 28:04 Inspecting reasoning traces and final answers 31:27 Tokenization and chat templates 35:58 Pre-tokenizing the training dataset 39:01 Sequence lengths and the 2,048-token limit 42:38 Creating training and validation splits 45:32 Loading the Qwen3-0.6B base model 46:24 Computing answer-token log probabilities 54:10 Cross-entropy loss for distillation 57:25 Training and validation loss helpers 1:02:03 Implementing the distillation training loop 1:07:26 Running the standalone training script 1:09:41 Plotting training and validation losses 1:11:31 Evaluating checkpoints on MATH-500 1:14:09 Comparing teachers and distillation results 1:16:34 Future directions for reasoning models 1:19:41 Bonus exercises and next steps
Sebastian Raschka39,937 просмотров • 27 дней назад
Reasoning from scratch, round number 5! This time, talking about log-probability scoring (also a great fundamental concept for loss functions like cross-entropy in pre-training and distillation) and self-refinement. 00:00 Introduction and inference-time scaling recap 05:02 Loading the pretrained LLM 08:00 Comparing and scoring model answers 10:18 Building a rule-based scorer 17:53 Token probabilities and sequence likelihood 26:47 Computing token probabilities in PyTorch 30:12 Token indexing and shifted targets 37:27 Log probabilities and numerical stability 45:57 Scoring answers with average log probabilities 56:24 How self-refinement works 59:07 Generating critiques and revised answers 1:01:00 Implementing the self-refinement loop 1:05:57 MATH-500 evaluation results 1:07:35 Takeaways and next steps
Sebastian Raschka59,636 просмотров • 1 месяц назад
Reasoning from scratch round 2: In this video, I cover the text generation process in LLMs and KV caching (to prepare the base model before adding reasoning techniques in the upcoming ones). 00:00 Introduction and reasoning model demo 01:55 How to work through the book 05:00 Chapter 2 overview 08:25 Checking PyTorch and hardware support 10:26 Apple silicon and MPS caveats 15:00 Cloud GPU options 16:08 Tokens and tokenization 18:20 Qwen3 and the Reasoning From Scratch package 23:05 Encoding and decoding text 26:24 Downloading weights and selecting a device 31:01 Loading the pretrained Qwen3 model 34:32 How LLMs generate text 36:47 Input tensors and batch dimensions 41:48 Running the model in inference mode 44:11 Logits and next-token predictions 49:21 Greedy decoding with argmax 52:28 Building a streaming text generator 01:01:28 Generating text and handling end-of-sequence tokens 01:06:00 Benchmarking text generation 01:14:34 How KV caching works 01:17:22 Adding KV caching and measuring the speedup 01:24:31 Model compilation with torch.compile 01:30:33 Combining compilation with KV caching 01:32:53 Comparing CPU and GPU performance 01:35:32 Recap and next steps
Sebastian Raschka68,300 просмотров • 2 месяцев назад
Thanks everyone for all the kind words and feedback. Super happy that you are enjoying Build a Reasoning Model (From Scratch)! Unfortunately, there's small typo in listing 6.5 on page 198 (see video below). The line "torch.manual_seed(0)" should be "torch.manual_seed(5)" This correction is needed to reproduce the generated response in listing 6.5 and the corresponding log-probability outputs later in Chapter 6. If you use 0, the generated response and the results that follow will be different. This will be fixed in the next printing. I am sorry about the oversight, and I hope this note saves you some debugging time.
Sebastian Raschka106,496 просмотров • 3 месяцев назад
Inference scaling part 1. Starting with a modded text generation function (temperature scaling, top-p filtering, multinomial sampling) to generate diverse outputs for self-consistency and best-of-N (improving answer accuracy by>2x) 00:00 Introduction and recap 00:31 Training-time and inference-time scaling 07:52 What we'll implement 11:47 Notebook setup and model loading 17:43 Building a flexible text generation function 24:40 Chain-of-thought prompting 28:26 Sampling and output diversity 33:43 Next-token logits and greedy decoding 38:20 Temperature scaling step by step 42:46 Softmax and token probabilities 47:42 Multinomial sampling 54:51 Adding temperature sampling to text generation 59:31 Top-p filtering step by step 1:10:23 Adding top-p filtering to text generation 1:13:43 Sampling and LLM watermarking 1:16:01 Self-consistency and majority voting 1:20:36 Implementing self-consistency 1:29:02 MATH-500 results 1:35:01 Accuracy and compute tradeoffs 1:36:50 Next steps and self-refinement
Sebastian Raschka44,580 просмотров • 1 месяц назад
Reasoning from scratch round 3: This time, I cover generating a verifier for... a) ...evaluation (base model versus any future model improvement) b) ...the reinforcement learning with verifiable rewards (RLVR) training later on 00:00 Introduction 01:21 Four approaches to LLM evaluation 07:20 Verifiers and reinforcement learning with verifiable rewards 10:52 Notebook setup and dependencies 13:43 Section 3.1 Building a math verifier 18:57 Section 3.2 Loading a pre-trained model to generate text 24:34 Generating and displaying model answers 29:23 Section 3.3 Implementing a wrapper for easier text generation 34:00 Section 3.4 Extracting the final answer box 37:29 Handling answers without boxes 43:17 Section 3.5 Normalizing the extracted answer 46:56 Section 3.6 Verifying mathematical equivalence 53:32 Implementing the equality check 57:48 Section 3.7 Grading answers 59:20 Building and testing the answer grader 1:03:18 Section 3.8 Loading the evaluation dataset (MATH-500) 1:07:51 Section 3.9 Evaluating the model 1:08:34 Prompt templates for evaluation 1:10:47 Prompt sensitivity and memorization 1:13:55 A minimal evaluation example 1:15:32 Building the evaluation loop 1:20:27 Comparing CPU, MPS, and CUDA results 1:21:54 Reproducibility and floating-point math 1:23:37 Base model vs. reasoning model 1:25:30 Summary and next steps
Sebastian Raschka43,229 просмотров • 2 месяцев назад
Updated & turned my Big LLM Architecture Comparison article into a narrated video lecture. The 11 LLM architectures covered in this video: 1. DeepSeek V3/R1 2. OLMo 2 3. Gemma 3 4. Mistral Small 3.1 5. Llama 4 6. Qwen3 7. SmolLM3 8. Kimi 2 9. GPT-OSS 10. Grok 2.5 11. GLM-4.5
Sebastian Raschka200,075 просмотров • 1 год назад
Some food for thought when designing benchmarks... So, here's a little computer-use (visual) comparison between GPT-5.6 Astra and Qwen3.8 Max. The task here was to recreate the image in the center using the Paint UI. Super interesting how the two different LLMs+Harnesses approached this totally differently by default. I.e., Astra tried to approach this by drawing and layering geometric shapes. Qwen approached this pixel by pixel. (Of course, the pixel-by-pixel result looks closer to the original, it's essentially a low-res version of that by nature.) So, the Qwen-generated image would surely score higher in the sense that it's closer to the original. But I wouldn’t conclude from this example that one LLM generalizes better than the other on other tasks. Also, I wouldn't say Qwen has better compute-use capabilities or better visual understanding than Astra. But it highlights an interesting point about how slippery benchmarks are when they only compare final results.
Sebastian Raschka25,821 просмотров • 1 месяц назад
Claude distillation has been a big topic this week while I am (coincidentally) writing Chapter 8 on model distillation. In that context, I shared some utilities to generate distillation data from all sorts of open-weight models via OpenRouter and Ollama:
Sebastian Raschka63,331 просмотров • 8 месяцев назад
Excited to be launching a series of short videos (5 min or less) to optimize the training performance of your PyTorch models! Covering - mixed-precision training - multi-GPU training strategies - Compiling PyTorch models - and finding good batch sizes 🔗
Sebastian Raschka120,844 просмотров • 3 лет назад
When doing machine learning and AI research (or writing books), making the code reproducible is usually desirable. Often, that's easier said than done! So, I recorded a video illustrating and dealing with 6 sources of randomness that occur when training deep neural networks and LLMs: 1. Model weight initialization 2. Dataset sampling and shuffling 3. Nondeterministic algorithms 4. Different runtime algorithms 5. Hardware and drivers 6. Randomness in generative AI models
Sebastian Raschka81,713 просмотров • 2 лет назад
I just added the new Llama 3.2 1B and 3B models to LitGPT, the open-source LLM library I help develop (focused on efficiency and code readability). LitGPT allows you to fine-tune and use these models on the cloud or a laptop. So, if you are looking for something to play with this weekend: # 1) Finetune the model litgpt finetune_lora meta-llama/Llama-3.2-1B \ --data JSON \ --data.json_path my_custom_dataset.json \ --train.epochs 1 \ --out_dir out/llama-3.2-finetuned \ --precision bf16-true # 2) Chat with the model litgpt chat out/llama-3.2-finetuned/final # 3) Serve the model via an API endpoint litgpt serve out/llama-3.2-finetuned/final
Sebastian Raschka65,529 просмотров • 2 лет назад
Больше нет контента для загрузки