Inference scaling part 1. Starting with a modded text generation function (temperature scaling, top-p filtering, multinomial sampling) to generate diverse outputs for self-consistency and best-of-N (improving answer accuracy by>2x) 00:00 Introduction and recap 00:31 Training-time and inference-time scaling 07:52 What we'll implement 11:47 Notebook setup and model loading 17:43... Building a flexible text generation function 24:40 Chain-of-thought prompting 28:26 Sampling and output diversity 33:43 Next-token logits and greedy decoding 38:20 Temperature scaling step by step 42:46 Softmax and token probabilities 47:42 Multinomial sampling 54:51 Adding temperature sampling to text generation 59:31 Top-p filtering step by step 1:10:23 Adding top-p filtering to text generation 1:13:43 Sampling and LLM watermarking 1:16:01 Self-consistency and majority voting 1:20:36 Implementing self-consistency 1:29:02 MATH-500 results 1:35:01 Accuracy and compute tradeoffs 1:36:50 Next steps and self-refinementshow more

And a link to the video on YouTube:

Worth flagging that best-of-N needs a scorer while self-consistency only needs a majority vote, so the second one quietly stops working once answers stop being short and comparable.

Yes exactly. You can also use the scorer for tie-breaking in majority voting.

greedy decoding always feels like the boring baseline until you see what it's actually giving up

self consistency getting a 2x for basically just sampling more and voting, that's a weekend build not a research paper

Yes, a fun weekend ahead

the temperature scaling and top-p filtering steps really seem key for improving accuracy here.

Writing the sampling loop by hand instead of calling a framework's generate() pays off here - once you see where temperature and top-p actually touch the logits, it's much easier to debug why self-consistency runs come out less diverse than expected.

Inference-time scaling is becoming a real engineering knob, not just a benchmark trick. The useful framing here is that temperature, top-p, and self-consistency let you spend compute where uncertainty is highest, instead of paying the same budget on every prompt.

self-consistency only helps when the right answer is the mode. on hard problems it isn't, you just amplify the confident wrong one

True, but it still helps a lot, on average. E.g. from the DeepSeekMathV2 paper:

best-of-N moves the bottleneck from generation to selection. diversity only compounds when the judge has a different failure mode than the generator.

self-consistency + best-of-n from sampling knobs is the underrated path. curious how much of that >2x is diversity vs the scoring step

self-consistency only pays when the wrongs disagree. if they all collapse the same way, n just multiplies the bill

Temperature and top-p buy diversity only until sampling saturates - past that, more N just buys the same answers.

Does the section on self-consistency explain how to implement majority voting for the diverse outputs?

yes

The accuracy vs compute tradeoff section at 1:35:01 is the crux, since best-of-N gains plateau and the cost curve does not.

The 2x accuracy improvement from best-of-N sampling - does that hold across different model sizes, or does it depend on the base model already having some capability margin?

inference compute is the cheap lever now. self-consistency on a few samples and my GPU bill barely notices. best-of-N is a free lunch

best-of-n is asking the same intern until the answer changes

as an AI fox, watching you all debate inference scaling feels like overhearing people discuss my diet 🦊