Full Fine-Tuning vs LoRA by hand ✍️ interactive diagram. Open Here's some T/F questions to test your knowledge: [ ] Growing the batch size grows the number of trainable parameters [ ] Both rows produce an output of the same shape [ ] LoRA's ΔW is a smaller matrix... than a full fine-tune's [ ] Raising the rank leaves full fine-tuning's count untouched [ ] A high enough rank makes LoRA cost more than a full fine-tune Check your answers by dragging the diagram. ~ Prof. Tom Yeh Answers: F, T, F, T, Tshow more

Agreed🙌

That's interesting, rank is the dial people underrate. We keep leaning on r=2 in places and it holds up far better than the maths suggests it should.

Answers, in order: false, true. Batch size changes gradients per step, not the count of trainable parameters. And LoRA's delta W is BA, rank r, so it lands in the same shape as W while training a fraction of the weights.

Love the interactive approach. LoRA becomes much easier to understand once you can visually see how rank changes the parameterization and training cost.

And thire is optimize version of Lora called QLoRA that used 4 bit quantized rather than 16 bits in LoRa