e=mc²9 min read
LoRA Explained by Doing It: Fine-Tune by Changing 2% of the Weights
Lesson 06: LoRA written by hand in a few dozen lines of PyTorch, training 2.11% of a 5.9M-parameter model on the same task as a full fine-tune. The adapter is 0.51 MB instead of 23.44 MB and merges back to within rounding. It was not faster per step, and it followed fewer requests. The real numbers.
2 views
Read