Skip to content

Tag

PEFT

2 posts

Application-Level Distillation: Learn From a Bigger Model's Own Decisions
e=mc²9 min read

Application-Level Distillation: Learn From a Bigger Model's Own Decisions

Lesson 09: a teacher model writes 420 realistic questions and labels them, 406 kept, for 49 cents; 183 join 20,000 templates. Three small models learn from them side by side: the from-scratch decider, BERT-mini with LoRA, and Qwen2.5-0.5B with LoRA. Coverage on teacher-written questions, before and after, with the precision that came with it.

2 views
Read
LoRA Explained by Doing It: Fine-Tune by Changing 2% of the Weights
e=mc²9 min read

LoRA Explained by Doing It: Fine-Tune by Changing 2% of the Weights

Lesson 06: LoRA written by hand in a few dozen lines of PyTorch, training 2.11% of a 5.9M-parameter model on the same task as a full fine-tune. The adapter is 0.51 MB instead of 23.44 MB and merges back to within rounding. It was not faster per step, and it followed fewer requests. The real numbers.

2 views
Read