e=mc²9 min read
Application-Level Distillation: Learn From a Bigger Model's Own Decisions
Lesson 09: a teacher model writes 420 realistic questions and labels them, 406 kept, for 49 cents; 183 join 20,000 templates. Three small models learn from them side by side: the from-scratch decider, BERT-mini with LoRA, and Qwen2.5-0.5B with LoRA. Coverage on teacher-written questions, before and after, with the precision that came with it.
2 views
Read