the learn-by-building vertical
Every next word = model × context²
Language models built from nothing and explained as they are built. Each lesson is a post you can read, a notebook you can run, and a model you can keep.
In order, 01 to 10. Each one is a complete build with its real numbers, written so that someone who has never trained anything can follow it.
Lesson 01: a 5.9-million-parameter GPT trained from random weights on children's stories: 1,500 steps in 1.2 minutes of training on a free Colab GPU (the whole notebook is about 5 minutes). Every piece explained through Momo the monkey and Ella the elephant, with the real numbers and an honest look at what it wrote.
Lesson 02: open up the 5.9-million-parameter model from Lesson 01 and read what all 24 attention heads look at. Query, key and value in plain words, the causal mask, why the KV cache exists, and the head that puts the most attention on a story's main character's name.
Lesson 03: the same small language model trained at 0.95M, 3.06M, 5.85M and 12.3M parameters on the same stories. Quiz perplexity falls from 16.0 to 7.4, and verbatim recitation of training stories switches on somewhere between the first and second size. Real numbers, and what they do not prove.
Lesson 04: fine-tune the 5.9M-parameter Tiny Storyteller on 300 stories about a new character. Its old quiz perplexity jumps from 9.09 to 156. Mixing two old windows into every batch of twenty holds it at 11.75. Real numbers, and what the model got wrong.
Lesson 05: teach a 5.9M-parameter story model to follow a request, with two special tokens, a chat template and prompt masking. On 50 held-out requests, the share of stories with the right character and place went from 12% to 44%. Here is what it learned and what it did not.
Lesson 06: LoRA written by hand in a few dozen lines of PyTorch, training 2.11% of a 5.9M-parameter model on the same task as a full fine-tune. The adapter is 0.51 MB instead of 23.44 MB and merges back to within rounding. It was not faster per step, and it followed fewer requests. The real numbers.
Lesson 07: the Lesson 01 GPT, shrunk to 0.83M parameters, writes one token after a question: the operation that answers it. Beside it, pankhllm's hashed logistic regression. On phrasings neither saw, a threshold alone holds 98% precision for neither; with pankhllm's known-words guard, the transformer does, at 41.2% coverage.
Lesson 08: the tiny decider learns to write the whole request as JSON, operation and parameters, and a grammar built from the catalog constrains every token. Validity goes to 100%. The number that matters is the one a grammar cannot fix: answers that are valid and wrong.
Lesson 09: a teacher model writes 420 realistic questions and labels them, 406 kept, for 49 cents; 183 join 20,000 templates. Three small models learn from them side by side: the from-scratch decider, BERT-mini with LoRA, and Qwen2.5-0.5B with LoRA. Coverage on teacher-written questions, before and after, with the precision that came with it.
Lesson 10: int8 quantization of the Stage C deciders, CPU latency and size, a stdlib server speaking pankhllm's decision shape, and a real run through pankhllm's own binary. The default int8 broke Qwen; the small decider got smaller and slower. And the contract, applied honestly: neither lane earns its place yet.
Ten lessons in three stages, from a storyteller built from nothing to a decision model for a real system. Each lesson loads what the one before it built. The first ten were built over one weekend from the Lesson 01 notebook, each one run and graded before it went on the board.
pankhllm is an LLM gateway that learns which calls never needed a model. Today it decides with a 262 KB logistic regression. The last four lessons train a small transformer for the same job and put it in the same table, on the same public template and benchmark questions, plus a teacher-written set of our own. These are the numbers to beat, from its published benchmark:
| Held-out set | Questions | Decided without a model | Precision | Decision p50 |
|---|---|---|---|---|
| Templates | 2,000 | 63.3% | 100% | 0.21 ms |
| Teacher-written A | 82 | 67.1% | 100% | 0.36 ms |
| Teacher-written B, held-out half | 184 | 61.4% | 96.5% | 0.23 ms |
The question the journey answers: can a transformer I trained myself decide more of the questions people actually type, at 98% precision or better, and at what cost in milliseconds and megabytes. If it cannot, that is the final lesson, with the numbers.
Build one from nothing, then open it up and look.
The whole loop: tokens, attention blocks, loss, AdamW, sampling.
What each head looks at, the causal mask, and why Ella's shelf exists.
Capacity against data; training loss against held-out loss, and why the overfitting gap never appears while recitation does.
The skill the journey is for, learned on a model small enough to see.
Continued training on new data; forgetting as a number, not a word; replay mixing.
Instruction and output pairs, a chat template, and masking the prompt out of the loss.
Low-rank adapters, what rank buys, adapters as swappable files.
pankhllm decides with a 262 KB logistic regression today. Its docs name the missing piece: a checkpoint fine-tuned to the catalog. These lessons build it and measure it against the baseline on the same public template and benchmark questions, and on a teacher-written set of our own.
Classification with an LLM: the next token is the answer; the UNSUPPORTED class; abstaining on a threshold.
Slot filling as generation; constrained decoding against the catalog; why validity is not precision: the structural check buys the first, grounding the enum words buys the second.
Teacher labels as training data; a few hundred real phrasings against 20,000 templates; from-scratch against two pretrained small models with LoRA, one BERT-class encoder and one Qwen-class decoder.
int8 export, CPU latency, size, and the serve-time contract: the lane must hold 98% precision.
Three forms, one build. You can stop at any of them.
A post on this page that tells the whole build in plain words, every strange term decoded the first time it appears, with the real numbers from the run.
A notebook that builds the same thing cell by cell. Lesson 01 runs top to bottom in about five minutes on a free Google Colab GPU, or on a Mac with Apple silicon.
The trained model, its tokenizer and a model card on the Hugging Face Hub (Lesson 01 is public now; later repos open as they are reviewed), so you can load what you built on another machine, or publish your own copy under your name.
The formula on the door, read as a sentence about language models.
A language model does one thing: given the words so far, it guesses the next one. A story, an answer, a whole chat is that guess repeated. E is the output.
A pile of dials and a fixed recipe for using them. Lesson 01's model has 5.9 million dials; the famous ones have hundreds of billions. Training turns the dials, a little at a time, until the guesses get good.
Attention lets every word look at every word before it and decide how much to listen. Compare each word with each word and the work grows with the square of the context length. That square is why long contexts are expensive, and why a model can connect a word to one it read a page ago.
What the first notebook builds, in order. Later lessons split some of these parts into lessons of their own.
| Part | What you build | Why |
|---|---|---|
| 0 | Momo, Ella, and a word list | Every idea and every strange word in plain language before any code |
| 1 | Tensors and gradients | The two things every neural network is made of |
| 2 | Data: TinyStories, plus 1,000 Momo stories | Simple English stories, made for small models |
| 3 | A bigram model | The simplest language model that can write anything |
| 4 | A real tokenizer (BPE) | Pieces of words instead of letters |
| 5 | Attention, by hand | The idea that makes transformers work, plus the KV cache |
| 6 | A tiny GPT | Stack the pieces into a full model |
| 7 | Training | Watch the loss fall; learn what perplexity means |
| 8 | Telling a story | Temperature, top-k, and an honest look at what it wrote |
| 9 | Publish to the Hub | Weights, config, tokenizer and a model card |
| 10 | Load it back later | Rebuild the model anywhere from the Hub |
Subscribe to e=mc² lessons