Skip to content

Tag

Tokenizers

2 posts

Instruction Tuning From Scratch: Turn a Story Generator Into a Prompt Follower
e=mc²9 min read

Instruction Tuning From Scratch: Turn a Story Generator Into a Prompt Follower

Lesson 05: teach a 5.9M-parameter story model to follow a request, with two special tokens, a chat template and prompt masking. On 50 held-out requests, the share of stories with the right character and place went from 12% to 44%. Here is what it learned and what it did not.

2 views
Read
Build an LLM From Scratch on a Free GPU in 5 Minutes: Tiny Storyteller
e=mc²9 min read

Build an LLM From Scratch on a Free GPU in 5 Minutes: Tiny Storyteller

Lesson 01: a 5.9-million-parameter GPT trained from random weights on children's stories: 1,500 steps in 1.2 minutes of training on a free Colab GPU (the whole notebook is about 5 minutes). Every piece explained through Momo the monkey and Ella the elephant, with the real numbers and an honest look at what it wrote.

2 views
Read