A learning project to implement the path from text tokenization to autoregressive training, while keeping the questions, failed assumptions, and verification evidence visible.
Write GPT-2 myself in PyTorch, one piece at a time, and understand what each part does. Whether it is right is checked by loading OpenAI's official GPT-2 124M weights and comparing its output with the reference implementation. Training stays small: it only has to run and show the loss going down, not reproduce the original results.
Development plan
01
In progress
Foundations and tokenizer
Text representation, byte-level BPE, vocabulary, and reversible encoding.
02
Planned
Data pipeline
Sequence windows, batching, targets, and deterministic data loading.
03
Planned
Transformer model
Embeddings, causal attention, MLP blocks, normalization, and residual paths.
04
Planned
Training and evaluation
Optimization, checkpoints, sampling, evaluation, and failure analysis.
Entries
As the project develops, I will add the questions, implementation notes, and test results from each stage here.
00
Warm-up
Published
Stage 0: a character-level bigram warm-up
Writing a character-level bigram language model from scratch: encoding characters, cutting out training examples, scoring by table lookup, cross-entropy, generation and the training loop. The code is explained line by line, along with the questions that came up while learning.
Before writing the tokenizer: what actually becomes a token?
A preview record for the first conceptual boundary in the project: the relationship between Unicode text, UTF-8 bytes, BPE merges, and the token IDs received by the model.