Project 01

Building GPT-2 from scratch

A learning project to implement the path from text tokenization to autoregressive training, while keeping the questions, failed assumptions, and verification evidence visible.

Status
In progress
Started
September 2026
Planned tools
Python · PyTorch · NumPy · GPT-2 reference implementation

What I want to learn

Write GPT-2 myself in PyTorch, one piece at a time, and understand what each part does. Whether it is right is checked by loading OpenAI's official GPT-2 124M weights and comparing its output with the reference implementation. Training stays small: it only has to run and show the loss going down, not reproduce the original results.

Development plan

  1. 01
    In progress

    Foundations and tokenizer

    Text representation, byte-level BPE, vocabulary, and reversible encoding.

  2. 02
    Planned

    Data pipeline

    Sequence windows, batching, targets, and deterministic data loading.

  3. 03
    Planned

    Transformer model

    Embeddings, causal attention, MLP blocks, normalization, and residual paths.

  4. 04
    Planned

    Training and evaluation

    Optimization, checkpoints, sampling, evaluation, and failure analysis.

Entries

As the project develops, I will add the questions, implementation notes, and test results from each stage here.

00

Warm-up

Published

Stage 0: a character-level bigram warm-up

Writing a character-level bigram language model from scratch: encoding characters, cutting out training examples, scoring by table lookup, cross-entropy, generation and the training loop. The code is explained line by line, along with the questions that came up while learning.

  • GPT-2
  • PyTorch
  • Bigram
Read entry
01

Tokenizer · preparation

Planned

Before writing the tokenizer: what actually becomes a token?

A preview record for the first conceptual boundary in the project: the relationship between Unicode text, UTF-8 bytes, BPE merges, and the token IDs received by the model.

  • GPT-2
  • Tokenizer
  • Byte-level BPE
Read entry