Project 01
A learning project to implement the path from text tokenization to autoregressive training, while keeping the questions, failed assumptions, and verification evidence visible.
This project has not started yet. For now, this page shows the plan and the structure future entries will follow.
Understand the full data and gradient path well enough to explain each boundary, reproduce a small model, and diagnose failures without treating the reference implementation as a black box.
Text representation, byte-level BPE, vocabulary, and reversible encoding.
Sequence windows, batching, targets, and deterministic data loading.
Embeddings, causal attention, MLP blocks, normalization, and residual paths.
Optimization, checkpoints, sampling, evaluation, and failure analysis.
As the project develops, I will add the questions, implementation notes, and test results from each stage here.
Planned
A preview record for the first conceptual boundary in the project: the relationship between Unicode text, UTF-8 bytes, BPE merges, and the token IDs received by the model.