karpathy/nanogpt
The simplest, fastest repository for training/finetuning medium-sized GPTs.
How nanogpt is put together
nanoGPT is a minimal, educational/research repository for training and finetuning medium-sized GPT models (e.g., GPT-2 124M on OpenWebText) written almost entirely in Python with PyTorch. The architecture is a flat script layout: `model.py` defines the GPT Transformer, `train.py` is a ~300-line training loop supporting single-GPU, CPU/MPS, and multi-GPU/multi-node via PyTorch DDP launched with `torchrun`, and `sample.py` generates text from checkpoints. Data flows from dataset-specific `prepare.py` scripts that download and tokenize corpora into flat binary token files (`train.bin`/`val.bin`), which are memory-mapped by the training loop. Configuration is handled unconventionally: `configurator.py` is exec'd inside `train.py` so config files are plain Python snippets that overwrite globals, with CLI `--key=value` overrides applied afterward.
Languages
Frameworks
Infrastructure
Major components
GPT model definition (model.py)
Defines the GPT Transformer architecture (~300 lines) including optional loading of OpenAI GPT-2 pretrained weights.
Training loop (train.py)
~300-line boilerplate training/finetuning loop with DDP support, gradient accumulation, checkpointing to --out_dir, and optional wandb logging.
Sampling/inference (sample.py)
Loads a trained checkpoint from an out_dir and generates text samples.
Configuration system (configurator.py + config/*.py)
Exec'd config files that override train.py globals, plus --key=value CLI overrides parsed with ast.literal_eval.
Data preparation scripts (data/*/prepare.py)
Download and tokenize datasets (OpenWebText, Shakespeare) into flat uint16 binary token streams (train.bin/val.bin).
Benchmarking (bench.py)
Standalone benchmarking script for measuring model/training performance.
Analysis notebooks
Jupyter notebooks for scaling-law analysis (scaling_laws.ipynb) and transformer parameter sizing (transformer_sizing.ipynb).
No digest this week
1 weeksWeek of
Dependencies and code review
Dependency advisories
Security Watch
No known advisories across 0 scanned dependencies.
No known advisories in the scanned dependencies.
Code review
No concrete code-level issues confirmed.
Get this report every week for your repos.
GitZoid learns each repo, reports what changed, and flags what needs attention. One flat price for the whole team.
$19 a month, flat · First 10 outputs free · No card required