nanochat-llm-training

Skill by ara.so — Daily 2026 Skills collection. nanochat is Karpathy's minimal, hackable harness for training LLMs end-to-end on a single GPU node. It covers tokenization, pretraining, SFT finetuning, RL, evaluation (DCLM CORE score), inference with KV cache, and a ChatGPT-like web UI. A single complexity dial ( --depth ) auto-configures all other hyperparameters (width, heads, LR, training horizon, weight decay) for compute-optimal training. You can reproduce GPT-2 capability (~$43,000 in 2019) for ~$48 on an 8×H100 node (~2 hours). nanochat uses uv for dependency management: The single most important parameter. Everything else is derived automatically: nanochat uses explicit dtype management via COMPUTE_DTYPE in nanochat/common.py . No torch.amp.autocast . How it works: Weights stored in fp32 (optimizer precision), custom Linear casts to COMPUTE_DTYPE in forward pass, embeddings stored directly in COMPUTE_DTYPE to save memory. This is expected.