Training LLMs from Scratch with nanochat

Large Language Models (LLMs) like ChatGPT have revolutionized AI, but the pipeline for training them is often complex, distributed across massive clusters, and expensive. Andrej Karpathy's nanochat breaks down this barrier by providing a minimal, hackable, and educational experimental harness for training LLMs. Designed to be maximally readable and simple, the repository covers the entire lifecycle of an LLM: tokenization, pretraining, finetuning, evaluation, and inference.

Unlike its predecessor nanoGPT, which focused primarily on the pretraining stage, nanochat provides a full-stack implementation, allowing users to train a language model and then immediately chat with it via a web interface or CLI. Remarkably, the tool provides a complete environment designed for pretraining and fine-tuning on a single GPU machine. For instance, an 8xH100 node can train a GPT-2 capability model in roughly two hours—bringing the compute cost down to less than $100. Source

In this tutorial, we will explore what makes nanochat unique, how to set up the environment, and the primary workflows for training and interacting with your own language models.

Understanding the Design Philosophy

The main goal of nanochat is not to build the next frontier model but to serve as an educational and experimental foundation. It is heavily optimized for a single node, which avoids the massive complexity of distributed, multi-node communication.

One of its standout features is the simplification of model scaling. Hyperparameter tuning is notoriously difficult when scaling up transformers. However, nanochat scales models seamlessly by adjusting a single --depth parameter instead of manually tuning every dimension. By adjusting the number of layers in the GPT transformer model, the repository automatically scales the transformer width, number of heads, learning rate schedules, and weight decay in an optimal manner. Source

Setting Up the Environment

Getting started with nanochat is straightforward. nanochat uses the uv package manager to handle Python dependencies.

To set up your local environment, clone the repository and run the setup commands. If you are using a CUDA-enabled GPU (such as an A100 or H100), you will want to install the gpu extras. Alternatively, CPU or Apple Silicon (MPS) users can install the cpu extras.

If you plan on modifying the code or contributing to the project, you can include the dev dependencies, which will install additional packages like pytest and matplotlib:

Running the GPT-2 Speedrun

The core milestone within nanochat is the "GPT-2 speedrun." The challenge is to train a model to match or exceed the original OpenAI GPT-2 capability (measured by the DCLM CORE metric) in the shortest possible wall-clock time.

A single shell script manages the complete GPT-2 training workflow from start to finish. If you have access to an 8xH100 GPU instance, you can reproduce this speedrun. Given the compute requirements, running this inside a screen or tmux session is highly recommended.

As of early 2026, the community has pushed this speedrun down to under two hours. The script handles everything from dataset preparation to the final steps of pretraining. If you're running on hardware with less VRAM (e.g., cards with less than 80GB), you will need to adjust the --device-batch-size downwards in the scripts to prevent Out-Of-Memory (OOM) errors.

Interacting with Your Custom Model

Training the model is only half the fun; interacting with it brings the training metrics to life. Once the training script finishes executing and the checkpoints are saved, you can load your model into a chat interface.

Users can interact with their locally trained model using a dedicated command-line chat application. Ensure your virtual environment is still active, and run the following command:

Once it's done, you can ask it to tell you a story, explain why the sky is blue, or write a poem. Because the GPT-2 capability model is relatively small by modern standards, its responses might exhibit some charming hallucinations, but it fully demonstrates the end-to-end viability of the training pipeline. Source