Hands-on — building LLMs from scratch, Karpathy pedagogy — AIML

Objectives

Identify the key rules and §§ that apply to hands-on — building llms from scratch, karpathy pedagogy.
Apply the AI.Practice.Build knowledge element in a typical exam scenario.
Recognize common distractors and partial-credit answers.

Read

**Andrej Karpathy's open-source teaching repos** are the canonical path from zero to transformer:

• **micrograd** — a tiny autograd engine (<100 lines) that builds a scalar autodiff graph with backprop. Grounds the idea of automatic differentiation. • **makemore** — bigram → MLP → WaveNet-style → Transformer for character-level name generation. Demonstrates how each architectural step improves log-likelihood. • **nanoGPT** — minimal GPT trained on OpenWebText/TinyStories. ~300 LOC core; trains on one GPU. Reproduces the key tricks of the GPT-2 paper. • **llm.c** — pure C/CUDA implementation of GPT-2 training. No framework dependency; illustrates what the low level looks like. • **nanochat** — nanoGPT-style 'full' ChatGPT-style assistant (tokenization → pretraining → SFT → inference) you can train for ~$100 on cloud GPUs.

**Exercise structure** (building your own transformer):

1. Implement **BPE tokenization** (or use tiktoken). 2. Write the **decoder-only transformer** block: RMSNorm, causal MHA with RoPE, SwiGLU MLP, residual. 3. Implement **cross-entropy loss** with label shift. 4. Train on TinyStories (40M param, 10K steps, one A100). Achieve <2.0 val loss for coherent story completion. 5. Fine-tune on an instruction dataset (Alpaca/OpenHermes/UltraChat subset). 6. Add **KV cache** for autoregressive generation; measure tokens/sec. 7. Quantize to Int4 with AWQ/GPTQ; compare quality/throughput. 8. Wrap as a tool-using agent calling the Claude API for the hard parts (RAG-style).

**Additional canonical courses:** Stanford CS224N (NLP), CS336 (LLMs from scratch), CMU 11-667 (LLMs), Hugging Face LLM course, Sebastian Raschka's 'Build a Large Language Model (From Scratch)' book. All permissively licensed or freely available.

Check yourself

3 quick questions — no score kept, just formative feedback.

Q1 · AI.LLM.Transformer
In scaled dot-product attention (Vaswani et al. 2017), the attention scores QKᵀ are divided by √d_k before softmax primarily to:

Answer choices A Reduce computational complexity. B Prevent the softmax from saturating as dimensionality grows. C Enforce positive attention weights. D Make attention permutation-equivariant.
Q2 · AI.LLM.Training
DPO (Direct Preference Optimization, Rafailov et al. 2023) differs from RLHF-PPO in that DPO:

Answer choices A Requires a larger reward model. B Uses no human preference data. C Directly optimizes the policy on preference pairs without training an explicit reward model. D Requires reinforcement learning through a simulator.
Q3 · AI.Foundations.DL
AdamW (Loshchilov & Hutter 2019) improves upon Adam by:

Answer choices A Using second-order gradients exclusively. B Decoupling weight decay from the adaptive gradient update. C Eliminating momentum. D Removing the bias-correction terms.

No related lessons found nearby.

Objectives

Read

Check yourself

Related lessons