LLM training — pretraining, SFT, RLHF, DPO, RLAIF — AIML

Objectives

Identify the key rules and §§ that apply to llm training — pretraining, sft, rlhf, dpo, rlaif.
Apply the AI.LLM.Training knowledge element in a typical exam scenario.
Recognize common distractors and partial-credit answers.

Read

**Pretraining** — next-token prediction over trillions of tokens (web + code + books + synthetic). Scaling laws (Chinchilla, Hoffmann et al. 2022): compute-optimal D ≈ 20 × N parameters. Modern frontier models train to D >> 20N for inference-time efficiency (Llama 3: 15T tokens on 8B/70B/405B).

**Post-training alignment pipeline** typically has three stages:

1. **Supervised fine-tuning (SFT)** on high-quality instruction-response pairs. Teaches format and basic helpfulness. ~10k-1M examples.

2. **Preference optimization:** • **RLHF (Christiano et al. 2017, InstructGPT — Ouyang et al. 2022)** — train reward model on human preferences (A > B), then PPO against RM with KL penalty to reference policy. • **DPO (Rafailov et al. 2023)** — direct preference optimization; closed-form optimal policy under Bradley-Terry assumption, no explicit RM needed. Cheaper, simpler, widely adopted. • **RLAIF (Bai et al. 2022 Constitutional AI)** — use an LLM as judge to generate preferences from a constitution. Enables scaling beyond human annotation throughput. • **KTO, IPO, SimPO** — DPO variants with different loss parameterizations.

3. **Online/iterative RL** — best-of-N rejection sampling, rubric-based rewards, test-time compute (o1/o3-style chain-of-thought with outcome rewards on math/code/science).

**Data quality > data quantity** past a threshold. Deduplication, toxicity filtering, language balance, synthetic data (model-generated with filtering) all crucial. Textbooks are high-signal (Phi family, Gunasekar et al.).

**Tokenization** — byte-pair encoding (BPE) dominant. SentencePiece, tiktoken (GPT), tokenizers crate. Context windows grew from 2K (GPT-3) to 128K (GPT-4) to 1M-10M (Gemini 1.5, Claude Opus 4.7) via long-context training and retrieval augmentation.

Check yourself

3 quick questions — no score kept, just formative feedback.

Q1 · AI.LLM.Transformer
In scaled dot-product attention (Vaswani et al. 2017), the attention scores QKᵀ are divided by √d_k before softmax primarily to:

Answer choices A Reduce computational complexity. B Prevent the softmax from saturating as dimensionality grows. C Enforce positive attention weights. D Make attention permutation-equivariant.
Q2 · AI.LLM.Training
DPO (Direct Preference Optimization, Rafailov et al. 2023) differs from RLHF-PPO in that DPO:

Answer choices A Requires a larger reward model. B Uses no human preference data. C Directly optimizes the policy on preference pairs without training an explicit reward model. D Requires reinforcement learning through a simulator.
Q3 · AI.LLM.Transformer
Rotary Position Embedding (RoPE) encodes token positions by:

Answer choices A Adding learned vectors to token embeddings. B Rotating query and key vectors in 2D planes by an angle proportional to position. C Adding a linear bias to attention scores based on distance. D Using one-hot position vectors concatenated to embeddings.

AI.LLM.Transformer Transformers and attention — the architecture behind LLMs

Objectives

Read

Check yourself

Related lessons