Deep learning — backprop, optimization, regularization
**Backpropagation** computes ∂L/∂θ via reverse-mode automatic differentiation. Modern frameworks (PyTorch 2.5, JAX) build a computation graph and walk it backward, computing gradients in O(forward) time.
**Optimizers:** • **SGD with momentum** — still used for CNN image training; best generalization for computer vision. • **Adam / AdamW** — adaptive per-parameter learning rate via first and second moment estimates. AdamW decouples weight decay from the adaptive term (Loshchilov & Hutter 2019). Default for transformers. • **Lion, Shampoo** — newer second-order-ish optimizers; Shampoo (Gupta et al.) uses Kronecker-factored preconditioners.
**Learning-rate schedules:** warmup (linear for first ~1-10% of steps), cosine decay, inverse square root. For LLM pretraining, warmup + cosine-to-zero is standard.
**Regularization:** • **Dropout** — zero activations with probability p; inverted dropout scales at train time. Less common in modern LLMs (layer/RMS norm + large batches serve similar role). • **Weight decay** (L2) — shrinks parameters. • **Gradient clipping** — cap ‖∇‖₂ (typ. 1.0) for stability. • **Label smoothing** — ε-mix target with uniform. • **Mixup, CutMix** — input/label interpolation, improves CV robustness.
**Mixed precision** (FP16/BF16 with loss scaling, FP8 in H100/H200) halves memory and doubles throughput. FSDP and DeepSpeed ZeRO stage-3 shard params/grads/opt-states across GPUs. Tensor/sequence/pipeline parallelism compose for frontier-scale training.
**Loss landscape:** modern wide networks have many nearly-equivalent low-loss minima connected by linear mode connectivity; explains why SGD variants find generalizing solutions.