Transformers and attention — the architecture behind LLMs — AIML

Objectives

Identify the key rules and §§ that apply to transformers and attention — the architecture behind llms.
Apply the AI.LLM.Transformer knowledge element in a typical exam scenario.
Recognize common distractors and partial-credit answers.

Read

**Scaled dot-product attention (Vaswani et al. 2017):**

Attn(Q,K,V) = softmax(QKᵀ / √d_k) V

Q (queries) and K (keys) are projected from hidden states; V (values) is also projected. √d_k scaling prevents softmax saturation as dimensionality grows.

**Multi-head attention (MHA):** h parallel attention heads on dim d_model/h each. Concat and project back to d_model. Allows heads to specialize on different relational patterns. **GQA (Grouped-Query Attention, Ainslie et al. 2023)** and **MQA** share K,V across groups of heads — reduce KV cache memory ~4-8× at small quality cost; standard in Llama 3, Claude, Mistral.

**Decoder-only transformer block:** RMSNorm (or LayerNorm) → self-attention (causal mask) → residual → RMSNorm → MLP (SwiGLU or GELU) → residual. Stacked N times.

**Positional encoding:** • **Sinusoidal** (original) — fixed; limited extrapolation. • **Learned absolute** — positions 0..max_len. • **RoPE** (Rotary PE, Su et al. 2021) — rotates Q,K in 2D planes by position. Used by Llama, Mistral, most modern LLMs. Extendable via NTK/YaRN/LongRoPE. • **ALiBi** — linear distance bias on attention scores.

**KV cache:** during autoregressive generation, cache computed K,V tensors for past tokens. Memory scales O(L × d × layers) per request — the dominant memory cost for long context. Paged attention (vLLM, Kwon et al. 2023), KV quantization (Int4/Int8), and context-parallel serving are active areas.

**FlashAttention (Dao et al. 2022, v2 2023, v3 2024):** IO-aware attention that tiles computation to fit in SRAM, avoiding materialization of the N×N attention matrix. 2-4× speedup and enables long context. Standard in production training/inference stacks.

**Encoder-decoder** transformers (T5, BART) remain strong for translation/summarization; decoder-only dominates general chat/coding.

Check yourself

3 quick questions — no score kept, just formative feedback.

Q1 · AI.LLM.Transformer
In scaled dot-product attention (Vaswani et al. 2017), the attention scores QKᵀ are divided by √d_k before softmax primarily to:

Answer choices A Reduce computational complexity. B Prevent the softmax from saturating as dimensionality grows. C Enforce positive attention weights. D Make attention permutation-equivariant.
Q2 · AI.LLM.Training
DPO (Direct Preference Optimization, Rafailov et al. 2023) differs from RLHF-PPO in that DPO:

Answer choices A Requires a larger reward model. B Uses no human preference data. C Directly optimizes the policy on preference pairs without training an explicit reward model. D Requires reinforcement learning through a simulator.
Q3 · AI.LLM.Transformer
Rotary Position Embedding (RoPE) encodes token positions by:

Answer choices A Adding learned vectors to token embeddings. B Rotating query and key vectors in 2D planes by an angle proportional to position. C Adding a linear bias to attention scores based on distance. D Using one-hot position vectors concatenated to embeddings.

AI.LLM.Training LLM training — pretraining, SFT, RLHF, DPO, RLAIF

Objectives

Read

Check yourself

Related lessons