Multimodal and vision-language models — AIML

Objectives

Identify the key rules and §§ that apply to multimodal and vision-language models.
Apply the AI.Multimodal knowledge element in a typical exam scenario.
Recognize common distractors and partial-credit answers.

Read

**Contrastive vision-language pretraining:** • **CLIP (Radford et al. 2021)** — dual encoder (ViT image + transformer text), InfoNCE contrastive loss. Zero-shot classification by prompting. • **SigLIP, EVA-CLIP, OpenCLIP** — follow-ups with better losses and scale.

**Vision-Language Models (VLMs) / MLLMs** (April 2026): • **Claude Opus 4.7 vision** — native multimodal; high OCR + chart/diagram reasoning. • **Gemini 2.5** — 1-2M context multimodal. • **GPT-4o, o3, o4** — omni-model with real-time voice/video. • **Llama 3.2 Vision (11B / 90B)** — open weights. • **Qwen2.5-VL, InternVL-2.5** — strong open Asian labs.

Architecture pattern: CLIP-style vision encoder → projector (MLP or Q-former) → LLM decoder. Trained with image-text pairs then instruction-tuned on VQA/Chart/Doc datasets.

**Generative image/video:** • **Diffusion models** (Ho et al. 2020 DDPM, Rombach et al. 2022 Latent Diffusion) — Stable Diffusion, Flux, Imagen. Rectified flow (SD3) simplifies training. • **Masked generative** — **Muse (Google 2023) / Muse Spark (Meta 2025)** — parallel prediction of visual tokens; orders of magnitude faster inference than autoregressive. • **Video** — Sora, Veo, Movie Gen. Still expensive; consistency and physics fidelity are open problems. • **Autoregressive pixel/token** — Parti, MAGVIT.

**3D + world models** — Genie 2/3 (Google DeepMind), Sora as a world model, Tesla FSD, Cosmos. **Embodied AI** (robotics): RT-2, Octo, Pi-Zero/Pi-0.5 from Physical Intelligence.

**Evaluation:** MMMU, MMBench, ChartQA, DocVQA, MathVista, RealWorldQA. Human evals still critical for generative outputs.

Check yourself

3 quick questions — no score kept, just formative feedback.

Q1 · AI.Multimodal
CLIP (Radford et al. 2021) is trained with which loss objective?

Answer choices A Mean squared error between image and text embeddings. B Cross-entropy next-token prediction. C Contrastive InfoNCE — pulling matched image-text pairs together and pushing mismatched pairs apart in a shared embedding space. D Adversarial GAN loss.
Q2 · AI.Multimodal
Contrastive pretraining methods such as CLIP enable:

Answer choices A Zero-shot classification via text prompts mapped into a shared embedding space. B Only language modeling. C Only speech recognition. D Direct robotic control.
Q3 · AI.LLM.Transformer
In scaled dot-product attention (Vaswani et al. 2017), the attention scores QKᵀ are divided by √d_k before softmax primarily to:

Answer choices A Reduce computational complexity. B Prevent the softmax from saturating as dimensionality grows. C Enforce positive attention weights. D Make attention permutation-equivariant.

No related lessons found nearby.

Objectives

Read

Check yourself

Related lessons