Inference, deployment, and the Claude API
**Inference optimization levers:** • **Quantization** — FP16/BF16 → Int8/Int4/FP8. Weight-only quantization (AWQ, GPTQ) preserves quality; activation quantization is harder. 4-bit (BitsAndBytes NF4) standard for local inference. • **Speculative decoding** — draft model proposes K tokens; target model verifies in one forward pass. 2-3× speedup for greedy/low-temp. • **Continuous batching** (vLLM, TGI) — pack tokens from concurrent requests into each forward pass. • **Prefix / prompt caching** — reuse attention state across requests sharing a prefix. Anthropic's **prompt caching** charges ~10% of base for cache reads, ~125% for writes; 5-min TTL. Huge cost savings for RAG/long system prompts. • **Distillation** — train a smaller model on a larger model's outputs (e.g., Gemma 3, Llama 3.2 1B/3B from 70B/405B).
**Claude API patterns (docs.anthropic.com):** ```python import anthropic client = anthropic.Anthropic() resp = client.messages.create( model='claude-opus-4-7', max_tokens=1024, system=[{'type':'text','text':'...', 'cache_control':{'type':'ephemeral'}}], messages=[{'role':'user','content':'...'}], ) ```
Capabilities (April 2026): • **Extended thinking** — dedicated reasoning budget; shows thinking tokens separately; higher accuracy on math/code. • **1M context** — Claude Opus 4.7 supports 1,000,000 tokens. • **Tool use** — function calling with structured JSON; parallel tool calls. • **Computer use** — agent can drive a GUI via screenshot + click/type actions. • **Batch API** — 50% discount, 24-hour turnaround. • **Files API + PDFs + vision** — multimodal input.
**Serving stacks:** vLLM (open, high-throughput), TensorRT-LLM (NVIDIA), SGLang (structured generation), llama.cpp (edge). Always use caching + streaming for chat UX.