On-device and edge AI — Apple Intelligence, Gemma 3, Phi — AIML

Objectives

Identify the key rules and §§ that apply to on-device and edge ai — apple intelligence, gemma 3, phi.
Apply the AI.Deployment.Edge knowledge element in a typical exam scenario.
Recognize common distractors and partial-credit answers.

Read

**Why on-device?** Privacy (data never leaves), latency (no round-trip), offline operation, lower per-inference cost, regulatory data-locality (HIPAA, GDPR, EU AI Act).

**Apple Intelligence (WWDC 2024 → ongoing):** • ~3B-parameter on-device Foundation Model + larger Private Cloud Compute (PCC) server model on Apple Silicon servers. • LoRA/adapter system for task specialization (summarize, rewrite, proofread, extract from email). • Semantic Index of personal data on-device; App Intents bring app actions into Siri. • PCC: verifiable enclave — images of software signed, public transparency log, no persistent storage, narrow attack surface. Security architecture paper published 2024.

**Google Gemma 3 (Mar 2025):** 1B / 4B / 12B / 27B open weights; multimodal (4B+); 128K context; strong math/code; permissive license allowing commercial use with limited restrictions.

**Microsoft Phi family:** Phi-3 Mini (3.8B), Phi-4 Mini (3.8B), Phi-4 (14B). Trained heavily on textbook-quality synthetic data; strong reasoning per parameter.

**Meta Llama 3.2 1B/3B** — tuned for mobile (Qualcomm + MediaTek NPUs). Llama 4 Scout/Maverick pushed larger.

**Deployment runtimes:** • **Apple MLX, Core ML, Metal** — Apple Silicon. • **ONNX Runtime, DirectML** — cross-platform. • **llama.cpp / GGUF** — CPU + NEON/AVX + any GPU via Metal/Vulkan/CUDA. • **TensorFlow Lite + XNNPACK** — Android. • **Qualcomm AI Engine Direct, MediaTek NeuroPilot** — mobile NPU. • **NVIDIA Jetson / TensorRT** — edge GPU.

**Practical sizing:** Int4 quantization lets a 3B model run in ~2 GB RAM at 30+ tokens/sec on a modern phone. 7B fits on consumer laptops; 70B needs a >64GB unified-memory box (M3/M4 Ultra Mac Studio) or multi-GPU workstation.

Check yourself

3 quick questions — no score kept, just formative feedback.

Q1 · AI.Deployment.Inference
Speculative decoding accelerates autoregressive inference by:

Answer choices A Using a smaller 'draft' model to propose K tokens, verified in one forward pass of the target model. B Quantizing the target model to 1-bit weights. C Skipping transformer layers. D Running inference on CPU.
Q2 · AI.Deployment.Inference
KV-cache memory scales as:

Answer choices A O(n·d·L)·B bytes per sequence — linear in context length. B O(1). C O(log n). D Unrelated to context.
Q3 · AI.Deployment.Inference
Quantizing LLM weights from FP16 to INT4 (e.g., GPTQ/AWQ):

Answer choices A Reduces memory ~4× and typically preserves quality with calibration; minor degradation common. B Always doubles accuracy. C Is never used. D Increases memory.

AI.Deployment.Inference Inference, deployment, and the Claude API

Objectives

Read

Check yourself

Related lessons