What is model drift in LLMs and how do you monitor it?

Model drift in LLMs is the phenomenon where a model's outputs change without an obvious version bump. There are three flavors: (a) explicit version drift — provider ships a new snapshot (claude-3-5-sonnet-20240620 to claude-3-5-sonnet-20241022); (b) silent serving drift — same alias, but inference stack changes (quantization, MoE routing, system-prompt defaults, safety filters); (c) distribution drift on your side — your input prompts changed and the model is fine, but your eval pipeline mistakes it for a regression.

Monitoring requires separating those three. The standard recipe: hold inputs constant (golden-prompt suite), hold the judging constant (deterministic graders + frozen judge model if using LLM-as-judge), and track per-metric time series. Useful metrics: MMLU subset pass-rate, HumanEval pass@1, GSM8K accuracy, JSON-schema validity rate, refusal rate, p50/p95 latency, cost per 1K output tokens. ModelWatch ships with all of these wired up across OpenAI, Anthropic, Google, and major open-weight providers, and publishes a free public scorecard.