JSON-mode output drift detection

JSON-mode drift is one of the most production-impactful failure modes because broken JSON crashes downstream parsers immediately. Three drift patterns to monitor.

(1) Schema validity rate. Send N=50–200 fixed structured-output prompts daily, parse the response with json.loads() and validate against a fixed JSON Schema. Track the percentage that parse + validate successfully. Frontier models with strict modes (gpt-4o's structured output, Claude tool-use) typically run 99–100 percent. A drop to 96 percent is a real alert. Watch for: extra prose before/after the JSON, mismatched braces, hallucinated fields, missing required fields, wrong types on optional fields.

(2) Schema-conformance distribution. Even when JSON parses, the *content* shape can drift. Are enum fields holding values from the allowed set? Are array lengths stable? Are numeric fields in expected ranges? Sample-by-sample diff against last week's outputs flags these — use distance metrics (Levenshtein on string fields, KL divergence on categorical distributions).

(3) Mode-specific quirks. OpenAI's response_format={"type": "json_object"} (loose mode) vs response_format={"type": "json_schema", "strict": true} (strict, Aug 2024+) behave very differently — loose mode has historically had silent drift in how strictly the system prompt JSON instruction is honored. Anthropic's tool-use JSON has had similar shifts. Monitor each mode separately.

ModelWatch tracks "JSON validity rate" and "schema conformance score" as first-class daily metrics per model and per mode, with alerts on drops >2 percentage points day-over-day.