Benchmarking on your own production data

Disclosure: I'm the CTO at the company that ran this, and the write-up is on our domain. Method's all below, happy to go deeper on any of it in the comments. TL;DR: We built a harness that replays recorded production requests through DeepSeek V4 Flash using exact saved configurations (temperature, schema constraints, tool definitions). We ran zero-cost structural validation checks first, then used Claude Sonnet 5 as a blind, randomized LLM judge using each task's own system prompt. \## The problem When building a feature, you pick a solid model, wire it up, and ship it. Six months later, three