Spent a sprint building automated rollback for model deployments. Turns out the hard part wasn't the rollback.

The rollback mechanism itself took maybe two days. Detect a regression against a baseline metric, trigger, revert to the last known-good version, done. Felt like the real engineering work of the sprint. What ate the rest of the week was everything the rollback assumed already existed and didn't. "Last known-good version" turned out to be a genuinely fuzzy concept once I went looking for it. Good by what metric, measured over what window, and was that version even still compatible with the current feature schema, since two upstream changes had landed since it was last serving traffic. The rollb