I ran 34 iterations of a model-improvement loop. Most of what I found were evaluation bugs. Feedback?

I’ve been building a bounded improvement loop for ANXEngine, an AIOps system we use at Anexum to rank daily incident risk across roughly 200 network devices. The setup is fairly simple: register one hypothesis, implement the smallest testable change, retrain a challenger plus an unchanged control, evaluate both, then KEEP, REVERT, or PAUSE. After 34 documented iterations, most of the useful findings were not model improvements. They were problems in my evaluation setup. The first version had overlap between training data and the later fitness window. PR-AUC looked like 0.72–0.76. Once I separa