Online Shift Detection and Conformal Adaptation for Deployed Safety Classifiers
arXiv stat.ML1mo4 min read
arXiv:2606.11949v2 Announce Type: replace-cross Abstract: Safety classifiers deployed in production operate under a stationarity assumption that fails silently: when input distributions drift, accuracy degrades with no error signal until ground-truth labels arrive. We present an online monitor that detects distributional shift in classifier scores via a sliding-window KS statistic with empirically calibrated alarm thresholds. In a pre-registered factorial evaluation (4 classifiers $\times$ 5 shift conditions $\times$ 20 seeds $\times$ 2 window sizes; 800 cells), the monitor achieves 86.6% vali
