[Research] We benchmarked four geo-experimentation packages on 8,000 simulated panels with known ground truth. None achieved nominal 95% coverage without substantially missing real effects.

Our research team benchmarked four open-source incrementality packages: CausalPy (Bayesian synthetic control), Meta GeoLift (augmented synthetic control with conformal inference), Google Matched Markets (time-based regression), and CausalImpact (Bayesian structural time series). We simulated panels where the true treatment effect is known and the headline result was that no tool delivered nominal 95% coverage together with adequate power. Coverage here means the share of runs where the tool's 95% interval contains the true effect we injected in the data. We ran this study because a lot of prac