[2410.09027] Variance reduction combining pre-experiment and in-experiment data

Statistics > Methodology arXiv:2410.09027 (stat) [Submitted on 11 Oct 2024 (v1), last revised 21 Mar 2026 (this version, v2)] Title:Variance reduction combining pre-experiment and in-experiment data Authors:Zhexiao Lin, Pablo Crespo View a PDF of the paper titled Variance reduction combining pre-experiment and in-experiment data, by Zhexiao Lin and 1 other authors View PDF Abstract:Online controlled experiments (A/B testing) are fundamental to data-driven decision-making in many companies. Improving the sensitivity of these experiments under fixed sample size constraints requires reducing the variance of the average treatment effect (ATE) estimator. Existing variance reduction techniques such as CUPED and CUPAC use pre-experiment data, but their effectiveness depends on how predictive those data are for outcomes measured during the experiment. In-experiment data are often more strongly correlated with the outcome, but using arbitrary post-treatment variables can introduce bias. In this paper, we propose a general, robust, and scalable framework that combines both pre-experiment and in-experiment data to achieve variance reduction. Our framework is simple, interpretable, and computationally efficient, making it practical for real-world deployment. We develop the asymptotic theory of the proposed estimator and provide consistent variance estimators. Empirical results from multiple online experiments conducted at Etsy demonstrate substantial additional variance reduction over...