![[2602.13848] Testing For Distribution Shifts with Conditional Conformal Test Martingales](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.13848] Testing For Distribution Shifts with Conditional Conformal Test Martingales
Summary
This paper introduces a novel sequential test for detecting distribution shifts using Conditional Conformal Test Martingales, enhancing detection speed and reliability.
Why It Matters
Understanding distribution shifts is crucial in machine learning, particularly in applications where data evolves over time. This research presents a more robust method that mitigates contamination issues seen in existing approaches, thus improving the reliability of models in dynamic environments.
Key Takeaways
- Proposes a new method for detecting distribution shifts that avoids contamination from post-shift observations.
- Ensures anytime-valid type-I error control while maintaining high detection power.
- Empirical results show faster shift detection compared to traditional methods.
Computer Science > Machine Learning arXiv:2602.13848 (cs) [Submitted on 14 Feb 2026] Title:Testing For Distribution Shifts with Conditional Conformal Test Martingales Authors:Shalev Shaer, Yarin Bar, Drew Prinster, Yaniv Romano View a PDF of the paper titled Testing For Distribution Shifts with Conditional Conformal Test Martingales, by Shalev Shaer and 3 other authors View PDF HTML (experimental) Abstract:We propose a sequential test for detecting arbitrary distribution shifts that allows conformal test martingales (CTMs) to work under a fixed, reference-conditional setting. Existing CTM detectors construct test martingales by continually growing a reference set with each incoming sample, using it to assess how atypical the new sample is relative to past observations. While this design yields anytime-valid type-I error control, it suffers from test-time contamination: after a change, post-shift observations enter the reference set and dilute the evidence for distribution shift, increasing detection delay and reducing power. In contrast, our method avoids contamination by design by comparing each new sample to a fixed null reference dataset. Our main technical contribution is a robust martingale construction that remains valid conditional on the null reference data, achieved by explicitly accounting for the estimation error in the reference distribution induced by the finite reference set. This yields anytime-valid type-I error control together with guarantees of asymptoti...
