![[2602.20571] CausalReasoningBenchmark: A Real-World Benchmark for Disentangled Evaluation of Causal Identification and Estimation](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.20571] CausalReasoningBenchmark: A Real-World Benchmark for Disentangled Evaluation of Causal Identification and Estimation
Summary
The CausalReasoningBenchmark introduces a new framework for evaluating automated causal inference, distinguishing between identification and estimation processes in causal analysis.
Why It Matters
This benchmark addresses the limitations of existing causal inference evaluations by separating the identification of causal strategies from numerical estimation. It provides a more nuanced understanding of model performance, which is crucial for developing robust automated systems in artificial intelligence and data science.
Key Takeaways
- CausalReasoningBenchmark evaluates causal inference by separating identification and estimation.
- It includes 173 queries across 138 real-world datasets, enhancing the evaluation process.
- Baseline results show a significant drop in identification-specification correctness, highlighting challenges in research design.
- The benchmark is publicly available, promoting further advancements in automated causal inference systems.
- Understanding the distinction between identification and estimation can improve model robustness.
Computer Science > Artificial Intelligence arXiv:2602.20571 (cs) [Submitted on 24 Feb 2026] Title:CausalReasoningBenchmark: A Real-World Benchmark for Disentangled Evaluation of Causal Identification and Estimation Authors:Ayush Sawarni, Jiyuan Tan, Vasilis Syrgkanis View a PDF of the paper titled CausalReasoningBenchmark: A Real-World Benchmark for Disentangled Evaluation of Causal Identification and Estimation, by Ayush Sawarni and 2 other authors View PDF HTML (experimental) Abstract:Many benchmarks for automated causal inference evaluate a system's performance based on a single numerical output, such as an Average Treatment Effect (ATE). This approach conflates two distinct steps in causal analysis: identification-formulating a valid research design under stated assumptions-and estimation-implementing that design numerically on finite data. We introduce CausalReasoningBenchmark, a benchmark of 173 queries across 138 real-world datasets, curated from 85 peer-reviewed research papers and four widely-used causal-inference textbooks. For each query a system must produce (i) a structured identification specification that names the strategy, the treatment, outcome, and control variables, and all design-specific elements, and (ii) a point estimate with a standard error. By scoring these two components separately, our benchmark enables granular diagnosis: it distinguishes failures in causal reasoning from errors in numerical execution. Baseline results with a state-of-the-art ...
