![[2602.15785] This human study did not involve human subjects: Validating LLM simulations as behavioral evidence](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.15785] This human study did not involve human subjects: Validating LLM simulations as behavioral evidence
Summary
This article discusses the use of large language models (LLMs) as synthetic participants in social science experiments, evaluating their validity in simulating human behavior.
Why It Matters
As researchers increasingly utilize LLMs for behavioral studies, understanding their limitations and potential for valid inference is crucial. This work clarifies the conditions under which LLM simulations can effectively replace human participants, impacting future research methodologies in social sciences.
Key Takeaways
- LLMs can serve as cost-effective substitutes for human participants in social science experiments.
- Heuristic approaches lack the formal guarantees needed for confirmatory research.
- Statistical calibration offers a more reliable method for using LLMs, preserving validity while reducing costs.
- The effectiveness of LLMs depends on their ability to accurately represent the relevant populations.
- Researchers should be cautious of over-relying on LLMs and consider their limitations.
Computer Science > Artificial Intelligence arXiv:2602.15785 (cs) [Submitted on 17 Feb 2026] Title:This human study did not involve human subjects: Validating LLM simulations as behavioral evidence Authors:Jessica Hullman, David Broska, Huaman Sun, Aaron Shaw View a PDF of the paper titled This human study did not involve human subjects: Validating LLM simulations as behavioral evidence, by Jessica Hullman and 3 other authors View PDF HTML (experimental) Abstract:A growing literature uses large language models (LLMs) as synthetic participants to generate cost-effective and nearly instantaneous responses in social science experiments. However, there is limited guidance on when such simulations support valid inference about human behavior. We contrast two strategies for obtaining valid estimates of causal effects and clarify the assumptions under which each is suitable for exploratory versus confirmatory research. Heuristic approaches seek to establish that simulated and observed human behavior are interchangeable through prompt engineering, model fine-tuning, and other repair strategies designed to reduce LLM-induced inaccuracies. While useful for many exploratory tasks, heuristic approaches lack the formal statistical guarantees typically required for confirmatory research. In contrast, statistical calibration combines auxiliary human data with statistical adjustments to account for discrepancies between observed and simulated responses. Under explicit assumptions, statisti...
