[2605.07247] EnvSimBench: A Benchmark for Evaluating and Improving LLM-Based Environment Simulation

Computer Science > Artificial Intelligence arXiv:2605.07247 (cs) [Submitted on 8 May 2026] Title:EnvSimBench: A Benchmark for Evaluating and Improving LLM-Based Environment Simulation Authors:Yi Liu, TingFeng Hui, Wei Zhang, Li Sun, Ningxin Su, Jian Wang, Sen Su View a PDF of the paper titled EnvSimBench: A Benchmark for Evaluating and Improving LLM-Based Environment Simulation, by Yi Liu and 6 other authors View PDF HTML (experimental) Abstract:Scalable AI agents training relies on interactive environments that faithfully simulate the consequences of agent actions. Manually crafted environments are expensive to build, brittle to extend, and fundamentally limited in diversity. A promising direction is to replace manually crafted environments with LLM-simulated counterparts. However, this paradigm hinges on an unexamined core assumption: LLMs can accurately simulate environmental feedback. In practice, LLM-simulated environments suffer from hallucinations, logical inconsistencies, and silent state drift failures that corrupt agent reward signals and compound the construction costs that the paradigm was designed to eliminate. To address this gap, we propose EnvSimBench with four contributions: 1) We provide the first formal definition and operationalization of Environment Simulation Ability (EnvSim Ability) as a quantifiable research objective. 2) We construct EnvSimBench, a rigorous benchmark covering 400 samples across 167 diverse environments, equipped with verifiable lab...