[2509.24282] SimuHome: A Temporal- and Environment-Aware Benchmark for Smart Home LLM Agents

Computer Science > Computation and Language arXiv:2509.24282 (cs) [Submitted on 29 Sep 2025 (v1), last revised 2 Mar 2026 (this version, v3)] Title:SimuHome: A Temporal- and Environment-Aware Benchmark for Smart Home LLM Agents Authors:Gyuhyeon Seo, Jungwoo Yang, Junseong Pyo, Nalim Kim, Jonggeun Lee, Yohan Jo View a PDF of the paper titled SimuHome: A Temporal- and Environment-Aware Benchmark for Smart Home LLM Agents, by Gyuhyeon Seo and 5 other authors View PDF HTML (experimental) Abstract:We introduce $\textbf{SimuHome}$, a high-fidelity smart home simulator and a benchmark of 600 episodes for LLM-based smart home agents. Existing smart home benchmarks treat the home as a static system, neither simulating how device operations affect environmental variables over time nor supporting workflow scheduling of device commands. SimuHome is grounded in the Matter protocol, the industry standard that defines how real smart home devices communicate and operate. Agents interact with devices through SimuHome's APIs and observe how their actions continuously affect environmental variables such as temperature and humidity. Our benchmark covers state inquiry, implicit user intent inference, explicit device control, and workflow scheduling, each with both feasible and infeasible requests. For workflow scheduling, the simulator accelerates time so that scheduled workflows can be evaluated immediately. An evaluation of 18 agents reveals that workflow scheduling is the hardest category, ...