[2603.01966] AMemGym: Interactive Memory Benchmarking for Assistants in Long-Horizon Conversations

Computer Science > Computation and Language arXiv:2603.01966 (cs) [Submitted on 2 Mar 2026] Title:AMemGym: Interactive Memory Benchmarking for Assistants in Long-Horizon Conversations Authors:Cheng Jiayang, Dongyu Ru, Lin Qiu, Yiyang Li, Xuezhi Cao, Yangqiu Song, Xunliang Cai View a PDF of the paper titled AMemGym: Interactive Memory Benchmarking for Assistants in Long-Horizon Conversations, by Cheng Jiayang and 6 other authors View PDF HTML (experimental) Abstract:Long-horizon interactions between users and LLM-based assistants necessitate effective memory management, yet current approaches face challenges in training and evaluation of memory. Existing memory benchmarks rely on static, off-policy data as context, limiting evaluation reliability and scalability. To address these gaps, we introduce AMemGym, an interactive environment enabling on-policy evaluation and optimization for memory-driven personalization. AMemGym employs structured data sampling to predefine user profiles, state-dependent questions, and state evolution trajectories, enabling cost-effective generation of high-quality, evaluation-aligned interactions. LLM-simulated users expose latent states through role-play while maintaining structured state consistency. Comprehensive metrics based on structured data guide both assessment and optimization of assistants. Extensive experiments reveal performance gaps in existing memory systems (e.g., RAG, long-context LLMs, and agentic memory) and corresponding reaso...