[2411.03292] Interaction2Code: Benchmarking MLLM-based Interactive Webpage Code Generation from Interactive Prototyping

Computer Science > Software Engineering arXiv:2411.03292 (cs) [Submitted on 5 Nov 2024 (v1), last revised 1 Mar 2026 (this version, v3)] Title:Interaction2Code: Benchmarking MLLM-based Interactive Webpage Code Generation from Interactive Prototyping Authors:Jingyu Xiao, Yuxuan Wan, Yintong Huo, Zixin Wang, Xinyi Xu, Wenxuan Wang, Zhiyao Xu, Yuhang Wang, Michael R. Lyu View a PDF of the paper titled Interaction2Code: Benchmarking MLLM-based Interactive Webpage Code Generation from Interactive Prototyping, by Jingyu Xiao and 8 other authors View PDF HTML (experimental) Abstract:Multimodal Large Language Models (MLLMs) have demonstrated remarkable performance on the design-to-code task, i.e., generating UI code from UI mock-ups. However, existing benchmarks only contain static web pages for evaluation and ignore the dynamic interaction, limiting the practicality, usability and user engagement of the generated webpages. To bridge these gaps, we present the first systematic investigation of MLLMs in generating interactive webpages. Specifically, we formulate the Interaction-to-Code task and establish the Interaction2Code benchmark, encompassing 127 unique webpages and 374 distinct interactions across 15 webpage types and 31 interaction categories. Through comprehensive experiments utilizing state-of-the-art (SOTA) MLLMs, evaluated via both automatic metrics and human assessments, we identify four critical limitations of MLLM on Interaction-to-Code task: (1) inadequate generatio...