[2306.05036] Mapping the Challenges of HCI: An Application and Evaluation of ChatGPT for Mining Insights at Scale

Computer Science > Human-Computer Interaction arXiv:2306.05036 (cs) [Submitted on 8 Jun 2023 (v1), last revised 24 Mar 2026 (this version, v5)] Title:Mapping the Challenges of HCI: An Application and Evaluation of ChatGPT for Mining Insights at Scale Authors:Jonas Oppenlaender, Joonas Hämäläinen View a PDF of the paper titled Mapping the Challenges of HCI: An Application and Evaluation of ChatGPT for Mining Insights at Scale, by Jonas Oppenlaender and 1 other authors View PDF HTML (experimental) Abstract:Large language models (LLMs) are increasingly used for analytical tasks, yet their effectiveness in real-world applications remains underexamined, partly due to the opacity of proprietary models. We evaluate ChatGPT (GPT-3.5 and GPT-4) on the practical task of extracting research challenges from a large scholarly corpus in Human-Computer Interaction (HCI). Using a two-step approach, we first apply GPT-3.5 to extract candidate challenges from the 879 papers in the 2023 ACM CHI Conference proceedings, then use GPT-4 to select the most relevant challenges per paper. This process yielded 4,392 research challenges across 113 topics, which we organized through topic modeling and present in an interactive visualization. We compare the identified challenges with previously established HCI grand challenges and the United Nations Sustainable Development Goals, finding both strong alignment in areas such as ethics and accessibility, and gaps in areas such as human-AI collaboration. A...