[2505.20730] Do LLMs Understand Collaborative Signals? Diagnosis and Repair

Computer Science > Information Retrieval arXiv:2505.20730 (cs) [Submitted on 27 May 2025 (v1), last revised 22 Mar 2026 (this version, v4)] Title:Do LLMs Understand Collaborative Signals? Diagnosis and Repair Authors:Shahrooz Pouryousef, Ali Montazeralghaem View a PDF of the paper titled Do LLMs Understand Collaborative Signals? Diagnosis and Repair, by Shahrooz Pouryousef and 1 other authors View PDF HTML (experimental) Abstract:Collaborative information from user-item interactions is a fundamental source of signal in successful recommender systems. Recently, researchers have attempted to incorporate this knowledge into large language model-based recommender approaches (LLMRec) to enhance their performance. However, there has been little fundamental analysis of whether LLMs can effectively reason over collaborative information. In this paper, we analyze the ability of LLMs to reason about collaborative information in recommendation tasks, comparing their performance to traditional matrix factorization (MF) models. We propose a simple and effective method to improve LLMs' reasoning capabilities using retrieval-augmented generation (RAG) over the user-item interaction matrix with four different prompting strategies. Our results show that the LLM outperforms the MF model whenever we provide relevant information in a clear and easy-to-follow format, and prompt the LLM to reason based on it. We observe that with this strategy, in almost all cases, the more information we provi...