[2603.22344] Errors in AI-Assisted Retrieval of Medical Literature: A Comparative Study

Computer Science > Information Retrieval arXiv:2603.22344 (cs) [Submitted on 21 Mar 2026] Title:Errors in AI-Assisted Retrieval of Medical Literature: A Comparative Study Authors:Jenny Gao (1), Yongfeng Zhang (2), Mary L Disis (3)Lanjing Zhang (4,5,6) ((1) College of Arts and Science, New York University, New York, NY (2) Department of Computer Sciences, School of Arts & Sciences, Rutgers University, Piscataway, NJ, (3) UW Medicine Cancer Vaccine Institute University of Washington, Seattle, WA, (4) Department of Chemical Biology, Ernest Mario School of Pharmacy, Rutgers University, Piscataway, NJ, (5) Department of Pathology, Princeton Medical Center, Plainsboro, NJ, (6) Rutgers Cancer Institute, New Brunswick, NJ) View a PDF of the paper titled Errors in AI-Assisted Retrieval of Medical Literature: A Comparative Study, by Jenny Gao (1) and 26 other authors View PDF Abstract:Large language models (LLMs) assisted literature retrieval may lead to erroneous references, but these errors have not been rigorously quantified. Therefore, we quantitatively assess errors in reference retrieval of widely used free-version LLM platforms and identify the factors associated with retrieval errors. We evaluated 2,000 references retrieved by 5 LLMs (Grok-2, ChatGPT GPT-4.1, Google Gemini Flash 2.5, Perplexity AI, and DeepSeek GPT-4) for 40 randomly-selected original articles (10 per journal) published Jan. 2024 to July 2025 from British Medical Journal (BMJ), Journal of the American Medica...