![[2510.19139] A Multi-faceted Analysis of Cognitive Abilities: Evaluating Prompt Methods with Large Language Models on the CONSORT Checklist](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2510.19139] A Multi-faceted Analysis of Cognitive Abilities: Evaluating Prompt Methods with Large Language Models on the CONSORT Checklist
Summary
This paper evaluates the cognitive abilities of large language models (LLMs) in assessing clinical trial reporting according to CONSORT standards, revealing significant miscalibration and overconfidence in model responses.
Why It Matters
As LLMs become increasingly integrated into healthcare, understanding their limitations in clinical contexts is crucial. This study highlights the need for improved evaluation methods to ensure reliable and explainable AI applications in medical settings, addressing a significant gap in current research.
Key Takeaways
- The study compares general and domain-specialized LLMs using three prompt strategies.
- Results indicate pronounced miscalibration and overconfidence in LLMs during clinical role-playing.
- Calibration errors exceeded clinically relevant thresholds, emphasizing the need for better evaluation methods.
- The findings advocate for improved calibration and transparent coding in medical AI.
- Strategic prompt engineering is essential for developing reliable AI in healthcare.
Computer Science > Artificial Intelligence arXiv:2510.19139 (cs) This paper has been withdrawn by Sohyeon Jeon [Submitted on 22 Oct 2025 (v1), last revised 25 Feb 2026 (this version, v3)] Title:A Multi-faceted Analysis of Cognitive Abilities: Evaluating Prompt Methods with Large Language Models on the CONSORT Checklist Authors:Sohyeon Jeon, Hyung-Chul Lee View a PDF of the paper titled A Multi-faceted Analysis of Cognitive Abilities: Evaluating Prompt Methods with Large Language Models on the CONSORT Checklist, by Sohyeon Jeon and 1 other authors No PDF available, click to view other formats Abstract:Despite the rapid expansion of Large Language Models (LLMs) in healthcare, robust and explainable evaluation of their ability to assess clinical trial reporting according to CONSORT standards remains an open challenge. In particular, uncertainty calibration and metacognitive reliability of LLM reasoning are poorly understood and underexplored in medical automation. This study applies a behavioral and metacognitive analytic approach using an expert-validated dataset, systematically comparing two representative LLMs - one general and one domain-specialized - across three prompt strategies. We analyze both cognitive adaptation and calibration error using metrics: Expected Calibration Error (ECE) and a baseline-normalized Relative Calibration Error (RCE) that enables reliable cross-model comparison. Our results reveal pronounced miscalibration and overconfidence in both models, es...
