[2603.25821] Doctorina MedBench: End-to-End Evaluation of Agent-Based Medical AI

Computer Science > Computation and Language arXiv:2603.25821 (cs) [Submitted on 26 Mar 2026] Title:Doctorina MedBench: End-to-End Evaluation of Agent-Based Medical AI Authors:Anna Kozlova, Stanislau Salavei, Pavel Satalkin, Hanna Plotnitskaya, Sergey Parfenyuk View a PDF of the paper titled Doctorina MedBench: End-to-End Evaluation of Agent-Based Medical AI, by Anna Kozlova and 4 other authors View PDF HTML (experimental) Abstract:We present Doctorina MedBench, a comprehensive evaluation framework for agent-based medical AI based on the simulation of realistic physician-patient interactions. Unlike traditional medical benchmarks that rely on solving standardized test questions, the proposed approach models a multi-step clinical dialogue in which either a physician or an AI system must collect medical history, analyze attached materials (including laboratory reports, images, and medical documents), formulate differential diagnoses, and provide personalized recommendations. System performance is evaluated using the D.O.T.S. metric, which consists of four components: Diagnosis, Observations/Investigations, Treatment, and Step Count, enabling assessment of both clinical correctness and dialogue efficiency. The system also incorporates a multi-level testing and quality monitoring architecture designed to detect model degradation during both development and deployment. The framework supports safety-oriented trap cases, category-based random sampling of clinical scenarios, and fu...