![[2602.20379] Case-Aware LLM-as-a-Judge Evaluation for Enterprise-Scale RAG Systems](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.20379] Case-Aware LLM-as-a-Judge Evaluation for Enterprise-Scale RAG Systems
Summary
The paper presents a case-aware evaluation framework for enterprise-scale Retrieval-Augmented Generation (RAG) systems, addressing the limitations of existing evaluation methods in multi-turn workflows.
Why It Matters
As enterprises increasingly rely on RAG systems for complex workflows, a tailored evaluation framework is crucial for ensuring effectiveness and operational alignment. This research highlights the need for metrics that reflect real-world enterprise challenges, improving system reliability and user satisfaction.
Key Takeaways
- Existing RAG evaluation frameworks are inadequate for multi-turn, case-based scenarios.
- The proposed framework includes eight metrics focused on operational constraints and workflow alignment.
- A severity-aware scoring protocol enhances diagnostic clarity and reduces score inflation.
- Deterministic prompting with JSON outputs allows for scalable evaluation and monitoring.
- Comparative studies reveal critical trade-offs that can guide system improvements.
Computer Science > Computation and Language arXiv:2602.20379 (cs) [Submitted on 23 Feb 2026] Title:Case-Aware LLM-as-a-Judge Evaluation for Enterprise-Scale RAG Systems Authors:Mukul Chhabra, Luigi Medrano, Arush Verma View a PDF of the paper titled Case-Aware LLM-as-a-Judge Evaluation for Enterprise-Scale RAG Systems, by Mukul Chhabra and 2 other authors View PDF HTML (experimental) Abstract:Enterprise Retrieval-Augmented Generation (RAG) assistants operate in multi-turn, case-based workflows such as technical support and IT operations, where evaluation must reflect operational constraints, structured identifiers (e.g., error codes, versions), and resolution workflows. Existing RAG evaluation frameworks are primarily designed for benchmark-style or single-turn settings and often fail to capture enterprise-specific failure modes such as case misidentification, workflow misalignment, and partial resolution across turns. We present a case-aware LLM-as-a-Judge evaluation framework for enterprise multi-turn RAG systems. The framework evaluates each turn using eight operationally grounded metrics that separate retrieval quality, grounding fidelity, answer utility, precision integrity, and case/workflow alignment. A severity-aware scoring protocol reduces score inflation and improves diagnostic clarity across heterogeneous enterprise cases. The system uses deterministic prompting with strict JSON outputs, enabling scalable batch evaluation, regression testing, and production mon...
