[2603.27355] LLM Readiness Harness: Evaluation, Observability, and CI Gates for LLM/RAG Applications

Computer Science > Artificial Intelligence arXiv:2603.27355 (cs) [Submitted on 28 Mar 2026] Title:LLM Readiness Harness: Evaluation, Observability, and CI Gates for LLM/RAG Applications Authors:Alexandre Cristovão Maiorano View a PDF of the paper titled LLM Readiness Harness: Evaluation, Observability, and CI Gates for LLM/RAG Applications, by Alexandre Cristov\~ao Maiorano View PDF HTML (experimental) Abstract:We present a readiness harness for LLM and RAG applications that turns evaluation into a deployment decision workflow. The system combines automated benchmarks, OpenTelemetry observability, and CI quality gates under a minimal API contract, then aggregates workflow success, policy compliance, groundedness, retrieval hit rate, cost, and p95 latency into scenario-weighted readiness scores with Pareto frontiers. We evaluate the harness on ticket-routing workflows and BEIR grounding tasks (SciFact and FiQA) with full Azure matrix coverage (162/162 valid cells across datasets, scenarios, retrieval depths, seeds, and models). Results show that readiness is not a single metric: on FiQA under sla-first at k=5, gpt-4.1-mini leads in readiness and faithfulness, while gpt-5.2 pays a substantial latency cost; on SciFact, models are closer in quality but still separable operationally. Ticket-routing regression gates consistently reject unsafe prompt variants, demonstrating that the harness can block risky releases instead of merely reporting offline scores. The result is a repro...