![[2602.16131] Empirical Cumulative Distribution Function Clustering for LLM-based Agent System Analysis](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.16131] Empirical Cumulative Distribution Function Clustering for LLM-based Agent System Analysis
Summary
This article presents a novel evaluation framework for LLM-based agents using empirical cumulative distribution functions (ECDFs) to assess response quality and distribution characteristics.
Why It Matters
As large language models (LLMs) are increasingly utilized in complex tasks, understanding the nuances of their outputs is crucial. This framework enhances evaluation methods, allowing for better insights into response quality beyond traditional metrics, which can improve agent design and application in various fields.
Key Takeaways
- Introduces a new evaluation framework using ECDFs for LLM responses.
- Clustering methods reveal distributional differences in response quality.
- Enhances understanding of agent performance beyond accuracy metrics.
- Provides insights into the impact of parameters like temperature and persona.
- Demonstrates practical applications through experiments on QA datasets.
Statistics > Machine Learning arXiv:2602.16131 (stat) [Submitted on 18 Feb 2026] Title:Empirical Cumulative Distribution Function Clustering for LLM-based Agent System Analysis Authors:Chihiro Watanabe, Jingyu Sun View a PDF of the paper titled Empirical Cumulative Distribution Function Clustering for LLM-based Agent System Analysis, by Chihiro Watanabe and 1 other authors View PDF HTML (experimental) Abstract:Large language models (LLMs) are increasingly used as agents to solve complex tasks such as question answering (QA), scientific debate, and software development. A standard evaluation procedure aggregates multiple responses from LLM agents into a single final answer, often via majority voting, and compares it against reference answers. However, this process can obscure the quality and distributional characteristics of the original responses. In this paper, we propose a novel evaluation framework based on the empirical cumulative distribution function (ECDF) of cosine similarities between generated responses and reference answers. This enables a more nuanced assessment of response quality beyond exact match metrics. To analyze the response distributions across different agent configurations, we further introduce a clustering method for ECDFs using their distances and the $k$-medoids algorithm. Our experiments on a QA dataset demonstrate that ECDFs can distinguish between agent settings with similar final accuracies but different quality distributions. The clustering a...
