![[2602.18230] [Re] Benchmarking LLM Capabilities in Negotiation through Scoreable Games](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.18230] [Re] Benchmarking LLM Capabilities in Negotiation through Scoreable Games
Summary
This paper evaluates the benchmarking of Large Language Models (LLMs) in negotiation tasks using Scoreable Games, assessing the reproducibility and usability of existing benchmarks.
Why It Matters
Understanding the capabilities of LLMs in negotiation is crucial for their application in real-world scenarios. This study addresses the challenges in evaluating these models, providing insights that can improve benchmarking practices and model comparisons.
Key Takeaways
- The study replicates and expands on existing benchmarks for LLMs in negotiation.
- Findings indicate ambiguity in model comparisons, questioning the objectivity of current benchmarks.
- Identifies limitations in experimental setups, particularly regarding information leakage and evaluation thoroughness.
- Highlights the importance of context in comparative evaluations of models.
- Introduces additional metrics to enhance negotiation quality assessments.
Computer Science > Machine Learning arXiv:2602.18230 (cs) [Submitted on 20 Feb 2026] Title:[Re] Benchmarking LLM Capabilities in Negotiation through Scoreable Games Authors:Jorge Carrasco Pollo, Ioannis Kapetangeorgis, Joshua Rosenthal, John Hua Yao View a PDF of the paper titled [Re] Benchmarking LLM Capabilities in Negotiation through Scoreable Games, by Jorge Carrasco Pollo and 3 other authors View PDF HTML (experimental) Abstract:Large Language Models (LLMs) demonstrate significant potential in multi-agent negotiation tasks, yet evaluation in this domain remains challenging due to a lack of robust and generalizable benchmarks. Abdelnabi et al. (2024) introduce a negotiation benchmark based on Scoreable Games, with the aim of developing a highly complex and realistic evaluation framework for LLMs. Our work investigates the reproducibility of claims in their benchmark, and provides a deeper understanding of its usability and generalizability. We replicate the original experiments on additional models, and introduce additional metrics to verify negotiation quality and evenness of evaluation. Our findings reveal that while the benchmark is indeed complex, model comparison is ambiguous, raising questions about its objectivity. Furthermore, we identify limitations in the experimental setup, particularly in information leakage detection and thoroughness of the ablation study. By examining and analyzing the behavior of a wider range of models on an extended version of the benc...