![[2602.15809] Decision Quality Evaluation Framework at Pinterest](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.15809] Decision Quality Evaluation Framework at Pinterest
Summary
The article presents a Decision Quality Evaluation Framework developed at Pinterest to enhance content moderation by evaluating the quality of decisions made by human agents and LLMs.
Why It Matters
As online platforms face increasing scrutiny over content safety, this framework provides a structured approach to assess moderation decisions, balancing cost, scale, and trustworthiness. It represents a significant shift towards data-driven methodologies in content management.
Key Takeaways
- Introduces a comprehensive framework for evaluating moderation decisions.
- Utilizes a Golden Set curated by experts as a benchmark for quality.
- Employs an automated sampling pipeline to enhance dataset coverage.
- Facilitates data-driven prompt optimization and policy management.
- Shifts content safety assessments from subjective to quantitative practices.
Statistics > Applications arXiv:2602.15809 (stat) [Submitted on 17 Feb 2026] Title:Decision Quality Evaluation Framework at Pinterest Authors:Yuqi Tian, Robert Paine, Attila Dobi, Kevin O'Sullivan, Aravindh Manickavasagam, Faisal Farooq View a PDF of the paper titled Decision Quality Evaluation Framework at Pinterest, by Yuqi Tian and 5 other authors View PDF HTML (experimental) Abstract:Online platforms require robust systems to enforce content safety policies at scale. A critical component of these systems is the ability to evaluate the quality of moderation decisions made by both human agents and Large Language Models (LLMs). However, this evaluation is challenging due to the inherent trade-offs between cost, scale, and trustworthiness, along with the complexity of evolving policies. To address this, we present a comprehensive Decision Quality Evaluation Framework developed and deployed at Pinterest. The framework is centered on a high-trust Golden Set (GDS) curated by subject matter experts (SMEs), which serves as a ground truth benchmark. We introduce an automated intelligent sampling pipeline that uses propensity scores to efficiently expand dataset coverage. We demonstrate the framework's practical application in several key areas: benchmarking the cost-performance trade-offs of various LLM agents, establishing a rigorous methodology for data-driven prompt optimization, managing complex policy evolution, and ensuring the integrity of policy content prevalence metric...