![[2602.12247] ExtractBench: A Benchmark and Evaluation Methodology for Complex Structured Extraction](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.12247] ExtractBench: A Benchmark and Evaluation Methodology for Complex Structured Extraction
Summary
ExtractBench introduces a benchmark and evaluation framework for extracting structured data from unstructured documents like PDFs, addressing critical gaps in current methodologies.
Why It Matters
As large language models (LLMs) are increasingly used for data extraction, ensuring their accuracy and reliability is vital. ExtractBench provides a standardized approach to evaluate PDF-to-JSON extraction, which is crucial for industries relying on structured data from unstructured sources. This framework can help improve the performance of AI models in real-world applications.
Key Takeaways
- ExtractBench offers an open-source benchmark for PDF-to-JSON extraction.
- It addresses gaps in evaluating schema breadth and nested extraction semantics.
- Baseline evaluations show that current LLMs struggle with complex schemas.
- The benchmark includes 35 PDFs and 12,867 evaluatable fields across various domains.
- Performance drops significantly with increased schema complexity, highlighting the need for better models.
Computer Science > Machine Learning arXiv:2602.12247 (cs) [Submitted on 12 Feb 2026 (v1), last revised 13 Feb 2026 (this version, v2)] Title:ExtractBench: A Benchmark and Evaluation Methodology for Complex Structured Extraction Authors:Nick Ferguson, Josh Pennington, Narek Beghian, Aravind Mohan, Douwe Kiela, Sheshansh Agrawal, Thien Hang Nguyen View a PDF of the paper titled ExtractBench: A Benchmark and Evaluation Methodology for Complex Structured Extraction, by Nick Ferguson and 6 other authors View PDF HTML (experimental) Abstract:Unstructured documents like PDFs contain valuable structured information, but downstream systems require this data in reliable, standardized formats. LLMs are increasingly deployed to automate this extraction, making accuracy and reliability paramount. However, progress is bottlenecked by two gaps. First, no end-to-end benchmark evaluates PDF-to-JSON extraction under enterprise-scale schema breadth. Second, no principled methodology captures the semantics of nested extraction, where fields demand different notions of correctness (exact match for identifiers, tolerance for quantities, semantic equivalence for names), arrays require alignment, and omission must be distinguished from hallucination. We address both gaps with ExtractBench, an open-source benchmark and evaluation framework for PDF-to-JSON structured extraction. The benchmark pairs 35 PDF documents with JSON Schemas and human-annotated gold labels across economically valuable domai...
