![[2602.13232] PlotChain: Deterministic Checkpointed Evaluation of Multimodal LLMs on Engineering Plot Reading](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.13232] PlotChain: Deterministic Checkpointed Evaluation of Multimodal LLMs on Engineering Plot Reading
Summary
PlotChain introduces a deterministic benchmark for evaluating multimodal large language models (MLLMs) on engineering plot reading, focusing on quantitative value recovery from plots.
Why It Matters
This research addresses the need for robust evaluation methods in multimodal AI, particularly in engineering applications where accurate data extraction from plots is critical. By providing a standardized protocol and dataset, it enhances reproducibility and diagnostic capabilities in AI model assessments.
Key Takeaways
- PlotChain benchmarks MLLMs on engineering plot reading tasks.
- It includes 15 plot families with 450 rendered plots for evaluation.
- Checkpoint-based diagnostics allow for failure localization in model predictions.
- Top models achieved pass rates above 78% on field-level evaluations.
- The dataset and evaluation tools are released for reproducibility.
Computer Science > Artificial Intelligence arXiv:2602.13232 (cs) [Submitted on 29 Jan 2026] Title:PlotChain: Deterministic Checkpointed Evaluation of Multimodal LLMs on Engineering Plot Reading Authors:Mayank Ravishankara View a PDF of the paper titled PlotChain: Deterministic Checkpointed Evaluation of Multimodal LLMs on Engineering Plot Reading, by Mayank Ravishankara View PDF HTML (experimental) Abstract:We present PlotChain, a deterministic, generator-based benchmark for evaluating multimodal large language models (MLLMs) on engineering plot reading-recovering quantitative values from classic plots (e.g., Bode/FFT, step response, stress-strain, pump curves) rather than OCR-only extraction or free-form captioning. PlotChain contains 15 plot families with 450 rendered plots (30 per family), where every item is produced from known parameters and paired with exact ground truth computed directly from the generating process. A central contribution is checkpoint-based diagnostic evaluation: in addition to final targets, each item includes intermediate 'cp_' fields that isolate sub-skills (e.g., reading cutoff frequency or peak magnitude) and enable failure localization within a plot family. We evaluate four state-of-the-art MLLMs under a standardized, deterministic protocol (temperature = 0 and a strict JSON-only numeric output schema) and score predictions using per-field tolerances designed to reflect human plot-reading precision. Under the 'plotread' tolerance policy, the ...
