![[2602.19509] Pyramid MoA: A Probabilistic Framework for Cost-Optimized Anytime Inference](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.19509] Pyramid MoA: A Probabilistic Framework for Cost-Optimized Anytime Inference
Summary
The article presents Pyramid MoA, a probabilistic framework designed to optimize inference costs in large language models (LLMs) while maintaining high accuracy. It employs a hierarchical Mixture-of-Agents architecture to dynamically manage computational resources.
Why It Matters
As LLMs become integral to various applications, the trade-off between performance and cost is critical. Pyramid MoA addresses this challenge by providing a scalable solution that enhances efficiency without sacrificing accuracy, making advanced AI more accessible.
Key Takeaways
- Pyramid MoA reduces inference costs by 61% while achieving high accuracy.
- The framework uses a lightweight Router to manage resource allocation dynamically.
- It allows for a tunable balance between performance and budget constraints.
- Achieves 93.0% accuracy on the GSM8K benchmark, closely matching Oracle models.
- Introduces minimal latency overhead, enhancing user experience.
Computer Science > Computation and Language arXiv:2602.19509 (cs) [Submitted on 23 Feb 2026] Title:Pyramid MoA: A Probabilistic Framework for Cost-Optimized Anytime Inference Authors:Arindam Khaled View a PDF of the paper titled Pyramid MoA: A Probabilistic Framework for Cost-Optimized Anytime Inference, by Arindam Khaled View PDF HTML (experimental) Abstract:Large Language Models (LLMs) face a persistent trade-off between inference cost and reasoning capability. While "Oracle" models (e.g., Llama-3-70B) achieve state-of-the-art accuracy, they are prohibitively expensive for high-volume deployment. Smaller models (e.g., 8B parameters) are cost-effective but struggle with complex tasks. In this work, we propose "Pyramid MoA", a hierarchical Mixture-of-Agents architecture that uses a lightweight Router to dynamically escalate queries only when necessary. By leveraging semantic agreement and confidence calibration among an ensemble of small models, our Router identifies "hard" problems with high precision. On the GSM8K benchmark, our system achieves 93.0% accuracy, effectively matching the Oracle baseline (98.0%) while reducing compute costs by 61%. We demonstrate that the system introduces negligible latency overhead (+0.82s) and allows for a tunable trade-off between performance and budget. Comments: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) Cite as: arXiv:2602.19509 [cs.CL] (or arXiv:2602.19509v1 [cs.CL] for thi...
