[2506.00077] Gaussian mixture models as a proxy for interacting language models

Computer Science > Computation and Language arXiv:2506.00077 (cs) [Submitted on 29 May 2025 (v1), last revised 3 Apr 2026 (this version, v4)] Title:Gaussian mixture models as a proxy for interacting language models Authors:Edward L. Wang, Mohammad Sharifi Kiasari, Tianyu Wang, Hayden Helm, Avanti Athreya, Carey Priebe, Vince Lyzinski View a PDF of the paper titled Gaussian mixture models as a proxy for interacting language models, by Edward L. Wang and 6 other authors View PDF HTML (experimental) Abstract:Large language models (LLMs) are powerful tools that, in a number of settings, overlap with the results of human pattern recognition and reasoning. Retrieval-augmented generation (RAG) further allows LLMs to produce tailored output depending on the contents of their RAG databases. However, LLMs depend on complex, computationally expensive algorithms. In this paper, we introduce interacting Gaussian mixture models (GMMs) as a proxy for interacting LLMs. We construct a model of interacting GMMs, complete with an analogue to RAG updating, under which GMMs can generate, exchange, and update data and parameters. We show that this interacting system of Gaussian mixture models, which can be implemented at minimal computational cost, mimics certain aspects of experimental simulations of interacting LLMs whose iterative responses depend on feedback from other LLMs. We build a Markov chain from this system of interacting GMMs; formalize and interpret the notion of polarization for ...