![[2411.04551] Measure-to-measure interpolation using Transformers](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2411.04551] Measure-to-measure interpolation using Transformers
Summary
This article explores the use of Transformers as measure-to-measure maps, detailing their ability to process arbitrary input and target measures, which enhances their applicability in various domains.
Why It Matters
Understanding the capabilities of Transformers in processing measures is crucial as their applications expand beyond traditional language tasks. This research provides insights into their expressive power, which can influence future developments in machine learning and optimization.
Key Takeaways
- Transformers can function as measure-to-measure maps, enhancing their versatility.
- The study provides a framework for matching arbitrary input and target measures.
- Understanding these capabilities can lead to improved applications in various fields.
Mathematics > Optimization and Control arXiv:2411.04551 (math) [Submitted on 7 Nov 2024 (v1), last revised 13 Feb 2026 (this version, v3)] Title:Measure-to-measure interpolation using Transformers Authors:Borjan Geshkovski, Philippe Rigollet, Domènec Ruiz-Balet View a PDF of the paper titled Measure-to-measure interpolation using Transformers, by Borjan Geshkovski and 2 other authors View PDF HTML (experimental) Abstract:Transformers are deep neural network architectures that underpin the recent successes of large language models. Unlike more classical architectures that can be viewed as point-to-point maps, a Transformer acts as a measure-to-measure map implemented as specific interacting particle system on the unit sphere: the input is the empirical measure of tokens in a prompt and its evolution is governed by the continuity equation. In fact, Transformers are not limited to empirical measures and can in principle process any input measure. As the nature of data processed by Transformers is expanding rapidly, it is important to investigate their expressive power as maps from an arbitrary measure to another arbitrary measure. To that end, we provide an explicit choice of parameters that allows a single Transformer to match $N$ arbitrary input measures to $N$ arbitrary target measures, under the minimal assumption that every pair of input-target measures can be matched by some transport map. Comments: Subjects: Optimization and Control (math.OC); Machine Learning (cs.LG);...
