![[2511.18945] MIST: Mutual Information Estimation Via Supervised Training](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2511.18945] MIST: Mutual Information Estimation Via Supervised Training
Summary
The paper presents MIST, a novel approach for estimating mutual information using a neural network trained on a large dataset of synthetic distributions, outperforming traditional methods in efficiency and accuracy.
Why It Matters
This research addresses the limitations of classical mutual information estimators by introducing a fully data-driven method that enhances flexibility and efficiency. The ability to provide well-calibrated uncertainty estimates is crucial for applications in machine learning and information theory, making it a significant contribution to the field.
Key Takeaways
- MIST uses a neural network to estimate mutual information from synthetic joint distributions.
- The method provides better performance than classical estimators across various sample sizes and dimensions.
- Quantile regression loss is employed to output uncertainty estimates, improving reliability.
- The framework allows for integration into larger learning systems, enhancing its practical applicability.
- Adaptability to different data modalities through normalizing flows expands its use cases.
Computer Science > Machine Learning arXiv:2511.18945 (cs) [Submitted on 24 Nov 2025 (v1), last revised 20 Feb 2026 (this version, v2)] Title:MIST: Mutual Information Estimation Via Supervised Training Authors:German Gritsai, Megan Richards, Maxime Méloux, Kyunghyun Cho, Maxime Peyrard View a PDF of the paper titled MIST: Mutual Information Estimation Via Supervised Training, by German Gritsai and 4 other authors View PDF HTML (experimental) Abstract:We propose a fully data-driven approach to designing mutual information (MI) estimators. Since any MI estimator is a function of the observed sample from two random variables, we parameterize this function with a neural network (MIST) and train it end-to-end to predict MI values. Training is performed on a large meta-dataset of 625,000 synthetic joint distributions with known ground-truth MI. To handle variable sample sizes and dimensions, we employ a two-dimensional attention scheme ensuring permutation invariance across input samples. To quantify uncertainty, we optimize a quantile regression loss, enabling the estimator to approximate the sampling distribution of MI rather than return a single point estimate. This research program departs from prior work by taking a fully empirical route, trading universal theoretical guarantees for flexibility and efficiency. Empirically, the learned estimators largely outperform classical baselines across sample sizes and dimensions, including on joint distributions unseen during training....
