![[2602.18955] Incremental Transformer Neural Processes](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.18955] Incremental Transformer Neural Processes
Summary
The paper introduces Incremental Transformer Neural Processes (incTNP), a model designed for efficient sequential data processing, achieving significant computational speedups while maintaining predictive performance.
Why It Matters
As data streams become increasingly prevalent in various applications, the ability to update models incrementally without full recomputation is crucial. This research addresses a significant gap in existing Transformer Neural Processes, enhancing their utility in real-time scenarios such as sensor data analysis and forecasting.
Key Takeaways
- incTNP reduces update time complexity from quadratic to linear.
- The model maintains predictive performance comparable to standard TNPs.
- Causal masking and Key-Value caching are key innovations in incTNP.
- Empirical evaluations demonstrate incTNP's effectiveness in real-world tasks.
- The model retains Bayesian consistency, crucial for streaming inference.
Computer Science > Machine Learning arXiv:2602.18955 (cs) [Submitted on 21 Feb 2026] Title:Incremental Transformer Neural Processes Authors:Philip Mortimer, Cristiana Diaconu, Tommy Rochussen, Bruno Mlodozeniec, Richard E. Turner View a PDF of the paper titled Incremental Transformer Neural Processes, by Philip Mortimer and 4 other authors View PDF HTML (experimental) Abstract:Neural Processes (NPs), and specifically Transformer Neural Processes (TNPs), have demonstrated remarkable performance across tasks ranging from spatiotemporal forecasting to tabular data modelling. However, many of these applications are inherently sequential, involving continuous data streams such as real-time sensor readings or database updates. In such settings, models should support cheap, incremental updates rather than recomputing internal representations from scratch for every new observation -- a capability existing TNP variants lack. Drawing inspiration from Large Language Models, we introduce the Incremental TNP (incTNP). By leveraging causal masking, Key-Value (KV) caching, and a data-efficient autoregressive training strategy, incTNP matches the predictive performance of standard TNPs while reducing the computational cost of updates from quadratic to linear time complexity. We empirically evaluate our model on a range of synthetic and real-world tasks, including tabular regression and temperature prediction. Our results show that, surprisingly, incTNP delivers performance comparable to -...