![[2602.16629] Almost Sure Convergence of Differential Temporal Difference Learning for Average Reward Markov Decision Processes](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.16629] Almost Sure Convergence of Differential Temporal Difference Learning for Average Reward Markov Decision Processes
Summary
This paper presents the almost sure convergence of differential temporal difference learning for average reward Markov decision processes, addressing limitations in existing convergence guarantees.
Why It Matters
Understanding the convergence of differential temporal difference learning is crucial for practitioners in reinforcement learning, as it enhances the theoretical foundation and practical application of these algorithms without reliance on local clocks, making them more accessible for real-world scenarios.
Key Takeaways
- Proves almost sure convergence of on-policy n-step differential TD learning without local clocks.
- Establishes sufficient conditions for off-policy n-step differential TD convergence.
- Strengthens theoretical foundations of differential TD learning for average reward scenarios.
Computer Science > Machine Learning arXiv:2602.16629 (cs) [Submitted on 18 Feb 2026] Title:Almost Sure Convergence of Differential Temporal Difference Learning for Average Reward Markov Decision Processes Authors:Ethan Blaser, Jiuqi Wang, Shangtong Zhang View a PDF of the paper titled Almost Sure Convergence of Differential Temporal Difference Learning for Average Reward Markov Decision Processes, by Ethan Blaser and 2 other authors View PDF Abstract:The average reward is a fundamental performance metric in reinforcement learning (RL) focusing on the long-run performance of an agent. Differential temporal difference (TD) learning algorithms are a major advance for average reward RL as they provide an efficient online method to learn the value functions associated with the average reward in both on-policy and off-policy settings. However, existing convergence guarantees require a local clock in learning rates tied to state visit counts, which practitioners do not use and does not extend beyond tabular settings. We address this limitation by proving the almost sure convergence of on-policy $n$-step differential TD for any $n$ using standard diminishing learning rates without a local clock. We then derive three sufficient conditions under which off-policy $n$-step differential TD also converges without a local clock. These results strengthen the theoretical foundations of differential TD and bring its convergence analysis closer to practical implementations. Subjects: Machine...
