K-Score: Kalman Filter as a Principled Alternative to Reward Normalization in Reinforcement Learning
Abstract
We propose a simple yet effective alternative to reward normalization in policy gradient reinforcement learning by integrating a 1D Kalman filter for online reward estimation. Instead of relying on fixed heuristics, our method recursively estimates the latent reward mean, smoothing high-variance returns and adapting to non-stationary environments. This approach incurs minimal overhead and requires no modification to existing policy architectures. Experiments on \textit{LunarLander} and \textit{CartPole} demonstrate that Kalman-filtered rewards significantly accelerate convergence and reduce training variance compared to standard normalization techniques. Code is available at https://github.com/Sumxiaa/Kalman_Normalization.
Keywords
Cite
@article{arxiv.2604.23056,
title = {K-Score: Kalman Filter as a Principled Alternative to Reward Normalization in Reinforcement Learning},
author = {Zixuan Xia and Quanxi Li},
journal= {arXiv preprint arXiv:2604.23056},
year = {2026}
}
Comments
Accepted in NewInML Workshop, The 42nd International Conference on Machine Learning (ICML 2025).\href{https://icml.cc/virtual/2025/affinity-event/39980}{Event Page}