Generalized Fitted Q-Iteration with Clustered Data
Machine Learning
2025-10-07 v1
Abstract
This paper focuses on reinforcement learning (RL) with clustered data, which is commonly encountered in healthcare applications. We propose a generalized fitted Q-iteration (FQI) algorithm that incorporates generalized estimating equations into policy learning to handle the intra-cluster correlations. Theoretically, we demonstrate (i) the optimalities of our Q-function and policy estimators when the correlation structure is correctly specified, and (ii) their consistencies when the structure is mis-specified. Empirically, through simulations and analyses of a mobile health dataset, we find the proposed generalized FQI achieves, on average, a half reduction in regret compared to the standard FQI.
Keywords
Cite
@article{arxiv.2510.03912,
title = {Generalized Fitted Q-Iteration with Clustered Data},
author = {Liyuan Hu and Jitao Wang and Zhenke Wu and Chengchun Shi},
journal= {arXiv preprint arXiv:2510.03912},
year = {2025}
}