Average Cost Optimality Inequality for Markov Decision Processes with Borel Spaces and Universally Measurable Policies
Abstract
We consider average-cost Markov decision processes (MDPs) with Borel state and action spaces and universally measurable policies. For the nonnegative cost model and an unbounded cost model with a Lyapunov-type stability character, we introduce a set of new conditions under which we prove the average cost optimality inequality (ACOI) via the vanishing discount factor approach. Unlike most existing results on the ACOI, our result does not require any compactness and continuity conditions on the MDPs. Instead, the main idea is to use the almost-uniform-convergence property of a pointwise convergent sequence of measurable functions as asserted in Egoroff's theorem. Our conditions are formulated in order to exploit this property. Among others, we require that for each state, on selected subsets of actions at that state, the state transition stochastic kernel is majorized by finite measures. We combine this majorization property of the transition kernel with Egoroff's theorem to prove the ACOI.
Keywords
Cite
@article{arxiv.2001.01357,
title = {Average Cost Optimality Inequality for Markov Decision Processes with Borel Spaces and Universally Measurable Policies},
author = {Huizhen Yu},
journal= {arXiv preprint arXiv:2001.01357},
year = {2020}
}
Comments
34 pages; to appear in SIAM Journal on Control and Optimization (this is the accepted version before the galley proof). The contents of this paper consist of (i) the author's results given previously in Section 3 of arXiv:1901.03374v1 and (ii) additional sections for an extended discussion and illustrative examples. arXiv admin note: text overlap with arXiv:1901.03374