Related papers: A Short Note on Stationary Distributions of Unicha…
This paper discusses the functional stability of closed-loop Markov Chains under optimal policies resulting from a discounted optimality criterion, forming Markov Decision Processes (MDPs). We investigate the stability of MDPs in the sense…
The synthesis problem for partially observable Markov decision processes (POMDPs) is to compute a policy that satisfies a given specification. Such policies have to take the full execution history of a POMDP into account, rendering the…
We consider a random model for directed graphs whereby an arc is placed from one vertex to another with a prescribed probability which may vary from arc to arc. Using perturbation bounds as well as Chernoff inequalities, we show that the…
This paper presents a Distributed Stochastic Model Predictive Control algorithm for networks of linear systems with multiplicative uncertainties and local chance constraints on the states and control inputs. The chance constraints are…
Because failures in distribution systems caused by extreme weather events directly result in consumers' outages, this paper proposes a state-based decision-making model with the objective of mitigating loss of load to improve the…
Human preferences are not always represented via complete linear orders: It is natural to employ partially-ordered preferences for expressing incomparable outcomes. In this work, we consider decision-making and probabilistic planning in…
We study the relation between different Markov Decision Process (MDP) frameworks in the machine learning and econometrics literatures, including the standard MDP, the entropy and general regularized MDP, and stochastic MDP, where the latter…
We investigate the classical active pure exploration problem in Markov Decision Processes, where the agent sequentially selects actions and, from the resulting system trajectory, aims at identifying the best policy as fast as possible. We…
In a general stochastic multistate promoter model of dynamic mRNA/protein interactions, we identify the stationary joint distribution of the promoter state, mRNA, and protein levels through an explicit `stick-breaking' construction of…
The existence of stationary distributions to distribution dependent stochastic differential equations are investigated by using the ergodicity of the associated decoupled equation and the Schauder fixed point theorem. By using Zvonkin's…
Locally Markov walks are natural generalizations of classical Markov chains, where instead of a particle moving independently of the past, it decides where to move next depending on the last action performed at the current location. We…
Markov decisions processes (MDPs) are becoming increasing popular as models of decision theoretic planning. While traditional dynamic programming methods perform well for problems with small state spaces, structured methods are needed for…
The online Markov decision process (MDP) is a generalization of the classical Markov decision process that incorporates changing reward functions. In this paper, we propose practical online MDP algorithms with policy iteration and…
We present new algorithms for computing and approximating bisimulation metrics in Markov Decision Processes (MDPs). Bisimulation metrics are an elegant formalism that capture behavioral equivalence between states and provide strong…
In this paper, we study a class of stochastic processes, called evolving network Markov chains, in evolving networks. Our approach is to transform the degree distribution problem of an evolving network to a corresponding problem of evolving…
This paper presents with justifications a technique that is useful for the study of piecewise deterministic Markov decision processes (PDMDPs) with general policies and unbounded transition intensities. This technique produces an auxiliary…
We are studying stationary random processes with conditional polynomial moments that allow a continuous path modification. Processes with continuous path modification, are important because they are relatively easy to simulate. One does not…
Maximum entropy (maxEnt) inference of state probabilities using state-dependent constraints is popular in the study of complex systems. In stochastic dynamical systems, the effect of state space topology and path-dependent constraints on…
Markov Decision Processes (MDPs) are a classical model for decision making in the presence of uncertainty. Often they are viewed as state transformers with planning objectives defined with respect to paths over MDP states. An increasingly…
We investigate a zero-range process where the underlying one-particle stationary distribution has multifractality. The multiparticle stationary probability measure can be written in a factorized form. If the number of the particles is…