相关论文: The time-dependent expected reward and deviation m…

We consider Poisson's equation for quasi-birth-and-death processes (QBDs) and we exploit the special transition structure of QBDs to obtain its solutions in two different forms. One is based on a decomposition through first passage times to…

We are interested in the analysis of very large continuous-time Markov chains (CTMCs) with many distinct rates. Such models arise naturally in the context of reliability analysis, e.g., of computer network performability analysis, of power…

This paper studies the expected value of multiplicative rewards, where rewards obtained in each step are multiplied (instead of the usual addition), in Markov chains (MCs) and Markov decision processes (MDPs). One of the key differences to…

This paper is concerned with the development of rigorous approximations to various expectations associated with Markov chains and processes having non-stationary transition probabilities. Such non-stationary models arise naturally in…

In this paper, we develop some matrix Poisson's equations satisfied by the mean and variance of the mixing time in an irreducible positive-recurrent discrete-time Markov chain with infinitely-many levels, and provide a computational…

We describe an exact approach for calculating transition probabilities and waiting times in finite-state discrete-time Markov processes. All the states and the rules for transitions between them must be known in advance. We can then…

We study the computational complexity of central analysis problems for One-Counter Markov Decision Processes (OC-MDPs), a class of finitely-presented, countable-state MDPs. OC-MDPs are equivalent to a controlled extension of (discrete-time)…

We propose a new approach for estimating the finite dimensional transition matrix of a Markov chain using a large number of independent sample paths observed at random times. The sample paths may be observed as few as two times, and the…

The goal of a traditional Markov decision process (MDP) is to maximize expected cumulative reward over a defined horizon (possibly infinite). In many applications, however, a decision maker may be interested in optimizing a specific…

In the study of large scale stochastic networks with resource management, differential equations and mean-field limits are two key techniques. Recent research shows that the expected fraction vector (that is, the tailed probability vector)…

Consider a system evolving according to an absorbing discrete-time Markov chain with known transition matrix. The state of the system is observed at two points in time, separated by an unknown number of generations. We are interested in…

In this article, we consider a continuous review (s, S) inventory system with failures of demand fulfillment (service) modeled as a Markov-modulated retrial queueing system. The inventory system features a single product that experiences…

We consider the problem of estimating the transition rate matrix of a continuous-time Markov chain from a finite-duration realisation of this process. We approach this problem in an imprecise probabilistic framework, using a set of prior…

A discrete-time two-dimensional quasi-birth-and-death process (2d-QBD process), $\{{\boldsymbol{Y}}_n\}=\{(X_{1,n},X_{2,n},J_n)\}$, is a two-dimensional skip-free random walk $\{(X_{1,n},X_{2,n})\}$ on $\mathbb{Z}_+^2$ with a supplemental…

It is interesting and challenging to study double-ended queues with First-Come-First-Match discipline under customers' impatient behavior and non-Poisson inputs. The system stability can be guaranteed by the customers' impatient behavior,…

Markov chains are the de facto finite-state model for stochastic dynamical systems, and Markov decision processes (MDPs) extend Markov chains by incorporating non-deterministic behaviors. Given an MDP and rewards on states, a classical…

Note that the serial structure of blockchain has many essential pitfalls, thus a data network structure and its DAG-based blockchain are introduced to resolve the blockchain pitfalls. From such a network perspective, analysis of the…

The paper addresses the problem of computing maximal conditional expected accumulated rewards until reaching a target state (briefly called maximal conditional expectations) in finite-state Markov decision processes where the condition is…

A Markov decision process can be parameterized by a transition kernel and a reward function. Both play essential roles in the study of reinforcement learning as evidenced by their presence in the Bellman equations. In our inquiry of various…

In this paper, we present a numerical framework for constructing bounds on stationary performance measures of random walks in the positive orthant using the Markov reward approach. These bounds are established in terms of stationary…