Related papers: A Generalized Fundamental Matrix for Computing Fun…
Markov cohort state-transition models have been the standard approach for simulating the prognosis of patients or, more generally, the life trajectories of individuals over a time period. Current approaches for estimating the variance of a…
We propose a principled kernel-based policy iteration algorithm to solve the continuous-state Markov Decision Processes (MDPs). In contrast to most decision-theoretic planning frameworks, which assume fully known state transition models, we…
Markov chains are the de facto finite-state model for stochastic dynamical systems, and Markov decision processes (MDPs) extend Markov chains by incorporating non-deterministic behaviors. Given an MDP and rewards on states, a classical…
This paper presents different recursive formulas for computing the marginals and the normalizing constant of a Gibbs distribution $\pi$: The common thread is the use of the underlying Markov properties of such processes. The procedures are…
Consider the batch-arrival $GI^X/M/c/N$ model with $c$ servers, general inter-arrival batch times, finite buffer, and exponential service times. Inter-arrival batch times, batch sizes, and service times are $i.i.d.$ and independent of each…
Under certain conditions, the dynamics of coarse-grained models of solvated proteins can be described using a Markov state model, which tracks the evolution of populations of configurations. The transition rates among states that appear in…
In banking practice, rating transition matrices have become the standard approach of deriving multi-year probabilities of default (PDs) from one-year PDs, the latter normally being available from Basel ratings. Rating transition matrices…
We consider the parameter estimation of Markov chain when the unknown transition matrix belongs to an exponential family of transition matrices. Then, we show that the sample mean of the generator of the exponential family is an…
Spectral methods have proven to be a highly effective tool in understanding the intrinsic geometry of a high-dimensional data set $\left\{x_i \right\}_{i=1}^{n} \subset \mathbb{R}^d$. The key ingredient is the construction of a Markov chain…
In this paper, we study the problem of principal component analysis with generative modeling assumptions, adopting a general model for the observed matrix that encompasses notable special cases, including spiked matrix recovery and phase…
In this paper, we present a numerical framework for constructing bounds on stationary performance measures of random walks in the positive orthant using the Markov reward approach. These bounds are established in terms of stationary…
A birth-death process is a continuous-time Markov chain that counts the number of particles in a system over time. In the general process with $n$ current particles, a new particle is born with instantaneous rate $\lambda_n$ and a particle…
We consider the voter model with $M$ states initially in the system. Using generating functions, we pose the spectral problem for the Markov transition matrix and solve for all eigenvalues and eigenvectors exactly. With this solution, we…
Multiparametric statistical model providing stable reconstruction of parameters by observations is considered. The only general method of this kind is the root model based on the representation of the probability density as a squared…
We define the projected entropy S(T) at a given temperature T in the context of an Ising model transition matrix calculation as the entropy associated with the distribution of Markov chain realizations in energy-magnetization, E-H, space.…
We study the complexity of central controller synthesis problems for finite-state Markov decision processes, where the objective is to optimize both the expected mean-payoff performance of the system and its stability. We argue that the…
Principal component analysis (PCA) is a well-established tool in machine learning and data processing. The principal axes in PCA were shown to be equivalent to the maximum marginal likelihood estimator of the factor loading matrix in a…
This paper presents a method for predicting stock returns using principal component analysis (PCA) and the hidden Markov model (HMM) and tests the results of trading stocks based on this approach. Principal component analysis is applied to…
This paper investigates the optimization problem of an infinite stage discrete time Markov decision process (MDP) with a long-run average metric considering both mean and variance of rewards together. Such performance metric is important…
Markov decision processes (MDPs) are standard models for probabilistic systems with non-deterministic behaviours. Long-run average rewards provide a mathematically elegant formalism for expressing long term performance. Value iteration (VI)…