Related papers: Optimal Markov Approximations and Generalized Embe…
Markov chains are a natural and well understood tool for describing one-dimensional patterns in time or space. We show how to infer $k$-th order Markov chains, for arbitrary $k$, from finite data by applying Bayesian methods to both…
Estimating entropy production from real observation data can be difficult due to finite resolution in both space and time and finite measurement statistics. We characterize the statistical error introduced by finite sample size and compare…
This article presents a short and concise description of stochastic approximation algorithms in reinforcement learning of Markov decision processes. The algorithms can also be used as a suboptimal method for partially observed Markov…
On a weakly Blackwell space we show how to define a Markov chain approximating problem, for the target problem. The approximating problem is proved to converge to the optimal reduced problem under different pseudometrics. A computational…
An important problem in applied dynamical systems is to compute the external forcing that provokes the largest response of a desired observable quantity. For this, we investigate the perturbation theory of Markov matrices in connection with…
The paper is devoted to studies of perturbed Markov chains commonly used for description of information networks. In such models, the matrix of transition probabilities for the corresponding Markov chain is usually regularised by adding a…
Markov parameters play a key role in system identification. There exists many algorithms where these parameters are estimated using least-squares in a first, pre-processing, step, including subspace identification and multi-step…
This brief paper presents simple simulation-based algorithms for obtaining an approximately optimal policy in a given finite set in large finite constrained Markov decision processes. The algorithms are adapted from playing strategies for…
In this paper, we consider the finite-state approximation of a discrete-time constrained Markov decision process (MDP) under the discounted and average cost criteria. Using the linear programming formulation of the constrained discounted…
In this work, we discuss low-parametric approaches for approximating SimRank matrices, which estimate the similarity between pairs of nodes in a graph. Although SimRank matrices and their computation require a significant amount of memory,…
We consider a hidden Markov model with multiple observation processes, one of which is chosen at each point in time by a policy---a deterministic function of the information state---and attempt to determine which policy minimises the…
Computing the probability of evidence even with known error bounds is NP-hard. In this paper we address this hard problem by settling on an easier problem. We propose an approximation which provides high confidence lower bounds on…
Time-homogeneous Markov chains are often used as disease progression models in studies of cost-effectiveness and optimal decision-making. Maximum likelihood estimation of these models can be challenging when data are collected at a time…
Entropy estimation is a fundamental problem in information theory that has applications in various fields, including physics, biology, and computer science. Estimating the entropy of discrete sequences can be challenging due to limited data…
Recent years have seen the rise of convolutional neural network techniques in exemplar-based image synthesis. These methods often rely on the minimization of some variational formulation on the image space for which the minimizers are…
This note summarizes the optimization formulations used in the study of Markov decision processes. We consider both the discounted and undiscounted processes under the standard and the entropy-regularized settings. For each setting, we…
This paper considers the problem of randomized influence maximization over a Markovian graph process: given a fixed set of nodes whose connectivity graph is evolving as a Markov chain, estimate the probability distribution (over this fixed…
Error estimates for the numerical solution of the master equation are presented. Estimates are based on adjoint methods. We find that a good estimate can often be computed without spending computational effort on a dual problem. Estimates…
Given a sequence of observations from a discrete-time, finite-state hidden Markov model, we would like to estimate the sampling distribution of a statistic. The bootstrap method is employed to approximate the confidence regions of a…
This article provides the first procedure for computing a fully data-dependent interval that traps the mixing time $t_{\text{mix}}$ of a finite reversible ergodic Markov chain at a prescribed confidence level. The interval is computed from…