Related papers: Skolem and Positivity Completeness of Ergodic Mark…
We consider Markov chains that obey the following general non-linear state space model: $\Phi_{k+1} = F(\Phi_k, \alpha(\Phi_k, U_{k+1}))$ where the function $F$ is $C^1$ while $\alpha$ is typically discontinuous and $\{U_k: k \in…
We consider general Markov chains with discrete time in an arbitrary measurable (phase) space and homogeneous in time. Markov chains are defined by the classical transition function which within the framework of the operator treatment…
We consider the problem of approximating the reachability probabilities in Markov decision processes (MDP) with uncountable (continuous) state and action spaces. While there are algorithms that, for special classes of such MDP, provide a…
We study the setting of \emph{performative reinforcement learning} where the deployed policy affects both the reward, and the transition of the underlying Markov decision process. Prior work~\parencite{MTR23} has addressed this problem…
A one-dimensional confined Nonlinear Random Walk is a tuple of $N$ diffeomorphisms of the unit interval driven by a probabilistic Markov chain. For generic such walks, we obtain a geometric characterization of their ergodic stationary…
Stationary ergodic processes with finite alphabets are estimated by finite memory processes from a sample, an n-length realization of the process, where the memory depth of the estimator process is also estimated from the sample using…
The goal of this paper is to develop a general method to establish conditional ergodicity of infinite-dimensional Markov chains. Given a Markov chain in a product space, we aim to understand the ergodic properties of its conditional…
This paper is a variation on the uniform spanning tree theme. We use random spanning forests to solve the following problem: for a Markov process on a finite set of size $n$, find a probability law on the subsets of any given size $m \leq…
How many parameters are required for a model to execute a given task? It has been argued that large language models, pre-trained via self-supervised learning, exhibit emergent capabilities such as multi-step reasoning as their number of…
In [ABM07], Abdulla et al. introduced the concept of decisiveness, an interesting tool for lifting good properties of finite Markov chains to denumerable ones. Later, this concept was extended to more general stochastic transition systems…
This paper studies the Linear Quadratic Regulator (LQR) problem for continuous-time Markov Jump Linear Systems (MJLS) governed by general finite-state Markov chains that may include transient, absorbing, or non-communicating states. The…
This paper investigates the stochastic linear quadratic (LQ, for short) optimal control problem of Markov regime switching system. The representation of the cost functional for the stochastic LQ optimal control problem of Markov regime…
We show that in a parametric family of linear recurrence sequences $a_1(\alpha) f_1(\alpha)^n + \ldots + a_k(\alpha) f_k(\alpha)^n$ with the coefficients $a_i$ and characteristic roots $f_i$, $i=1, \ldots,k$, given by rational functions…
Markov Chains offer ideal conditions for the study and mathematical modelling of a certain kind of situations depending on random variables. The basic concepts of the corresponding theory were introduced by Markov in 1907 on coding literary…
The law of the iterated logarithm (LIL) for the time-homogeneous Markov process with a unique invariant measure characterizes the almost sure maximum possible fluctuation of time averages around the ergodic limit. Whether a numerical…
The Skolem Problem asks to determine whether a given integer linear recurrence sequence has a zero term. This problem arises across a wide range of topics in computer science, including loop termination, formal languages, automata theory,…
It is well known that the distributions of hitting times in Markov chains are quite irregular, unless the limit as time tends to infinity is considered. We show that nevertheless for a typical finite irreducible Markov chain and for…
The paper deals with finite-state Markov decision processes (MDPs) with integer weights assigned to each state-action pair. New algorithms are presented to classify end components according to their limiting behavior with respect to the…
Understanding and predicting how complex systems respond to external perturbations is a central challenge in nonequilibrium statistical physics. Here we consider continuous-time Markov networks, which we subject to perturbations along a…
We introduce the notion of quantum Markov decision process (qMDP) as a semantic model of nondeterministic and concurrent quantum programs. It is shown by examples that qMDPs can be used in analysis of quantum algorithms and protocols. We…