Related papers: Skolem and Positivity Completeness of Ergodic Mark…

While reachability analysis is one of the most promising approaches for formal verification of dynamic systems, a major disadvantage preventing a more widespread application is the requirement to manually tune algorithm parameters such as…

Stochastic optimal control problems have a long tradition in applied probability, with the questions addressed being of high relevance in a multitude of fields. Even though theoretical solutions are well understood in many scenarios, their…

The framework of decision-making, modeled as a Markov Decision Process (MDP), typically assumes a single objective. However, practical scenarios often involve tradeoffs between multiple objectives. We address this in the Linear Quadratic…

The embedding problem for Markov chains is a famous problem in probability theory and only partial results are available up till now. In this paper, we propose a variant of the embedding problem called the reversible embedding problem which…

We solve the Chapman-Kolmogorov equation and study the exact splitting probabilities of the general stochastic process which describes polymer translocation through membrane pores within the broad class of Markov chains. Transition…

We revisit the classical problem of approximating a stochastic differential equation by a discrete-time and discrete-space Markov chain. Our construction iterates Caratheodory's theorem over time to match the moments of the increments…

This paper studies the exponential stability of random matrix products driven by a general (possibly unbounded) state space Markov chain. It is a cornerstone in the analysis of stochastic algorithms in machine learning (e.g. for parameter…

To overcome the curses of dimensionality and modeling of Dynamic Programming (DP) methods to solve Markov Decision Process (MDP) problems, Reinforcement Learning (RL) methods are adopted in practice. Contrary to traditional RL algorithms…

Large language model (LLM) agents increasingly operate as sequential software systems, but their reliability is often summarized by scalar benchmark metrics. Metrics such as pass$@k$, pass$^k$, and the reliability decay curve (RDC) are…

This paper presents a simple algorithm to check whether reachability probabilities in parametric Markov chains are monotonic in (some of) the parameters. The idea is to construct - only using the graph structure of the Markov chain and…

Let $X_n$ be a discrete time Markov chain with state space $S$ (countably infinite, in general) and initial probability distribution $\mu^{(0)} = (P(X_0=i_1),P(X_0=i_2),\cdots,)$. What is the probability of choosing in random some $k \in…

Consider a stochastic process $\{X(t)\}$ on a finite state space $ {\sf X}=\{1,\dots, d\}$. It is conditionally Markov, given a real-valued `input process' $\{\zeta(t)\}$. This is assumed to be small, which is modeled through the scaling,…

Lossy channel systems (LCSs) are systems of finite state automata that communicate via unreliable unbounded fifo channels. In order to circumvent the undecidability of model checking for nondeterministic LCSs, probabilistic models have been…

Discrete-time discrete-state finite Markov chains are versatile mathematical models for a wide range of real-life stochastic processes. One of most common tasks in studies of Markov chains is computation of the stationary distribution.…

We consider a broad class of dynamic programming (DP) problems that involve a partially linear structure and some positivity properties in their system equation and cost function. We address deterministic and stochastic problems, possibly…

Consider an ergodic Markov chain on a countable state space for which the return times have exponential tails. We show that the stationary version of any such chain is a finitary factor of an i.i.d. process. A key step is to show that any…

We propose a novel randomized linear programming algorithm for approximating the optimal policy of the discounted Markov decision problem. By leveraging the value-policy duality and binary-tree data structures, the algorithm adaptively…

Many applications in networked control require intermittent access of a controller to a system, as in event-triggered systems or information constrained control applications. Motivated by such applications and extending previous work on…

We revisit the source coding problem for a Markov chain under the assumption that the transmission times and how fast the Markov chain transitions its state happen at the same time-scale. Specifically, we assume that the transmission of…

Using ideas borrowed from topological dynamics and ergodic theory we introduce topological and metric versions of the recurrence property for general Markov chains. The main question of interest here is how large is the set of recurrent…