Related papers: A derivation of the master equation from path entr…
We develop the stochastic approach to thermodynamics based on the stochastic dynamics, which can be discrete (master equation) continuous (Fokker-Planck equation), and on two assumptions concerning entropy. The first is the definition of…
The entropy maximum approach (Maxent) was developed as a minimization of the subjective uncertainty measured by the Boltzmann--Gibbs--Shannon entropy. Many new entropies have been invented in the second half of the 20th century. Now there…
Markov processes are popular mathematical models, studied by theoreticians for their intriguing properties, and applied by practitioners for their flexible structure. With this book we teach how to model and analyze Markov processes. We…
In the development of stochastic integration and the theory of semimartingales, Markov processes have been a constant source of inspiration. Despite this historical interweaving, it turned out that semimartingales should be considered the…
For integers $n\geq r$, we treat the $r$th largest of a sample of size $n$ as an $\mathbb{R}^\infty$-valued stochastic process in $r$ which we denote $\mathbf{M}^{(r)}$. We show that the sequence regarded in this way satisfies the Markov…
We are interested in risk constraints for infinite horizon discrete time Markov decision processes (MDPs). Starting with average reward MDPs, we show that increasing concave stochastic dominance constraints on the empirical distribution of…
Motivated by applications in systems biology, we seek a probabilistic framework based on Markov processes to represent intracellular processes. We review the formal relationships between different stochastic models referred to in the…
We consider the problem of conditioning a Markov process on a rare event and of representing this conditioned process by a conditioning-free process, called the effective or driven process. The basic assumption is that the rare event used…
This paper proposes a new formulation for the dynamic resource allocation problem, which converts the traditional MDP model with known parameters and no capacity constraints to a new model with uncertain parameters and a resource capacity…
This work studies discrete-time discounted Markov decision processes with continuous state and action spaces and addresses the inverse problem of inferring a cost function from observed optimal behavior. We first consider the case in which…
Stochastic optimization is a vital field in the realm of mathematical optimization, finding applications in diverse areas ranging from operations research to machine learning. In this paper, we introduce a novel first-order optimization…
Traditional thermodynamic trade-off relations usually apply to quantities that depend linearly on probability distributions. In contrast, many important information-theoretic measures, such as entropies, are nonlinear and therefore…
In this paper, we are interested in optimal decisions in a partially observable Markov universe. Our viewpoint departs from the dynamic programming viewpoint: we are directly approximating an optimal strategic tree depending on the…
Many systems are partially stochastic in nature. We have derived data driven approaches for extracting stochastic state machines (Markov models) directly from observed data. This chapter provides an overview of our approach with numerous…
This paper examines a variety of classical optimization problems, including well-known minimization tasks and more general variational inequalities. We consider a stochastic formulation of these problems, and unlike most previous work, we…
We consider a pair of correlated processes {Z_n} and {S_n} (two sided), where the former is observable and the later is hidden. The uncertainty in the estimation of Z_n upon its finite past history is H(Z_n|Z_0^{n-1}), and for estimation of…
We present an elementary state augmentation method for a class of static risk measure applied to the total cost for both Markov decision processes and stochastic optimal control, such that dynamic programming equations can be derived on the…
We classify the rare events of structured, memoryful stochastic processes and use this to analyze sequential and parallel generators for these events. Given a stochastic process, we introduce a method to construct a new process whose…
We present an investigation of stochastic evolution in which a family of evolution equations in $L^1$ are driven by continuous-time Markov processes. These are examples of so-called piecewise deterministic Markov processes (PDMP's) on the…
We consider Markov jump processes on a graph described by a rate matrix that depends on various control parameters. We derive explicit expressions for the static responses of edge currents and steady-state probabilities. We show that they…