Related papers: Stochastic weight matrix dynamics during learning …

Herein we develop a dynamical foundation for fractional Brownian Motion. A clear relation is established between the asymptotic behaviour of the correlation function and diffusion in a dynamical system. Then, assuming that scaling is…

We propose an algorithm based on modulable hidden variables and adaptive step lengths, inspired by heuristic statistical physics and the replica method, to study the effect of mutual correlations and the emergent Wigner-Dyson distribution…

We take a Bayesian perspective to illustrate a connection between training speed and the marginal likelihood in linear models. This provides two major insights: first, that a measure of a model's training speed can be used to estimate its…

Machine learning systems often acquire biases by leveraging undesired features in the data, impacting accuracy variably across different sub-populations. Current understanding of bias formation mostly focuses on the initial and final stages…

In large-scale learning algorithms, the momentum term is usually included in the stochastic sub-gradient method to improve the learning speed because it can navigate ravines efficiently to reach a local minimum. However, step-size and…

The algorithms used to train neural networks, like stochastic gradient descent (SGD), have close parallels to natural processes that navigate a high-dimensional parameter space -- for example protein folding or evolution. Our study uses a…

In the last decade there has been increasing interest in the fields of random matrices, interacting particle systems, stochastic growth models, and the connections between these areas. For instance, several objects appearing in the limit of…

This paper proposes a framework to study the convergence of stochastic optimization and learning algorithms. The framework is modeled over the different challenges that these algorithms pose, such as (i) the presence of random additive…

We study the averaging-based distributed optimization solvers over random networks. We show a general result on the convergence of such schemes using weight-matrices that are row-stochastic almost surely and column-stochastic in expectation…

We consider the general class of time-homogeneous stochastic dynamical systems, both discrete and continuous, and study the problem of learning a representation of the state that faithfully captures its dynamics. This is instrumental to…

Weighted automata over the nonnegative reals form a fundamental model for quantitative languages. We show that, up to scaling, this model collapses to probabilistic automata. Concretely, we prove that every weighted automaton whose…

We study the problem of learning-to-learn: inferring a learning algorithm that works well on tasks sampled from an unknown distribution. As class of algorithms we consider Stochastic Gradient Descent on the true risk regularized by the…

Simple stochastic momentum methods are widely used in machine learning optimization, but their good practical performance is at odds with an absence of theoretical guarantees of acceleration in the literature. In this work, we aim to close…

The molecular dynamics lattice gas method maps a molecular dynamics simulation onto a lattice gas using a coarse-graining procedure. This is a novel fundamental approach to derive the lattice Boltzmann method by taking a Boltzmann average…

Diffusion with stochastic transport is investigated here when the random driving process is a very general Gaussian process, including Fractional Brownian motion. The purpose is the comparison with a deterministic PDE, which in certain…

Boltzmann machines are undirected graphical models with two-state stochastic variables, in which the logarithms of the clique potentials are quadratic functions of the node states. They have been widely studied in the neural computing…

A central question in computational neuroscience is how structure determines function in neural networks. The emerging high-quality large-scale connectomic datasets raise the question of what general functional principles can be gleaned…

Many physical systems are described by probability distributions that evolve in both time and space. Modeling these systems is often challenging to due large state space and analytically intractable or computationally expensive dynamics. To…

We develop a Monte Carlo wave function algorithm for the quantum linear Boltzmann equation, a Markovian master equation describing the quantum motion of a test particle interacting with the particles of an environmental background gas. The…

We discuss a possibility that the entire universe on its most fundamental level is a neural network. We identify two different types of dynamical degrees of freedom: "trainable" variables (e.g. bias vector or weight matrix) and "hidden"…