Related papers: Noise-induced degeneration in online learning
Traditional analyses of gradient descent optimization show that, when the largest eigenvalue of the loss Hessian - often referred to as the sharpness - is below a critical learning-rate threshold, then training is 'stable' and training loss…
Driven by the need to solve increasingly complex optimization problems in signal processing and machine learning, there has been increasing interest in understanding the behavior of gradient-descent algorithms in non-convex environments.…
We analyze the effect of weak-noise-induced transitions on the dynamics of the FitzHugh-Nagumo neuron model in a bistable state consisting of a stable fixed point and a stable unforced limit cycle. Bifurcation and slow-fast analysis give…
The natural gradient allows for more efficient gradient descent by removing dependencies and biases inherent in a function's parameterization. Several papers present the topic thoroughly and precisely. It remains a very difficult idea to…
Variational quantum algorithms rely on gradient based optimization to iteratively minimize a cost function evaluated by measuring output(s) of a quantum processor. A barren plateau is the phenomenon of exponentially vanishing gradients in…
Many applications in machine learning or signal processing involve nonsmooth optimization problems. This nonsmoothness brings a low-dimensional structure to the optimal solutions. In this paper, we propose a randomized proximal gradient…
We systematically investigate the interplay of disorder and time-homogeneous collisional noise in shaping the transport dynamics of an anisotropic XXZ spin chain. Using stochastic collision models to simulate interaction with the…
Gradient-based optimization of neural differential equations and other parameterized dynamical systems fundamentally relies on the ability to differentiate numerical solutions with respect to model parameters. In stiff systems, it has been…
We investigate the influence of nonstationary 1/f^mu noise, produced by interacting defects, on a quantum two-level system. Adopting a simple phenomenological model for this noise we describe exactly the corresponding dephasing in various…
Nonlinear damping, the change in damping rate with the amplitude of oscillations plays an important role in many electrical, mechanical and even biological oscillators. In novel technologies such as carbon nanotubes, graphene membranes or…
Diffusion models (DMs) are a powerful generative framework that have attracted significant attention in recent years. However, the high computational cost of training DMs limits their practical applications. In this paper, we start with a…
Feature learning is thought to be one of the fundamental reasons for the success of deep neural networks. It is rigorously known that in two-layer fully-connected neural networks under certain conditions, one step of gradient descent on the…
We study the complexity of training neural network models with one hidden nonlinear activation layer and an output weighted sum layer. We analyze Gradient Descent applied to learning a bounded target function on $n$ real-valued inputs. We…
This paper studies the stability of low-rank implicit regularization in perturbed deep matrix factorization, where the target matrix is corrupted by a noise matrix. We first derive sufficient spectral conditions under which gradient descent…
Recursion is a fundamental concept in the design of filters and audio systems. In particular, artificial reverberation systems that use delay networks depend on recursive paths to control both echo density and the decay rate of modal…
An important challenge in quantum science is to fully understand the efficiency of energy flow in networks. Here we present a simple and intuitive explanation for the intriguing observation that optimally efficient networks are not purely…
A law of large numbers for the empirical distribution of parameters of a one-layer artificial neural networks with sparse connectivity is derived for a simultaneously increasing number of both, neurons and training iterations of the…
We theoretically describe how weak signals may be efficiently transmitted throughout more than one frequency range in noisy excitable media by kind of stochastic multiresonance. This serves us here to reinterpret recent experiments in…
In this paper, we focus on online representation learning in non-stationary environments which may require continuous adaptation of model architecture. We propose a novel online dictionary-learning (sparse-coding) framework which…
The performance of stochastic gradient descent (SGD) depends critically on how learning rates are tuned and decreased over time. We propose a method to automatically adjust multiple learning rates so as to minimize the expected error at any…