Related papers: On a variable step size modification of Hines' met…
Distributed optimization and learning algorithms are designed to operate over large scale networks enabling processing of vast amounts of data effectively and efficiently. One of the main challenges for ensuring a smooth learning process in…
To overcome the tradeoff of the conventional normalized least mean square (NLMS) algorithm between fast convergence rate and low steady-state misalignment, this paper proposes a variable step size (VSS) NLMS algorithm by devising a new…
In this work, we systematically investigate linear multi-step methods for differential equations with memory. In particular, we focus on the numerical stability for multi-step methods. According to this investigation, we give some…
The automatic selection of an appropriate time step size has been considered extensively in the literature. However, most of the strategies developed operate under the assumption that the computational cost (per time step) is independent of…
We address the weak numerical solution of stochastic differential equations driven by independent Brownian motions (SDEs for short). This paper develops a new methodology to design adaptive strategies for determining automatically the…
The stiffness of the Hodgkin-Huxley (HH) equations during an action potential (spike) limits the use of large time steps. We observe that the neurons can be evolved independently between spikes, $i.e.,$ different neurons can be evolved with…
We consider two formulations for distributed optimization wherein $N$ agents in a generic connected network solve a problem of common interest: distributed personalized optimization and consensus optimization. A new method termed DINAS…
In this work we study the problem of step size selection for numerical schemes, which guarantees that the numerical solution presents the same qualitative behavior as the original system of ordinary differential equations, by means of tools…
In continual learning, a learner has to keep learning from the data over its whole life time. A key issue is to decide what knowledge to keep and what knowledge to let go. In a neural network, this can be implemented by using a step-size…
The paper considers distributed stochastic optimization over randomly switching networks, where agents collaboratively minimize the average of all agents' local expectation-valued convex cost functions. Due to the stochasticity in gradient…
An algorithm is presented for momentum gradient descent optimization based on the first-order differential equation of the Newtonian dynamics. The fictitious mass is introduced to the dynamics of momentum for regularizing the adaptive…
We compare six numerical integrators' performance when simulating a regular spiking cortical neuron model whose 74-compartments are equipped with eleven membrane ion channels and Calcium dynamics. Four methods are explicit and two are…
Step size in continuous cellular automata (CA) plays an important role in the stability and behavior of self-organizing patterns. Continous CA dynamics are defined by formula very similar to numerical estimation of physics-based ordinary…
A scheme for stabilizing stochastic approximation iterates by adaptively scaling the step sizes is proposed and analyzed. This scheme leads to the same limiting differential equation as the original scheme and therefore has the same…
Training a neural network with the gradient descent algorithm gives rise to a discrete-time nonlinear dynamical system. Consequently, behaviors that are typically observed in these systems emerge during training, such as convergence to an…
Neurons are the central biological objects in understanding how the brain works. The famous Hodgkin-Huxley model, which describes how action potentials of a neuron are initiated and propagated, consists of four coupled nonlinear…
Gradient-based iterative optimization methods are the workhorse of modern machine learning. They crucially rely on careful tuning of parameters like learning rate and momentum. However, one typically sets them using heuristic approaches…
We present a new numerical multiscale integrator for stiff and highly oscillatory dynamical systems. The new algorithm can be seen as an improved version of the seamless Heterogeneous Multiscale Method by E, Ren, and Vanden-Eijnden and the…
We study a fixed step-size noisy distributed gradient descent algorithm for solving optimization problems in which the objective is a finite sum of smooth but possibly non-convex functions. Random perturbations are introduced to the…
Many time-dependent differential equations are equipped with invariants. Preserving such invariants under discretization can be important, e.g., to improve the qualitative and quantitative properties of numerical solutions. Recently,…