Related papers: Input-Output Stability of Gradient Descent: A Disc…
This paper introduces and studies the notion of output-input stability, which represents a variant of the minimum-phase property for general smooth nonlinear control systems. The definition of output-input stability does not rely on a…
We consider a class of non-linear dynamics on a graph that contains and generalizes various models from network systems and control and study convergence to uniform agreement states using gradient methods. In particular, under the…
This paper provides a constructive passivity-based control approach to solve the set-point regulation problem for input-affine continuous nonlinear systems while considering saturation in the inputs. As customarily in passivity-based…
The paper studies a distributed gradient descent (DGD) process and considers the problem of showing that in nonconvex optimization problems, DGD typically converges to local minima rather than saddle points. The paper considers…
Stochastic Gradient Descent (SGD) is the workhorse algorithm of deep learning technology. At each step of the training phase, a mini batch of samples is drawn from the training dataset and the weights of the neural network are adjusted…
In a variety of problems originating in supervised, unsupervised, and reinforcement learning, the loss function is defined by an expectation over a collection of random variables, which might be part of a probabilistic model or the external…
The problem of stopping stochastic gradient descent (SGD) in an online manner, based solely on the observed trajectory, is a challenging theoretical problem with significant consequences for applications. While SGD is routinely monitored as…
This paper describes a robust linear time-invariant output-feedback control strategy to reduce turbulent fluctuations, and therefore skin-friction drag, in wall-bounded turbulent fluid flows, that nonetheless gives performance guarantees in…
Stochastic coordinate descent algorithms are efficient methods in which each iterate is obtained by fixing most coordinates at their values from the current iteration, and approximately minimizing the objective with respect to the remaining…
Finding latent structures in data is drawing increasing attention in diverse fields such as image and signal processing, fluid dynamics, and machine learning. In this work we examine the problem of finding the main modes of gradient flows.…
A dynamic backstepping method is proposed to design controllers for nonlinear systems in the pure-feedback form, for which the traditional backstepping method suffers from solving the implicit nonlinear algebraic equation. The idea of this…
We study the worst-case convergence rates of the proximal gradient method for minimizing the sum of a smooth strongly convex function and a non-smooth convex function whose proximal operator is available. We establish the exact worst-case…
Stochastic optimization methods have been hugely successful in making large-scale optimization problems feasible when computing the full gradient is computationally prohibitive. Using the theory of modified equations for numerical…
When training the parameters of a linear dynamical model, the gradient descent algorithm is likely to fail to converge if the squared-error loss is used as the training loss function. Restricting the parameter space to a smaller subset and…
Stochastic gradient descent procedures have gained popularity for parameter estimation from large data sets. However, their statistical properties are not well understood, in theory. And in practice, avoiding numerical instability requires…
This article investigates the robustness of gradient descent algorithms under perturbations. The concept of small-disturbance input-to-state stability (ISS) for discrete-time nonlinear dynamical systems is introduced, along with its…
Nonlinear control-affine systems with time-varying vector fields are considered in the paper. We propose a unified control design scheme with oscillating inputs for solving the trajectory tracking and stabilization problems. This…
In the vanishing learning rate regime, stochastic gradient descent (SGD) is now relatively well understood. In this work, we propose to study the basic properties of SGD and its variants in the non-vanishing learning rate regime. The focus…
Many existing region-of-attraction (ROA) analysis tools find difficulty in addressing feedback systems with large-scale neural network (NN) policies and/or high-dimensional sensing modalities such as cameras. In this paper, we tailor the…
This work shows that applying Gradient Descent (GD) with a fixed step size to minimize a (possibly nonconvex) quadratic function is equivalent to running the Power Method (PM) on the gradients. The connection between GD with a fixed step…