Related papers: Characterizing the learning dynamics in extremum s…
In this work, we investigate the use of data-driven equation discovery for dynamical systems to model and forecast continuous-time dynamics of unconstrained optimization problems. To avoid expensive evaluations of the objective function and…
This paper deals with the gradient extremum seeking control for static scalar maps with actuators governed by distributed diffusion partial differential equations (PDEs). To achieve the real-time optimization objective, we design a…
The aim of global optimization is to find the global optimum of arbitrary classes of functions, possibly highly multimodal ones. In this paper we focus on the subproblem of global optimization for differentiable functions and we propose an…
In this work, we propose an optimization algorithm which we call norm-adapted gradient descent. This algorithm is similar to other gradient-based optimization algorithms like Adam or Adagrad in that it adapts the learning rate of stochastic…
In this paper, we study gradient-based classical extremum seeking (ES) for uncertain n-dimensional (nD) static quadratic maps in the presence of known large constant distinct input delays and large output constant delay with a small…
Neural networks trained with standard objectives exhibit behaviors characteristic of probabilistic inference: soft clustering, prototype specialization, and Bayesian uncertainty tracking. These phenomena appear across architectures -- in…
We study the connection between gradient-based meta-learning and convex op-timisation. We observe that gradient descent with momentum is a special case of meta-gradients, and building on recent results in optimisation, we prove convergence…
This paper studies the distributed optimization problem when the objective functions might be nondifferentiable and subject to heterogeneous set constraints. Unlike existing subgradient methods, we focus on the case when the exact…
In a variety of problems originating in supervised, unsupervised, and reinforcement learning, the loss function is defined by an expectation over a collection of random variables, which might be part of a probabilistic model or the external…
The paper deals with a well-known extremum seeking scheme by proving uniformity properties with respect to the amplitudes of the dither signal and of the cost function. Those properties are then used to show that the scheme guarantees the…
Decentralized optimization is a powerful paradigm that finds applications in engineering and learning design. This work studies decentralized composite optimization problems with non-smooth regularization terms. Most existing gradient-based…
We use differential equations based approaches to provide some {\it \textbf{physics}} insights into analyzing the dynamics of popular optimization algorithms in machine learning. In particular, we study gradient descent, proximal gradient…
In this paper, we investigate the problem of source seeking with a unicycle in the presence of local extrema. Our study is motivated by the fact that most of the existing source seeking methods follow the gradient direction of the signal…
Optimization in machine learning typically deals with the minimization of empirical objectives defined by training data. However, the ultimate goal of learning is to minimize the error on future data (test error), for which the training…
We adapt the gradient sampling algorithm to the local scoring algorithm to solve complex estimation problems based on an optimization of an objective function. This overcomes non-differentiability and non-smoothness of the objective…
Adaptive gradient methods, e.g. \textsc{Adam}, have achieved tremendous success in machine learning. Scaling the learning rate element-wisely by a certain form of second moment estimate of gradients, such methods are able to attain rapid…
We consider unconstrained randomized optimization of convex objective functions. We analyze the Random Pursuit algorithm, which iteratively computes an approximate solution to the optimization problem by repeated optimization over a…
Extremum seeking feedback is a powerful method to steer a dynamical system to an extremum of a partially or completely unknown map. It often requires advanced system-theoretic tools to understand the qualitative behavior of extremum seeking…
Gradient boosting is a prediction method that iteratively combines weak learners to produce a complex and accurate model. From an optimization point of view, the learning procedure of gradient boosting mimics a gradient descent on a…
Randomly initialized first-order optimization algorithms are the method of choice for solving many high-dimensional nonconvex problems in machine learning, yet general theoretical guarantees cannot rule out convergence to critical points of…