Related papers: The Gradient descent method from the perspective o…
Using backpropagation to compute gradients of objective functions for optimization has remained a mainstay of machine learning. Backpropagation, or reverse-mode differentiation, is a special case within the general family of automatic…
Recent advances in convex optimization have leveraged computer-assisted proofs to develop optimized first-order methods that improve over classical algorithms. However, each optimized method is specially tailored for a particular problem…
We propose a categorical semantics of gradient-based machine learning algorithms in terms of lenses, parametrised maps, and reverse derivative categories. This foundation provides a powerful explanatory and unifying framework: it…
We review recent works on analyzing the dynamics of gradient-based algorithms in a prototypical statistical inference problem. Using methods and insights from the physics of glassy systems, these works showed how to understand…
While first-order optimization methods are usually designed to efficiently reduce the function value $f(x)$, there has been recent interest in methods efficiently reducing the magnitude of $\nabla f(x)$, and the findings show that the two…
Gradient-based optimization methods are commonly used to identify local optima in high-dimensional spaces. When derivatives cannot be evaluated directly, stochastic estimators can provide approximate gradients. However, these estimators'…
In this paper, we consider nonconvex decentralised optimisation and learning over a network of distributed agents. We develop an ADMM algorithm based on the Randomised Block Coordinate Douglas-Rachford splitting method which enables agents…
Shape optimization with constraints given by partial differential equations (PDE) is a highly developed field of optimization theory. The elegant adjoint formalism allows to compute shape gradients at the computational cost of a further PDE…
Recently, there has been an increasing interest in using tools from dynamical systems to analyze the behavior of simple optimization algorithms such as gradient descent and accelerated variants. This paper strengthens such connections by…
Gradient-based algorithms are one of the methods of choice for the optimisation of Markov Decision Processes. In this article we will present a novel approximate Newton algorithm for the optimisation of such models. The algorithm has…
For a given {\it misfit function}, a specified optimality measure of a model, its gradient describes the manner in which one may alter properties of the system to march towards a stationary point. The adjoint method, arising from…
This paper provides a theoretical and numerical comparison of classical first-order splitting methods for solving smooth convex optimization problems and cocoercive equations. From a theoretical point of view, we compare convergence rates…
The natural gradient descent optimisation technique is an efficient optimising protocol for broad classes of classical and quantum systems that takes the underlying geometry of the parameter manifold into account by means of using either…
Gradient-based iterative optimization methods are the workhorse of modern machine learning. They crucially rely on careful tuning of parameters like learning rate and momentum. However, one typically sets them using heuristic approaches…
An algorithm and associated strategy for solving polynomial systems within the optimization framework is presented. The algorithm and strategy are named, respectively, the penetrating gradient algorithm and the deepest descent strategy. The…
One of the most important parts of Artificial Neural Networks is minimizing the loss functions which tells us how good or bad our model is. To minimize these losses we need to tune the weights and biases. Also to calculate the minimum value…
Scientific studies often require the precise calculation of derivatives. In many cases an analytical calculation is not feasible and one resorts to evaluating derivatives numerically. These are error-prone, especially for higher-order…
In this paper, we propose a novel approach to automatically determine the batch size in stochastic gradient descent methods. The choice of the batch size induces a trade-off between the accuracy of the gradient estimate and the cost in…
This paper proposes a novel gradient based scalable procedure for $\mathcal{H}_{\infty}-$control design. We compute the gradient using algebraic Riccati equation and then couple it with a novel Armijo rule inspired step-size selection…
The numerical solution of algebraic tensor equations is a largely open and challenging task. Assuming that the operator is symmetric and positive definite, we propose two new gradient-descent type methods for tensor equations that…