Related papers: Automatic Adjoint Differentiation for special func…
Alternating Direction Method of Multipliers (ADMM) is a popular method for solving large-scale Machine Learning problems. Stochastic ADMM was proposed to reduce the per iteration computational complexity, which is more suitable for big data…
Several machine learning applications involve the optimization of higher-order derivatives (e.g., gradients of gradients) during training, which can be expensive in respect to memory and computation even with automatic differentiation. As a…
We present a new method for approximating real-valued functions on ${\mathbb R}^+$ by linear combinations of exponential functions with complex coefficients. The approach is based on a multi-point Pad\'e approximation of the Laplace…
Adjoint functors and projectivization in representation theory of partially ordered sets are used to generalize the algorithms of differentiation by a maximal and by a minimal point. Conceptual explanations are given for the combinatorial…
In this contribution, we present a full overview of the continuous stochastic gradient (CSG) method, including convergence results, step size rules and algorithmic insights. We consider optimization problems in which the objective function…
In this paper we demonstrate a new technique for deriving discrete adjoint and tangent linear models of finite element models. The technique is significantly more efficient and automatic than standard algorithmic differentiation techniques.…
We propose a mini-batching scheme for improving the theoretical complexity and practical performance of semi-stochastic gradient descent applied to the problem of minimizing a strongly convex composite function represented as the sum of an…
Recovering the underlying Directed Acyclic Graph (DAG) structures from observational data presents a formidable challenge, partly due to the combinatorial nature of the DAG-constrained optimization problem. Recently, researchers have…
The shape gradient quantifies the change in some figure of merit resulting from differential perturbations to a shape. Shape gradients can be applied to gradient-based optimization, sensitivity analysis, and tolerance calculation. An…
The alternating direction method of multipliers (ADMM) is commonly used for distributed model fitting problems, but its performance and reliability depend strongly on user-defined penalty parameters. We study distributed ADMM methods that…
Stochastic Gradient Descent (SGD) is an important algorithm in machine learning. With constant learning rates, it is a stochastic process that, after an initial phase of convergence, generates samples from a stationary distribution. We show…
An inexact accelerated stochastic Alternating Direction Method of Multipliers (AS-ADMM) scheme is developed for solving structured separable convex optimization problems with linear constraints. The objective function is the sum of a…
We characterize the best $L_{2}$ approximation to a multivariate function by linear combinations of ridge functions multiplied by some fixed weight functions. In the special case when the weight functions are constants, we propose explicit…
Douglas-Rachford splitting and its equivalent dual formulation ADMM are widely used iterative methods in composite optimization problems arising in control and machine learning applications. The performance of these algorithms depends on…
In neural networks with binary activations and or binary weights the training by gradient descent is complicated as the model has piecewise constant response. We consider stochastic binary networks, obtained by adding noises in front of…
Additive regression models have a long history in multivariate nonparametric regression. They provide a model in which each regression function depends only on a single explanatory variable allowing to obtain estimators at the optimal…
Due to their conjugate posteriors, Gaussian process priors are attractive for estimating the drift of stochastic differential equations with continuous time observations. However, their performance strongly depends on the choice of the…
We introduce a family of pairwise stochastic gradient estimators for gradients of expectations, which are related to the log-derivative trick, but involve pairwise interactions between samples. The simplest example of our new estimator,…
To a finite dimensional representation of a complex Lie group $G$, an associative algebra of adjoint covariant polynomial maps from the direct sum of $m$ copies of the Lie algebra $\mathfrak{g}$ of $G$ into an algebra of complex matrices is…
We develop algorithms to compute the differential Galois group $G$ associated to a parameterized second-order homogeneous linear differential equation of the form \[ \tfrac{\partial^2}{\partial x^2} Y + r_1 \tfrac{\partial}{\partial x} Y +…