Related papers: Descent modulus and applications
We introduce $\mathbf{G}$radient Descent with $\mathbf{A}$daptive $\mathbf{M}$omentum $\mathbf{S}$caling ($\mathbf{Grams}$), a novel optimization algorithm that decouples the direction and magnitude of parameter updates in deep learning.…
When smoothing a function $f$ via convolution with some kernel, it is often desirable to adapt the amount of smoothing locally to the variation of $f$. For this purpose, the constant smoothing coefficient of regular convolutions needs to be…
An influential line of recent work has focused on the generalization properties of unregularized gradient-based learning procedures applied to separable linear classification with exponentially-tailed loss functions. The ability of such…
We consider the problem of maximizing non-negative non-decreasing set functions. Although most of the recent work focus on exploiting submodularity, it turns out that several objectives we encounter in practice are not submodular.…
Consider a class of functions of one real variable with the following uniqueness property: if a function f(x) from the class vanishes on a set of positive measure, then f is the zero function. In many instances, we would like to have a…
Many statistical $M$-estimators are based on convex optimization problems formed by the combination of a data-dependent loss function with a norm-based regularizer. We analyze the convergence rates of projected gradient and composite…
Shadowing trajectories are model trajectories consistent with a sequence of observations of a system, given a distribution of observational noise. The existence of such trajectories is a desirable property of any forecast model. Gradient…
Many tasks in machine learning and signal processing can be solved by minimizing a convex function of a measure. This includes sparse spikes deconvolution or training a neural network with a single hidden layer. For these problems, we study…
In deep learning, classification tasks are formalized as optimization problems often solved via the minimization of the cross-entropy. However, recent advancements in the design of objective functions allow the usage of the $f$-divergence…
In this paper, we address the optimization problem of minimizing $Q(df_x)$ over a Hadamard manifold ${\cal M}$, where $f$ is a convex function on ${\cal M}$, $df_x$ is the differential of $f$ at $x \in {\cal M}$, and $Q$ is a function on…
A generalization of the classical Sard theorem in the plane is the following. Let $f$ be a function defined on a subset $A\subset{\mathbb R}^2$. If $f$ has modulus of continuity $\omega(r)\lesssim r^2$, then $f(A)\subset{\mathbb R}$ has…
We study the iteration complexity of stochastic gradient descent (SGD) for minimizing the gradient norm of smooth, possibly nonconvex functions. We provide several results, implying that the $\mathcal{O}(\epsilon^{-4})$ upper bound of…
We consider the gradient method with variable step size for minimizing functions that are definable in o-minimal structures on the real field and differentiable with locally Lipschitz gradients. We prove that global convergence holds if…
We study the type of solutions to which stochastic gradient descent converges when used to train a single hidden-layer multivariate ReLU network with the quadratic loss. Our results are based on a dynamical stability analysis. In the…
Stochastic Gradient Descent (SGD) based methods have been widely used for training large-scale machine learning models that also generalize well in practice. Several explanations have been offered for this generalization performance, a…
In this paper, a sequential search method for finding the global minimum of an objective function is presented, The descent gradient search is repeated until the global minimum is obtained. The global minimum is located by a process of…
We define a number of natural (from geometric and combinatorial points of view) deformation spaces of valuations on finite graphs, and study functions over these deformation spaces. These functions include both direct metric invariants…
We develop new sub-optimality bounds for gradient descent (GD) that depend on the conditioning of the objective along the path of optimization rather than on global, worst-case constants. Key to our proofs is directional smoothness, a…
In many naturally occurring optimization problems one needs to ensure that the definition of the optimization problem lends itself to solutions that are tractable to compute. In cases where exact solutions cannot be computed tractably, it…
Convolutional neural networks are widely used in imaging and image recognition. Learning such networks from training data leads to the minimization of a non-convex function. This makes the analysis of standard optimization methods such as…