相关论文: Normalized Gradients for All

We provide the first proof of convergence for normalized error feedback algorithms across a wide range of machine learning problems. Despite their popularity and efficiency in training deep neural networks, traditional analyses of error…

We prove an analogue for a one-phase free boundary problem of the classical gradient bound for solutions to the minimal surface equation. It follows, in particular, that every energy-minimizing free boundary that is a graph is also smooth.…

Understanding why trained Transformers generalize well is a fundamental problem in modern machine learning theory, and complexity-based generalization bounds provide a principled way to study this question. While existing norm-based bounds…

We evaluate natural gradient, an algorithm originally proposed in Amari (1997), for learning deep models. The contributions of this paper are as follows. We show the connection between natural gradient and three other recently proposed…

In this paper, we prove new complexity bounds for zeroth-order methods in non-convex optimization with inexact observations of the objective function values. We use the Gaussian smoothing approach of Nesterov and Spokoiny [2015] and extend…

We present a strikingly simple proof that two rules are sufficient to automate gradient descent: 1) don't increase the stepsize too fast and 2) don't overstep the local curvature. No need for functional values, no line search, no…

Variational methods for revealing visual concepts learned by convolutional neural networks have gained significant attention during the last years. Being based on noisy gradients obtained via back-propagation such methods require the…

In this paper, we deal with the problem of optimizing a black-box smooth function over a full-dimensional smooth convex set. We study sets of feasible curves that allow to properly characterize stationarity of a solution and possibly carry…

Motivated by a wide variety of applications, ranging from stochastic optimization to dimension reduction through variable selection, the problem of estimating gradients accurately is of crucial importance in statistics and learning theory.…

In this paper, we consider linear elliptic systems from composite materials where the coefficients depend on the shape and might have the discontinuity between the subregions. We derive a function which is related to the gradient of the…

We provide improved error bounds for kernel-based numerical differentiation in terms of growth functions when kernels are of a finite smoothness, such as polyharmonic splines, thin plate splines or Wendland kernels. In contrast to existing…

We derive a new generalization of the nonlinear variational wave equation. We prove existence of local, smooth solutions for this system. As a limiting case, we recover the nonlinear variational wave equation.

Integrated gradients is prevalent within machine learning to address the black-box problem of neural networks. The explanations given by integrated gradients depend on a choice of base-point. The choice of base-point is not a priori obvious…

Universal methods for optimization are designed to achieve theoretically optimal convergence rates without any prior knowledge of the problem's regularity parameters or the accurarcy of the gradient oracle employed by the optimizer. In this…

We study the higher H\"older regularity of local weak solutions to a class of nonlinear nonlocal elliptic equations with kernels that satisfy a mild continuity assumption. An interesting feature of our main result is that the obtained…

The goal of this paper is to debunk and dispel the magic behind black-box optimizers and stochastic optimizers. It aims to build a solid foundation on how and why the techniques work. This manuscript crystallizes this knowledge by deriving…

We introduce an algorithm to solve linear inverse problems regularized with the total (gradient) variation in a gridless manner. Contrary to most existing methods, that produce an approximate solution which is piecewise constant on a fixed…

In this paper, we investigate interior gradient estimates for solutions to the mean curvature equation $$ \dive \left( \frac{\nabla u}{\sqrt{1 + |\nabla u|^2}} \right) = f(\nabla u)$$ under various nonlinear assumptions on the right-hand…

The key to generalization is controlling the complexity of the network. However, there is no obvious control of complexity -- such as an explicit regularization term -- in the training of deep networks for classification. We will show that…

Zeroth-order (ZO) optimization is popular in real-world applications that accessing the gradient information is expensive or unavailable. Recently, adaptive ZO methods that normalize gradient estimators by the empirical standard deviation…