相关论文: Towards Quantifying the Preconditioning Effect of …
We prove a \emph{query complexity} lower bound on rank-one principal component analysis (PCA). We consider an oracle model where, given a symmetric matrix $M \in \mathbb{R}^{d \times d}$, an algorithm is allowed to make $T$ \emph{exact}…
Zhang et al. introduced a novel modification of Goldstein's classical subgradient method, with an efficiency guarantee of $O(\varepsilon^{-4})$ for minimizing Lipschitz functions. Their work, however, makes use of a nonstandard subgradient…
Adaptive optimization methods are widely recognized as among the most popular approaches for training Deep Neural Networks (DNNs). Techniques such as Adam, AdaGrad, and AdaHessian utilize a preconditioner that modifies the search direction…
Recent research has focused on $\ell_1$ penalized least squares (Lasso) estimators for high-dimensional linear regressions in which the number of covariates $p$ is considerably larger than the sample size $n$. However, few studies have…
Recent work [4] analyses the local convergence of Adam in a neighbourhood of an optimal solution for a twice-differentiable function. It is found that the learning rate has to be sufficiently small to ensure local stability of the optimal…
We present a sampling strategy suitable for optimization problems characterized by high-dimensional design spaces and noisy outputs. Such outputs can arise, for example, in time-averaged objectives that depend on chaotic states. The…
We study the complexity of producing $(\delta,\epsilon)$-stationary points of Lipschitz objectives which are possibly neither smooth nor convex, using only noisy function evaluations. Recent works proposed several stochastic zero-order…
We observe that the traditional use of DP with the Adam optimizer introduces a bias in the second moment estimation, due to the addition of independent noise in the gradient computation. This bias leads to a different scaling for low…
Adaptive optimizers with decoupled weight decay, such as AdamW, are the de facto standard for pre-training large transformer-based generative models. Yet the quadratic nature of the $\ell_2$ penalty embedded in weight decay drives all…
The determinant can be computed by classical circuits of depth $O(\log^2 n)$, and therefore it can also be computed in classical space $O(\log^2 n)$. Recent progress by Ta-Shma [Ta13] implies a method to approximate the determinant of…
Adaptive optimization algorithms, particularly Adam and its variant AdamW, are fundamental components of modern deep learning. However, their training dynamics lack comprehensive theoretical understanding, with limited insight into why…
In this paper we prove an infinite dimensional KAM theorem, in which the assumptions on the derivatives of perturbation in \cite{GT} are weakened from polynomial decay to logarithmic decay. As a consequence, we apply it to 1d quantum…
Adiabatic transport provides a powerful way to manipulate quantum states. By preparing a system in a readily initialised state and then slowly changing its Hamiltonian, one may achieve quantum states that would otherwise be inaccessible.…
The adaptive moment estimation algorithm Adam (Kingma and Ba) is a popular optimizer in the training of deep neural networks. However, Reddi et al. have recently shown that the convergence proof of Adam is problematic and proposed a variant…
We perform a detailed study of the dynamics of a nonlinear, one-dimensional oscillator driven by a periodic force under hysteretic damping, whose linear version was originally proposed and analyzed by Bishop in [1]. We first add a small…
We present details and expand on the framework leading to the recently introduced degenerate adiabatic perturbation theory [Phys. Rev. Lett. 104, 170406 (2010)], and on the formulation of the degenerate adiabatic theorem, along with its…
We consider a partial-feedback variant of the well-studied online PCA problem where a learner attempts to predict a sequence of $d$-dimensional vectors in terms of a quadratic loss, while only having limited feedback about the environment's…
The purpose of this paper is to study the existence of weak solutions for some classes of hemivariational problems in the Euclidean space $\mathbb{R}^d$ ($d\geq 3$). These hemivariational inequalities have a variational structure and,…
Unconstrained optimization problems become more common in scientific computing and engineering applications with the rapid development of artificial intelligence, and numerical methods for solving them more quickly and efficiently have been…
Bilevel optimization minimizes an objective function, defined by an upper-level problem whose feasible region is the solution of a lower-level problem. We study the oracle complexity of finding an $\epsilon$-stationary point with…