Related papers: Norm-based convergence bounds for nonsymmetric alg…
This work provides the first convergence analysis for the Randomized Block Coordinate Descent method for minimizing a function that is both H\"older smooth and block H\"older smooth. Our analysis applies to objective functions that are…
We study the convergence of a random iterative sequence of a family of operators on infinite dimensional Hilbert spaces, inspired by the Stochastic Gradient Descent (SGD) algorithm in the case of the noiseless regression, as studied in [1].…
Let $\mathbb{B}(\mathcal{H})$ denote the $C^{\ast}$-algebra of all bounded linear operators on a Hilbert space $\big(\mathcal{H}, \langle\cdot, \cdot\rangle\big)$. Given a positive operator $A\in\B(\h)$, and a number $\lambda\in [0,1]$, a…
Adam-type methods, the extension of adaptive gradient methods, have shown great performance in the training of both supervised and unsupervised machine learning models. In particular, Adam-type optimizers have been widely used empirically…
Motivated by the training of Generative Adversarial Networks (GANs), we study methods for solving minimax problems with additional nonsmooth regularizers. We do so by employing \emph{monotone operator} theory, in particular the…
Dual averaging and gradient descent with their stochastic variants stand as the two canonical recipe books for first-order optimization: Every modern variant can be viewed as a descendant of one or the other. In the convex regime, these…
In this paper we investigate how standard nonlinear programming algorithms can be used to solve constrained optimization problems in a distributed manner. The optimization setup consists of a set of agents interacting through a…
Employing two distinct types of regularization terms, we propose two regularized extragradient methods for solving equilibrium problems on Hadamard manifolds. The sequences generated by these extragradient algorithms converge to a solution…
The problem of synchronization over a group $\mathcal{G}$ aims to estimate a collection of group elements $G^*_1, \dots, G^*_n \in \mathcal{G}$ based on noisy observations of a subset of all pairwise ratios of the form $G^*_i {G^*_j}^{-1}$.…
The alternating direction method of multipliers (ADMM) were extensively investigated in the past decades for solving separable convex optimization problems. Fewer researchers focused on exploring its convergence properties for the nonconvex…
The rate of convergence of the classical Thresholding Greedy Algorithm with respect to bases is studied in this paper. We bound the error of approximation by the product of both norms -- the norm of $f$ and the $A_1$-norm of $f$. We obtain…
In a recent paper Birke and Bissantz (2008) considered the problem of nonparametric estimation in inverse regression models with convolution-type operators. For multivariate predictors nonparametric methods suffer from the curse of…
We introduce randomized algorithms to Clifford's Geometric Algebra, generalizing randomized linear algebra to hypercomplex vector spaces. This novel approach has many implications in machine learning, including training neural networks to…
Let $H$ be a complex separable Hilbert space and $B(H)$ the algebra of all bounded linear operators on $H$. In this paper, we give considerable generalizations of the inequalities for norms of commutators of normal operators. Let $S, T \in…
Non-standard distributional approximations have received considerable attention in recent years. They often provide more accurate approximations in small samples, and theoretical improvements in some cases. This paper shows that the…
A classical approach to accurately estimating the covariance matrix \Sigma of a p-variate normal distribution is to draw a sample of size n > p and form a sample covariance matrix. However, many modern applications operate with much smaller…
In some cases, computational benefit can be gained by exploring the hyper parameter space using a deterministic set of grid points instead of a Markov chain. We view this as a numerical integration problem and make three unique…
We obtain non-symmetric upper and lower bounds on the rate of convergence of general monotone approximation/numerical schemes for parabolic Hamilton Jacobi Bellman Equations by introducing a new notion of consistency. We apply our general…
One of the most popular training algorithms for deep neural networks is the Adaptive Moment Estimation (Adam) introduced by Kingma and Ba. Despite its success in many applications there is no satisfactory convergence analysis: only local…
Modern machine learning architectures are often highly expressive. They are usually over-parameterized and can interpolate the data by driving the empirical loss close to zero. We analyze the convergence of Local SGD (or FedAvg) for such…