Related papers: NS-RGS: Newton-Schulz based Riemannian gradient me…
Projected gradient descent and its Riemannian variant belong to a typical class of methods for low-rank matrix estimation. This paper proposes a new Nesterov's Accelerated Riemannian Gradient algorithm by efficient orthographic retraction…
As a representation learning method, nearest regularized subspace(NRS) algorithm is an effective tool to obtain both accuracy and speed for PolSAR image classification. However, existing NRS methods use the polarimetric feature vector but…
I show how a renormalization group (RG) method can be used to incrementally integrate the information in cosmological large-scale structure data sets (including CMB, galaxy redshift surveys, etc.). I show numerical tests for Gaussian…
We consider the problem of minimizing the average of a large number of smooth but possibly non-convex functions. In the context of most machine learning applications, each loss function is non-negative and thus can be expressed as the…
Over the past decade, stochastic algorithms have emerged as scalable and efficient tools for solving large-scale ill-posed inverse problems by randomly selecting subsets of equations at each iteration. However, due to the ill-posedness and…
SGD (Stochastic Gradient Descent) is a popular algorithm for large scale optimization problems due to its low iterative cost. However, SGD can not achieve linear convergence rate as FGD (Full Gradient Descent) because of the inherent…
Heavy-tailed noise in nonconvex stochastic optimization has garnered increasing research interest, as empirical studies, including those on training attention models, suggest it is a more realistic gradient noise condition. This paper…
The conjugate gradient (CG) method is an efficient iterative method for solving large-scale strongly convex quadratic programming (QP). In this paper we propose some generalized CG (GCG) methods for solving the $\ell_1$-regularized…
We propose a second renormalization group (SRG) in the triad representation of tensor networks. The SRG method improves two parts of the triad tensor renormalization group, which are the decomposition of intermediate tensors and the…
Recent advances in quantum simulator experiments enable unprecedented access to quantum many-body states through snapshot measurements of individual many-body configurations. Here, we introduce an exact renormalization group (RG)…
We present a family of algorithms, called descent algorithms, for optimizing convex and non-convex functions. We also introduce a new first-order algorithm, called rescaled gradient descent (RGD), and show that RGD achieves a faster…
Parallel implementations of stochastic gradient descent (SGD) have received significant research attention, thanks to excellent scalability properties of this algorithm, and to its efficiency in the context of training deep neural networks.…
A quasi-Newton method with cubic regularization is designed for solving Riemannian unconstrained nonconvex optimization problems. The proposed algorithm is fully adaptive with at most ${\cal O} (\epsilon_g^{-3/2})$ iterations to achieve a…
We propose a renormalization group (RG) approach to compare and collapse eigenvalue densities of random matrix models of complex systems across different system sizes. The approach is to fix a natural spectral scale by letting the model…
In this paper, we show that simple {Stochastic} subGradient Decent methods with multiple Restarting, named {\bf RSGD}, can achieve a \textit{linear convergence rate} for a class of non-smooth and non-strongly convex optimization problems…
We study finite-sum nonconvex optimization problems, where the objective function is an average of $n$ nonconvex functions. We propose a new stochastic gradient descent algorithm based on nested variance reduction. Compared with…
We present a variational renormalization group (RG) approach using a deep generative model based on normalizing flows. The model performs hierarchical change-of-variables transformations from the physical space to a latent space with…
We present a pedagogical discussion of Similarity Renormalization Group (SRG) methods, in particular the In-Medium SRG (IMSRG) approach for solving the nuclear many-body problem. These methods use continuous unitary transformations to…
We introduce a hybrid approach to applying the density matrix renormalization group (DMRG) to continuous systems, combining a grid approximation along one direction with a finite Gaussian basis set along the remaining two directions. This…
We propose an online learning algorithm for a class of machine learning models under a separable stochastic approximation framework. The essence of our idea lies in the observation that certain parameters in the models are easier to…