Related papers: A stable one-synchronization variant of reorthogon…
In distributed and federated learning algorithms, communication overhead is often reduced by performing multiple local updates between communication rounds. However, due to data heterogeneity across nodes and the local gradient noise within…
For a matrix ${\bf A}$ with linearly independent columns, this work studies to use its normalization $\bar{\bf A}$ and ${\bf A}$ itself to approximate its orthonormalization $\bf V$. We theoretically analyze the order of the approximation…
We propose a modified form of a tensor renormalization group algorithm for evaluating partition functions of classical statistical mechanical models on 2D lattices. This algorithm coarse-grains only the rows and columns of the lattice…
We perform a backward stability analysis of preconditioned sketched GMRES [Nakatsukasa and Tropp, SIAM J. Matrix Anal. Appl, 2024] for solving linear systems $Ax=b$, and show that the backward stability at iteration $i$ depends on the…
Although Federated Learning has been widely studied in recent years, there are still high overhead expenses in each communication round for large-scale models such as Vision Transformer. To lower the communication complexity, we propose a…
Synchronous mini-batch SGD is state-of-the-art for large-scale distributed machine learning. However, in practice, its convergence is bottlenecked by slow communication rounds between worker nodes. A natural solution to reduce communication…
Guessing Random Additive Noise Decoding (GRAND) is a recently proposed decoding method searching for the error pattern applied to the transmitted codeword. Ordered reliability bit GRAND (ORBGRAND) uses soft channel information to reorder…
Distributed Learning often suffers from Byzantine failures, and there have been a number of works studying the problem of distributed stochastic optimization under Byzantine failures, where only a portion of workers, instead of all the…
This paper addresses the problem of synchronizing orthogonal matrices over directed graphs. For synchronized transformations (or matrices), composite transformations over loops equal the identity. We formulate the synchronization problem as…
Principal component analysis (PCA) is a key statistical technique for multivariate data analysis. For large data sets the common approach to PCA computation is based on the standard NIPALS-PCA algorithm, which unfortunately suffers from…
Orthogonal geometric constructions are the basis of many many quantum error-correcting codes (QEC), but strict orthogonality constraints limit design flexibility and resource efficiency. We introduce a quasi-orthogonal geometric framework…
In the present study, we establish two new block variants of the Conjugate Orthogonal Conjugate Gradient (COCG) and the Conjugate A-Orthogonal Conjugate Residual (COCR) Krylov subspace methods for solving complex symmetric linear systems…
We develop and analyze a broad family of stochastic/randomized algorithms for inverting a matrix. We also develop specialized variants maintaining symmetry or positive definiteness of the iterates. All methods in the family converge…
Block coordinate descent (BCD) methods are prevalent in large scale optimization problems due to the low memory and computational costs per iteration, the predisposition to parallelization, and the ability to exploit the structure of the…
The so-called Burer-Monteiro method is a well-studied technique for solving large-scale semidefinite programs (SDPs) via low-rank factorization. The main idea is to solve rank-restricted, albeit non-convex, surrogates instead of the SDP.…
The block coordinate descent (BCD) method is widely used for minimizing a continuous function f of several block variables. At each iteration of this method, a single block of variables is optimized, while the remaining variables are held…
Block-coordinate descent (BCD) is a popular framework for large-scale regularized optimization problems with block-separable structure. Existing methods have several limitations. They often assume that subproblems can be solved exactly at…
This paper addresses two fundamental challenges in distributed online convex optimization: communication efficiency and optimization under limited feedback. We propose Online Compressed Gradient Tracking with one-point Bandit Feedback…
It is known that under fixed-rate information constraints, adaptive quantizers can be used to stabilize an open-loop-unstable linear system on $\mathbb{R}^n$ driven by unbounded noise. These adaptive schemes can be designed so that they…
The Gram-Schmidt Process (GSP) is used to convert a non-orthogonal basis (a set of linearly independent vectors, matrices, etc) into an orthonormal basis (a set of orthogonal, unit-length vectors, bi or tri dimensional matrices). The…