Related papers: Communication-optimal parallel and sequential QR a…
Gradient compression alleviates expensive communication in distributed deep learning by sending fewer values and its corresponding indices, typically via Allgather (AG). Training with high compression ratio (CR) achieves high accuracy like…
Recently, there has been an increasing interest in designing distributed convex optimization algorithms under the setting where the data matrix is partitioned on features. Algorithms under this setting sometimes have many advantages over…
Communication compression is an essential strategy for alleviating communication overhead by reducing the volume of information exchanged between computing nodes in large-scale distributed stochastic optimization. Although numerous…
The sheer sizes of modern datasets are forcing data-structure designers to consider seriously both parallel construction and compactness. To achieve those goals we need to design a parallel algorithm with good scalability and with low…
An extended formulation of a polyhedron $P$ is a linear description of a polyhedron $Q$ together with a linear map $\pi$ such that $\pi(Q)=P$. These objects are of fundamental importance in polyhedral combinatorics and optimization theory,…
Decentralized federated learning (DFL) is a variant of federated learning, where edge nodes only communicate with their one-hop neighbors to learn the optimal model. However, as information exchange is restricted in a range of one-hop in…
This paper studies the network optimization problem about which a group of agents cooperates to minimize a global function under practical constraints of finite bandwidth communication. Particularly, we propose an adaptive encoding-decoding…
In this paper we propose a new efficient message passing algorithm for decoding LDPC transmitted over a channel with strong phase noise. The algorithm performs approximate bayesian inference on a factor graph representation of the channel…
Consider a distributed control problem with a communication channel connecting the observer of a linear stochastic system to the controller. The goal of the controller is to minimize a quadratic cost function in the state variables and…
The algorithms in the current sequential numerical linear algebra libraries (e.g. LAPACK) do not parallelize well on multicore architectures. A new family of algorithms, the tile algorithms, has recently been introduced. Previous research…
The dominant contribution to communication complexity in factorizing a matrix using QR with column pivoting is due to column-norm updates that are required to process pivot decisions. We use randomized sampling to approximate this process…
Scaling foundation model training with Distributed Data Parallel (DDP) methods is bandwidth-limited. Existing infrequent communication methods like Local SGD were designed to synchronize only model parameters and cannot be trivially applied…
We resolve the min-max complexity of distributed stochastic convex optimization (up to a log factor) in the intermittent communication setting, where $M$ machines work in parallel over the course of $R$ rounds of communication to optimize…
We give lower bounds on the communication complexity required to solve several computational problems in a distributed-memory parallel machine, namely standard matrix multiplication, stencil computations, comparison sorting, and the Fast…
Federated Learning (FL) with quantization and deliberately added noise over wireless networks is a promising approach to preserve user differential privacy (DP) while reducing wireless resources. Specifically, an FL process can be fused…
Distributed parameter estimation for large-scale systems is an active research problem. The goal is to derive a distributed algorithm in which each agent obtains a local estimate of its own subset of the global parameter vector, based on…
We describe several features of parallel or distributed asynchronous iterative algorithms such as unbounded delays, possible out of order messages or flexible communication. We concentrate on the concept of macroiteration sequence which was…
In this paper we investigate the optimal latency of communications. Focusing on fixed rate communication without any feedback channel, this paper encompasses low-latency strategies with which one hop and multi-hop communication issues are…
Linear regression is fundamental to statistical analysis and machine learning, but its application to large-scale datasets necessitates distributed computing. The problem also arises in quantum computing, where handling extensive data…
The current computer architecture has moved towards the multi/many-core structure. However, the algorithms in the current sequential dense numerical linear algebra libraries (e.g. LAPACK) do not parallelize well on multi/many-core…