Related papers: Reducing Communication in Algebraic Multigrid with…

Communication-aware control is essential to reduce costs and complexity in large-scale networks. However, it is challenging to simultaneously determine a sparse communication topology and achieve high performance and robustness. This work…

We consider the problem of how to reduce the cost of communication that is required for the parallel training of a neural network. The state-of-the-art method, Bulk Synchronous Parallel Stochastic Gradient Descent (BSP-SGD), requires many…

Synchronous mini-batch SGD is state-of-the-art for large-scale distributed machine learning. However, in practice, its convergence is bottlenecked by slow communication rounds between worker nodes. A natural solution to reduce communication…

Isogeometric analysis (IgA) offers enhanced approximation capabilities for the discretization of elliptic boundary-value problems, yet it results in large, sparse, and increasingly ill-conditioned linear systems due to higher…

Strength-of-connection algorithms play a key role in algebraic multigrid (AMG). Specifically, they determine which matrix nonzeros are classified as weak and so ignored when coarsening matrix graphs and defining interpolation sparsity…

Communication-avoiding and pipelined variants of Krylov solvers are critical for the scalability of linear system solvers on future exascale architectures. We present low synchronization variants of iterated classical (CGS) and modified…

Problems arising in Earth's mantle convection involve finding the solution to Stokes systems with large viscosity contrasts. These systems contain localized features which, even with adaptive mesh refinement, result in linear systems that…

High network communication cost for synchronizing gradients and parameters is the well-known bottleneck of distributed training. In this work, we propose TernGrad that uses ternary gradients to accelerate distributed deep learning in data…

In this paper, we review the parallel and distributed optimization algorithms based on alternating direction method of multipliers (ADMM) for solving "big data" optimization problem in smart grid communication networks. We first introduce…

Collective communications, namely the patterns allgatherv, reduce_scatter, and allreduce in message-passing systems are optimised based on measurements at the installation time of the library. The algorithms used are set up in an…

We study an element agglomeration coarsening strategy that requires data redistribution at coarse levels when the number of coarse elements becomes smaller than the used computational units (cores). The overall procedure generates coarse…

The multi-robot unlabeled motion planning problem of concurrently assigning robots to goals and generating safe trajectories is central in many collaborative tasks. Recent Graph Neural Network methods offer scalable decentralized solutions…

First-order optimization methods, such as stochastic gradient descent (SGD) and its variants, are widely used in machine learning applications due to their simplicity and low per-iteration costs. However, they often require larger numbers…

Adaptive gradient methods including Adam, AdaGrad, and their variants have been very successful for training deep learning models, such as neural networks. Meanwhile, given the need for distributed computing, distributed optimization…

The future of main memory appears to lie in the direction of new technologies that provide strong capacity-to-performance ratios, but have write operations that are much more expensive than reads in terms of latency, bandwidth, and energy.…

Algebraic multigrid (AMG) solvers and preconditioners are some of the fastest numerical methods to solve linear systems, particularly in a parallel environment, scaling to hundreds of thousands of cores. Most AMG methods and theory assume a…

The setup cost of a modern solver such as DD-\alpha AMG (Wuppertal Multigrid) is a significant contribution to the total time spent on solving the Dirac equation, and in HMC it can even be dominant. We present an improved implementation of…

Training LLMs relies on distributed implementations using multiple GPUs to compute gradients in parallel with sharded optimizers. However, synchronizing gradients in data parallel setups introduces communication overhead that grows with the…

Topology optimization for large scale problems continues to be a computational challenge. Several works exist in the literature to address this topic, and all make use of iterative solvers to handle the linear system arising from the Finite…

Parallel implementations of linear iterative solvers generally alternate between phases of data exchange and phases of local computation. Increasingly large problem sizes on more heterogeneous systems make load balancing and network layout…