Related papers: A Quantitative Survey of Communication Optimizatio…

In this paper, a deep learning (DL) framework for the optimization of the resource allocation in multi-channel cellular systems with device-to-device (D2D) communication is proposed. Thereby, the channel assignment and discrete transmit…

In this paper, a distributed convex optimization algorithm, termed \emph{distributed coordinate dual averaging} (DCDA) algorithm, is proposed. The DCDA algorithm addresses the scenario of a large distributed optimization problem with…

Deep Neural Network (DNN) models are usually trained sequentially from one layer to another, which causes forward, backward and update locking's problems, leading to poor performance in terms of training time. The existing parallel…

We study distributed optimization problems over a network when the communication between the nodes is constrained, and so information that is exchanged between the nodes must be quantized. This imperfect communication poses a fundamental…

Decentralized optimization with time-varying networks is an emerging paradigm in machine learning. It saves remarkable communication overhead in large-scale deep training and is more robust in wireless scenarios especially when nodes are…

Recently there has been a surge of research on improving the communication efficiency of distributed training. However, little work has been done to systematically understand whether the network is the bottleneck and to what extent. In this…

Distributed optimization algorithms are widely used in many industrial machine learning applications. However choosing the appropriate algorithm and cluster size is often difficult for users as the performance and convergence rate of…

Many problems of interest for cyber-physical network systems can be formulated as Mixed-Integer Linear Programs in which the constraints are distributed among the agents. In this paper we propose a distributed algorithmic framework to solve…

In training of modern large natural language processing (NLP) models, it has become a common practice to split models using 3D parallelism to multiple GPUs. Such technique, however, suffers from a high overhead of inter-node communication.…

Decentralized state estimation in a communication-constrained sensor network is considered. The exchanged estimates are dimension-reduced to reduce the communication load using a linear mapping to a lower-dimensional space. The mean squared…

Large language models (LLMs), based on transformer architectures, have revolutionized numerous domains within artificial intelligence, science, and engineering due to their exceptional scalability and adaptability. However, the exponential…

Training large language models (LLMs) with increasingly long and varying sequence lengths introduces severe load imbalance challenges in large-scale data-parallel training. Recent frameworks attempt to mitigate these issues through data…

We consider the problem of clustering graph nodes over large-scale dynamic graphs, such as citation networks, images and web networks, when graph updates such as node/edge insertions/deletions are observed distributively. We propose…

This paper compares centralized and distributed methods to solve the power minimization problem with quality-of-service (QoS) constraints in the downlink (DL) of multi-cell Massive multiple-input multiple-output (MIMO) systems. In…

With the prevalence of Large Learning Models (LLM), Split Federated Learning (SFL), which divides a learning model into server-side and client-side models, has emerged as an appealing technology to deal with the heavy computational burden…

Pre-training models are an important tool in Natural Language Processing (NLP), while the BERT model is a classic pre-training model whose structure has been widely adopted by followers. It was even chosen as the reference model for the…

Despite the notable success of deep neural networks (DNNs) in solving complex tasks, the training process still remains considerable challenges. A primary obstacle is the substantial time required for training, particularly as high…

Resource allocation is a fundamental problem in Industrial Internet of Things (IIoT) systems, in which devices work together under limited communication bandwidth to complete diverse tasks. This paper proposes a communication-efficient…

Distributed Quantum Computing (DQC) enables scalability by interconnecting multiple QPUs. Among various DQC implementations, quantum data centers (QDCs), which utilize reconfigurable optical switch networks to link QPUs across different…

Communication overhead is one of the major obstacles to train large deep learning models at scale. Gradient sparsification is a promising technique to reduce the communication volume. However, it is very challenging to obtain real…