Related papers: Revisiting Parameter Server in LLM Post-Training

Communication is a crucial phase in the context of distributed training. Because parameter server (PS) frequently experiences network congestion, recent studies have found that training paradigms without a centralized server outperform the…

We propose a new data-centric synchronization framework for carrying out of machine learning (ML) tasks in a distributed environment. Our framework exploits the iterative nature of ML algorithms and relaxes the application agnostic bulk…

We consider distributed optimization under communication constraints for training deep learning models. We propose a new algorithm, whose parameter updates rely on two forces: a regular gradient step, and a corrective direction dictated by…

Training large language models (LLMs) increasingly relies on geographically distributed accelerators, causing prohibitive communication costs across regions and uneven utilization of heterogeneous hardware. We propose HALoS, a hierarchical…

Orthogonal signal-division multiplexing (OSDM) is an attractive alternative to conventional orthogonal frequency-division multiplexing (OFDM) due to its enhanced ability in peak-to-average power ratio (PAPR) reduction. Combining OSDM with…

Recently, there has been gradually more attention paid to Out-of-Distribution (OOD) performance prediction, whose goal is to predict the performance of trained models on unlabeled OOD test datasets, so that we could better leverage and…

Modern large language model (LLM) training is inherently dynamic: resource fluctuations, RLHF phase shifts, and cluster elasticity continually reshape the optimal parallelism layout, posing a significant challenge to existing training…

Ride-hailing platforms face significant challenges in optimizing order dispatching and driver repositioning operations in dynamic urban environments. Traditional approaches based on combinatorial optimization, rule-based heuristics, and…

Video diffusion models (VDMs) perform attention computation over the 3D spatio-temporal domain. Compared to large language models (LLMs) processing 1D sequences, their memory consumption scales cubically, necessitating parallel serving…

Parameter-shared pre-trained language models (PLMs) have emerged as a successful approach in resource-constrained environments, enabling substantial reductions in model storage and memory costs without significant performance compromise.…

We present OptiReduce, a new collective-communication system for the cloud with bounded, predictable completion times for deep-learning jobs in the presence of varying computation (stragglers) and communication (congestion and gradient…

The evolution of 5G and Beyond networks has enabled new applications with stringent end-to-end latency requirements, but providing reliable low-latency service with high throughput over public wireless networks is still a significant…

In this paper, we study the scheduling problem for downlink transmission in a multi-channel (e.g., OFDM-based) wireless network. We focus on a single cell, with the aim of developing a unifying framework for designing low-complexity…

The conjunction of edge intelligence and the ever-growing Internet-of-Things (IoT) network heralds a new era of collaborative machine learning, with federated learning (FL) emerging as the most prominent paradigm. With the growing interest…

Today, we have required to accommodate a large number of users under a single base station. This can be possible only if we have some flexibility over the spectrum. Previously we have lots of multiplexing methods to accommodate large number…

Data-parallel (DP) load balancing has emerged as a first-order bottleneck in large-scale LLM serving. When a model is sharded across devices via tensor parallelism (TP) or expert parallelism (EP) and replicated across many DP workers, every…

Recently, a new precoding technique called orthogonal time-frequency signaling (OTFS) has been proposed for time- and frequency-selective communication channels. OTFS precodes a data frame with a complete set of spreading sequences and…

Multiple Input Multiple Output (MIMO) Orthogonal Frequency Division Multiplexing (OFDM) is a promising transmission scheme for high performance broadband wireless communications. However, this technique suffers from a major drawback which…

Windowed orthogonal frequency-division multiplexing (OFDM) and wavelet OFDM have been proposed as medium access techniques for broadband communications over the power line network by the standard IEEE 1901. Windowed OFDM has been…

Data Parallelism (DP), Tensor Parallelism (TP), and Pipeline Parallelism (PP) are the three strategies widely adopted to enable fast and efficient Large Language Model (LLM) training. However, these approaches rely on data-intensive…