Related papers: Short-circuiting Rings for Low-Latency AllReduce
Many large datasets exhibit power-law statistics: The web graph, social networks, text data, click through data etc. Their adjacency graphs are termed natural graphs, and are known to be difficult to partition. As a consequence most…
We consider the problem of distilling efficient network topologies for collective communications. We provide an algorithmic framework for constructing direct-connect topologies optimized for the latency vs. bandwidth trade-off associated…
All-gather collective communication is one of the most important communication primitives in parallel and distributed computation, which plays an essential role in many HPC applications such as distributed Deep Learning (DL) with model and…
Secure multi-party computation (MPC) allows users to offload machine learning inference on untrusted servers without having to share their privacy-sensitive data. Despite their strong security properties, MPC-based private inference has not…
We propose that clusters interconnected with network topologies having minimal mean path length will increase their overall performance for a variety of applications. We approach our heuristic by constructing clusters of up to 36 nodes…
Distributed inference of large language models (LLMs) using tensor parallelism can introduce communication overheads of $20$% even over GPUs connected via NVLink, a high-speed GPU interconnect. Several techniques have been proposed to…
We study the complexity of finding communication trees with the lowest possible completion time for rooted, irregular gather and scatter collective communication operations in fully connected, $k$-ported communication networks under a…
Low-power wireless networks play an important role in the Internet of Things. Typically, these networks consist of a very large number of lossy and low-capacity devices, challenging the current state of the art in protocol design. In this…
Hybrid switching - in which a high bandwidth circuit switch (optical or wireless) is used in conjunction with a low bandwidth packet switch - is a promising alternative to interconnect servers in today's large scale data-centers. Circuit…
Natural, social, and artificial multi-agent systems usually operate in dynamic environments, where the ability to respond to changing circumstances is a crucial feature. An effective collective response requires suitable information…
Virtual topologies in peer-to-peer networks can reduce the traffic consumed by altering the logical connectivity of peers without altering the underlying network. However, such sparsely connected virtual topologies do not focus on the needs…
Smart grid is an energy network that integrates advanced power equipment, communication technology and control technology. It can transmit two-way power and data among all components of the grid at the same time. The existing smart grid…
In order to satisfy their ever increasing capacity and compute requirements, machine learning models are distributed across multiple nodes using numerous parallelism strategies. As a result, collective communications are often on the…
As LLM reasoning performance plateau, improving inference-time compute efficiency is crucial to mitigate overthinking and long thinking traces even for simple queries. Prior approaches including length regularization, adaptive routing, and…
This work presents joint iterative power allocation and interference suppression algorithms for spread spectrum networks which employ multiple hops and the amplify-and-forward cooperation strategy for both the uplink and the downlink. We…
Transformers have emerged as the architecture of choice for many state-of-the-art AI models, showcasing exceptional performance across a wide range of AI applications. However, the memory demands imposed by Transformers limit their ability…
We present and analyze a new robust feedback and transmit strategy for multiuser MIMO downlink communication systems, termed Rate Approximation (RA). RA combines the flexibility and robustness needed for reliable communications with the…
We present a distributed self-adjusting algorithm for skip graphs that minimizes the average routing costs between arbitrary communication pairs by performing topological adaptation to the communication pattern. Our algorithm is fully…
Linear sequence modeling approaches, such as linear attention, provide advantages like linear-time training and constant-memory inference over sequence lengths. However, existing sequence parallelism (SP) methods are either not optimized…
Training deep neural networks on large datasets can often be accelerated by using multiple compute nodes. This approach, known as distributed training, can utilize hundreds of computers via specialized message-passing protocols such as Ring…