Related papers: When Do Redundant Requests Reduce Latency ?
Serving systems for Large Language Models (LLMs) improve throughput by processing several requests concurrently. However, multiplexing hardware resources between concurrent requests involves non-trivial scheduling decisions. Practical…
Redundancy for straggler mitigation, originally in data download and more recently in distributed computing context, has been shown to be effective both in theory and practice. Analysis of systems with redundancy has drawn significant…
Several recently proposed techniques achieve latency reduction by trading it off for some amount of additional bandwidth usage. But how would one quantify whether the tradeoff is actually beneficial in a given system? We develop an economic…
We consider a large distributed service system consisting of $n$ homogeneous servers with infinite capacity FIFO queues. Jobs arrive as a Poisson process of rate $\lambda n/k_n$ (for some positive constant $\lambda$ and integer $k_n$). Each…
The maximum possible throughput (or the rate of job completion) of a multi-server system is typically the sum of the service rates of individual servers. Recent work shows that launching multiple replicas of a job and canceling them as soon…
Modern computationally-heavy applications are often time-sensitive, demanding distributed strategies to accelerate them. On the other hand, distributed computing suffers from the bottleneck of slow workers in practice. Distributed coded…
In most of modern enterprise systems, redundancy configuration is often considered to provide availability during the part of such systems is being patched. However, the redundancy may increase the attack surface of the system. In this…
To facilitate load balancing, distributed systems store data redundantly. We evaluate the load balancing performance of storage schemes in which each object is stored at $d$ different nodes, and each node stores the same number of objects.…
Redundancy for straggler mitigation, originally in data download and more recently in distributed computing context, has been shown to be effective both in theory and practice. Analysis of systems with redundancy has drawn significant…
A current trend in networking and cloud computing is to provide compute resources over widely dispersed places exemplified by initiatives like Network Function Virtualisation. This paves the way for a widespread service deployment and can…
To improve the utility of learning applications and render machine learning solutions feasible for complex applications, a substantial amount of heavy computations is needed. Thus, it is essential to delegate the computations among several…
Motivated by the growing demand for serving large language model inference requests, we study distributed load balancing for global serving systems with network latencies. We consider a fluid model in which continuous flows of requests…
In a multi-server system, how can one get better performance than random assignment of jobs to servers if queue-states cannot be queried by the dispatcher? A replication strategy has recently been proposed where $d$ copies of each arriving…
Traditional approaches to replication require client requests to be ordered before making them durable by copying them to replicas. As a result, clients must wait for two round-trip times (RTTs) before updates complete. In this paper, we…
The increased model capacity of Diffusion Transformers (DiTs) and the demand for generating higher resolutions of images and videos have led to a significant rise in inference latency, impacting real-time performance adversely. While prior…
Caching is crucial for system performance, but the delayed hit phenomenon, where requests queue during lengthy fetches after a cache miss, significantly degrades user-perceived latency in modern high-throughput systems. While prior works…
With significant increases in mobile device traffic slated for the foreseeable future, numerous technologies must be embraced to satisfy such demand. Notably, one of the more intriguing approaches has been blending on-device caching and…
Partial sharing allows providers to possibly pool a fraction of their resources when full pooling is not beneficial to them. Recent work in systems without sharing has shown that redundancy can improve performance considerably. In this…
Serverless computing has gained a significant traction in recent times because of its simplicity of development, deployment and fine-grained billing. However, while implementing complex services comprising databases, file stores, or more…
We consider the following distributed service model: jobs with unit mean, general distribution, and independent processing times arrive as a renewal process of rate $\lambda n$, with $0<\lambda<1$, and are immediately dispatched to one of…