Related papers: SkyMemory: A LEO Edge Cache for Transformer Infere…

LLM inference is essential for applications like text summarization, translation, and data analysis, but the high cost of GPU instances from Cloud Service Providers (CSPs) like AWS is a major burden. This paper proposes InferSave, a…

The Key-Value (KV) cache is integral to efficient autoregressive inference in large language models (LLMs), yet its unbounded growth in stateful multi-turn scenarios presents major challenges. This paper examines the interplay between KV…

Inter-satellite-link-enabled low-Earth-orbit (LEO) satellite constellations are evolving toward networked architectures that support constellation-level cooperation, enabling multiple satellites to jointly serve user terminals through…

With the skyrocketing costs of GPUs and their virtual instances in the cloud, there is a significant desire to use CPUs for large language model (LLM) inference. KV cache update, often implemented as allocation, copying, and in-place…

Satellite communication systems are a fundamental component in support of Europe's ambition to deploy smart and sustainable networks and services for the success of its digital economy. To cope with the 5G and beyond ever increasing demand…

Low Earth Orbit (LEO) satellite networks connect millions of devices on Earth and offer various services, such as data communications, remote sensing, and data harvesting. As the number of services increases, LEO satellite networks will…

Large Language Models (LLMs) have achieved impressive results across various tasks, yet their high computational demands pose deployment challenges, especially on consumer-grade hardware. Mixture of Experts (MoE) models provide an efficient…

The rapid development of communication technologies in the past decades has provided immense vertical opportunities for individuals and enterprises. However, conventional terrestrial cellular networks have unfortunately neglected the huge…

Utilizing Low Earth Orbit (LEO) satellite networks equipped with Inter-Satellite Links (ISL) is envisioned to provide lower delay compared to traditional optical networks. However, LEO satellites have constrained energy resources as they…

Large low-Earth orbit (LEO) satellite networks are being built to provide low-latency broadband Internet access to a global subscriber base. In addition to network transmissions, researchers have proposed embedding compute resources in…

As emerging massive constellations are intended to provide seamless connectivity for remote areas using hundreds of small low Earth orbit (LEO) satellites, new methodologies have great importance to study the performance of these networks.…

Computation in-memory is a promising non-von Neumann approach aiming at completely diminishing the data transfer to and from the memory subsystem. Although a lot of architectures have been proposed, compiler support for such architectures…

Existing analyses of ergodic capacity in satellite mega-constellations often rely on restrictive serving time assumptions or become intractable under realistic handover strategies. This paper develops a framework for characterising the…

To mitigate inter-satellite interference and payload limits in LEO mega-constellations, satellite clusters, groups of small cooperative satellites have been proposed to improve performance and reduce interference. The typical configuration…

Disaggregating memory from compute offers the opportunity to better utilize stranded memory in cloud data centers. It is important to cache data in the compute nodes and maintain cache coherence across multiple compute nodes. However, the…

Large Language Models (LLMs) demonstrate substantial potential across a diverse array of domains via request serving. However, as trends continue to push for expanding context sizes, the autoregressive nature of LLMs results in highly…

In this paper, we propose a terrestrial-satellite network (TSN) architecture to integrate the ultra-dense low earth orbit (LEO) networks and the terrestrial networks to achieve efficient data offloading. In TSN, each ground user can access…

Large language models (LLMs) are becoming increasingly capable at small parameter scales. At the same time, conventional cloud-centric deployment introduces challenges around data privacy, latency, and cost that are acute in operational…

Large Language Models are increasingly being deployed in datacenters. Serving these models requires careful memory management, as their memory usage includes static weights, dynamic activations, and key-value caches. While static weights…

How to efficiently serve LLMs in practice has become exceptionally challenging due to their prohibitive memory and computation requirements. In this study, we investigate optimizing the KV cache, whose memory footprint poses a critical…