Related papers: SkyMemory: A LEO Edge Cache for Transformer Infere…
The widespread of Large Language Models (LLMs) marks a significant milestone in generative AI. Nevertheless, the increasing context length and batch size in offline LLM inference escalate the memory requirement of the key-value (KV) cache,…
The integration of Low Earth Orbit (LEO) satellite constellations into 5G and Beyond is essential to achieve efficient global connectivity. As LEO satellites are a global infrastructure with predictable dynamics, a pre-planned fair and…
In recent years, the emergence of large-scale Low-Earth-Orbit (LEO) satellite constellations has introduced unprecedented opportunities for global connectivity. However, routing efficiency and inter-shell communication remain key challenges…
The rapid advancement of low Earth orbit (LEO) satellite communication systems has significantly enhanced global connectivity, offering high-capacity, low-latency services crucial for next-generation applications. However, the dense…
LLMs are increasingly used world-wide from daily tasks to agentic systems and data analytics, requiring significant GPU resources. LLM inference systems, however, are slow compared to database systems, and inference performance and…
The rapid adoption of large language models (LLMs) is pushing AI accelerators toward increasingly powerful and specialized designs. Instead of further complicating software development with deeply hierarchical scratchpad memories (SPMs) and…
Quantum networking seeks to enable global entanglement distribution through terrestrial and free space channels; however, the exponential loss in these channels necessitates quantum repeaters with efficient, long lived quantum memories…
With numerous ongoing deployments owned by private companies and startups, dense satellite constellations deployed in low Earth orbit (LEO) will play a major role in the near future of wireless communications. In addition, the 3rd…
Low Earth Orbit (LEO) satellite networks introduce unique congestion control (CC) challenges due to frequent handovers, rapidly changing round-trip times (RTTs), and non-congestive loss. This paper presents the first comprehensive,…
During LLM inference, KVCache memory usage grows linearly with sequence length and batch size and often exceeds GPU capacity. Recent proposals offload KV states to host memory and reduce transfers using top-k attention. But their…
Caching is crucial in hybrid satellite-terrestrial networks to reduce latency, optimize throughput, and improve data availability by storing frequently accessed content closer to users, especially in bandwidth-limited satellite systems,…
Key-Value (KV) cache has become a de facto component of modern Large Vision-Language Models (LVLMs) for inference. While it enhances decoding efficiency in Large Language Models (LLMs), its direct adoption in LVLMs introduces substantial…
Large language models (LLMs) have shown exceptional performance and vast potential across diverse tasks. However, the deployment of LLMs with high performance in low-resource environments has garnered significant attention in the industry.…
Large low earth orbit (LEO) satellite networks such as SpaceX's Starlink constellation promise to deliver low-latency, high-bandwidth Internet access with global coverage. As an alternative to terrestrial fiber as a global Internet…
With the widespread deployment of long-context large language models (LLMs), there has been a growing demand for efficient support of high-throughput inference. However, as the key-value (KV) cache expands with the sequence length, the…
The construction of Low Earth Orbit (LEO) satellite constellations has recently spurred tremendous attention from academia and industry. 5G and 6G standards have specified LEO satellite network as a key component of 5G and 6G networks.…
Transformer-based large language models (LLMs) have already achieved remarkable results on long-text tasks, but the limited GPU memory (VRAM) resources struggle to accommodate the linearly growing demand for key-value (KV) cache as the…
Large language model (LLM) inference has been a prevalent demand in daily life and industries. The large tensor sizes and computing complexities in LLMs have brought challenges to memory, computing, and databus. This paper proposes a…
Low Earth orbit (LEO) satellite mega-constellations with hundreds or thousands of satellites and inter-satellite links (ISLs) have the potential to provide global end-to-end connectivity. Furthermore, if the physical distance between source…
The number of satellites, especially those operating in low-earth orbit (LEO), is exploding in recent years. Additionally, the use of COTS hardware into those satellites enables a new paradigm of computing: orbital edge computing (OEC). OEC…