Conventional LLM inference architectures suffer from high energy and latency due to frequent data movement across memory hierarchies. We propose Ouroboros, a wafer-scale SRAM-based Computing-in-Memory (CIM) architecture that executes all operations in situ, eliminating off-chip migration. To maximize its limited first-level capacity, we introduce three innovations: Token-Grained Pipelining: Replaces sequence-level pipelining to mitigate length variations, boosting utilization and reducing activation storage. Distributed Dynamic KV Cache Management: Decouples memory from compute to leverage fragmented SRAM for efficient KV storage. Communication-Aware Mapping: Optimizes core allocation for locality and fault tolerance across the wafer. Experimental results show Ouroboros achieves average gains of 4.1× in throughput and 4.2× in energy efficiency, peaking at 9.1× and 17× for the 13B model. (*Due to the notification of arXiv "The Abstract field cannot be longer than 1,920 characters", the appeared Abstract is shortened. For the full Abstract, please download the Article.)
@article{arxiv.2603.02737,
title = {Ouroboros: Wafer-Scale SRAM CIM with Token-Grained Pipelining for Large Language Model Inference},
author = {Yiqi Liu and Yudong Pan and Mengdi Wang and Shixin Zhao and Haonan Zhu and Yinhe Han and Lei Zhang and Ying Wang},
journal= {arXiv preprint arXiv:2603.02737},
year = {2026}
}