Modern GPU workloads, especially large language model (LLM) inference, suffer from kernel launch overheads and coarse synchronization that limit inter-kernel parallelism. Recent megakernel techniques fuse multiple operators into a single persistent kernel to eliminate launch gaps and expose inter-kernel parallelism, but struggle to handle dynamic shapes and data-dependent computation in real workloads. We present Event Tensor, a unified compiler abstraction for dynamic megakernels. Event Tensor encodes dependencies between tiled tasks, and enables first-class support for both shape and data-dependent dynamism. Built atop this abstraction, our Event Tensor Compiler (ETC) applies static and dynamic scheduling transformations to generate high-performance persistent kernels. Evaluations show that ETC achieves state-of-the-art LLM serving latency while significantly reducing system warmup overhead.
@article{arxiv.2604.13327,
title = {Event Tensor: A Unified Abstraction for Compiling Dynamic Megakernel},
author = {Hongyi Jin and Bohan Hou and Guanjie Wang and Ruihang Lai and Jinqi Chen and Zihao Ye and Yaxing Cai and Yixin Dong and Xinhao Cheng and Zhihao Zhang and Yilong Zhao and Yingyi Huang and Lijie Yang and Jinchen Jiang and Gabriele Oliaro and Jianan Ji and Xupeng Miao and Vinod Grover and Todd C. Mowry and Zhihao Jia and Tianqi Chen},
journal= {arXiv preprint arXiv:2604.13327},
year = {2026}
}
Comments
16 pages. 18 figures. accepted in MLSys 2026. References corrected