Adaptive and Efficient Log Parsing as a Cloud Service
Abstract
Logs are a critical data source for cloud systems, enabling advanced features like monitoring, alerting, and root cause analysis. However, the massive scale and diverse formats of unstructured logs pose challenges for adaptable, efficient, and accurate parsing methods. This paper introduces ByteBrain-LogParser, an innovative log parsing framework designed specifically for cloud environments. ByteBrain-LogParser employs a hierarchical clustering algorithm to allow real-time precision adjustments, coupled with optimizations such as positional similarity distance, deduplication, and hash encoding to enhance performance. Experiments on large-scale datasets show that it processes 229,000 logs per second on average, achieving an 840% speedup over the fastest baseline while maintaining accuracy comparable to state-of-the-art methods. Real-world evaluations further validate its efficiency and adaptability, demonstrating its potential as a robust cloud-based log parsing solution.
Cite
@article{arxiv.2504.09113,
title = {Adaptive and Efficient Log Parsing as a Cloud Service},
author = {Zeyan Li and Jie Song and Tieying Zhang and Tao Yang and Xiongjun Ou and Yingjie Ye and Pengfei Duan and Muchen Lin and Jianjun Chen},
journal= {arXiv preprint arXiv:2504.09113},
year = {2025}
}
Comments
Accepted by SIGMOD'25 Industry track