Fast Botnet Detection From Streaming Logs Using Online Lanczos Method
Abstract
Botnet, a group of coordinated bots, is becoming the main platform of malicious Internet activities like DDOS, click fraud, web scraping, spam/rumor distribution, etc. This paper focuses on design and experiment of a new approach for botnet detection from streaming web server logs, motivated by its wide applicability, real-time protection capability, ease of use and better security of sensitive data. Our algorithm is inspired by a Principal Component Analysis (PCA) to capture correlation in data, and we are first to recognize and adapt Lanczos method to improve the time complexity of PCA-based botnet detection from cubic to sub-cubic, which enables us to more accurately and sensitively detect botnets with sliding time windows rather than fixed time windows. We contribute a generalized online correlation matrix update formula, and a new termination condition for Lanczos iteration for our purpose based on error bound and non-decreasing eigenvalues of symmetric matrices. On our dataset of an ecommerce website logs, experiments show the time cost of Lanczos method with different time windows are consistently only 20% to 25% of PCA.
Keywords
Cite
@article{arxiv.1812.07810,
title = {Fast Botnet Detection From Streaming Logs Using Online Lanczos Method},
author = {Zheng Chen and Xinli Yu and Chi Zhang and Jin Zhang and Cui Lin and Bo Song and Jianliang Gao and Xiaohua Hu and Wei-Shih Yang and Erjia Yan},
journal= {arXiv preprint arXiv:1812.07810},
year = {2018}
}