We leverage a streaming architecture based on ELK, Spark and Hadoop in order to collect, store, and analyse database connection logs in near real-time. The proposed system investigates outliers using unsupervised learning; widely adopted clustering and classification algorithms for log data, highlighting the subtle variances in each model by visualisation of outliers. Arriving at a novel solution to evaluate untagged, unfiltered connection logs, we propose an approach that can be extrapolated to a generalised system of analysing connection logs across a large infrastructure comprising thousands of individual nodes and generating hundreds of lines in logs per second.
@article{arxiv.1812.01941,
title = {Anomaly Detection for Network Connection Logs},
author = {Swapneel Mehta and Prasanth Kothuri and Daniel Lanza Garcia},
journal= {arXiv preprint arXiv:1812.01941},
year = {2018}
}