Public information contains valuable Cyber Threat Intelligence (CTI) that is used to prevent future attacks. While standards exist for sharing this information, much appears in non-standardized news articles or blogs. Monitoring online sources for threats is time-consuming and source selection is uncertain. Current research focuses on extracting Indicators of Compromise from known sources, rarely addressing new source identification. This paper proposes a CTI-focused crawler using multi-armed bandit (MAB) and various crawling strategies. It employs SBERT to identify relevant documents while dynamically adapting its crawling path. Our system ThreatCrawl achieves a harvest rate exceeding 25% and expands its seed by over 300% while maintaining topical focus. Additionally, the crawler identifies previously unknown but highly relevant overview pages, datasets, and domains.
@article{arxiv.2504.18375,
title = {Bandit on the Hunt: Dynamic Crawling for Cyber Threat Intelligence},
author = {Philipp Kuehn and Dilara Nadermahmoodi and Markus Bayer and Christian Reuter},
journal= {arXiv preprint arXiv:2504.18375},
year = {2025}
}