Duplicate marking is a critical preprocessing step in gene sequence analysis to flag redundant reads arising from polymerase chain reaction(PCR) amplification and sequencing artifacts. Although Picard MarkDuplicates is widely recognized as the gold-standard tool, its single-threaded implementation and reliance on global sorting result in significant computational and resource overhead, limiting its efficiency on large-scale datasets. Here, we introduce FastDup: a high-performance, scalable solution that follows the speculation-and-test mechanism. FastDup achieves up to 20x throughput speedup and guarantees 100\% identical output compared to Picard MarkDuplicates. FastDup is a C++ program available from GitHub (https://github.com/zzhofict/FastDup.git) under the MIT license.
@article{arxiv.2505.06127,
title = {FastDup: a scalable duplicate marking tool using speculation-and-test mechanism},
author = {Zhonghai Zhang and Yewen Li and Ke Meng and Chunming Zhang and Guangming Tan},
journal= {arXiv preprint arXiv:2505.06127},
year = {2025}
}