Lightweight merging of compressed indices based on BWT variants
Abstract
In this paper we propose a flexible and lightweight technique for merging compressed indices based on variants of Burrows-Wheeler transform (BWT), thus addressing the need for algorithms that compute compressed indices over large collections using a limited amount of working memory. Merge procedures make it possible to use an incremental strategy for building large indices based on merging indices for progressively larger subcollections. Starting with a known lightweight algorithm for merging BWTs [Holt and McMillan, Bionformatics 2014], we show how to modify it in order to merge, or compute from scratch, also the Longest Common Prefix (LCP) array. We then expand our technique for merging compressed tries and circular/permuterm compressed indices, two compressed data structures for which there were hitherto no known merging algorithms.
Keywords
Cite
@article{arxiv.1903.01465,
title = {Lightweight merging of compressed indices based on BWT variants},
author = {Lavinia Egidi and Giovanni Manzini},
journal= {arXiv preprint arXiv:1903.01465},
year = {2019}
}
Comments
23 pages. A preliminary version appeared in Proc. SPIRE 2017, Springer Verlag LNCS 10508. arXiv admin note: text overlap with arXiv:1609.04618