Prefix-free parsing for merging big BWTs
Data Structures and Algorithms
2025-06-09 v2
Abstract
When building Burrows-Wheeler Transforms (BWTs) of truly huge datasets, prefix-free parsing (PFP) can use an unreasonable amount of memory. In this paper we show how if a dataset can be broken down into small datasets that are not very similar to each other -- such as collections of many copies of genomes of each of several species, or collections of many copies of each of the human chromosomes -- then we can drastically reduce PFP's memory footprint by building the BWTs of the small datasets and then merging them into the BWT of the whole dataset.
Cite
@article{arxiv.2506.03294,
title = {Prefix-free parsing for merging big BWTs},
author = {Diego Diaz-Dominguez and Travis Gagie and Veronica Guerrini and Ben Langmead and Zsuzsanna Liptak and Giovanni Manzini and Francesco Masillo and Vikram Shivakumar},
journal= {arXiv preprint arXiv:2506.03294},
year = {2025}
}