A faster algorithm for efficient longest common substring calculation for non-parametric entropy estimation in sequential data
Abstract
Non-parametric entropy estimation on sequential data is a fundamental tool in signal processing, capturing information flow within or between processes to measure predictability, redundancy, or similarity. Methods based on longest common substrings (LCS) provide a non-parametric estimate of typical set size but are often inefficient, limiting use on real-world data. We introduce LCSFinder, a new algorithm that improves the worst-case performance of LCS calculations from cubic to log-linear time. Although built on standard algorithmic constructs - including sorted suffix arrays and persistent binary search trees - the details require care to provide the matches required for entropy estimation on dynamically growing sequences. We demonstrate that LCSFinder achieves dramatic speedups over existing implementations on real and simulated data, enabling entropy estimation at scales previously infeasible in practical signal processing.
Cite
@article{arxiv.2510.13330,
title = {A faster algorithm for efficient longest common substring calculation for non-parametric entropy estimation in sequential data},
author = {Bridget Smart and Max Ward and Matthew Roughan},
journal= {arXiv preprint arXiv:2510.13330},
year = {2025}
}
Comments
Package available at https://pypi.org/project/LCSFinder, codebase available at https://github.com/bridget-smart/LCSFinder. It also includes a heuristic variant offering excellent practical performance with slower worst-case complexity, and has been integrated into the ProcessEntropy (https://pypi.org/project/ProcessEntropy/) package