Tutorial: $\varphi$-Transductions in OpenFst via the Gallic Semiring
Formal Languages and Automata Theory
2025-06-24 v1 Computation and Language
Abstract
OpenFst, a popular finite-state transducer library, supports -transitions but, due to an implementation constraint, they cannot be used with transducers in a straightforward way. In this short tutorial, we describe how one can use other functionality provided by OpenFst (namely, the Gallic semiring) to correctly implement -transductions and demonstrate it by implementing the MaxMatch (WordPiece) tokenization algorithm (Devlin et al., 2019; Song et al., 2021). Accompanying self-contained code examples are provided. https://www.openfst.org/twiki/pub/Contrib/FstContrib/phi_transduction_tutorial_code.tgz
Cite
@article{arxiv.2506.17942,
title = {Tutorial: $\varphi$-Transductions in OpenFst via the Gallic Semiring},
author = {Marco Cognetta and Cyril Allauzen},
journal= {arXiv preprint arXiv:2506.17942},
year = {2025}
}
Comments
8 pages, 2 figures, code included