We look at a decision taken early in training a subword tokenizer, namely whether it should be the word-initial token that carries a special mark, or the word-final one. Based on surface-level considerations of efficiency and cohesion, as well as morphological coverage, we find that a Unigram LM tokenizer trained on pre-tokenized English text is better off marking the word-initial token, while one trained on raw text benefits from marking word ends. Our findings generalize across domains.
@article{arxiv.2208.01561,
title = {Lost in Space Marking},
author = {Cassandra L. Jacobs and Yuval Pinter},
journal= {arXiv preprint arXiv:2208.01561},
year = {2022}
}