English
Related papers

Related papers: Tokenisation is NP-Complete

200 papers

We show that the computational problem CONSENSUS-HALVING is PPA-complete, the first PPA-completeness result for a problem whose definition does not involve an explicit circuit. We also show that an approximate version of this problem is…

Computational Complexity · Computer Science 2017-11-15 Aris Filos-Ratsikas , Paul W. Goldberg

Consider a problem where we are given a bipartite graph H with vertices arranged on two horizontal lines in the plane, such that the two sets of vertices placed on the two lines form a bipartition of H. We additionally require that H admits…

Computational Complexity · Computer Science 2017-12-27 Grzegorz Guśpiel

By considering a discrete tape where each cell corresponds to an integer, thus to a possible sum, a pseudo-polynomial solution can be given to subset sum problem, which is an NP-complete problem and a cornerstone application for this study,…

Computational Complexity · Computer Science 2024-01-08 Yigit Oktar

We study the problem of synthesizing string to string transformations from a set of input/output examples. The transformations we consider are expressed using deterministic finite automata (DFA) that read pairs of letters, one letter from…

Formal Languages and Automata Theory · Computer Science 2018-06-06 Jad Hamza , Viktor Kunčak

This paper proposes a method to optimize tokenization for the performance improvement of already trained downstream models. Our method generates tokenization results attaining lower loss values of a given downstream model on the training…

Computation and Language · Computer Science 2023-04-24 Tatsuya Hiraoka , Tomoya Iwakura

It is well known that computing a minimum DFA consistent with a given set of positive and negative examples is NP-hard. Previous work has identified conditions on the input sample under which the problem becomes tractable or remains hard.…

Formal Languages and Automata Theory · Computer Science 2026-01-21 Radu Cosmin Dumitru , Ryo Yoshinaka , Ayumi Shinohara

Itemset mining is one of the most studied tasks in knowledge discovery. In this paper we analyze the computational complexity of three central itemset mining problems. We prove that mining confident rules with a given item in the head is…

Databases · Computer Science 2020-12-09 Christian Bessiere , Mohamed-Bachir Belaid , Nadjib Lazaar

In this note we introduce a notion of a generically (strongly generically) NP-complete problem and show that the randomized bounded version of the halting problem is strongly generically NP-complete.

Computational Complexity · Computer Science 2016-06-06 Alexei Miasnikov , Alexander Ushakov

Tokenization is fundamental in assembly code analysis, impacting intrinsic characteristics like vocabulary size, semantic coverage, and extrinsic performance in downstream tasks. Despite its significance, tokenization in the context of…

Artificial Intelligence · Computer Science 2025-11-07 Ahmed Mostafa , Raisul Arefin Nahid , Samuel Mulder

We investigate the constrained synchronization problem for weakly acyclic, or partially ordered, input automata. We show that, for input automata of this type, the problem is always in NP. Furthermore, we give a full classification of the…

Formal Languages and Automata Theory · Computer Science 2021-08-17 Stefan Hoffmann

Automatic segmentation of text into minimal content-bearing units is an unsolved problem even for languages like English. Spaces between words offer an easy first approximation, but this approximation is not good enough for machine…

cmp-lg · Computer Science 2008-02-03 I. Dan Melamed

We study the problem of anonymizing tables containing personal information before releasing them for public use. One of the formulations considered in this context is the $k$-anonymization problem: given a table, suppress a minimum number…

Computational Complexity · Computer Science 2010-04-28 Venkatesan T. Chakaravarthy , Vinayaka Pandit , Yogish Sabharwal

Motivated by satisfiability of constraints with function symbols, we consider numerical inequalities on non-negative integers. The constraints we consider are a conjunction of a linear system Ax = b and a conjunction of (non-)convex…

Logic in Computer Science · Computer Science 2022-10-21 Rodrigo Raya , Jad Hamza , Viktor Kunčak

Auto-completion is one of the most prominent features of modern information systems. The existing solutions of auto-completion provide the suggestions based on the beginning of the currently input character sequence (i.e. prefix). However,…

Information Retrieval · Computer Science 2016-11-24 Pengfei Xu , Jiaheng Lu

Given a graph where every vertex has exactly one labeled token, how can we most quickly execute a given permutation on the tokens? In (sequential) token swapping, the goal is to use the shortest possible sequence of swaps, each of which…

With distributed computing and mobile applications becoming ever more prevalent, synchronizing diverging replicas of the same data is a common problem. Reconciliation -- bringing two replicas of the same data structure as close as possible…

Information Theory · Computer Science 2022-08-10 Elod P. Csirmaz , Laszlo Csirmaz

We prove that path puzzles with complete row and column information--or equivalently, 2D orthogonal discrete tomography with Hamiltonicity constraint--are strongly NP-complete, ASP-complete, and #P-complete. Along the way, we newly…

Computational Geometry · Computer Science 2019-02-12 Jeffrey Bosboom , Erik D. Demaine , Martin L. Demaine , Adam Hesterberg , Roderick Kimball , Justin Kopinsky

This paper talk about the complexity of computation by Turing Machine. I take attention to the relation of symmetry and order structure of the data, and I think about the limitation of computation time. First, I make general problem named…

Computational Complexity · Computer Science 2010-09-24 Koji Kobayashi

In recent years, language models have become increasingly larger and more complex. However, the input representations for these models continue to rely on simple and greedy subword tokenization methods. In this paper, we propose a novel…

Computation and Language · Computer Science 2023-06-14 David Samuel , Lilja Øvrelid

Many natural combinatorial problems can be expressed as constraint satisfaction problems. This class of problems is known to be NP-complete in general, but certain restrictions on the form of the constraints can ensure tractability. The…

Computational Complexity · Computer Science 2020-10-05 Dmitriy Zhuk