Modular invariance and uniqueness of $T\bar{T}$ deformed CFT

Any two dimensional quantum field theory that can be consistently defined on a torus is invariant under modular transformations. In this paper we study families of quantum field theories labeled by a dimensionful parameter $t$, that have the additional property that the energy of a state at finite $t$ is a function only of $t$ and of the energy and momentum of the corresponding state at $t=0$, where the theory becomes conformal. We show that under this requirement, the partition sum of the theory at $t=0$ uniquely determines the partition sum (and thus the spectrum) of the perturbed theory, to all orders in $t$, to be that of a $T\bar T$ deformed CFT. Non-perturbatively, we find that for one sign of $t$ (for which the energies are real) the partition sum is uniquely determined, while for the other sign we find non-perturbative ambiguities. We characterize these ambiguities and comment on their possible relations to holography.