Spin liquid versus long range magnetic order in the frustrated body-centered tetragonal lattice

An $SU({\rm n})$-symmetric generalization of the Heisenberg model for quantum spin $S$ operators is used to investigate the geometrically frustrated body-centered tetragonal (BCT) lattice with antiferromagnetic interlayer coupling $J_1$ and intralayer first and second neighbor coupling $J_2$ and $J_3$. Using complementary representations of the spin operators, we study the phase diagram characterizing the ground state of the system. For small n, we find that the most stable solutions correspond to four different families of possible long range magnetic orders that are governed by $J_1$, $J_2$, and $J_3$. First, some possible instabilities of these phases are identified for $n=2$ in large $S$ expansions up to the linear spin-wave corrections. Then, using a fermionic representation of the $SU({\rm n})$ spin operators for $S=1/2$, we find that purely magnetic orders occur for ${\rm n}\le 3$ while spin-liquid (SL) solutions are stabilized for ${\rm n}\ge 10$. The SL solution governed by $J_1$ breaks the lattice translation symmetry. This Modulated SL is associated to a commensurate ordering wave vector $(1,1,1)$. For $4\le {\rm n}\le 9$, we show how competition between $J_1$, $J_2$, and $J_3$ can tune the ground state from beeing magnetically ordered to a SL state. We discuss the relevance of this scenario for correlated systems with BCT crystal structure.