Nematic Spin Liquid Phase in a Frustrated Spin-1 System on the Square Lattice

Frustration in quantum spin systems promote a variety of novel quantum phases. An important example is the frustrated spin-$1$ model on the square lattice with the nearest-neighbor bilinear ($J_1$) and biquadratic ($K_1$) interactions. We provide strong evidence for a nematic spin liquid phase in a range of $K_1/J_1$ near the SU(3)-symmetric point ($J_1 = K_1$), based on the linear flavor-wave theory and extensive density matrix renormalization group calculation. This phase displays no spin dipolar or quadrupolar order, preserves translational symmetry but spontaneously breaks $C_4$ lattice rotational symmetry, and possesses fluctuations peaked at the wavevector $(\pi, 2\pi/3)$. The spin excitation gap drops rapidly with system size and appears to be gapless, and the nematic order is attributed to the dominant $(\pi, 2\pi/3)$ fluctuations. Our results provide a novel mechanism for electronic nematic order and, more generally, open up a new avenue to explore frustration-induced exotic ground states.