Quantum phase diagram of the extended spin-3/2 Kitaev-Heisenberg model: A DMRG study

Recently there has been considerable excitement surrounding the promising realization of high-spin Kitaev material, such as the quasi-2D compound CrI$_3$ and CrGeTe$_3$. However, the stability of quantum spin liquids (QSL) against single ion anisotropy (SIA) in these materials and the global quantum phase diagram of the extended spin-3/2 Kitaev model with finite SIA remain unclear. In this study, we perform large-scale density matrix renormalization group (DMRG) to explore the quantum phase diagram of the generalized spin-3/2 Kitaev-Heisenberg (K-H) model accompanied with SIA $A_c$. In the $A_c=0$ limit, the spin-3/2 K-H model exhibits a quantum phase diagram similar to that of a spin-1/2 system, including two QSLs around antiferromagnetic and ferromagnetic Kitaev models. For models with finite $A_c$, we map out the quantum phase diagram around two Kitaev points and observe distinct types of in-plane vortex orders developed from these two QSL phases. Interestingly, series of nearly degenerate vortex configurations are discovered in each vortex phases. Using linear spin-wave theory, we demonstrate that these vortex configurations can be understood as a consequence of the quantum correction on a continuous family of degenerate classical states.