Mott Physics and Topological Phase Transition in Correlated Dirac Fermions

We investigate the interplay between the strong correlation and the spin-orbital coupling in the Kane-Mele-Hubbard model and obtain the qualitative phase diagram via the variational cluster approach. We identify, through an increase of the Hubbard $U$, the transition from the topological band insulator to either the spin liquid phase or the easy-plane antiferromagnetic insulating phase, depending on the strength of the spin-orbit coupling. A nontrivial evolution of the bulk bands in the topological quantum phase transition is also demonstrated.