In this paper, a novel high order semi-Lagrangian (SL) spectral volume (SV) method is proposed and studied for nonlinear Vlasov-Poisson (VP) simulations via operator splitting. The proposed algorithm combines both advantages of semi-Lagrangian and spectral volume approaches, exhibiting strong stability, robustness under large time steps, arbitrary high-order accuracy in space, local mass conservation, and positivity preservation. Numerical study of the SLSV method applied to the one-dimensional and two-dimensional transport equations, the Vlasov-Poisson system, the classical benchmark problems including Landau damping and two-stream instabilities is conducted, confirming the effectiveness, accuracy, and robustness of our algorithm in addressing complex nonlinear phenomena.