Multistate memory systems have the ability to store and process more data in the same physical space as binary memory systems, making them a potential alternative to existing binary memory systems. In the past, it has been demonstrated that voltage-controlled magnetic anisotropy (VCMA) based writing is highly energy-efficient compared to other writing methods used in non-volatile nano-magnetic binary memory systems. In this study, we introduce a new, VCMA-based and skyrmion-mediated non-volatile ternary memory system using a perpendicular magnetic tunnel junction (p-MTJ) in the presence of room temperature thermal perturbation. We have also shown that ternary states {-1, 0, +1} can be implemented with three magnetoresistance values obtained from a p-MTJ corresponding to ferromagnetic up, down, and skyrmion state, with 99% switching probability in the presence of room temperature thermal noise in an energy-efficient way, requiring ~3 fJ energy on an average for each switching operation. Additionally, we show that our proposed ternary memory demonstrates an improvement in area and energy by at least 2X and ~60X respectively, compared to state-of-the-art spin-transfer torque (STT)-based non-volatile magnetic multistate memories. Furthermore, these three states can be potentially utilized for energy-efficient, high-density in-memory quantized deep neural network implementation.