Angular instabilities of a homogeneous neutrino gas from (pseudo)scalar nonstandard self-interactions

We demonstrate that in the presence of (pseudo)scalar four-fermion nonstandard interactions, a homogeneous and isotropic gas of Majorana neutrinos placed in a strong magnetic field can exhibit an angular flavor instability mixing the two neutrino helicities (i.e., a `neutrino-antineutrino' instability). This instability is most pronounced for the inverted mass hierarchy and high neutrino number densities; at the same time, even a tiny transition magnetic moment of the order of $10^{-24}\mu_{\text{B}}$ is quite sufficient to trigger it. We study the properties of the neutrino-antineutrino mode using the linear stability analysis and then perform a numerical simulation to analyze the late-time properties of an oscillating neutrino gas with V-A and (pseudo)scalar interactions. The latter analysis reveals three phases of a chaotic, `thermalized' late-time state of neutrinos: along with a low-density phase with the SO(3) symmetry broken by V-A interactions and a symmetric high-density phase, another, (pseudo)scalar-induced phase with a broken rotational symmetry appears. The phases can be distinguished by the average late-time probabilities of (anti)neutrino flavors and/or by the neutrino-to-antineutrino ratio, the latter being a key quantity for the (pseudo)scalar phase. We thus show that observations of these quantities from a supernova could place new constraints on the (pseudo)scalar nonstandard neutrino interactions.