Superfluid phase transition of nanoscale-confined helium-3

We theoretically investigate the superfluid phase transition of helium-3 under nanoscale confinement of one spatial dimension realized in recent experiments. Instead of the 3x3 complex matrix order parameter found in the three-dimensional system, the quasi two-dimensional superfluid is described by a reduced 3x2 complex matrix. It features a nodal quasiparticle spectrum, regardless of the value of the order parameter. The origin of the 3x2 order parameter is first illustrated via the two-particle Cooper problem, where Cooper pairs in the $p_x$ and $p_y$ orbitals are shown to have a lower bound state energy than those in $p_z$ orbitals, hinting at their energetically favorable role at the phase transition. We then compute the Landau free energy under confinement within the mean-field approximation and show that the critical temperature for condensation of the 3x2 order parameter is larger than for other competing phases. Through exact minimization of the mean-field free energy, we show that mean-field theory predicts precisely two energetically degenerate superfluid orders to emerge at the transition that are not related by symmetry: the A-phase and the planar phase. Beyond the mean-field approximation, we show that strong-coupling corrections favor the A-phase observed in experiment, whereas weak-coupling perturbative renormalization group predicts the planar phase to be stable.