We investigate theoretically the superfluid phase transition of helium-3 under nanoscale confinement of one spatial dimension realized in recent experiments. Instead of the 3×3 complex matrix order parameter found in the three-dimensional system, the quasi-two-dimensional superfluid is described by a reduced 3×2 complex matrix. It features a nodal quasiparticle spectrum, regardless of the value of the order parameter. The origin of the 3×2 order parameter is first illustrated via the two-particle Cooper problem, where Cooper pairs in the px and py orbitals are shown to have a lower bound-state energy than those in pz orbitals, hinting at their energetically favorable role at the phase transition. We then compute the Landau free energy under confinement within the mean-field approximation and show that the critical temperature for condensation of the 3×2 order parameter is larger than for other competing phases. Through exact minimization of the mean-field free energy, we show that mean-field theory predicts precisely two energetically degenerate superfluid orders to emerge at the transition that are not related by symmetry: The A-phase and the planar phase. Beyond the mean-field approximation, we show that strong-coupling corrections favor the A-phase observed in experiment, whereas weak-coupling perturbative renormalization group predicts the planar phase to be stable.