Dalitz analysis of $D^{0}\to K^{-}\pi^{+}\eta$ decays at Belle

We present the results of the first Dalitz plot analysis of the decay $D^{0}\to K^{-}\pi^{+}\eta$. The analysis is performed on a data set corresponding to an integrated luminosity of 953 $\rm{fb}^{-1}$ collected by the Belle detector at the asymmetric-energy $e^{+}e^{-}$ KEKB collider. The Dalitz plot is well described by a combination of the six resonant decay channels $\bar{K}^{*}(892)^0\eta$, $K^{-}a_0(980)^+$, $K^{-}a_2(1320)^+$, $\bar{K}^{*}(1410)^0\eta$, $K^{*}(1680)^-\pi^{+}$ and $K_2^{*}(1980)^-\pi^{+}$, together with $K\pi$ and $K\eta$ S-wave components. The decays $K^{*}(1680)^{-}\to K^{-}\eta$ and $K_{2}^{*}(1980)^{-}\to K^{-}\eta$ are observed for the first time. We measure ratio of the branching fractions, $\frac{\mathcal{B}(D^{0}\to K^{-}\pi^{+}\eta)}{\mathcal{B}(D^{0}\to K^{-}\pi^{+})}=0.500\pm0.002{\rm(stat)}\pm0.020{\rm(syst)}\pm0.003{\rm (\mathcal{B}_{PDG})}$. Using the Dalitz fit result, the ratio $\frac{\mathcal{B}(K^{*}(1680)\to K\eta)}{\mathcal{B}(K^{*}(1680)\to K\pi)}$ is measured to be $0.11\pm0.02{\rm(stat)}^{+0.06}_{-0.04}{\rm(syst)}\pm0.04{\rm(\mathcal{B}_{\text{PDG}})}$; this is much lower than the theoretical expectations ($\approx1$) made under the assumption that $K^{*}(1680)$ is a pure $1^{3}D_1$ state. The product branching fraction $\mathcal{B}(D^0\to [K_2^{*}(1980)^-\to K^{-}\eta]\pi^{+})=(2.2^{+1.7}_{-1.9})\times10^{-4}$ is determined. In addition, the $\pi\eta^{\prime}$ contribution to the $a_0(980)^{\pm}$ resonance shape is confirmed with 10.1$\sigma$ statistical significance using the three-channel Flatt\'{e} model. We also measure $\mathcal{B}(D^0\to\bar{K}^{*}(892)^0\eta)=(1.41^{+0.13}_{-0.12})\%$. This is consistent with, and more precise than, the current world average $(1.02\pm0.30)\%$, deviates with a significance of more than $3\sigma$ from the theoretical predictions of (0.51-0.92)%.