Detecting the third family of compact stars with normalizing flows

We explore the anomaly detection framework based on Normalizing Flows (NF) models introduced in \cite{PhysRevC.106.065802} to detect the presence of a large (destabilising) dense matter phase transition in neutron star (NS) observations of masses and radii, and relate the feasibility of detection with parameters of the underlying mass-radius sequence, which is a functional of the dense matter equation of state. Once trained on simulated data featuring continuous $M(R)$ solutions (i.e., no phase transitions), NF is used to determine the likelihood of a first-order phase transition in a given set of $M(R)$ observations featuring a discontinuity, i.e., perform the anomaly detection. Different mock test sets, featuring two branch solutions in the $M(R)$ diagram, were parameterized by the NS mass at which the phase transition occurs, $M_c$, and the radius difference between the heaviest hadronic star and lightest hybrid star, $\Delta R$. We analyze the impact of these parameters on the NF performance in detecting the presence of a first-order phase transition. Among the results, we report that given a set of 15 stars with radius uncertainty of $0.2$ km, a detection of a two-branch solution is possible with 95\% accuracy if $\Delta R > 0.4$ km.