Exceptional sensitivity near the bistable transition point of a hybrid quantum system

Phase transitions can dramatically alter system dynamics, unlocking new behavior and improving performance. Exceptional points (EPs), where the eigenvalues and corresponding eigenvectors of a coupled linear system coalesce, are particularly relevant for sensing applications as they can increase sensor response to external perturbations to a range of phenomena from optical phase shifts to gravitational waves. However, the coalescence of eigenstates at linear EPs amplifies noise, negating the signal-to-noise ratio (SNR) enhancement. Here, we overcome this limitation using nonlinearity, which exhibits exceptional SNR around a bistable transition point (BP). We couple a state-of-the-art diamond quantum sensor to a nonlinear Van der Pol oscillator, forming a self-oscillating hybrid system that exhibits both a single-valued and bistable phase. The boundaries between these phases are marked by both adiabatic and deterministic non-adiabatic transitions that enable chiral state switching and state coalescence at the BP. Crucially, NV magnetometry performed near the BP exhibits a 17x enhancement in SNR, achieving a record sensitivity of 170 fT/\sqrt{Hz}. This result surpasses the sensitivity limit of an ideal, thermally-limited electron magnetometer and resolves a long-standing debate regarding EP-like physics in advanced quantum sensing.