Rigidity in dynamics and M\"obius disjointness

Let $(X, T)$ be a topological dynamical system. We show that if each invariant measure of $(X, T)$ gives rise to a measure-theoretic dynamical system that is either: a. rigid along a sequence of "bounded prime volume" or b. admits a polynomial rate of rigidity on a linearly dense subset in $C(X)$, then $(X, T)$ satisfies Sarnak's conjecture on M\"obius disjointness. We show that the same conclusion also holds if there are countably many invariant ergodic measures, and each of them satisfies a. or b. This recovers some earlier results and implies Sarnak's conjecture in the following new cases: for almost every interval exchange map of $d$ intervals with $d \geq 2$, for $C^{2+\epsilon}$-smooth skew products over rotations and $C^{2+\epsilon}$-smooth flows (without fixed points) on the torus. In particular, these are improvements of earlier results of respectively Chaika-Eskin, Wang and Huang-Wang-Ye. We also discuss some purely arithmetic consequences for the Liouville function.