Can a quantum nondemolition measurement improve the sensitivity of an atomic magnetometer?

Noise properties of an idealized atomic magnetometer that utilizes spin squeezing induced by a continuous quantum nondemolition measurement are considered. Such a magnetometer measures spin precession of $N$ atomic spins by detecting optical rotation of far-detuned light. Fundamental noise sources include the quantum projection noise and the photon shot-noise. For measurement times much shorter than the spin-relaxation time observed in the absence of light ($\tau_{\rm rel}$) divided by $\sqrt{N}$, the optimal sensitivity of the magnetometer scales as $N^{-3/4}$, so an advantage over the usual sensitivity scaling as $N^{-1/2}$ can be achieved. However, at longer measurement times, the optimized sensitivity scales as $N^{-1/2}$, as for a usual shot-noise limited magnetometer. If strongly squeezed probe light is used, the Heisenberg uncertainty limit may, in principle, be reached for very short measurement times. However, if the measurement time exceeds $\tau_{\rm rel}/N$, the $N^{-1/2}$ scaling is again restored.