Sets whose differences avoid squares modulo m

We prove that if $\varepsilon(m)\to 0$ arbitrarily slowly, then for almost all $m$ and any $A\subset\mathbb{Z}_m$ such that $A-A$ does not contain non-zero quadratic residues we have $|A|\leq m^{1/2-\varepsilon(m)}.$