Exact $\mathbb{Z}_2$ electromagnetic duality of $\mathbb{Z}_2$ toric code is non-Clifford

The 2D $\mathbb{Z}_2$ toric code admits a global symmetry exchanging electric and magnetic quasiparticles, known as electromagnetic duality. Known realizations include lattice translation symmetry, an exact $\mathbb{Z}_4$ symmetry generated by a Clifford circuit, and an exact $\mathbb{Z}_2$ symmetry generated by a non-Clifford circuit. We show that a Clifford electromagnetic duality cannot realize an exact internal $\mathbb{Z}_2$ symmetry. This is proved rigorously for symmetries with coarse translation invariance by $l$ lattice units for generic odd $l$. Therefore an exact internal $\mathbb{Z}_2$ electromagnetic duality must be non-Clifford, whereas generic internal Clifford realization necessarily has $\mathbb{Z}_{2^m}$ algebra with $m\ge 2$. Our result suggests an unexpected connection between the algebra of exact electromagnetic duality and Clifford hierarchy of circuits.