Cosmological birefringence, a rotation by an angle α of the polarization of photons as they propagate over cosmological distances, is constrained by the cosmic microwave background (CMB) to be ∣α∣≲1∘ (1σ) out to redshifts z≃1100 for a rotation that is uniform across the sky. However, the rotation angle α(θ,ϕ) may vary as a function of position (θ,ϕ) on the sky. Here I discuss how a position-dependent rotation can be sought in current and future AGN data. An upper limit \VEVα21/2≲3.7∘ to the scatter in the position-angle--polarization offsets in a sample of only N=9 AGN already constrains the rotation spherical-harmonic coefficients to (4π)−1/2αlm≲3.7∘ and constrains the power spectrum for α in models where it is a stochastic field. Future constraints can be improved with more sources and by analyzing well-mapped sources with a tensor-harmonic decomposition of the polarization analogous to that used in CMB polarization and weak gravitational lensing.