Non-Wiener groups with a Gelfand pair

Let $G$ be a non-amenable locally compact group and $K$ a compact subgroup of $G$ such that $(G,K)$ is a Gelfand pair. We show that if $G$ admits a suitable boundary representation which is topologically irreducible and not unitarizable, then $G$ is not a Wiener group in the sense that its Fourier transform does not satisfy the analogue of Wiener's Tauberian theorem. As an application, we show that if $G$ is a closed non-compact boundary transitive group of automorphisms of a connected locally finite graph with infinitely many ends, or a split reductive algebraic group over a non-archimedean local field, then $G$ is not Wiener.