How symmetric is too symmetric for large quantum speedups?

Suppose a Boolean function $f$ is symmetric under a group action $G$ acting on the $n$ bits of the input. For which $G$ does this mean $f$ does not have an exponential quantum speedup? Is there a characterization of how rich $G$ must be before the function $f$ cannot have enough structure for quantum algorithms to exploit? In this work, we make several steps towards understanding the group actions $G$ which are "quantum intolerant" in this way. We show that sufficiently transitive group actions do not allow a quantum speedup, and that a "well-shuffling" property of group actions -- which happens to be preserved by several natural transformations -- implies a lack of super-polynomial speedups for functions symmetric under the group action. Our techniques are motivated by a recent paper by Chailloux (2018), which deals with the case where $G=S_n$. Our main application is for graph symmetries: we show that any Boolean function $f$ defined on the adjacency matrix of a graph (and symmetric under relabeling the vertices of the graph) has a power $6$ relationship between its randomized and quantum query complexities, even if $f$ is a partial function. In particular, this means no graph property testing problems can have super-polynomial quantum speedups, settling an open problem of Ambainis, Childs, and Liu (2011).