The dual group of a spherical variety

Let $X$ be a spherical variety for a connected reductive group $G$. Work of Gaitsgory-Nadler strongly suggests that the Langlands dual group $G^\vee$ of $G$ has a subgroup whose Weyl group is the little Weyl group of $X$. Sakellaridis-Venkatesh defined a refined dual group $G^\vee_X$ and verified in many cases that there exists an isogeny $\phi$ from $G^\vee_X$ to $G^\vee$. In this paper, we establish the existence of $\phi$ in full generality. Our approach is purely combinatorial and works (despite the title) for arbitrary $G$-varieties.