Determining the automorphism group of a hyperelliptic curve

In this note we discuss techniques for determining the automorphism group of a genus $g$ hyperelliptic curve $\X_g$ defined over an algebraically closed field $k$ of characteristic zero. The first technique uses the classical $GL_2 (k)$-invariants of binary forms. This is a practical method for curves of small genus, but has limitations as the genus increases, due to the fact that such invariants are not known for large genus. The second approach, which uses dihedral invariants of hyperelliptic curves, is a very convenient method and works well in all genera. First we define the normal decomposition of a hyperelliptic curve with extra automorphisms. Then dihedral invariants are defined in terms of the coefficients of this normal decomposition. We define such invariants independently of the automorphism group $\Aut (\X_g)$. However, to compute such invariants the curve is required to be in its normal form. This requires solving a nonlinear system of equations. We find conditions in terms of classical invariants of binary forms for a curve to have reduced automorphism group $A_4$, $S_4$, $A_5$. As far as we are aware, such results have not appeared before in the literature.