On Peterson's open problem and representations of the general linear groups

Fix $\mathbb Z/2$ is the prime field of two elements and write $\mathcal A_2$ for the mod $2$ Steenrod algebra. Denote by $GL_d:= GL(d, \mathbb Z/2)$ the general linear group of rank $d$ over $\mathbb Z/2$ and by $\mathscr P_d$ the polynomial algebra $\mathbb Z/2[x_1, x_2, \ldots, x_d]$ as a connected unstable $\mathcal A_2$-module on $d$ generators of degree one. We study the Peterson "hit problem" of finding the minimal set of $\mathcal A_2$-generators for $\mathscr P_d.$ It is equivalent to determining a $\mathbb Z/2$-basis for the space of "cohits" $Q\mathscr P_d := \mathbb Z/2\otimes_{\mathcal A_2} \mathscr P_d \cong \mathscr P_d/\mathcal A_2^+\mathscr P_d.$ This $Q\mathscr P_d$ is also a representation of $GL_d$ over $\mathbb Z/2.$ The problem for $d= 5$ is not yet completely solved, and unknown in general. In this work, we give an explicit solution to the hit problem of five variables in the generic degree $n = r(2^t -1) + 2^ts$ with $r = d = 5,\ s =8$ and $t$ an arbitrary non-negative integer. An application of this study to the cases $t = 0$ and $t = 1$ shows that the Singer algebraic transfer of rank $5$ is an isomorphism in the bidegrees $(5, 5+(13.2^{0} - 5))$ and $(5, 5+(13.2^{1} - 5)).$ Moreover, the result when $t\geq 2$ was also discussed. Here, the Singer transfer of rank $d$ is a $\mathbb Z/2$-algebra homomorphism from $GL_d$-coinvariants of certain subspaces of $Q\mathscr P_d$ to the cohomology groups of the Steenrod algebra, ${\rm Ext}_{\mathcal A_2}^{d, d+*}(\mathbb Z/2, \mathbb Z/2).$ It is one of the useful tools for studying mysterious Ext groups and the Kervaire invariant one problem.