Tree Orbits under Permutation Group Action: Algorithm, Enumeration and Application to Viral Assembly

This paper uses combinatorics and group theory to answer questions about the assembly of icosahedral viral shells. Although the geometric structure of the capsid (shell) is fairly well understood in terms of its constituent subunits, the assembly process is not. For the purpose of this paper, the capsid is modeled by a polyhedron whose facets represent the monomers. The assembly process is modeled by a rooted tree, the leaves representing the facets of the polyhedron, the root representing the assembled polyhedron, and the internal vertices representing intermediate stages of assembly (subsets of facets). Besides its virological motivation, the enumeration of orbits of trees under the action of a finite group is of independent mathematical interest. If $G$ is a finite group acting on a finite set $X$, then there is a natural induced action of $G$ on the set $\mathcal{T}_X$ of trees whose leaves are bijectively labeled by the elements of $X$. If $G$ acts simply on $X$, then $|X| := |X_n| = n \cdot |G|$, where $n$ is the number of $G$-orbits in $X$. The basic combinatorial results in this paper are (1) a formula for the number of orbits of each size in the action of $G$ on $\mathcal{T}_{X_n}$, for every $n$, and (2) a simple algorithm to find the stabilizer of a tree $\tau \in \mathcal{T}_X$ in $G$ that runs in linear time and does not need memory in addition to its input tree.