Enumerative invariants in self-dual categories. II. Homological invariants

This is the second paper in a series on enumerative invariants counting self-dual objects in self-dual categories, and is a sequal to (arXiv:2302.00038). Ordinary enumerative invariants in abelian categories can be seen as invariants for the structure group $\mathrm{GL} (n)$, and our theory is an extension of this to structure groups $\mathrm{O} (n)$ and $\mathrm{Sp} (2n)$. Examples of our theory include counting principal orthogonal or symplectic bundles, and counting self-dual quiver representations. In the present paper, we propose a conjectural picture on homological enumerative invariants counting self-dual objects in self-dual categories, which are homology classes lying in the ordinary homology of moduli stacks. This is a self-dual analogue of the conjectures of Gross-Joyce-Tanaka (arXiv:2005.05637). We study algebraic structures arising from the homology of these moduli stacks, including vertex algebras and twisted modules, and we formulate wall-crossing formulae for the invariants using these algebraic structures. We also provide a partial proof of our conjecture in the case of self-dual quiver representations.