Modularity of $d$-elliptic loci with level structure

We consider the generating series of special cycles on $\mathcal{A}_1(N)\times \mathcal{A}_g(N)$, with full level $N$ structure, valued in the cohomology of degree $2g$. The modularity theorem of Kudla-Millson for locally symmetric spaces implies that these series are modular. When $N=1$, the images of these loci in $\mathcal{A}_g$ are the $d$-elliptic Noether-Lefschetz loci, which are conjectured to be modular. In the appendix, it is shown that the resulting modular forms are nonzero for $g=2$ when $N\geq 11$ and $N\neq 12$.