Density of Neural Network Classes on Compact Subsets of Topological Vector Spaces

We prove density results for neural-network classes on compact sets $K\subset X$, where $X$ is a topological vector space whose continuous dual $X^*$ separates points. Let $\Psi:\mathbb R\to\mathbb R$ be a continuous squashing function. We show that the class $$\Sigma_X(\Psi) = \left\{ \sum_{j=1}^{N}\omega_j\Psi(f_j(x)+b_j): N\in\mathbb N,\ \omega_j,b_j\in\mathbb R,\ f_j\in X^* \right\}$$ is dense in $C(K)$ with respect to the uniform norm. As a consequence, if $\mu$ is a Radon probability measure supported on $K$, then $\Sigma_X(\Psi)$ is dense in $L^p(K,\mu)$ for every $1\le p<\infty$.