English

Graph-SND: Sparse Aggregation for Behavioral Diversity in Multi-Agent Reinforcement Learning

Machine Learning 2026-05-07 v1 Multiagent Systems

Abstract

System Neural Diversity (SND) measures behavioral heterogeneity in multi-agent reinforcement learning by averaging pairwise distances over all (n2)\binom{n}{2} agent pairs, making each call quadratic in team size. We introduce Graph-SND, which replaces this complete-graph average with a weighted average over the edges of an arbitrary graph GG. Three regimes follow: G=KnG=K_n recovers SND exactly; a fixed sparse GG defines a localized diversity measure at O(E)O(|E|) cost; and random edge samples yield an unbiased Horvitz-Thompson estimator and a normalized sample mean with O(1/m)O(1/\sqrt{m}) concentration in the sampled edge count mm. For fixed sparse graphs we prove forwarding-index distortion bounds for expanders and a spectral refinement under low-rank distance structure; for random dd-regular graphs we prove an unconditional probabilistic O~(Dmax/n)\widetilde{\mathcal{O}}(D_{\max}/\sqrt{n}) bound. On VMAS we verify recovery, unbiasedness, concentration, and wall-clock scaling, with a PettingZoo TVD panel checking non-Gaussian transfer. In a 500-iteration n=100n=100 PPO run, Bernoulli-0.10.1 Graph-SND tracks full SND while reducing per-call metric time by about 10×10\times, and frozen-policy GPU timing up to n=500n=500 follows the predicted (n2)/E\binom{n}{2}/|E| speedup. Random dd-regular expanders empirically achieve SNDGu/SND[0.9987,1.0013]\mathrm{SND}_{G}^{\mathrm{u}}/\mathrm{SND} \in [0.9987, 1.0013] at Θ(nlogn)\Theta(n \log n) edges. In DiCo diversity control at n=50n=50, Bernoulli-0.10.1 Graph-SND preserves set-point tracking with paired reward differences indistinguishable from zero across nine matched cells while cutting per-call metric cost by 9.5×{\sim}9.5\times. Together, these results show that the SND aggregation bottleneck can be removed without changing the metric's semantics, yielding a drop-in sparse alternative that scales beyond complete-graph SND and supports both passive measurement and closed-loop diversity control.

Keywords

Cite

@article{arxiv.2605.05020,
  title  = {Graph-SND: Sparse Aggregation for Behavioral Diversity in Multi-Agent Reinforcement Learning},
  author = {Shawn Ray},
  journal= {arXiv preprint arXiv:2605.05020},
  year   = {2026}
}

Comments

22 pages, 12 figures, 7 tables

R2 v1 2026-07-01T12:52:58.856Z