While Visual Multi-Agent Systems (VMAS) promise to enhance comprehensive abilities through inter-agent collaboration, empirical evidence reveals a counter-intuitive "scaling wall": increasing agent turns often degrades performance while exponentially inflating token costs. We attribute this failure to the information bottleneck inherent in text-centric communication, where converting perceptual and thinking trajectories into discrete natural language inevitably induces semantic loss. To this end, we propose L2-VMAS, a novel model-agnostic framework that enables inter-agent collaboration with dual latent memories. Furthermore, we decouple the perception and thinking while dynamically synthesizing dual latent memories. Additionally, we introduce an entropy-driven proactive triggering that replaces passive information transmission with efficient, on-demand memory access. Extensive experiments among backbones, sizes, and multi-agent structures demonstrate that our method effectively breaks the "scaling wall" with superb scalability, improving average accuracy by 2.7-5.4% while reducing token usage by 21.3-44.8%. Codes: https://github.com/YU-deep/L2-VMAS.
@article{arxiv.2602.00471,
title = {Dual Latent Memory for Visual Multi-agent System},
author = {Xinlei Yu and Chengming Xu and Zhangquan Chen and Bo Yin and Cheng Yang and Yongbo He and Yihao Hu and Jiangning Zhang and Cheng Tan and Xiaobin Hu and Shuicheng Yan},
journal= {arXiv preprint arXiv:2602.00471},
year = {2026}
}