MALLM: Multi-Agent Large Language Models Framework
Abstract
Multi-agent debate (MAD) has demonstrated the ability to augment collective intelligence by scaling test-time compute and leveraging expertise. Current frameworks for multi-agent debate are often designed towards tool use, lack integrated evaluation, or provide limited configurability of agent personas, response generators, discussion paradigms, and decision protocols. We introduce MALLM (Multi-Agent Large Language Models), an open-source framework that enables systematic analysis of MAD components. MALLM offers more than 144 unique configurations of MAD, including (1) agent personas (e.g., Expert, Personality), (2) response generators (e.g., Critical, Reasoning), (3) discussion paradigms (e.g., Memory, Relay), and (4) decision protocols (e.g., Voting, Consensus). MALLM uses simple configuration files to define a debate. Furthermore, MALLM can load any textual Hugging Face dataset (e.g., MMLU-Pro, WinoGrande) and provides an evaluation pipeline for easy comparison of MAD configurations. MALLM enables researchers to systematically configure, run, and evaluate debates for their problems, facilitating the understanding of the components and their interplay.
Cite
@article{arxiv.2509.11656,
title = {MALLM: Multi-Agent Large Language Models Framework},
author = {Jonas Becker and Lars Benedikt Kaesberg and Niklas Bauer and Jan Philip Wahle and Terry Ruas and Bela Gipp},
journal= {arXiv preprint arXiv:2509.11656},
year = {2025}
}
Comments
Accepted at EMNLP 2025 (Demo)