Hybrid model architectures that combine computational primitives (e.g., Attention, MLP) in different ratios have shown promising performance beyond Transformers. Some studies have shown that different interleavings of primitives can affect model quality as well. However, prior works explore the hybrid model architecture design space manually. Due to the large design space and training costs, discovering hybrid models that combine key computational primitives for pre-training is challenging. In this work, we take a principled approach in designing a modular hybrid model architecture search framework -- Composer. Composer explores model architectures at a small scale and extrapolates the top-performing model architectures to a larger scale using our proposed scaling strategies. Using Composer, we discover new hybrid LLM architectures that outperform Llama 3.2. Compared to Llama 3.2 and previous state-of-the-art baselines, the new model architectures consistently reduce validation loss at parameter scales of 350M-3B and improve evaluation accuracy on the downstream tasks by up to 2.8-8.3% (1.1-3.1% on average) while improving both training and inference efficiency.
@article{arxiv.2510.00379,
title = {Composer: A Search Framework for Hybrid Neural Architecture Design},
author = {Bilge Acun and Prasoon Sinha and Newsha Ardalani and Sangmin Bae and Alicia Golden and Chien-Yu Lin and Meghana Madhyastha and Fei Sun and Neeraja J. Yadwadkar and Carole-Jean Wu},
journal= {arXiv preprint arXiv:2510.00379},
year = {2026}
}