The impressive advancement of Large Language Models (LLMs) in English has not been matched across all languages. In particular, LLM performance in Arabic lags behind, due to data scarcity, linguistic diversity of Arabic and its dialects, morphological complexity, etc. Progress is further hindered by the quality of Arabic benchmarks, which typically rely on static, publicly available data, lack comprehensive task coverage, or do not provide dedicated platforms with blind test sets. This makes it challenging to measure actual progress and to mitigate data contamination. Here, we aim to bridge these gaps. In particular, we introduce BALSAM, a comprehensive, community-driven benchmark aimed at advancing Arabic LLM development and evaluation. It includes 78 NLP tasks from 14 broad categories, with 52K examples divided into 37K test and 15K development, and a centralized, transparent platform for blind evaluation. We envision BALSAM as a unifying platform that sets standards and promotes collaborative research to advance Arabic LLM capabilities.
@article{arxiv.2507.22603,
title = {BALSAM: A Platform for Benchmarking Arabic Large Language Models},
author = {Rawan Al-Matham and Kareem Darwish and Raghad Al-Rasheed and Waad Alshammari and Muneera Alhoshan and Amal Almazrua and Asma Al Wazrah and Mais Alheraki and Firoj Alam and Preslav Nakov and Norah Alzahrani and Eman alBilali and Nizar Habash and Abdelrahman El-Sheikh and Muhammad Elmallah and Haonan Li and Hamdy Mubarak and Mohamed Anwar and Zaid Alyafeai and Ahmed Abdelali and Nora Altwairesh and Maram Hasanain and Abdulmohsen Al Thubaity and Shady Shehata and Bashar Alhafni and Injy Hamed and Go Inoue and Khalid Elmadani and Ossama Obeid and Fatima Haouari and Tamer Elsayed and Emad Alghamdi and Khalid Almubarak and Saied Alshahrani and Ola Aljarrah and Safa Alajlan and Areej Alshaqarawi and Maryam Alshihri and Sultana Alghurabi and Atikah Alzeghayer and Afrah Altamimi and Abdullah Alfaifi and Abdulrahman AlOsaimy},
journal= {arXiv preprint arXiv:2507.22603},
year = {2025}
}