挑战大型语言模型在意大利语中的能力：一个社区倡议
计算与语言
2025-12-05 v1
摘要
大型语言模型（LLM）的快速进展已变革自然语言处理领域，并扩大了其在研究和社会层面的影响。然而，针对英语之外语言的这些模型系统评估仍有限。"Challenging the Abilities of LAnguage Models in ITAlian"（CALAMITA）是一个规模庞大的合作性基准测试计划，专注于意大利语，由意大利计算语言学协会（AILC）主导。与现有工作不同，CALAMITA 侧重于方法论：它联合来自 academia、industry 和 public sector 的 80 多位贡献者，设计、记录和评估覆盖语言能力、常识推理、事实一致性、公平性、摘要、翻译和代码生成等多样化任务。通过这一过程，我们不仅组装了包含 20 多个任务和近 100 个子任务的基准，还建立了支持异构数据集和指标的集中评估管道。我们报告了四个开源 LLM 的结果，揭示其在各项能力上的系统性优势与不足，以及任务特定评估的挑战。除了定量结果，CALAMITA 还暴露了方法论教训：细粒度、任务代表性指标的必要性、统一管道的重要性，以及广泛社区参与的收益与局限。CALAMITA 被设想为一个滚动基准，持续集成新的任务和模型。这使其既是一个资源——目前最全面且最具多样性的意大利语基准——也是可持续、社区驱动的评估框架。我们认为，这种组合为其他语言和社区寻求包容且严谨的 LLM 评估实践提供了蓝图。
引用
@article{arxiv.2512.04759,
title = {Challenging the Abilities of Large Language Models in Italian: a Community Initiative},
author = {Malvina Nissim and Danilo Croce and Viviana Patti and Pierpaolo Basile and Giuseppe Attanasio and Elio Musacchio and Matteo Rinaldi and Federico Borazio and Maria Francis and Jacopo Gili and Daniel Scalena and Begoña Altuna and Ekhi Azurmendi and Valerio Basile and Luisa Bentivogli and Arianna Bisazza and Marianna Bolognesi and Dominique Brunato and Tommaso Caselli and Silvia Casola and Maria Cassese and Mauro Cettolo and Claudia Collacciani and Leonardo De Cosmo and Maria Pia Di Buono and Andrea Esuli and Julen Etxaniz and Chiara Ferrando and Alessia Fidelangeli and Simona Frenda and Achille Fusco and Marco Gaido and Andrea Galassi and Federico Galli and Luca Giordano and Mattia Goffetti and Itziar Gonzalez-Dios and Lorenzo Gregori and Giulia Grundler and Sandro Iannaccone and Chunyang Jiang and Moreno La Quatra and Francesca Lagioia and Soda Marem Lo and Marco Madeddu and Bernardo Magnini and Raffaele Manna and Fabio Mercorio and Paola Merlo and Arianna Muti and Vivi Nastase and Matteo Negri and Dario Onorati and Elena Palmieri and Sara Papi and Lucia Passaro and Giulia Pensa and Andrea Piergentili and Daniele Potertì and Giovanni Puccetti and Federico Ranaldi and Leonardo Ranaldi and Andrea Amelio Ravelli and Martina Rosola and Elena Sofia Ruzzetti and Giuseppe Samo and Andrea Santilli and Piera Santin and Gabriele Sarti and Giovanni Sartor and Beatrice Savoldi and Antonio Serino and Andrea Seveso and Lucia Siciliani and Paolo Torroni and Rossella Varvara and Andrea Zaninello and Asya Zanollo and Fabio Massimo Zanzotto and Kamyar Zeinalipour and Andrea Zugarini},
journal= {arXiv preprint arXiv:2512.04759},
year = {2025}
}