English

Adapting BigScience Multilingual Model to Unseen Languages

Computation and Language 2022-04-12 v1 Artificial Intelligence Machine Learning
Authors: Zheng-Xin Yong , Vassilina Nikoulina
View on arXiv ↗ PDF ↗

Abstract

We benchmark different strategies of adding new languages (German and Korean) into the BigScience's pretrained multilingual language model with 1.3 billion parameters that currently supports 13 languages. We investigate the factors that affect the language adaptability of the model and the trade-offs between computational costs and expected performance.

Cite

@article{arxiv.2204.04873,
  title  = {Adapting BigScience Multilingual Model to Unseen Languages},
  author = {Zheng-Xin Yong and Vassilina Nikoulina},
  journal= {arXiv preprint arXiv:2204.04873},
  year   = {2022}
}

Related papers

View all related →
Computation and Language · Computer Science
Comparison of Multilingual Self-Supervised and Weakly-Supervised Speech Pre-Training for Adaptation to Unseen Languages
Andrew Rouditchenko, Sameer Khurana, Samuel Thomas, Rogerio Feris +5
2023-06-01
Computation and Language · Computer Science
Orion-14B: Open-source Multilingual Large Language Models
Du Chen, Yi Huang, Xiaopu Li, Yongqiang Li +6
2024-01-24
Computation and Language · Computer Science
MuBench: Assessment of Multilingual Capabilities of Large Language Models Across 61 Languages
Wenhan Han, Yifan Zhang, Zhixun Chen, Binbin Liu +6
2025-06-25
Computation and Language · Computer Science
Detecting Languages Unintelligible to Multilingual Models through Local Structure Probes
Louis Clouâtre, Prasanna Parthasarathi, Amal Zouaq, Sarath Chandar
2022-11-10
Computation and Language · Computer Science
X-LLaVA: Optimizing Bilingual Large Vision-Language Alignment
Dongjae Shin, Hyeonseok Lim, Inho Won, Changsu Choi +5
2024-08-05
Computation and Language · Computer Science
Optimizing Language Augmentation for Multilingual Large Language Models: A Case Study on Korean
ChangSu Choi, Yongbin Jeong, Seoyoon Park, InHo Won +10
2024-03-22
Computation and Language · Computer Science
How Many Languages Make Good Multilingual Instruction Tuning? A Case Study on BLOOM
Shaoxiong Ji, Pinzhen Chen
2024-12-10
Computation and Language · Computer Science
When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Models
Benjamin Muller, Antonis Anastasopoulos, Benoît Sagot, Djamé Seddah
2021-04-20
Computation and Language · Computer Science
Linguistically-Informed Multilingual Instruction Tuning: Is There an Optimal Set of Languages to Tune?
Gürkan Soykan, Gözde Gül Şahin
2024-10-11
Computation and Language · Computer Science
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
Abteen Ebrahimi, Katharina Kann
2021-06-07
Computation and Language · Computer Science
The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset
Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki +50
2023-03-08
Computation and Language · Computer Science
MiniLingua: A Small Open-Source LLM for European Languages
Anna Aksenova, Boris Zverkov, Nicola Dainese, Alexander Nikitin +1
2025-12-16
Computation and Language · Computer Science
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
Xinyi Wang, Sebastian Ruder, Graham Neubig
2022-04-07
Computation and Language · Computer Science
Match the Script, Adapt if Multilingual: Analyzing the Effect of Multilingual Pretraining on Cross-lingual Transferability
Yoshinari Fujinuma, Jordan Boyd-Graber, Katharina Kann
2022-03-22
Computation and Language · Computer Science
Quantifying Language Disparities in Multilingual Large Language Models
Songbo Hu, Ivan Vulić, Anna Korhonen
2025-08-26
Computation and Language · Computer Science
Cross-Lingual Unlearning of Selective Knowledge in Multilingual Language Models
Minseok Choi, Kyunghyun Min, Jaegul Choo
2024-10-04
Computation and Language · Computer Science
RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMs
John Dang, Arash Ahmadian, Kelly Marchisio, Julia Kreutzer +2
2024-07-04
Computation and Language · Computer Science
Towards Building Multilingual Language Model for Medicine
Pengcheng Qiu, Chaoyi Wu, Xiaoman Zhang, Weixiong Lin +4
2024-06-04
Computation and Language · Computer Science
Poro 34B and the Blessing of Multilinguality
Risto Luukkonen, Jonathan Burdge, Elaine Zosa, Aarne Talman +4
2025-06-11
Computation and Language · Computer Science
Large vocabulary speech recognition for languages of Africa: multilingual modeling and self-supervised learning
Sandy Ritchie, You-Chi Cheng, Mingqing Chen, Rajiv Mathews +3
2022-10-05
Computation and Language · Computer Science
Scaling Speech Technology to 1,000+ Languages
Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello +12
2023-05-24
Computation and Language · Computer Science
MultiPragEval: Multilingual Pragmatic Evaluation of Large Language Models
Dojun Park, Jiwoo Lee, Seohyun Park, Hyeyun Jeong +4
2024-10-01
Computation and Language · Computer Science
When Is Multilinguality a Curse? Language Modeling for 250 High- and Low-Resource Languages
Tyler A. Chang, Catherine Arnett, Zhuowen Tu, Benjamin K. Bergen
2023-11-16
Computation and Language · Computer Science
HAE-RAE Bench: Evaluation of Korean Knowledge in Language Models
Guijin Son, Hanwool Lee, Suwan Kim, Huiseo Kim +5
2024-03-21
Computation and Language · Computer Science
K-EXAONE Technical Report
Eunbi Choi, Kibong Choi, Seokhee Hong, Junwon Hwang +61
2026-01-12
R2 v1 2026-06-24T10:44:02.386Z