AstroLLaMA: Towards Specialized Foundation Models in Astronomy
Instrumentation and Methods for Astrophysics2023-09-13v1Cosmology and Nongalactic AstrophysicsAstrophysics of GalaxiesHigh Energy Astrophysical PhenomenaComputation and LanguageMachine Learning
Large language models excel in many human-language tasks but often falter in highly specialized domains like scholarly astronomy. To bridge this gap, we introduce AstroLLaMA, a 7-billion-parameter model fine-tuned from LLaMA-2 using over 300,000 astronomy abstracts from arXiv. Optimized for traditional causal language modeling, AstroLLaMA achieves a 30% lower perplexity than Llama-2, showing marked domain adaptation. Our model generates more insightful and scientifically relevant text completions and embedding extraction than state-of-the-arts foundation models despite having significantly fewer parameters. AstroLLaMA serves as a robust, domain-specific model with broad fine-tuning potential. Its public release aims to spur astronomy-focused research, including automatic paper summarization and conversational agent development.
@article{arxiv.2309.06126,
title = {AstroLLaMA: Towards Specialized Foundation Models in Astronomy},
author = {Tuan Dung Nguyen and Yuan-Sen Ting and Ioana Ciucă and Charlie O'Neill and Ze-Chang Sun and Maja Jabłońska and Sandor Kruk and Ernest Perkowski and Jack Miller and Jason Li and Josh Peek and Kartheik Iyer and Tomasz Różański and Pranav Khetarpal and Sharaf Zaman and David Brodrick and Sergio J. Rodríguez Méndez and Thang Bui and Alyssa Goodman and Alberto Accomazzi and Jill Naiman and Jesse Cranney and Kevin Schawinski and UniverseTBD},
journal= {arXiv preprint arXiv:2309.06126},
year = {2023}
}
Comments
6 pages, 3 figures, submitted to IJCNLP-AACL 2023. Comments are welcome. The model can be found on Hugging Face - https://huggingface.co/universeTBD/astrollama