We present specialized Large Language Models for theoretical High-Energy Physics, obtained as 20 fine-tuned variants of the 8-billion parameter Llama-3.1 model. Each variant was trained on arXiv abstracts (through August 2024) from different combinations of hep-th, hep-ph and gr-qc. For a comparative study, we also trained models on datasets that contained abstracts from disparate fields such as the q-bio and cs categories. All models were fine-tuned using two distinct Low-Rank Adaptation fine-tuning approaches and varying dataset sizes, and outperformed the base model on hep-th abstract completion tasks. We compare performance against leading commercial LLMs (ChatGPT, Claude, Gemini, DeepSeek) and derive insights for further developing specialized language models for High-Energy Theoretical Physics.
@article{arxiv.2508.03716,
title = {FeynTune: Large Language Models for High-Energy Theory},
author = {Paul Richmond and Prarit Agarwal and Borun Chowdhury and Vasilis Niarchos and Constantinos Papageorgakis},
journal= {arXiv preprint arXiv:2508.03716},
year = {2026}
}
Comments
16 pages; v2: Human evaluation discussion updated, additional training hyperparameters and inference settings included and references added