As opposed to scaling-up protein language models (PLMs), we seek improving performance via protein-specific optimization. Although the proportionality between the language model size and the richness of its learned representations is validated, we prioritize accessibility and pursue a path of data-efficient, cost-reduced, and knowledge-guided optimization. Through over twenty experiments ranging from masking, architecture, and pre-training data, we derive insights from protein-specific experimentation into building a model that interprets the language of life, optimally. We present Ankh, the first general-purpose PLM trained on Google's TPU-v4 surpassing the state-of-the-art performance with fewer parameters (<10% for pre-training, <7% for inference, and <30% for the embedding dimension). We provide a representative range of structure and function benchmarks where Ankh excels. We further provide a protein variant generation analysis on High-N and One-N input data scales where Ankh succeeds in learning protein evolutionary conservation-mutation trends and introducing functional diversity while retaining key structural-functional characteristics. We dedicate our work to promoting accessibility to research innovation via attainable resources.
@article{arxiv.2301.06568,
title = {Ankh: Optimized Protein Language Model Unlocks General-Purpose Modelling},
author = {Ahmed Elnaggar and Hazem Essam and Wafaa Salah-Eldin and Walid Moustafa and Mohamed Elkerdawy and Charlotte Rochereau and Burkhard Rost},
journal= {arXiv preprint arXiv:2301.06568},
year = {2023}
}