SGPT: GPT Sentence Embeddings for Semantic Search
Abstract
Decoder transformers have continued increasing in scale reaching hundreds of billions of parameters. Due to their scale the same decoder sets state-of-the-art results on various language tasks via prompting or fine-tuning. Yet, these large foundation models remain unusable for the related fields of semantic search and sentence embeddings. This prevents possibly new state-of-the-art results and forces organizations to train and maintain separate models. To this end, we propose SGPT to use decoders for sentence embeddings and semantic search via prompting or fine-tuning. At 5.8 billion parameters SGPT improves on the previously best sentence embeddings by a margin of 7% and outperforms a concurrent method with 175 billion parameters as measured on the BEIR search benchmark. Code, models and result files are freely available at https://github.com/Muennighoff/sgpt.
Cite
@article{arxiv.2202.08904,
title = {SGPT: GPT Sentence Embeddings for Semantic Search},
author = {Niklas Muennighoff},
journal= {arXiv preprint arXiv:2202.08904},
year = {2022}
}
Comments
19 pages, 3 figures, 12 tables. v2 corrects a misreported nDCG@10 number for the SGPT-BE-5.8B model. v3 updates SGPT-BE-5.8B scores based on retrained models with larger batch sizes v4 removes a superfluous table. v5 adds OpenAI scores on USEB and makes the paper easier to read