Revisiting Word Embeddings in the LLM Era
Abstract
Large Language Models (LLMs) have recently shown remarkable advancement in various NLP tasks. As such, a popular trend has emerged lately where NLP researchers extract word/sentence/document embeddings from these large decoder-only models and use them for various inference tasks with promising results. However, it is still unclear whether the performance improvement of LLM-induced embeddings is merely because of scale or whether underlying embeddings they produce significantly differ from classical encoding models like Word2Vec, GloVe, Sentence-BERT (SBERT) or Universal Sentence Encoder (USE). This is the central question we investigate in the paper by systematically comparing classical decontextualized and contextualized word embeddings with the same for LLM-induced embeddings. Our results show that LLMs cluster semantically related words more tightly and perform better on analogy tasks in decontextualized settings. However, in contextualized settings, classical models like SimCSE often outperform LLMs in sentence-level similarity assessment tasks, highlighting their continued relevance for fine-grained semantics.
Cite
@article{arxiv.2402.11094,
title = {Revisiting Word Embeddings in the LLM Era},
author = {Yash Mahajan and Matthew Freestone and Naman Bansal and Sathyanarayanan Aakur and Shubhra Kanti Karmaker Santu},
journal= {arXiv preprint arXiv:2402.11094},
year = {2025}
}
Comments
This is an updated version of the older version: arXiv:2402.11094. We accidentally submitted this article as a new submission (arXiv:2502.19607), which we have requested to withdraw. This version has 30 pages and 22 figures