Correspondence Analysis and PMI-Based Word Embeddings: A Comparative Study
Abstract
Popular word embedding methods such as GloVe and Word2Vec are related to the factorization of the pointwise mutual information (PMI) matrix. In this paper, we establish a formal connection between correspondence analysis (CA) and PMI-based word embedding methods. CA is a dimensionality reduction method that uses singular value decomposition (SVD), and we show that CA is mathematically close to the weighted factorization of the PMI matrix. We further introduce variants of CA for word-context matrices, namely CA applied after a square-root transformation (ROOT-CA) and after a fourth-root transformation (ROOTROOT-CA). We analyze the performance of these methods and examine how their success or failure is influenced by extreme values in the decomposed matrix. Although our primary focus is on traditionalstatic word embedding methods, we also include a comparison with a transformer-based encoder (BERT) to situate the results relative to contextual embeddings. Empirical evaluations across multiple corpora and word-similarity benchmarks show that ROOT-CA and ROOTROOT-CA perform slightly better overall than standard PMI-based methods and achieve results competitive with BERT.
Keywords
Cite
@article{arxiv.2405.20895,
title = {Correspondence Analysis and PMI-Based Word Embeddings: A Comparative Study},
author = {Qianqian Qi and Ayoub Bagheri and David J. Hessen and Peter G. M. van der Heijden},
journal= {arXiv preprint arXiv:2405.20895},
year = {2026}
}