Improving Clustering on Occupational Text Data through Dimensionality Reduction
Abstract
In this study, we focused on proposing an optimal clustering mechanism for the occupations defined in the well-known US-based occupational database, O*NET. Even though all occupations are defined according to well-conducted surveys in the US, their definitions can vary for different firms and countries. Hence, if one wants to expand the data that is already collected in O*NET for the occupations defined with different tasks, a map between the definitions will be a vital requirement. We proposed a pipeline using several BERT-based techniques with various clustering approaches to obtain such a map. We also examined the effect of dimensionality reduction approaches on several metrics used in measuring performance of clustering algorithms. Finally, we improved our results by using a specialized silhouette approach. This new clustering-based mapping approach with dimensionality reduction may help distinguish the occupations automatically, creating new paths for people wanting to change their careers.
Cite
@article{arxiv.2507.07582,
title = {Improving Clustering on Occupational Text Data through Dimensionality Reduction},
author = {Iago Xabier Vázquez García and Damla Partanaz and Emrullah Fatih Yetkin},
journal= {arXiv preprint arXiv:2507.07582},
year = {2025}
}
Comments
Preprint, 10 figures