Second-order Co-occurrence Sensitivity of Skip-Gram with Negative Sampling
Computation and Language
2019-06-10 v2
Abstract
We simulate first- and second-order context overlap and show that Skip-Gram with Negative Sampling is similar to Singular Value Decomposition in capturing second-order co-occurrence information, while Pointwise Mutual Information is agnostic to it. We support the results with an empirical study finding that the models react differently when provided with additional second-order information. Our findings reveal a basic property of Skip-Gram with Negative Sampling and point towards an explanation of its success on a variety of tasks.
Cite
@article{arxiv.1906.02479,
title = {Second-order Co-occurrence Sensitivity of Skip-Gram with Negative Sampling},
author = {Dominik Schlechtweg and Cennet Oguz and Sabine Schulte im Walde},
journal= {arXiv preprint arXiv:1906.02479},
year = {2019}
}
Comments
BlackboxNLP 2019, 6 pages