Hard cases of pronoun resolution have been used as a long-standing benchmark for commonsense reasoning. In the recent literature, pre-trained language models have been used to obtain state-of-the-art results on pronoun resolution. Overall, four categories of training and evaluation objectives have been introduced. The variety of training datasets and pre-trained language models used in these works makes it unclear whether the choice of training objective is critical. In this work, we make a fair comparison of the performance and seed-wise stability of four models that represent the four categories of objectives. Our experiments show that the objective of sequence ranking performs the best in-domain, while the objective of semantic similarity between candidates and pronoun performs the best out-of-domain. We also observe a seed-wise instability of the model using sequence ranking, which is not the case when the other objectives are used.
@article{arxiv.2010.02570,
title = {Does the Objective Matter? Comparing Training Objectives for Pronoun Resolution},
author = {Yordan Yordanov and Oana-Maria Camburu and Vid Kocijan and Thomas Lukasiewicz},
journal= {arXiv preprint arXiv:2010.02570},
year = {2020}
}