Most existing pre-trained language representation models (PLMs) are sub-optimal in sentiment analysis tasks, as they capture the sentiment information from word-level while under-considering sentence-level information. In this paper, we propose SentiWSP, a novel Sentiment-aware pre-trained language model with combined Word-level and Sentence-level Pre-training tasks. The word level pre-training task detects replaced sentiment words, via a generator-discriminator framework, to enhance the PLM's knowledge about sentiment words. The sentence level pre-training task further strengthens the discriminator via a contrastive learning framework, with similar sentences as negative samples, to encode sentiments in a sentence. Extensive experimental results show that SentiWSP achieves new state-of-the-art performance on various sentence-level and aspect-level sentiment classification benchmarks. We have made our code and model publicly available at https://github.com/XMUDM/SentiWSP.
@article{arxiv.2210.09803,
title = {Sentiment-Aware Word and Sentence Level Pre-training for Sentiment Analysis},
author = {Shuai Fan and Chen Lin and Haonan Li and Zhenghao Lin and Jinsong Su and Hang Zhang and Yeyun Gong and Jian Guo and Nan Duan},
journal= {arXiv preprint arXiv:2210.09803},
year = {2022}
}