Using Deep Mixture-of-Experts to Detect Word Meaning Shift for TempoWiC

Ze Chen; Kangxu Wang; Zijian Cai; Jiewen Zheng; Jiarong He; Max Gao; Jason Zhang

Using Deep Mixture-of-Experts to Detect Word Meaning Shift for TempoWiC

Computation and Language 2022-11-08 v1

Authors: Ze Chen , Kangxu Wang , Zijian Cai , Jiewen Zheng , Jiarong He , Max Gao , Jason Zhang

Abstract

This paper mainly describes the dma submission to the TempoWiC task, which achieves a macro-F1 score of 77.05% and attains the first place in this task. We first explore the impact of different pre-trained language models. Then we adopt data cleaning, data augmentation, and adversarial training strategies to enhance the model generalization and robustness. For further improvement, we integrate POS information and word semantic representation using a Mixture-of-Experts (MoE) approach. The experimental results show that MoE can overcome the feature overuse issue and combine the context, POS, and word semantic features well. Additionally, we use a model ensemble method for the final prediction, which has been proven effective by many research works.

Keywords

mixture of experts machine translation machine learning

Cite

@article{arxiv.2211.03466,
  title  = {Using Deep Mixture-of-Experts to Detect Word Meaning Shift for TempoWiC},
  author = {Ze Chen and Kangxu Wang and Zijian Cai and Jiewen Zheng and Jiarong He and Max Gao and Jason Zhang},
  journal= {arXiv preprint arXiv:2211.03466},
  year   = {2022}
}

Using Deep Mixture-of-Experts to Detect Word Meaning Shift for TempoWiC

Abstract

Keywords

Cite

Related papers