Problems and Countermeasures in Natural Language Processing Evaluation
Abstract
Evaluation in natural language processing guides and promotes research on models and methods. In recent years, new evalua-tion data sets and evaluation tasks have been continuously proposed. At the same time, a series of problems exposed by ex-isting evaluation have also restricted the progress of natural language processing technology. Starting from the concept, com-position, development and meaning of natural language evaluation, this article classifies and summarizes the tasks and char-acteristics of mainstream natural language evaluation, and then summarizes the problems and causes of natural language pro-cessing evaluation. Finally, this article refers to the human language ability evaluation standard, puts forward the concept of human-like machine language ability evaluation, and proposes a series of basic principles and implementation ideas for hu-man-like machine language ability evaluation from the three aspects of reliability, difficulty and validity.
Cite
@article{arxiv.2104.09712,
title = {Problems and Countermeasures in Natural Language Processing Evaluation},
author = {Qingxiu Dong and Zhifang Sui and Weidong Zhan and Baobao Chang},
journal= {arXiv preprint arXiv:2104.09712},
year = {2021}
}