Neural models have achieved significant results on the text-to-SQL task, in which most current work assumes all the input questions are legal and generates a SQL query for any input. However, in the real scenario, users can input any text that may not be able to be answered by a SQL query. In this work, we propose TriageSQL, the first cross-domain text-to-SQL question intention classification benchmark that requires models to distinguish four types of unanswerable questions from answerable questions. The baseline RoBERTa model achieves a 60% F1 score on the test set, demonstrating the need for further improvement on this task. Our dataset is available at https://github.com/chatc/TriageSQL.
@article{arxiv.2010.12634,
title = {Did You Ask a Good Question? A Cross-Domain Question Intention Classification Benchmark for Text-to-SQL},
author = {Yusen Zhang and Xiangyu Dong and Shuaichen Chang and Tao Yu and Peng Shi and Rui Zhang},
journal= {arXiv preprint arXiv:2010.12634},
year = {2020}
}