On the Importance of Attention in Meta-Learning for Few-Shot Text Classification
Abstract
Current deep learning based text classification methods are limited by their ability to achieve fast learning and generalization when the data is scarce. We address this problem by integrating a meta-learning procedure that uses the knowledge learned across many tasks as an inductive bias towards better natural language understanding. Based on the Model-Agnostic Meta-Learning framework (MAML), we introduce the Attentive Task-Agnostic Meta-Learning (ATAML) algorithm for text classification. The essential difference between MAML and ATAML is in the separation of task-agnostic representation learning and task-specific attentive adaptation. The proposed ATAML is designed to encourage task-agnostic representation learning by way of task-agnostic parameterization and facilitate task-specific adaptation via attention mechanisms. We provide evidence to show that the attention mechanism in ATAML has a synergistic effect on learning performance. In comparisons with models trained from random initialization, pretrained models and meta trained MAML, our proposed ATAML method generalizes better on single-label and multi-label classification tasks in miniRCV1 and miniReuters-21578 datasets.
Keywords
Cite
@article{arxiv.1806.00852,
title = {On the Importance of Attention in Meta-Learning for Few-Shot Text Classification},
author = {Xiang Jiang and Mohammad Havaei and Gabriel Chartrand and Hassan Chouaib and Thomas Vincent and Andrew Jesson and Nicolas Chapados and Stan Matwin},
journal= {arXiv preprint arXiv:1806.00852},
year = {2018}
}
Comments
13 pages, 4 figures, submitted to NIPS