In multitask retrieval, a single retriever is trained to retrieve relevant contexts for multiple tasks. Despite its practical appeal, naive multitask retrieval lags behind task-specific retrieval in which a separate retriever is trained for each task. We show that it is possible to train a multitask retriever that outperforms task-specific retrievers by promoting task specialization. The main ingredients are: (1) a better choice of pretrained model (one that is explicitly optimized for multitasking) along with compatible prompting, and (2) a novel adaptive learning method that encourages each parameter to specialize in a particular task. The resulting multitask retriever is highly performant on the KILT benchmark. Upon analysis, we find that the model indeed learns parameters that are more task-specialized compared to naive multitasking without prompting or adaptive learning.
@article{arxiv.2307.00342,
title = {Improving Multitask Retrieval by Promoting Task Specialization},
author = {Wenzheng Zhang and Chenyan Xiong and Karl Stratos and Arnold Overwijk},
journal= {arXiv preprint arXiv:2307.00342},
year = {2023}
}