Universal Representations: A Unified Look at Multiple Task and Domain Learning
Abstract
We propose a unified look at jointly learning multiple vision tasks and visual domains through universal representations, a single deep neural network. Learning multiple problems simultaneously involves minimizing a weighted sum of multiple loss functions with different magnitudes and characteristics and thus results in unbalanced state of one loss dominating the optimization and poor results compared to learning a separate model for each problem. To this end, we propose distilling knowledge of multiple task/domain-specific networks into a single deep neural network after aligning its representations with the task/domain-specific ones through small capacity adapters. We rigorously show that universal representations achieve state-of-the-art performances in learning of multiple dense prediction problems in NYU-v2 and Cityscapes, multiple image classification problems from diverse domains in Visual Decathlon Dataset and cross-domain few-shot learning in MetaDataset. Finally we also conduct multiple analysis through ablation and qualitative studies.
Cite
@article{arxiv.2204.02744,
title = {Universal Representations: A Unified Look at Multiple Task and Domain Learning},
author = {Wei-Hong Li and Xialei Liu and Hakan Bilen},
journal= {arXiv preprint arXiv:2204.02744},
year = {2022}
}
Comments
Multi-task Learning, Multi-domain Learning, Cross-domain Few-shot Learning, Universal Representation Learning, Balanced Optimization, Dense Prediction, Code and models are available at https://github.com/VICO-UoE/UniversalRepresentations. arXiv admin note: text overlap with arXiv:2103.13841