Efficient Conditional Pre-training for Transfer Learning
Abstract
Almost all the state-of-the-art neural networks for computer vision tasks are trained by (1) pre-training on a large-scale dataset and (2) finetuning on the target dataset. This strategy helps reduce dependence on the target dataset and improves convergence rate and generalization on the target task. Although pre-training on large-scale datasets is very useful, its foremost disadvantage is high training cost. To address this, we propose efficient filtering methods to select relevant subsets from the pre-training dataset. Additionally, we discover that lowering image resolutions in the pre-training step offers a great trade-off between cost and performance. We validate our techniques by pre-training on ImageNet in both the unsupervised and supervised settings and finetuning on a diverse collection of target datasets and tasks. Our proposed methods drastically reduce pre-training cost and provide strong performance boosts. Finally, we improve standard ImageNet pre-training by 1-3% by tuning available models on our subsets and pre-training on a dataset filtered from a larger scale dataset.
Cite
@article{arxiv.2011.10231,
title = {Efficient Conditional Pre-training for Transfer Learning},
author = {Shuvam Chakraborty and Burak Uzkent and Kumar Ayush and Kumar Tanmay and Evan Sheehan and Stefano Ermon},
journal= {arXiv preprint arXiv:2011.10231},
year = {2021}
}