We present an extension to masked autoencoders (MAE) which improves on the representations learnt by the model by explicitly encouraging the learning of higher scene-level features. We do this by: (i) the introduction of a perceptual similarity term between generated and real images (ii) incorporating several techniques from the adversarial training literature including multi-scale training and adaptive discriminator augmentation. The combination of these results in not only better pixel reconstruction but also representations which appear to capture better higher-level details within images. More consequentially, we show how our method, Perceptual MAE, leads to better performance when used for downstream tasks outperforming previous methods. We achieve 78.1% top-1 accuracy linear probing on ImageNet-1K and up to 88.1% when fine-tuning, with similar results for other downstream tasks, all without use of additional pre-trained models or data.
@article{arxiv.2212.14504,
title = {Improving Visual Representation Learning through Perceptual Understanding},
author = {Samyakh Tukra and Frederick Hoffman and Ken Chatfield},
journal= {arXiv preprint arXiv:2212.14504},
year = {2023}
}
Comments
v2: add additional details on MSG-MAE. In Proc CVPR 2023