Top-k Training of GANs: Improving GAN Performance by Throwing Away Bad Samples
Abstract
We introduce a simple (one line of code) modification to the Generative Adversarial Network (GAN) training algorithm that materially improves results with no increase in computational cost: When updating the generator parameters, we simply zero out the gradient contributions from the elements of the batch that the critic scores as `least realistic'. Through experiments on many different GAN variants, we show that this `top-k update' procedure is a generally applicable improvement. In order to understand the nature of the improvement, we conduct extensive analysis on a simple mixture-of-Gaussians dataset and discover several interesting phenomena. Among these is that, when gradient updates are computed using the worst-scoring batch elements, samples can actually be pushed further away from their nearest mode. We also apply our method to recent GAN variants and improve state-of-the-art FID for conditional generation from 9.21 to 8.57 on CIFAR-10.
Keywords
Cite
@article{arxiv.2002.06224,
title = {Top-k Training of GANs: Improving GAN Performance by Throwing Away Bad Samples},
author = {Samarth Sinha and Zhengli Zhao and Anirudh Goyal and Colin Raffel and Augustus Odena},
journal= {arXiv preprint arXiv:2002.06224},
year = {2020}
}
Comments
NeurIPS 2020. Samarth Sinha and Zhengli Zhao contributed equally as joint first authors