Notes on Latent Structure Models and SPIGOT
Machine Learning
2019-07-25 v1 Machine Learning
Abstract
These notes aim to shed light on the recently proposed structured projected intermediate gradient optimization technique (SPIGOT, Peng et al., 2018). SPIGOT is a variant of the straight-through estimator (Bengio et al., 2013) which bypasses gradients of the argmax function by back-propagating a surrogate "gradient." We provide a new interpretation to the proposed gradient and put this technique into perspective, linking it to other methods for training neural networks with discrete latent variables. As a by-product, we suggest alternate variants of SPIGOT which will be further explored in future work.
Cite
@article{arxiv.1907.10348,
title = {Notes on Latent Structure Models and SPIGOT},
author = {André F. T. Martins and Vlad Niculae},
journal= {arXiv preprint arXiv:1907.10348},
year = {2019}
}
Comments
7 pages