Iteratively Training Look-Up Tables for Network Quantization
Abstract
Operating deep neural networks (DNNs) on devices with limited resources requires the reduction of their memory as well as computational footprint. Popular reduction methods are network quantization or pruning, which either reduce the word length of the network parameters or remove weights from the network if they are not needed. In this article we discuss a general framework for network reduction which we call `Look-Up Table Quantization` (LUT-Q). For each layer, we learn a value dictionary and an assignment matrix to represent the network weights. We propose a special solver which combines gradient descent and a one-step k-means update to learn both the value dictionaries and assignment matrices iteratively. This method is very flexible: by constraining the value dictionary, many different reduction problems such as non-uniform network quantization, training of multiplierless networks, network pruning or simultaneous quantization and pruning can be implemented without changing the solver. This flexibility of the LUT-Q method allows us to use the same method to train networks for different hardware capabilities.
Keywords
Cite
@article{arxiv.1911.04951,
title = {Iteratively Training Look-Up Tables for Network Quantization},
author = {Fabien Cardinaux and Stefan Uhlich and Kazuki Yoshiyama and Javier Alonso Garcia and Lukas Mauch and Stephen Tiedemann and Thomas Kemp and Akira Nakamura},
journal= {arXiv preprint arXiv:1911.04951},
year = {2023}
}
Comments
Copyright 2019 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works