Learning Sparse Neural Networks via $\ell_0$ and T$\ell_1$ by a Relaxed Variable Splitting Method with Application to Multi-scale Curve Classification

Fanghui Xue; Jack Xin

Learning Sparse Neural Networks via $\ell_0$ and T$\ell_1$ by a Relaxed Variable Splitting Method with Application to Multi-scale Curve Classification

Machine Learning 2019-02-21 v1 Optimization and Control

Authors: Fanghui Xue , Jack Xin

Abstract

We study sparsification of convolutional neural networks (CNN) by a relaxed variable splitting method of $\ell_0$ and transformed- $\ell_1$ (T $\ell_1$ ) penalties, with application to complex curves such as texts written in different fonts, and words written with trembling hands simulating those of Parkinson's disease patients. The CNN contains 3 convolutional layers, each followed by a maximum pooling, and finally a fully connected layer which contains the largest number of network weights. With $\ell_0$ penalty, we achieved over 99 \% test accuracy in distinguishing shaky vs. regular fonts or hand writings with above 86 \% of the weights in the fully connected layer being zero. Comparable sparsity and test accuracy are also reached with a proper choice of T $\ell_1$ penalty.

Keywords

sparse neural networks convolutional neural networks ℓ0 penalty transformed-ℓ1 penalty variable splitting curve classification parkinson's handwriting

Cite

@article{arxiv.1902.07419,
  title  = {Learning Sparse Neural Networks via $\ell_0$ and T$\ell_1$ by a Relaxed Variable Splitting Method with Application to Multi-scale Curve Classification},
  author = {Fanghui Xue and Jack Xin},
  journal= {arXiv preprint arXiv:1902.07419},
  year   = {2019}
}

Learning Sparse Neural Networks via $\ell_0$ and T$\ell_1$ by a Relaxed Variable Splitting Method with Application to Multi-scale Curve Classification

Abstract

Keywords

Cite

Related papers