Hard ASH: Sparsity and the right optimizer make a continual learner

Santtu Keskinen

Hard ASH: Sparsity and the right optimizer make a continual learner

Machine Learning 2024-04-30 v1 Computer Vision and Pattern Recognition

Authors: Santtu Keskinen

Abstract

In class incremental learning, neural networks typically suffer from catastrophic forgetting. We show that an MLP featuring a sparse activation function and an adaptive learning rate optimizer can compete with established regularization techniques in the Split-MNIST task. We highlight the effectiveness of the Adaptive SwisH (ASH) activation function in this context and introduce a novel variant, Hard Adaptive SwisH (Hard ASH) to further enhance the learning retention.

Keywords

sparse learning

Cite

@article{arxiv.2404.17651,
  title  = {Hard ASH: Sparsity and the right optimizer make a continual learner},
  author = {Santtu Keskinen},
  journal= {arXiv preprint arXiv:2404.17651},
  year   = {2024}
}

Comments

ICLR 2024 TinyPaper

Related papers

View all related →

Computer Vision and Pattern Recognition · Computer Science

HyperSparse Neural Networks: Shifting Exploration to Exploitation through Adaptive Regularization