Exposing the Functionalities of Neurons for Gated Recurrent Unit Based Sequence-to-Sequence Model
Neural and Evolutionary Computing
2023-03-28 v1 Artificial Intelligence
Machine Learning
Abstract
The goal of this paper is to report certain scientific discoveries about a Seq2Seq model. It is known that analyzing the behavior of RNN-based models at the neuron level is considered a more challenging task than analyzing a DNN or CNN models due to their recursive mechanism in nature. This paper aims to provide neuron-level analysis to explain why a vanilla GRU-based Seq2Seq model without attention can achieve token-positioning. We found four different types of neurons: storing, counting, triggering, and outputting and further uncover the mechanism for these neurons to work together in order to produce the right token in the right position.
Keywords
Cite
@article{arxiv.2303.15072,
title = {Exposing the Functionalities of Neurons for Gated Recurrent Unit Based Sequence-to-Sequence Model},
author = {Yi-Ting Lee and Da-Yi Wu and Chih-Chun Yang and Shou-De Lin},
journal= {arXiv preprint arXiv:2303.15072},
year = {2023}
}
Comments
9 pages (excluding reference), 10 figures