An LSTM Based Architecture to Relate Speech Stimulus to EEG
Abstract
Modeling the relationship between natural speech and a recorded electroencephalogram (EEG) helps us understand how the brain processes speech and has various applications in neuroscience and brain-computer interfaces. In this context, so far mainly linear models have been used. However, the decoding performance of the linear model is limited due to the complex and highly non-linear nature of the auditory processing in the human brain. We present a novel Long Short-Term Memory (LSTM)-based architecture as a non-linear model for the classification problem of whether a given pair of (EEG, speech envelope) correspond to each other or not. The model maps short segments of the EEG and the envelope to a common embedding space using a CNN in the EEG path and an LSTM in the speech path. The latter also compensates for the brain response delay. In addition, we use transfer learning to fine-tune the model for each subject. The mean classification accuracy of the proposed model reaches 85%, which is significantly higher than that of a state of the art Convolutional Neural Network (CNN)-based model (73%) and the linear model (69%).
Keywords
Cite
@article{arxiv.2002.10988,
title = {An LSTM Based Architecture to Relate Speech Stimulus to EEG},
author = {Mohammad Jalilpour Monesi and Bernd Accou and Jair Montoya-Martinez and Tom Francart and Hugo Van Hamme},
journal= {arXiv preprint arXiv:2002.10988},
year = {2020}
}
Comments
3 figures, 6 pages