English

Continuous Speech Recognition using EEG and Video

Machine Learning 2020-01-01 v5 Audio and Speech Processing Image and Video Processing Machine Learning
Authors: Gautam Krishna , Mason Carnahan , Co Tran , Ahmed H Tewfik
View on arXiv ↗ PDF ↗

Abstract

In this paper we investigate whether electroencephalography (EEG) features can be used to improve the performance of continuous visual speech recognition systems. We implemented a connectionist temporal classification (CTC) based end-to-end automatic speech recognition (ASR) model for performing recognition. Our results demonstrate that EEG features are helpful in enhancing the performance of continuous visual speech recognition systems.

Keywords

eeg decoding eeg classification automatic speech recognition

Cite

@article{arxiv.1912.07730,
  title  = {Continuous Speech Recognition using EEG and Video},
  author = {Gautam Krishna and Mason Carnahan and Co Tran and Ahmed H Tewfik},
  journal= {arXiv preprint arXiv:1912.07730},
  year   = {2020}
}

Comments

On preparation for submission to EUSIPCO 2020. arXiv admin note: text overlap with arXiv:1911.11610, arXiv:1911.04261

Related papers

View all related →
Audio and Speech Processing · Electrical Eng. & Systems
Improving EEG based Continuous Speech Recognition
Gautam Krishna, Co Tran, Mason Carnahan, Yan Han +1
2019-12-25
Audio and Speech Processing · Electrical Eng. & Systems
Continuous Silent Speech Recognition using EEG
Gautam Krishna, Co Tran, Mason Carnahan, Ahmed Tewfik
2020-05-06
Audio and Speech Processing · Electrical Eng. & Systems
EEG based Continuous Speech Recognition using Transformers
Gautam Krishna, Co Tran, Mason Carnahan, Ahmed H Tewfik
2020-05-06
Audio and Speech Processing · Electrical Eng. & Systems
Advancing Speech Recognition With No Speech Or With Noisy Speech
Gautam Krishna, Co Tran, Mason Carnahan, Ahmed H Tewfik
2020-03-17
Audio and Speech Processing · Electrical Eng. & Systems
State-of-the-art Speech Recognition using EEG and Towards Decoding of Speech Spectrum From EEG
Gautam Krishna, Yan Han, Co Tran, Mason Carnahan +1
2020-03-06
Audio and Speech Processing · Electrical Eng. & Systems
Understanding effect of speech perception in EEG based speech recognition systems
Gautam Krishna, Co Tran, Mason Carnahan, Ahmed Tewfik
2020-06-03
Audio and Speech Processing · Electrical Eng. & Systems
Improving EEG based continuous speech recognition using GAN
Gautam Krishna, Co Tran, Mason Carnahan, Ahmed Tewfik
2020-06-03
Machine Learning · Computer Science
Speech Recognition with no speech or with noisy speech
Gautam Krishna, Co Tran, Jianguo Yu, Ahmed H Tewfik
2019-03-05
Audio and Speech Processing · Electrical Eng. & Systems
Speech Recognition With No Speech Or With Noisy Speech Beyond English
Gautam Krishna, Co Tran, Yan Han, Mason Carnahan +1
2020-03-02
Audio and Speech Processing · Electrical Eng. & Systems
Robust End-to-End Speaker Verification Using EEG
Yan Han, Gautam Krishna, Co Tran, Mason Carnahan +1
2020-06-11
Computation and Language · Computer Science
Robust end-to-end deep audiovisual speech recognition
Ramon Sanabria, Florian Metze, Fernando De La Torre
2016-11-22
Audio and Speech Processing · Electrical Eng. & Systems
End-to-End Multimodal Speech Recognition
Shruti Palaskar, Ramon Sanabria, Florian Metze
2018-04-27
Audio and Speech Processing · Electrical Eng. & Systems
Constrained Variational Autoencoder for improving EEG based Speech Recognition Systems
Gautam Krishna, Co Tran, Mason Carnahan, Ahmed Tewfik
2020-06-05
Sound · Computer Science
Improving Voice Separation by Incorporating End-to-end Speech Recognition
Naoya Takahashi, Mayank Kumar Singh, Sakya Basak, Parthasaarathy Sudarsanam +2
2020-05-05
Audio and Speech Processing · Electrical Eng. & Systems
Advancing Speech Synthesis using EEG
Gautam Krishna, Co Tran, Mason Carnahan, Ahmed Tewfik
2020-05-05
Sound · Computer Science
Unified End-to-End Speech Recognition and Endpointing for Fast and Efficient Speech Systems
Shaan Bijwadia, Shuo-yiin Chang, Bo Li, Tara Sainath +2
2023-02-16
Sound · Computer Science
Speech recognition for air traffic control via feature learning and end-to-end training
Peng Fan, Dongyue Guo, Yi Lin, Bo Yang +1
2021-11-05
Sound · Computer Science
Brain Signals to Rescue Aphasia, Apraxia and Dysarthria Speech Recognition
Gautam Krishna, Mason Carnahan, Shilpa Shamapant, Yashitha Surendranath +5
2021-07-20
Audio and Speech Processing · Electrical Eng. & Systems
Hierarchical Conditional End-to-End ASR with CTC and Multi-Granular Subword Units
Yosuke Higuchi, Keita Karube, Tetsuji Ogawa, Tetsunori Kobayashi
2022-02-09
Audio and Speech Processing · Electrical Eng. & Systems
Speech Synthesis using EEG
Gautam Krishna, Co Tran, Yan Han, Mason Carnahan
2020-05-05
Audio and Speech Processing · Electrical Eng. & Systems
End-to-End Automatic Speech Recognition Integrated With CTC-Based Voice Activity Detection
Takenori Yoshimura, Tomoki Hayashi, Kazuya Takeda, Shinji Watanabe
2020-03-16
Audio and Speech Processing · Electrical Eng. & Systems
Speaker Identification using EEG
Gautam Krishna, Co Tran, Mason Carnahan, Ahmed Tewfik
2020-03-11
Computation and Language · Computer Science
Advances in Joint CTC-Attention based End-to-End Speech Recognition with a Deep CNN Encoder and RNN-LM
Takaaki Hori, Shinji Watanabe, Yu Zhang, William Chan
2017-06-12
Computation and Language · Computer Science
Continual Learning in Machine Speech Chain Using Gradient Episodic Memory
Geoffrey Tyndall, Kurniawati Azizah, Dipta Tanaya, Ayu Purwarianti +2
2024-11-28
Computer Vision and Pattern Recognition · Computer Science
Speaker-Adapted End-to-End Visual Speech Recognition for Continuous Spanish
David Gimeno-Gómez, Carlos-D. Martínez-Hinarejos
2023-11-22
R2 v1 2026-06-23T12:47:50.924Z