English

From Silent Signals to Natural Language: A Dual-Stage Transformer-LLM Approach

Computation and Language 2025-09-08 v1 Artificial Intelligence

Abstract

Silent Speech Interfaces (SSIs) have gained attention for their ability to generate intelligible speech from non-acoustic signals. While significant progress has been made in advancing speech generation pipelines, limited work has addressed the recognition and downstream processing of synthesized speech, which often suffers from phonetic ambiguity and noise. To overcome these challenges, we propose an enhanced automatic speech recognition framework that combines a transformer-based acoustic model with a large language model (LLM) for post-processing. The transformer captures full utterance context, while the LLM ensures linguistic consistency. Experimental results show a 16% relative and 6% absolute reduction in word error rate (WER) over a 36% baseline, demonstrating substantial improvements in intelligibility for silent speech interfaces.

Keywords

Cite

@article{arxiv.2509.04507,
  title  = {From Silent Signals to Natural Language: A Dual-Stage Transformer-LLM Approach},
  author = {Nithyashree Sivasubramaniam},
  journal= {arXiv preprint arXiv:2509.04507},
  year   = {2025}
}