Interpretable Modeling of Articulatory Temporal Dynamics from real-time MRI for Phoneme Recognition
Abstract
Real-time Magnetic Resonance Imaging (rtMRI) visualizes vocal tract action, offering a comprehensive window into speech articulation. However, its signals are high dimensional and noisy, hindering interpretation. We investigate compact representations of spatiotemporal articulatory dynamics for phoneme recognition from midsagittal vocal tract rtMRI videos. We compare three feature types: (1) raw video, (2) optical flow, and (3) six linguistically-relevant regions of interest (ROIs) for articulator movements. We evaluate models trained independently on each representation, as well as multi-feature combinations. Results show that multi-feature models consistently outperform single-feature baselines, with the lowest phoneme error rate (PER) of 0.34 obtained by combining ROI and raw video. Temporal fidelity experiments demonstrate a reliance on fine-grained articulatory dynamics, while ROI ablation studies reveal strong contributions from tongue and lips. Our findings highlight how rtMRI-derived features provide accuracy and interpretability, and establish strategies for leveraging articulatory data in speech processing.
Keywords
Cite
@article{arxiv.2509.15689,
title = {Interpretable Modeling of Articulatory Temporal Dynamics from real-time MRI for Phoneme Recognition},
author = {Jay Park and Hong Nguyen and Sean Foley and Jihwan Lee and Yoonjeong Lee and Dani Byrd and Shrikanth Narayanan},
journal= {arXiv preprint arXiv:2509.15689},
year = {2026}
}