Recreating Neural Activity During Speech Production with Language and Speech Model Embeddings
Abstract
Understanding how neural activity encodes speech and language production is a fundamental challenge in neuroscience and artificial intelligence. This study investigates whether embeddings from large-scale, self-supervised language and speech models can effectively reconstruct high-gamma neural activity characteristics, key indicators of cortical processing, recorded during speech production. We leverage pre-trained embeddings from deep learning models trained on linguistic and acoustic data to represent high-level speech features and map them onto these high-gamma signals. We analyze the extent to which these embeddings preserve the spatio-temporal dynamics of brain activity. Reconstructed neural signals are evaluated against high-gamma ground-truth activity using correlation metrics and signal reconstruction quality assessments. The results indicate that high-gamma activity can be effectively reconstructed using large language and speech model embeddings in all study participants, generating Pearson's correlation coefficients ranging from 0.79 to 0.99.
Cite
@article{arxiv.2505.14074,
title = {Recreating Neural Activity During Speech Production with Language and Speech Model Embeddings},
author = {Owais Mujtaba Khanday and Pablo Rodroguez San Esteban and Zubair Ahmad Lone and Marc Ouellet and Jose Andres Gonzalez Lopez},
journal= {arXiv preprint arXiv:2505.14074},
year = {2025}
}
Comments
Accepted for presentation at Interspeech2025