面向脑电信号的语言神经表征学习与句子检索

从非侵入性脑信号解码语言信息由于其广泛的应用潜力，正在获得日益关注的研究注意力。最近，许多工作采用基于生成的框架来解码脑电图（EEG）信号，通过利用预训练大语言模型（LLM）的强大的生成能力来将其解码为句子。然而，这种方法存在若干缺陷，阻碍了脑机接口（BCI）语言应用的进一步发展。具体而言，脑电编码器学习从脑电数据中获取语义信息的能力值得怀疑，LLM解码器倾向于基于其训练记忆生成句子这一问题难以避免。这 necessitates a novel approach for converting EEG signals into sentences. In this paper, we propose a novel two-step pipeline that addresses these limitations and enhances the validity of linguistic EEG decoding research. We first confirm that word-level semantic information can be learned from EEG data recorded during natural reading by training a Conformer encoder via a masked contrastive objective for word-level classification. To achieve sentence decoding results, we employ a training-free retrieval method to retrieve sentences based on the predictions from the EEG encoder. Extensive experiments and ablation studies were conducted in this paper for a comprehensive evaluation of the proposed approach. Visualization of the top prediction candidates reveals that our model effectively groups EEG segments into semantic categories with similar meanings, thereby validating its ability to learn patterns from unspoken EEG recordings. Despite the exploratory nature of this work, these results suggest that our method holds promise for providing more reliable solutions for converting EEG signals into text.