Robust Target Speaker Diarization and Separation via Augmented Speaker Embedding Sampling
Abstract
Traditional speech separation and speaker diarization approaches rely on prior knowledge of target speakers or a predetermined number of participants in audio signals. To address these limitations, recent advances focus on developing enrollment-free methods capable of identifying targets without explicit speaker labeling. This work introduces a new approach to train simultaneous speech separation and diarization using automatic identification of target speaker embeddings, within mixtures. Our proposed model employs a dual-stage training pipeline designed to learn robust speaker representation features that are resilient to background noise interference. Furthermore, we present an overlapping spectral loss function specifically tailored for enhancing diarization accuracy during overlapped speech frames. Experimental results show significant performance gains compared to the current SOTA baseline, achieving 71% relative improvement in DER and 69% in cpWER.
Cite
@article{arxiv.2508.06393,
title = {Robust Target Speaker Diarization and Separation via Augmented Speaker Embedding Sampling},
author = {Md Asif Jalal and Luca Remaggi and Vasileios Moschopoulos and Thanasis Kotsiopoulos and Vandana Rajan and Karthikeyan Saravanan and Anastasis Drosou and Junho Heo and Hyuk Oh and Seokyeong Jeong},
journal= {arXiv preprint arXiv:2508.06393},
year = {2025}
}
Comments
Accepted to Interspeech 2025