Speech Synthesis from Intracranial Stereotactic Electroencephalography Using a Neural Vocoder

Arthur, Frigyes Viktor and Csapó, Tamás Gábor (2024) Speech Synthesis from Intracranial Stereotactic Electroencephalography Using a Neural Vocoder. INFOCOMMUNICATIONS JOURNAL : A PUBLICATION OF THE SCIENTIFIC ASSOCIATION FOR INFOCOMMUNICATIONS (HTE), 16 (1). pp. 47-55. ISSN 2061-2079

Preview

Text InfocomJournal_2024_1_6.pdf Download (2MB) | Preview

Abstract

Speech is one of the most important human biosignals. However, only some speech production characteristics are fully understood, which are required for a successful speech based Brain-Computer Interface (BCI). A proper brain-to- speech system that can generate the speech of full sentences intelligibly and naturally poses a great challenge. In our study, we used the SingleWordProduction-Dutch-iBIDS dataset, in which speech and intracranial stereotactic electroencephalography (sEEG) signals of the brain were recorded simultaneously during a single word production task. We apply deep neural networks (FC-DNN, 2D-CNN, and 3D-CNN) on the ten speakers’ data for sEEG-to-Mel spectrogram prediction. Next, we synthesize speech using the WaveGlow neural vocoder. Our objective and subjective evaluations have shown that the DNN based approaches with neural vocoder outperform the baseline linear regression model using Griffin-Lim. The synthesized samples resemble the original speech but are still not intelligible, and the results are clearly speaker dependent. In the long term, speech-based BCI applications might be useful for the speaking impaired or those having neurological disorders.

Actions (login required)

Edit Item