Exploring Language Dependency in Ultrasound-to-Speech Synthesis

Ibrahimov, Ibrahim and Zainkó, Csaba and Gosztolya, Gábor (2025) Exploring Language Dependency in Ultrasound-to-Speech Synthesis. In: 13th edition of the Speech Synthesis Workshop. International Speech Communication Association (ISCA), pp. 163-167.

Preview

Text 2025-ssw-ssi-language-dependency.pdf Download (2MB) | Preview

Abstract

Articulation-to-speech synthesis using ultrasound tongue imaging is a promising approach for Silent Speech Interfaces. However, its effectiveness is hindered by challenges such as session and speaker dependency, dataset scarcity and language variability. This study explores the language dependency of an ultrasound-to-speech synthesis system, consisting of a 2D-CNN to map ultrasound tongue images to mel spectrograms and a HiFi-GAN vocoder. The CNNs were trained on Azerbaijani recordings collected from three native speakers, each recorded in a single session containing both Azerbaijani (L1) and English (L2) sentences, and were then used to generate mel spectrograms for both languages. While the CNNs showed language dependency with lower mean squared error on L1, the mel-cepstral distortion of the synthesized speech did not reflect this, revealing the language bias of the vocoder. These results demonstrate the importance of considering language-specific factors in silent speech synthesis.

Actions (login required)

Edit Item