Data Augmentation Methods on Ultrasound Tongue Images for Articulation-to-Speech Synthesis

Ibrahimov, Ibrahim and Gosztolya, Gábor and Csapó, Tamás Gábor (2023) Data Augmentation Methods on Ultrasound Tongue Images for Articulation-to-Speech Synthesis. In: 12th ISCA Speech Synthesis Workshop (SSW2023). ISCA, [s.l.], pp. 230-235.

Preview

Text Ibrahimov-et-al-ssw2023.pdf Download (1MB) | Preview

Abstract

Articulation-to-Speech Synthesis (ATS) focuses on converting articulatory biosignal information into audible speech, nowadays mostly using DNNs, with a future target application of a Silent Speech Interface. Ultrasound Tongue Imaging (UTI) is an affordable and non-invasive technique that has become popular for collecting articulatory data. Data augmentation has been shown to improve the generalization ability of DNNs, e.g. to avoid overfitting, introduce variations into the existing dataset, or make the network more robust against various noise types on the input data. In this paper, we compare six different data augmentation methods on the UltraSuite-TaL corpus during UTIbased ATS using CNNs. Validation mean squared error is used to evaluate the performance of CNNs, while by the synthesized speech samples, the performace of direct ATS is measured using MCD and PESQ scores. Although we did not find large differences in the outcome of various data augmentation techniques, the results of this study suggest that while applying data augmentation techniques on UTI poses some challenges due to the unique nature of the data, it provides benefits in terms of enhancing the robustness of neural networks. In general, articulatory control might be beneficial in TTS as well.

Actions (login required)

Edit Item