Two-Stage Sequence-to-Sequence Neural Voice Conversion with Low-to-High Definition Spectrogram Mapping
In this study, we propose a voice conversion technique with two-stage conversion, which is realized by using two models consisting of U-Net and pix2pix. Using U-Net, we tried to reproduce intonation of a target speaker by performing low-dimensional feature conversion considering the time direction. We introduced pix2pix for the task of spectrogram enhancement. The pix2pix is trained to map from low definition spectrogram to high definition spectrogram (low-to-high spectrogram mapping). Low definition spectrogram is reconstructed from low dimensional mel-cepstrum converted by U-Net and high definition spectrogram is extracted from natural speech. In objective evaluations, we showed that the proposed method was effective in improvement of mel-cepstral distance (MCD) and Log F0 RMSE. Subjective evaluations revealed that the use of the proposed method had a certain effect in improving speech individuality while maintaining the same level of naturalness as the conventional method.
KeywordsDNN-based voice conversion U-Net Pix2pix CNN Two-stage conversion
Part of this work was supported by JSPS KAKENHI Grant Numbers JP16K13253 and JP17H00823.
- 1.Alec, R., Luke, M., Soumith, C.: Unsupervised representation learning with deep convolutional generative adversarial networks. In: International Conference on Learning Representations (2016)Google Scholar
- 2.Desai, S., Raghavendra, E.V., Yegnanarayana, B., Black, A.W., Prahallad, K.: Voice conversion using artificial neural networks. In: 2009 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3893–3896. IEEE (2009)Google Scholar
- 4.Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: IEEE/CVF International Conference on Computer Vision and Pattern Recognition (2017)Google Scholar
- 5.Kain, A., Macon, M.W.: Spectral voice conversion for text-to-speech synthesis. In: 1998 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 285–288. IEEE (1998)Google Scholar
- 6.Kaneko, T., Kameoka, H., Hiramatsu, K., Kashino, K.: Sequence-to-sequence voice conversion with similarity metric learned using generative adversarial networks. In: Proceedings of the INTERSPEECH, pp. 1283–1287 (2017)Google Scholar
- 7.Kingma, D., Ba, J.: Adam: a method for stochastic optimization. In: International Conference on Learning Representations (2015)Google Scholar
- 8.Kobayashi, K., Toda, T.: sprocket: open-source voice conversion software. In: Proceedings of the Odyssey 2018 The Speaker and Language Recognition Workshop, pp. 203–210 (2018)Google Scholar
- 9.Masanobu, A.: A segment-based approach to voice conversion. In: 1991 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), vol. 2, pp. 765–768 (1991)Google Scholar
- 10.Masci, J., Meier, U., Cireşan, D., Schmidhuber, J.: Stacked convolutional auto-encoders for hierarchical feature extraction. In: International Conference on Artificial Neural Networks, pp. 52–59. Springer (2011)Google Scholar
- 11.Mirza, M., Osindero, S.: Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 (2014)
- 14.Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: International Conference on Medical Image Computing and Computer-assisted Intervention, pp. 234–241. Springer (2015)Google Scholar
- 16.Sun, L., Kang, S., Li, K., Meng, H.: Voice conversion using deep bidirectional long short-term memory based recurrent neural networks. In: 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4869–4873. IEEE (2015)Google Scholar