Phone duration modeling: overview of techniques and performance optimization via feature selection in the context of emotional speech

Abstract

Accurate modeling of prosody is prerequisite for the production of synthetic speech of high quality. Phone duration, as one of the key prosodic parameters, plays an important role for the generation of emotional synthetic speech with natural sounding. In the present work we offer an overview of various phone duration modeling techniques, and consequently evaluate ten models, based on decision trees, linear regression, lazy-learning algorithms and meta-learning algorithms, which over the past decades have been successfully used in various modeling tasks. Furthermore, we study the opportunity for performance optimization by applying two feature selection techniques, the RReliefF and the Correlation-based Feature Selection, on a large set of numerical and nominal linguistic features extracted from text, such as: phonetic, phonologic and morphosyntactic ones, which have been reported successful on the phone and syllable duration modeling task. We investigate the practical usefulness of these phone duration modeling techniques on a Modern Greek emotional speech database, which consists of five categories of emotional speech: anger, fear, joy, neutral, sadness. The experimental results demonstrated that feature selection significantly improves the accuracy of phone duration prediction regardless of the type of machine learning algorithm used for phone duration modeling. Specifically, in four out of the five categories of emotional speech, feature selection contributed to the improvement of the phone duration modeling, when compared to the case without feature selection. The M5p trees based phone duration model was observed to achieve the best phone duration prediction accuracy in terms of RMSE and MAE.

This is a preview of subscription content, access via your institution.

References

  1. Aha, D., Kibler, D., & Albert, M. (1991). Instance-based learning algorithms. Journal of Machine Learning, 6, 37–66.

    Google Scholar 

  2. Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19, 716–723.

    MATH  Article  MathSciNet  Google Scholar 

  3. Allen, J., Hunnicutt, S., & Klatt, D. H. (1987). From text to speech: the MITalk system. Cambridge: Cambridge University Press.

    Google Scholar 

  4. Arvaniti, A., & Baltazani, M. (2000). Greek ToBI: a system for the annotation of Greek speech corpora. In Proceedings of the 2nd international conference on language resources and evaluation (pp. 555–562). Athens, Greece.

  5. Atkeson, C. G., Moorey, A. W., & Schaal, S. (1996). Locally weighted learning. Artificial Intelligence Review, 11, 11–73.

    Article  Google Scholar 

  6. Barbosa, P. A., & Bailly, G. (1994). Characterisation of rhythmic patterns for text-to-speech synthesis. Speech Communication, 15, 127–137.

    Article  Google Scholar 

  7. Bartkova, K., & Sorin, C. (1987). A model of segmental duration for speech synthesis in French. Speech Communication, 6, 245–260.

    Article  Google Scholar 

  8. Bell, A., Jurafsky, D., Fosler-Lussier, E., Girand, C., Gregory, M., & Gildea, D. (2003). Effects of disfluencies, predictability, and utterance position on word form variation in English conversation. Journal of the Acoustical Society of America, 113(2), 1001–1024.

    Article  Google Scholar 

  9. Black, A. (2003). Unit selection and emotional speech. In Proceedings of EUROSPEECH’03 (pp. 1649–1652). Geneva, Switzerland.

  10. Breiman, L. (1996). Bagging predictors. Journal of Machine Learning, 24(2), 123–140.

    MATH  MathSciNet  Google Scholar 

  11. Burkhardt, F., & Sendlmeier, W. F. (2000). Verification of acoustical correlates of emotional speech using formant-synthesis. In Proceedings of the ISCA workshop on speech & emotion (pp. 151–156). Northern Ireland.

  12. Campbell, W. N. (1992). Syllable based segment duration. In G. Bailly, C. Benoit, & T. R. Sawallis (Eds.), Talking machines: theories, models and designs (pp. 211–224). Amsterdam: Elsevier.

    Google Scholar 

  13. Carlson, R., & Granstrom, B. (1986). A search for durational rules in real speech database. Phonetica, 43, 140–154.

    Article  Google Scholar 

  14. Chien, J. T., & Huang, C. H. (2003). Bayesian learning of speech duration models. IEEE Transactions on Speech and Audio Processing, 11(6), 558–567.

    Article  Google Scholar 

  15. Chung, H. (2002). Duration models and the perceptual evaluation of spoken Korean. In Proceedings of speech prosody (pp. 219–222). France.

  16. Cordoba, R., Montero, J. M., Gutierrez-Ariola, J., & Pardo, J. M. (2001). Duration modeling in a restricted-domain female-voice synthesis in Spanish using neural networks. In Proceedings of ICASSP’01 (pp. 793–796). Utah, USA.

  17. Crystal, T. H., & House, A. S. (1988). Segmental durations in connected-speech signals: current results. Journal of the Acoustical Society of America, 83(4), 1553–1573.

    Article  Google Scholar 

  18. Dutoit, T. (1997). An introduction to text-to-speech synthesis. Dordrecht: Kluwer Academic.

    Google Scholar 

  19. Epitropakis, G., Tambakas, D., Fakotakis, N., & Kokkinakis, G. (1993). Duration modelling for the Greek language. In Proceedings of EUROSPEECH’93 (pp. 1995–1998). Berlin, Germany.

  20. Febrer, A., Padrell, J., & Bonafonte, A. (1998). Modeling phone duration: application to Catalan TTS. In Workshop of speech synthesis (pp. 43–46). Australia.

  21. Friedman, J. H. (2001). Greedy function approximation: a gradient boosting machine. Annals of Statistics, 29(5), 1189–1232.

    MATH  Article  MathSciNet  Google Scholar 

  22. Friedman, J. H. (2002). Stochastic gradient boosting. Computational Statistics and Data Analysis, 38(4), 367–378.

    MATH  Article  MathSciNet  Google Scholar 

  23. Gilad-Bachrach, R., Navot, A., & Tishby, N. (2004). Margin based feature selection—theory and algorithms. In P. Tadepalli, R. Givan, & K. Driessens (Eds.), Proceedings of the 21st international conference on machine learning (pp. 43–50). Banff: Morgan Kaufmann.

    Google Scholar 

  24. Goldberg, D. E. (1989). Genetic algorithms in search, optimization and machine learning. Boston: Addison–Wesley/Longman.

    Google Scholar 

  25. Goubanova, O., & King, S. (2008). Bayesian network for phone duration prediction. Speech Communication, 50, 301–311.

    Article  Google Scholar 

  26. Goubanova, O., & Taylor, P. (2000). Using Bayesian belief networks for modeling duration in text-to-speech systems. In Proceedings of the ICSLP’00 (pp. 427–431). Beijing, China.

  27. Gregory, M., Bell, A., Jurafsky, D., & Raymond, W. (2001). Frequency and predictability effects on the duration of content words in conversation. Journal of the Acoustical Society of America, 110(5), 27–38.

    Google Scholar 

  28. Hall, M. A. (1999). Correlation-based feature subset selection for machine learning. PhD thesis, Department of Computer Science, University of Waikato, Waikato, New Zealand.

  29. Hall, M. A. (2000). Correlation-based feature selection for discrete and numeric class machine learning. In P. Langley (Ed.), Proceedings of the 17th international conference on machine learning (pp. 359–366). San Francisco: Morgan Kaufmann.

    Google Scholar 

  30. Heuft, B., Portele, T., & Rauth, M. (1996). Emotions in time domain synthesis. In Proceedings of ICSLP’96 (pp. 1974–1977). Philadelphia, USA.

  31. Inanoglu, Z., & Young, S. (2009). Data-driven emotion conversion in spoken English. Speech Communication, 51, 268–283.

    Article  Google Scholar 

  32. Iida, A., Campbell, N., Iga, S., Higuchi, F., & Yasumura, M. (2000). A speech synthesis system for assisting communication. In Proceedings of the ISCA workshop on speech & emotion (pp. 167–172). Northern Ireland.

  33. Iwahashi, N., & Sagisaka, Y. (2000). Statistical modeling of speech segment duration by constrained tree regression. IEICE Transactions on Information and Systems, E83-D(7), 1550–1559.

    Google Scholar 

  34. Jiang, D. N., Zhang, W., Shen, L., & Cai, L. H. (2005). Prosody analysis and modeling for emotional speech synthesis. In Proceedings of ICASSP’05 (pp. 281–284). Philadelphia, USA.

  35. Kääriäinen, M., & Malinen, T. (2004). Selective rademacher penalization and reduced error pruning of decision trees. Journal of Machine Learning Research, 5, 1107–1126.

    Google Scholar 

  36. Kira, K., & Rendell, L. A. (1992). A practical approach to feature selection. In Sleeman, & P. Edwards (Eds.), Proceedings of the 9th international conference on machine learning (pp. 249–256). Aberdeen, Scotland. San Francisco: Morgan Kaufmann.

    Google Scholar 

  37. Klatt, D. H. (1976). Linguistic uses of segmental duration in English: Acoustic and perceptual evidence. Journal of the Acoustical Society of America, 59, 1209–1221.

    Article  Google Scholar 

  38. Klatt, D. H. (1979). Synthesis by rule of segmental durations in English sentences. In B. Lindlom & S. Ohman (Eds.), Frontiers of speech communication research (pp. 287–300). New York: Academic Press.

    Google Scholar 

  39. Klatt, D. H. (1987). Review of text-to-speech conversion for English. Journal of the Acoustical Society of America, 82(3), 737–793.

    Article  Google Scholar 

  40. Kohler, K. J. (1988). Zeistrukturierung in der Sprachsynthese. ITG-Tagung Digitalc Sprachverarbeitung, 6, 165–170.

    MathSciNet  Google Scholar 

  41. Kominek, J., & Black, A. W. (2003). CMU ARCTIC databases for speech synthesis, CMU-LTI-03-177, Language Technologies Institute, School of Computer Science, Carnegie Mellon University.

  42. Kononenko, I. (1994). Estimating attributes: analysis and extensions of relief. In F. Bergadano & L. De Raedt (Eds.), Proceedings of the European conference machine learning (pp. 171–182). New York: Springer.

    Google Scholar 

  43. Krishna, N. S., & Murthy, H. A. (2004). Duration modeling of Indian languages Hindi and Telugu. In Proceedings of the 5th ISCA speech synthesis workshop (pp. 197–202). Pittsburgh, USA.

  44. Krishna, N. S., Talukdar, P. P., Bali, K., & Ramakrishnan, A. G. (2004). Duration modeling for Hindi text-to-speech synthesis system. In Proceedings of ICSLP’04 (pp. 789–792). Jeju Island, Korea.

  45. Lazaridis, A., Zervas, P., & Kokkinakis, G. (2007). Segmental duration modeling for Greek speech synthesis. In Proceedings of ICTAI’07 (pp. 518–521). Patras, Greece.

  46. Lee, S., & Oh, Y. H. (1999a). Tree-based modeling of prosodic phrasing and segmental duration for Korean TTS systems. Speech Communication, 28, 283–300.

    Article  Google Scholar 

  47. Lee, S., & Oh, Y. H. (1999b). CART-based modelling of Korean segmental duration. In Proceedings of the oriental COCOSDA’99 (pp. 109–112). Taipei, Taiwan.

  48. Möbius, B., & Santen, P. H. J. (1996). Modeling segmental duration in German text-to-speech synthesis. In Proceedings of ICSLP’96 (pp. 2395–2398). Philadelphia, USA.

  49. Murray, I. R., & Arnott, J. L. (1995). Implementation and testing of a system for producing emotion-by-rule in synthetic speech. Speech Communication, 16, 369–390.

    Article  Google Scholar 

  50. Oatley, K., & Johnson-Laird, P. (1998). The communicative theory of emotions. In J. Jenkins, K. Oatley, & N. Stein (Eds.), Human emotions: a reader (pp. 84–87). Oxford: Blackwell.

    Google Scholar 

  51. Olive, J. P., & Liberman, M. Y. (1985). Text to speech—an overview. Journal of the Acoustical Society of America, 78(1), S6.

    Article  Google Scholar 

  52. Quinlan, R. J. (1992). Learning with continuous classes. In Proceedings of the 5th Australian Joint Conference on Artificial Intelligence (pp. 343–348). Hobart, Tasmania.

  53. Rank, E., & Pirker, H. (1998). Generating Emotional Speech with a Concatenative Synthesizer. In Proceedings of ICSLP’98 (pp. 671–674). Sydney, Australia.

  54. Rao, K. S., & Yegnanarayana, B. (2007). Modeling durations of syllables using neural networks. Computer Speech & Language, 21(2), 282–295.

    Article  Google Scholar 

  55. Riley, M. (1992). Tree-based modelling for speech synthesis. In G. Bailly, C. Benoit, & T. R. Sawallis (Eds.), Talking machines: theories, models and designs (pp. 265–273). Amsterdam: Elsevier.

    Google Scholar 

  56. Robnik-Sikonja, M., & Kononenko, I. (1997). An adaptation of relief for attribute estimation in regression. In D. H. Fisher (Ed.), Proceedings of the 14th international conference on machine learning (pp. 296–304). San Francisco: Morgan Kaufmann.

    Google Scholar 

  57. Silverman, K., Beckman, M., Pitrelli, J., Ostendorf, M., Wightman, C., Price, P., Pierrehumbert, J., & Hirschberg, J. (1992). ToBI: a standard for labeling English prosody. In Proceedings of ICSLP’92 (pp. 867–870). Banff, Alberta, Canada.

  58. Simoes, A. R. M. (1990). Predicting sound segment duration in connected speech: an acoustical study of Brazilian Portuguese. In Proceedings of the workshop on speech synthesis (pp. 173–176). Autrans, France.

  59. Takeda, K., Sagisaka, Y., & Kuwabara, H. (1989). On sentence-level factors governing segmental duration in Japanese. Journal of Acoustic Society of America, 86(6), 2081–2087.

    Article  Google Scholar 

  60. Tesser, F., Cosi, P., Drioli, C., & Tisato, G. (2005). Emotional festival-mbrola TTS synthesis. In Proceedings of INTERSPEECH’05 (pp. 505–508). Lisboa, Portugal.

  61. Teixeira, J. P., & Freitas, D. (2003). Segmental durations predicted with a neural network. In Proceedings of EUROSPEECH’03 (pp. 169–172). Geneva, Switzerland, September.

  62. van Santen, J. P. H. (1992). Contextual effects on vowel durations. Speech Communication, 11, 513–546.

    Article  Google Scholar 

  63. van Santen, J. P. H. (1994). Assignment of segmental duration in text-to-speech synthesis. Computer Speech & Language, 8(2), 95–128.

    Article  Google Scholar 

  64. Wang, Y., & Witten, I. H. (1997). Induction of model trees for predicting continuous classes. In Proceedings of the 9th European conference on machine learning (pp. 128–137). University of Economics, Faculty of Informatics and Statistics, Prague, Czech.

  65. Wang, L., Zhao, Y., Chu, M., Zhou, J., & Cao, Z. (2004). Refining segmental boundaries for TTS database using fine contextual-dependent boundary models. In Proceedings of ICASSP’04 (pp. 641–644). Montreal, Canada.

  66. Witten, H. I., & Frank, E. (2005). Data mining: practical machine learning tools and techniques (2nd ed.) San Francisco: Morgan Kaufmann.

    Google Scholar 

  67. Yamagishi, J., Kawai, H., & Kobayashi, T. (2008). Phone duration modeling using gradient tree boosting. Speech Communication, 50(5), 405–415.

    Article  Google Scholar 

  68. Yegnanarayana, B. (1999). Artificial neural networks. New Delhi: Prentice Hall.

    Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Alexandros Lazaridis.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Lazaridis, A., Ganchev, T., Kostoulas, T. et al. Phone duration modeling: overview of techniques and performance optimization via feature selection in the context of emotional speech. Int J Speech Technol 13, 175–188 (2010). https://doi.org/10.1007/s10772-010-9077-x

Download citation

Keywords

  • Phone duration modeling
  • Statistical modeling
  • Feature selection
  • Emotional speech
  • Text-to-speech synthesis