Skip to main content
SpringerLink
Log in
SpringerLink
Log in

Detection of Prosodic Boundaries in Speech Using Wav2Vec 2.0

  • Conference paper
  • First Online:
Text, Speech, and Dialogue (TSD 2022)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 13502))

Included in the following conference series:

Abstract

Prosodic boundaries in speech are of great relevance to both speech synthesis and audio annotation. In this paper, we apply the wav2vec 2.0 framework to the task of detecting these boundaries in speech signal, using only acoustic information. We test the approach on a set of recordings of Czech broadcast news, labeled by phonetic experts, and compare it to an existing text-based predictor, which uses the transcripts of the same data. Despite using a relatively small amount of labeled data, the wav2vec2 model achieves an accuracy of 94% and F1 measure of 83% on within-sentence prosodic boundaries (or 95% and 89% on all prosodic boundaries), outperforming the text-based approach. However, by combining the outputs of the two different models we can improve the results even further.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Since the publication of [20], the NRS annotations have undergone a round of revisions and the model was updated accordingly. The text-based results in Sect. 5.3 will thus differ from those listed in the aforementioned paper.

  2. 2.

    Czech language TRransformer from Unlabeled Speech,

    available from: https://huggingface.co/fav-kky/wav2vec2-base-cs-80k-ClTRUS.

References

  1. Baevski, A., Zhou, Y., Mohamed, A., Auli, M.: wav2vec 2.0: A framework for self-supervised learning of speech representations. In: Advances in Neural Information Processing Systems, vol. 33, pp. 12449–12460 (2020)

    Google Scholar 

  2. Beckman, M.E., Ayers Elam, G.: Guidelines for ToBI Labelling, Version 3. The Ohio State University Research Foundation, Ohio State University (1997)

    Google Scholar 

  3. Bredin, H.: TristouNet: triplet loss for speaker turn embedding. In: Proceedings of ICASSP 2017, pp. 5430–5434 (2017)

    Google Scholar 

  4. Christodoulides, G., Avanzi, M., Simon, A.C.: Automatic labelling of prosodic prominence, phrasing and disfluencies in French speech by simulating the perception of Naïve and expert listeners. In: Proceedings of InterSpeech 2017, pp. 3936–3940 (2017)

    Google Scholar 

  5. Cooper, E., Huang, W.C., Toda, T., Yamagishi, J.: Generalization ability of MOS prediction networks. In: Proceedings of ICASSP 2022, pp. 8442–8446 (2022)

    Google Scholar 

  6. Frazier, L., Carlson, K., Clifton, C., Jr.: Prosodic phrasing is central to language comprehension. Trends Cogn. Sci. 10(6), 244–249 (2006)

    Article  Google Scholar 

  7. Futamata, K., Park, B., Yamamoto, R., Tachibana, K.: Phrase break prediction with bidirectional encoder representations in Japanese text-to-speech synthesis. In: Proceedings of InterSpeech 2021, pp. 3126–3130 (2021)

    Google Scholar 

  8. Gallwitz, F., Niemann, H., Nöth, E., Warnke, V.: Integrated recognition of words and prosodic phrase boundaries. Speech Commun. 36(1), 81–95 (2002)

    Article  Google Scholar 

  9. Kocharov, D., Kachkovskaia, T., Skrelin, P.: Prosodic boundary detection using syntactic and acoustic information. Comput. Speech Lang. 53, 231–241 (2019)

    Article  Google Scholar 

  10. Lehečka, J., Švec, J., Pražák, A., Psutka, J.: Exploring capabilities of monolingual audio transformers using large datasets in automatic speech recognition of Czech. In: To appear at Interspeech 2022 (2022)

    Google Scholar 

  11. Lin, B., Wang, L., Feng, X., Zhang, J.: Joint detection of sentence stress and phrase boundary for prosody. In: Proceedings of InterSpeech 2020, pp. 4392–4396 (2020)

    Google Scholar 

  12. Raffel, C., et al.: Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 21, 1–67 (2020)

    MathSciNet  MATH  Google Scholar 

  13. Read, I., Cox, S.: Stochastic and syntactic techniques for predicting phrase breaks. Comput. Speech Lang. 21(3), 519–542 (2007)

    Article  Google Scholar 

  14. Rosenberg, A.: AuToBI - a tool for automatic ToBI annotation. In: Proceedings of InterSpeech 2010, pp. 146–149 (2010)

    Google Scholar 

  15. Schuppler, B., Ludusan, B.: An analysis of prosodic boundary detection in German and Austrian German read speech. In: Proceedings of Speech Prosody 2020, pp. 990–994 (2020)

    Google Scholar 

  16. Suni, A., Simko, J., Vainio, M.: Boundary detection using continuous wavelet analysis. In: Proceedings of Speech Prosody 2016, pp. 267–271 (2016)

    Google Scholar 

  17. Taylor, P.: Text-to-Speech Synthesis, 1st edn. Cambridge University Press, New York (2009)

    Book  Google Scholar 

  18. Taylor, P., Black, A.W.: Assigning phrase breaks from part-of-speech sequences. Comput. Speech Lang. 12(2), 99–117 (1998)

    Article  Google Scholar 

  19. Tihelka, D., Hanzlíček, Z., Jůzová, M., Vít, J., Matoušek, J., Grůber, M.: Current state of text-to-speech system ARTIC: a decade of research on the field of speech technologies. In: Sojka, P., Horák, A., Kopeček, I., Pala, K. (eds.) TSD 2018. LNCS (LNAI), vol. 11107, pp. 369–378. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00794-2_40

    Chapter  Google Scholar 

  20. Volín, J., Řezáčková, M., Matoušek, J.: Human and transformer-based prosodic phrasing in two speech genres. In: Karpov, A., Potapova, R. (eds.) SPECOM 2021. LNCS (LNAI), vol. 12997, pp. 761–772. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87802-3_68

    Chapter  Google Scholar 

  21. Wolf, T., et al.: Transformers: state-of-the-art natural language processing. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38–45 (2020)

    Google Scholar 

  22. Yang, S.W., et al.: SUPERB: speech processing Universal PERformance benchmark. In: Proceedings InterSpeech 2021, pp. 1194–1198 (2021)

    Google Scholar 

  23. Zhang, Y., et al.: Pushing the limits of semi-supervised learning for automatic speech recognition. arXiv preprint arXiv:2010.10504 (2020)

  24. Zou, Y., et al.: Fine-grained prosody modeling in neural speech synthesis using ToBI representation. In: Proceedings of InterSpeech 2021, pp. 3146–3150 (2021)

    Google Scholar 

Download references

Acknowledgements

This research was supported by the Czech Science Foundation (GA CR), project No. GA21-14758S, and by the grant of the University of West Bohemia, project No. SGS-2022-017. Computational resources were supplied by the project “e-Infrastruktura CZ” (e-INFRA CZ LM2018140) supported by the Ministry of Education, Youth and Sports of the Czech Republic.

Author information

Authors and Affiliations

  1. New Technologies for the Information Society and Department of Cybernetics, Faculty of Applied Sciences, University of West Bohemia, Pilsen, Czech Republic

    Marie Kunešová & Markéta Řezáčková

Authors
  1. Marie Kunešová
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Markéta Řezáčková
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Marie Kunešová .

Editor information

Editors and Affiliations

  1. Faculty of Informatics, Masaryk University, Brno, Czech Republic

    Petr Sojka

  2. Faculty of Informatics, Masaryk University, Brno, Czech Republic

    Aleš Horák

  3. Faculty of Informatics, Masaryk University, Brno, Czech Republic

    Ivan Kopeček

  4. Faculty of Informatics, Masaryk University, Brno, Czech Republic

    Karel Pala

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Kunešová, M., Řezáčková, M. (2022). Detection of Prosodic Boundaries in Speech Using Wav2Vec 2.0. In: Sojka, P., Horák, A., Kopeček, I., Pala, K. (eds) Text, Speech, and Dialogue. TSD 2022. Lecture Notes in Computer Science(), vol 13502. Springer, Cham. https://doi.org/10.1007/978-3-031-16270-1_31

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-16270-1_31

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-16269-5

  • Online ISBN: 978-3-031-16270-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation