Abstract
Affective speech analysis is an ongoing topic of research. A relatively new problem in this field is the analysis of affective vocal bursts, which are non-verbal vocalisations such as laughs or sighs. The current state of the art in the analysis of affective vocal bursts is predominantly based on wav2vec2 or HuBERT features. In this paper, we investigate the application of the wav2vec2 successor data2vec and the extension wav2vec2phoneme in combination with a multi-task learning pipeline to tackle different analysis problems at once, e.g., type of burst, country of origin, and conveyed emotion. Finally, we present an ablation study to validate our approach. We discovered that data2vec appears to be the best option if time and lightweightness are critical factors. On the other hand, wav2vec2phoneme is the most appropriate choice if overall performance is the primary criterion.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Anuchitanukul, A., Specia, L.: Burst2vec: an adversarial multi-task approach for predicting emotion, age, and origin from vocal bursts. arXiv preprint arXiv:2206.12469 (2022)
Atmaja, B.T., Sasou, A.: Predicting affective vocal bursts with finetuned wav2vec 2.0. arXiv preprint arXiv:2209.13146 (2022)
Atmaja, B.T., Sasou, A., et al.: Jointly predicting emotion, age, and country using pre-trained acoustic embedding. arXiv preprint arXiv:2207.10333 (2022)
Baevski, A., Hsu, W.N., Xu, Q., Babu, A., Gu, J., Auli, M.: Data2vec: a general framework for self-supervised learning in speech, vision and language. arXiv preprint arXiv:2202.03555 (2022)
Baevski, A., Zhou, Y., Mohamed, A., Auli, M.: wav2vec 2.0: a framework for self-supervised learning of speech representations. In: Advances in Neural Information Processing Systems vol. 33, pp. 12449–12460 (2020)
Baird, A., Tzirakis, P., Batliner, A., Schuller, B., Keltner, D., Cowen, A.: The ACII 2022 affective vocal bursts workshop and competition: Understanding a critically understudied modality of emotional expression. arXiv preprint arXiv:2207.03572v1 (2022). https://doi.org/10.48550/arXiv.2207.03572
Baird, A., et al.: The ICML 2022 expressive vocalizations workshop and competition: Recognizing, generating, and personalizing vocal bursts. arXiv preprint arXiv:2205.01780v3 (2022). https://doi.org/10.48550/ARXIV.2205.01780
Cordaro, D.T., Keltner, D., Tshering, S., Wangchuk, D., Flynn, L.M.: The voice conveys emotion in ten globalized cultures and one remote village in Bhutan. Emotion 16(1), 117 (2016)
Cowen, A., et al.: The Hume vocal burst competition dataset (H-VB) | raw data [exvo: updated 02.28.22] [data set]. Zenodo (2022). https://doi.org/10.5281/zenodo.6308780
Cowen, A.S., Elfenbein, H.A., Laukka, P., Keltner, D.: Mapping 24 emotions conveyed by brief human vocalization. Am. Psychol. 74(6), 698 (2019)
Hallmen, T., Mertes, S., Schiller, D., André, E.: An efficient multitask learning architecture for affective vocal burst analysis (2022)
Hendrycks, D., Gimpel, K.: Gaussian error linear units (GELUs). arXiv preprint arXiv:1606.08415 (2016)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Hsu, W.N., Bolte, B., Tsai, Y.H.H., Lakhotia, K., Salakhutdinov, R., Mohamed, A.: HuBERT: self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Trans. Audio Speech Lang. Proc. 29, 3451–3460 (2021)
Karas, V., Triantafyllopoulos, A., Song, M., Schuller, B.W.: Self-supervised attention networks and uncertainty loss weighting for multi-task emotion recognition on vocal bursts. arXiv preprint arXiv:2209.07384 (2022)
Kendall, A., Gal, Y., Cipolla, R.: Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7482–7491 (2018)
Kwon, J., Kim, J., Park, H., Choi, I.K.: ASAM: adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks. In: International Conference on Machine Learning, pp. 5905–5914. PMLR (2021)
Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)
Nguyen, D.K., Pant, S., Ho, N.H., Lee, G.S., Kim, S.H., Yang, H.J.: Fine-tuning wav2vec for vocal-burst emotion recognition. arXiv preprint arXiv:2210.00263 (2022)
Panayotov, V., Chen, G., Povey, D., Khudanpur, S.: LibriSpeech: an ASR corpus based on public domain audio books. In: 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5206–5210. IEEE (2015)
Phutela, D.: The importance of non-verbal communication. IUP J. Soft Skills 9(4), 43 (2015)
Purohit, T., Mahmoud, I.B., Vlasenko, B., Doss, M.M.: Comparing supervised and self-supervised embedding for exvo multi-task learning track. arXiv preprint arXiv:2206.11968 (2022)
Scherer, K.R.: Expression of emotion in voice and music. J. Voice 9(3), 235–248 (1995)
Schröder, M.: Experimental study of affect bursts. Speech Commun. 40(1–2), 99–116 (2003)
Sharma, R., Vuong, T., Lindsey, M., Dhamyal, H., Singh, R., Raj, B.: Self-supervision and learnable STRFs for age, emotion, and country prediction. arXiv preprint arXiv:2206.12568 (2022)
Syed, M.S.S., Syed, Z.S., Syed, A.: Classification of vocal bursts for ACII 2022 A-VB-Type competition using convolutional network networks and deep acoustic embeddings. arXiv preprint arXiv:2209.14842 (2022)
Trinh, D.L., Vo, M.C., Kim, S.H., Yang, H.J., Lee, G.S.: Self-relation attention and temporal awareness for emotion recognition via vocal burst. Sensors 23(1), 200 (2022)
Xu, Q., Baevski, A., Auli, M.: Simple and effective zero-shot cross-lingual phoneme recognition. arXiv preprint arXiv:2109.11680 (2021)
Acknowledgements
This work was partially funded by the KodiLL project (FBM2020, Stiftung Innovation in der Hochschullehre), project TherapAI (DFG, German Research Foundation, grant number 493169211) and project Panorama (DFG, German Research Foundation, grant number 442607480).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Hallmen, T., Mertes, S., Schiller, D., Lingenfelser, F., André, E. (2023). Phoneme-Based Multi-task Assessment of Affective Vocal Bursts. In: Conte, D., Fred, A., Gusikhin, O., Sansone, C. (eds) Deep Learning Theory and Applications. DeLTA 2023. Communications in Computer and Information Science, vol 1875. Springer, Cham. https://doi.org/10.1007/978-3-031-39059-3_14
Download citation
DOI: https://doi.org/10.1007/978-3-031-39059-3_14
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-39058-6
Online ISBN: 978-3-031-39059-3
eBook Packages: Computer ScienceComputer Science (R0)