Abstract
Modern virtual agents require knowledge about their environment, the interaction itself, and their interlocutors’ behavior in order to be able to show appropriate nonverbal behavior as well as to adapt dialog policies accordingly. Recent achievements in the area of automatic behavior recognition and understanding can provide information about the interactants’ multimodal nonverbal behavior and subsequently their affective states. In this paper, we introduce a perception markup language (PML) which is a first step towards a standardized representation of perceived nonverbal behaviors. PML follows several design concepts, namely compatibility and synergy, modeling uncertainty, multiple interpretative layers, and extensibility, in order to maximize its usefulness for the research community. We show how we can successfully integrate PML in a fully automated virtual agent system for healthcare applications.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Pocketsphinx: A Free, Real-Time Continuous Speech Recognition System for Hand-Held Devices, vol. 1 (2006)
Fairbanks, L.A., et al.: Nonverbal interaction of patients and therapists during psychiatric interviews. J. Abnorm. Psychol. 91(2), 109–119 (1982)
Hall, J.A.: et al. Nonverbal behavior in clinician-patient interaction. Appl. Prev. Psychol. 4(1), 21–37 (1995)
Heylen, D., Kopp, S., Marsella, S.C., Pelachaud, C., Vilhjálmsson, H.H.: The Next Step towards a Function Markup Language. In: Prendinger, H., Lester, J.C., Ishizuka, M. (eds.) IVA 2008. LNCS (LNAI), vol. 5208, pp. 270–280. Springer, Heidelberg (2008)
Kendon, A. (ed.): Nonverbal Communication, Interaction, and Gesture. Selections from Semiotica Series. Walter de Gruyter (1981)
Kopp, S., Krenn, B., Marsella, S.C., Marshall, A.N., Pelachaud, C., Pirker, H., Thórisson, K.R., Vilhjálmsson, H.H.: Towards a Common Framework for Multimodal Generation: The Behavior Markup Language. In: Gratch, J., Young, M., Aylett, R.S., Ballin, D., Olivier, P. (eds.) IVA 2006. LNCS (LNAI), vol. 4133, pp. 205–217. Springer, Heidelberg (2006)
Lee, J., Marsella, S.: Nonverbal Behavior Generator for Embodied Conversational Agents. In: Gratch, J., Young, M., Aylett, R.S., Ballin, D., Olivier, P. (eds.) IVA 2006. LNCS (LNAI), vol. 4133, pp. 243–255. Springer, Heidelberg (2006)
Ozkan, D., Morency, L.-P.: Modeling wisdom of crowds using latent mixture of discriminative experts. In: Proc. of ACL HLT 2011, pp. 335–340. Association for Computational Linguistics (2011)
Pentland, A.: Honest Signals - How they shape our world. MIT Press (2008)
Perez, J.E., Riggio, R.E.: Nonverbal social skills and psychopathology. Nonverbal Behavior in Clinical Settings, pp. 17–44. Oxford University Press (2003)
Pestonjee, D.M., Pandey, S.C.: A preliminary study of psychological aftereffects of post-traumatic stress disorder (ptsd) caused by earthquake: the ahmedabad experience. Technical Report WP2001-04-01, Indian Institute of Management (2001)
Saragih, J.M., et al.: Face alignment through subspace constrained mean-shifts. In: Proc. of ICCV 2009, pp. 1034–1041. IEEE (2009)
Scherer, S., et al.: A generic framework for the inference of user states in human computer interaction: How patterns of low level communicational cues support complex affective states. JMUI, Special Issue on: Conceptual frameworks for Multimodal Social Signal Processing, 1–25 (2012)
Scherer, S., et al.: Investigating fuzzy-input fuzzy-output support vector machines for robust voice quality classification. In: CSL (2012)
Scherer, S., et al.: Spotting laughter in naturalistic multiparty conversations: a comparison of automatic online and offline approaches using audiovisual data. ACM TiiS: Special Issue on Affective Interaction in Natural Environments 2(1), 4:1–4:31 (2012)
Schröder, M.: The SEMAINE API: A component integration framework for a naturally interacting and emotionally competent Embodied Conversational Agent. PhD thesis, Saarland University (2011)
Suma, E.A., et al.: Faast: The flexible action and articulated skeleton toolkit. In: Virtual Reality, pp. 247–248 (2011)
Thiebaux, M., et al.: Smartbody: behavior realization for embodied conversational agents. In: Proc. of AAMAS 2008, pp. 151–158 (2008)
Traum, D.R., Larsson, S.: The information state approach to dialogue management. In: Kuppevelt, J., Smith, R.W., Ide, N. (eds.) Current and New Directions in Discourse and Dialogue, vol. 22, pp. 325–353. Springer (2003)
Wagner, J., Lingenfelser, F., Bee, N., André, E.: Social signal interpretation (ssi). KI - Kuenstliche Intelligenz 25, 251–256 (2011), doi:10.1007/s13218-011-0115-x
Waxer, P.: Nonverbal cues for depression. Journal of Abnormal Psychology 83(3), 319–322 (1974)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Scherer, S. et al. (2012). Perception Markup Language: Towards a Standardized Representation of Perceived Nonverbal Behaviors. In: Nakano, Y., Neff, M., Paiva, A., Walker, M. (eds) Intelligent Virtual Agents. IVA 2012. Lecture Notes in Computer Science(), vol 7502. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-33197-8_47
Download citation
DOI: https://doi.org/10.1007/978-3-642-33197-8_47
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-33196-1
Online ISBN: 978-3-642-33197-8
eBook Packages: Computer ScienceComputer Science (R0)