Abstract
While emotional intelligence plays a key role in facilitating satisfactory interactions between people, its application is both underexplored and underexploited in human–computer interaction. Self-service technology is increasingly being incorporated by goods and service providers, however user satisfaction is still less than ideal. Studies have been carried out in which an affective embodied agent has been found to reduce frustration in users of interactive computer systems. This paper presents a preliminary study as a part of research aimed at designing and implementing an agent which detects negative emotions in a human user and expresses its own emotional reaction with the aim of improving the user’s mood and therefore their level of satisfaction in the context of a self-service interaction. We describe a study to determine customer facial expressions using facial Action Units (AUs) during interactions with self-service supermarket checkouts. Our preliminary results indicate that AU 23 and AU 24 were displayed with particular frequency.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
ASDA: ASDA VA. http://groceries.asda.com/asda-estore/index.jsp?utmsource=homepage&utmmedium=cta&utmcampaign=startShoppingDeflt#. Cited 14 Feb 2011
Berry, D.C., Butler, L.T., de Rosis, F.: Evaluating a realistic agent in an advice-giving task. Int. J. hum.-comput. Stud. 63(3), 304–327 (2005)
Branco, P., Firth, P., Encarnacao, L.M., Bonato, P.: Faces of emotion in human-computer Interaction. In: CHI’05 Extended Abstracts on Human Factors in Computing Systems (CHI’05), New York, 2005
Buenaposada, J.M., Munoz, E., Baumela, L.: Recognising facial expressions in video sequences. Pattern Anal. Appl. 11(1), 101–116 (2008)
Carrol, J.M., Russell, J.A.: Facial expressions in Hollywood’s portrayal of emotion. J. Pers. Soc. Psychol. 72(1), 164–176 (1997)
Ekman, P., Friesen, W.V.: Facial Action Coding System: A technique for the measurement of facial movement. Consulting Psychologists Press, Palo Alto (1978)
Ekman, P.: Universals and cultural differences in the judgments of facial expressions of emotion. J. Pers. Soc. Psychol. 53(4), 712–717 (1987)
Essa, I.A., Pentland, A.P.: Coding, analysis, interpretation, and recognition of facial expressions. IEEE Trans. Pattern Anal. Mach. Intell. 19(7), 757–763 (1995)
Gunes, H., Piccardi, M.: Fusing face and body gesture for machine recognition of emotions. In: IEEE Workshop on Robot and Human Interactive Communication, ROMAN, 2005
HUMAINE Association: Databases. http://emotion-research.net/wiki/Databases. Cited 28 April 2011
IKEA, IKEA Help Center: http://193.108.42.79/ikea-us/cgi-bin/ikea-us.cgi (2011). Cited 14 Feb 2011
Ioannou, S.V., Raouzaiou, A.T., Tzouvaras, V.A., Mailis, T.P., Karpouzis, K.C., Kollias, S.D.: Emotion recognition through facial expression analysis based on a neurofuzzy network. Neural Netw. 18(4), 423–435 (2005)
Jaksic, N., Branco, P., Stephenson, P., Encarna, L.M.: The effectiveness of social agents in reducing user frustration. In: CHI ’06 Extended Abstracts on Human Factors in Computing Systems. ACM: Montral, Quebec, 2006
Johnson, A.J., Pepperman, K., Quist, L.: The role of emotion in self-service technologies: How can the U-Scan system be improved to positively influence users’ emotions and shopping experiences. http://74.125.155.132/scholar?q=cache:6zCQBXL0EjIJ:scholar.google.com/+%22self+service%22+checkout+anger&hl=en&assdt=2001 (2008). Cited 15 Dec 2010
Kanade, T., Cohn, J.F., Tian, Y.: Comprehensive database for facial expression analysis. In: Fourth IEEE International Conference on Automatic Face and Gesture Recognition, Grenoble, 2000
Klein, J., Moon, Y., Picard, R.W.: This computer responds to user frustration. In: CHI ’99 Extended Abstracts on Human Factors in Computing Systems. ACM: Pittsburgh, 1999
Kotsia, I., Pitas, I.: Facial expression recognition in image sequences using geometric deformation features and support vector machines. IEEE trans. Image Process. 16(1), 172–187 (2007)
Kumano, S., Otsuka, K., Yamato, J.: Pose-invariant facial expression recognition using variable-intensity templates. Int. J. Comput. Vis. 83(2), 178–194 (2009)
Lin, J., Hsieh, P.: The role of technology readiness in customers’ perception and adoption of self-service technologies. Int. J. Serv. ind. manag. 17(5), 497–517 (2006)
Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: Computer Vision and Pattern Recognition Workshop on Human-Communicative Behaviour, 2010
Marks and Spencer: Ask Hanna. http://money.marksandspencer.com/help/overview (2011). Cited 14 Feb 2011
Meuter, M.L., Ostrom, A.L., Roundtree, R.I., Bitner, M.J.: Self-service technologies: understanding customer satisfaction with technology-based service encounters. J. Mark. 64(3), 50–64 (2000)
Michel, P., El Kaliouby, R.: Real time facial expression recognition in video using support vector machines. In: ICMI’03, Vancouver, November 2003. ACM (2003)
Pantic, M.: Affective computing. In: Pagani, M. (ed.) Encyclopedia of Multimedia Technology and Networking, 1, pp. 8–14. Idea Group Reference, Hershey (2005)
Pantic, M., Sebe, N., Cohn, J.F., Huang, T.: Affective multimodal human–computer interaction. In: Proceedings of the 13th Annual ACM International Conference on Multimedia. ACM: Hilton, Singapore, 2005
Picard, R.W.: Affective Computing. MIT Press, Cambridge (1997)
Picard, R.W.: Toward machine emotional intelligence: analysis of affective physiological state. IEEE Trans. Pattern Anal. Mach. Intell. 23(10), 1175–1191 (2001)
Pujari, D.: Self-service with a smile? Self-service technology (SST) encounters among Canadian business-to-business. Int. J. Serv. Ind. Manag. 15(2), 200–219 (2004)
de Rosis, F., Pelachaud, C., Poggi, I., Carofiglio, V., de Carolis, B.: From Greta’s mind to her face: modelling the dynamics of affective states in a conversational embodied agent. Int. J. hum.-Comput. Stud. 59(1–2), 81–118 (2003)
Rozin, P., Lowery, L., Imada, S., Haidit, J.: The CAD triad hypothesis: a mapping between three moral emotions (contempt, anger, disgust) and three moral codes (community, autonomy, divinity). J. Pers. Soc. Psychol. 76(4), 574–586 (1999)
Salovey, P., Mayer, J.D.: Emotional intelligence. Imagination Cogn. Pers. 9, 185–211 (1990)
Satiyan, M., Nagarajan, R.: Recognition of facial expression using Haar-like feature extraction method. In: International Conference on Intelligent and Advanced Systems (ICIAS). Kuala Lumpur, 2010
Schmidt, R.M., Ruth, C.R., Walters, C.M., Robinson, T.B., Shiyanov, P.A., Goodyear, C.D., Funke, M.A., McIntire, L.K.: Facial expressions of emotion and the assessment of performance. Air Force Research Laboratory, 711th Human Performance Wing, Human Effectiveness Directorate, Wright-Patterson Air Force Base, OH: Wright-Patterson Air Force Base, Ohio, 2010
de Silva, L.C., Ng. P.C.: Bimodal emotion recognition. In: Proceedings of Fourth IEEE International Conference on Automatic Face and Gesture Recogntion. Grenoble, 2000
Stone, D.: User Interface Design and Evaluation. Open University, UK (2005)
Tian, Y., Kanade, T., Cohn, J.F.: Recognizing action units for facial expression analysis. IEEE Trans. Pattern Anal. Mach. Intell. 23(2), 97–115 (2001)
Valstar, M., Pantic, M.: Fully automatic facial action unit detection and temporal analysis. In: Conference on Computer Vision and Pattern Recognition Workshop (CVPRW’06). IEEE (2006)
Vinciarelli, A., Pantic, M., Bourlard, H., Pentland, A.: Social signal processing: state-of-the-art and future perspectives of an emerging domain. In: 16th ACM international Conference on Multimedia. Vancouver, British Columbia, Canada 2008
Wang, C., Chen, G., Liu, C., Liu, B.: Design an empathic virtual human to encourage and persuade learners in e-learning systems. In: Proceedings of the First ACM International Workshop on Multimedia Technologies for Distance Learning. ACM: Beijing, (2009)
Yang, P., Liu, Q., Metaxas, D.N.: Boosting coded dynamic features for facial action units and facial expression recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, Minneapolis, 2007
Zhang, P., Li, N.: The importance of affective quality. Commun. ACM 48(9), 105–108 (2005)
Acknowledgments
The researchers would like to thank the supermarket for allowing us to carry out the filming session. We would also like to thank the participant who kindly allowed us to use their image in this paper.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Martin, C.J., Archibald, J., Ball, L., Carson, L. (2013). Towards an Affective Self-Service Agent. In: Kudělka, M., Pokorný, J., Snášel, V., Abraham, A. (eds) Proceedings of the Third International Conference on Intelligent Human Computer Interaction (IHCI 2011), Prague, Czech Republic, August, 2011. Advances in Intelligent Systems and Computing, vol 179. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-31603-6_1
Download citation
DOI: https://doi.org/10.1007/978-3-642-31603-6_1
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-31602-9
Online ISBN: 978-3-642-31603-6
eBook Packages: EngineeringEngineering (R0)