A systematic literature review about technologies for self-reporting emotional information

  • Carolina Fuentes
  • Valeria Herskovic
  • Iyubanit Rodríguez
  • Carmen Gerea
  • Maíra Marques
  • Pedro O. Rossel
Original Research

Abstract

Emotional information is complex to manage by humans and computers alike, so it is difficult for users to express emotional information through technology. Two main approaches are used to gather this type of information: objective (e.g. through sensors or facial recognition) and subjective (reports by users themselves). Subjective methods are less intrusive and may be more accurate, although users may fail to report their emotions or not be entirely truthful about them. The goal of this study is to identify trends in the area of interfaces for the self-report of human emotions, under-served populations of users, and avenues of future research. A systematic literature review was conducted on six search engines, resulting in a set of 863 papers, which were filtered in a systematic way until we established a corpus of 40 papers. We studied the technologies used for emotional self-report as well as the issues regarding these technologies, such as privacy, interaction mechanisms, and how they are evaluated.

Keywords

Systematic literature review Emotions Interfaces Self-report 

References

  1. Angelini L, Caon M, Lalanne D et al (2015) Towards an anthropomorphic lamp for affective interaction. In: proceedings of the ninth international conference on tangible, embedded, and embodied interaction. ACM, New York, pp 661–666Google Scholar
  2. Bannon L (2011) Reimagining HCI: toward a more human-centered perspective. Interactions 18:50–57. doi:10.1145/1978822.1978833 CrossRefGoogle Scholar
  3. Bardzell S, Bardzell J, Pace T (2009) Understanding affective interaction: Emotion, engagement, and internet videos. In: 2009 3rd international conference on affective computing and intelligent interaction and workshops (ACII 2009), pp 1–8. doi:10.1109/ACII.2009.5349551
  4. Barrett LF (2004) Feelings or words? understanding the content in self-report ratings of experienced emotion. J Pers Soc Psychol 87:266–281CrossRefGoogle Scholar
  5. Bialoskorski LSS, Westerink JHDM, Broek EL (2009) Mood swings: an affective interactive art system. In: Nijholt A, Reidsma D, Hondorp H (eds) Intelligent technologies for interactive entertainment. Springer, Berlin, pp 181–186CrossRefGoogle Scholar
  6. Broekens J, Brinkman WP (2009) AffectButton: towards a standard for dynamic affective user feedback. In: 2009 3rd international conference on affective computing and intelligent interaction and workshops, pp 1–8Google Scholar
  7. Broekens J, Brinkman W-P (2013) AffectButton: a method for reliable and valid affective self-report. Int J Hum Comput Stud 71:641–667. doi:10.1016/j.ijhcs.2013.02.003 CrossRefGoogle Scholar
  8. Broekens J, Pronker A, Neuteboom M (2010) Real time labeling of affect in music using the Affectbutton. In: Proceedings of the 3rd international workshop on affective interaction in natural environments. ACM, New York, pp 21–26Google Scholar
  9. Caon M, Khaled OA, Mugellini E et al (2013) Ubiquitous interaction for computer mediated communication of emotions. In: 2013 humaine association conference on affective computing and intelligent interaction (ACII). IEEE, pp 717–718Google Scholar
  10. Chambel EOM (2013) Accessing movies based on emotional impact. Multimed Syst 19:559–576. doi:10.1007/s00530-013-0303-7 CrossRefGoogle Scholar
  11. Chen L, Chen G-C, Xu C-Z et al (2008) EmoPlayer: a media player for video clips with affective annotations. Interact Comput 20:17–28. doi:10.1016/j.intcom.2007.06.003 CrossRefGoogle Scholar
  12. Chi T-S, Yeh L-Y, Hsu C-C (2011) Robust emotion recognition by spectro-temporal modulation statistic features. J Ambient Intell Humaniz Comput 3:47–60. doi:10.1007/s12652-011-0088-5 CrossRefGoogle Scholar
  13. Choi S, Yamasaki T, Aizawa K (2015) An interactive system based on yes–no questions for affective image retrieval. In: Proceedings of the 1st international workshop on affect and sentiment in multimedia. ACM, New York, pp 45–50Google Scholar
  14. Conner TS, Reid KA (2012) Effects of intensive mobile happiness reporting in daily life. Soc Psychol Pers Sci 3:315–323. doi:10.1177/1948550611419677 CrossRefGoogle Scholar
  15. Desmet P (2005) Measuring emotion: development and application of an instrument to measure emotional responses to products. In: Blythe M, Overbeeke K, Monk A, Wright P (eds) Funology. Springer, The Netherlands, pp 111–123Google Scholar
  16. Diener H, Oertel K (2006) Experimental approach to affective interaction in games. In: Proceedings of the first international conference on technologies for e-learning and digital entertainment. Springer, Berlin, pp 507–518Google Scholar
  17. Doryab A, Frost M, Faurholt-Jepsen M et al (2015) Impact factor analysis: combining prediction with parameter ranking to reveal the impact of behavior on health outcome. Pers Ubiquit Comput 19:355–365CrossRefGoogle Scholar
  18. Ekman P, Friesen WV (1986) A new pan-cultural facial expression of emotion. Motiv Emot 10:159–168CrossRefGoogle Scholar
  19. Fogg BJ (1998) Persuasive computers: perspectives and research directions. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp 225–232Google Scholar
  20. Frost M, Doryab A, Faurholt-Jepsen M, et al (2013) Supporting disease insight through data analysis: refinements of the monarca self-assessment system. In: Proceedings of the 2013 ACM international joint conference on pervasive and ubiquitous computing. ACM, New York, pp 133–142Google Scholar
  21. Fuentes C, Gerea C, Herskovic V et al (2015a) User interfaces for self-reporting emotions: a systematic literature review. In: Ubiquitous computing and ambient intelligence. Sensing, processing, and using environmental information—9th international conference, UCAmI 2015. Springer, pp 321–333Google Scholar
  22. Fuentes C, Rodríguez I, Herskovic V (2015b) EmoBall: a study on a tangible interface to self-report emotional information considering digital competences. In: Volume 9456 of the series lecture notes in computer science, pp 189–200Google Scholar
  23. Gallacher S, O’Connor J, Bird J et al (2015) Mood squeezer: lightening up the workplace through playful and lightweight interactions. In: Proceedings of the 18th ACM conference on computer supported cooperative work and social computing. ACM, pp 891–902Google Scholar
  24. Gross T (2009) Towards a new human-centred computing methodology for cooperative ambient intelligence. J Ambient Intell Humaniz Comput 1:31–42. doi:10.1007/s12652-009-0004-4 CrossRefGoogle Scholar
  25. Hastings J, Brass A, Caine C et al (2014) Evaluating the emotion ontology through use in the self-reporting of emotional responses at an academic conference. J Biomed Semant 5:1–17. doi:10.1186/2041-1480-5-38 CrossRefGoogle Scholar
  26. Isbister K, Höök K, Sharp M, Laaksolahti J (2006) The sensual evaluation instrument: developing an affective evaluation tool. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp 1163–1172Google Scholar
  27. Isbister K, Höök K, Laaksolahti J, Sharp M (2007) The sensual evaluation instrument: developing a trans-cultural self-report measure of affect. Int J Hum Comput Stud 65:315–328CrossRefGoogle Scholar
  28. Ishii H (2008) The tangible user interface and its evolution. Commun ACM 51:32–36CrossRefGoogle Scholar
  29. Jacko JA (2012) Human computer interaction handbook: fundamentals, evolving technologies, and emerging applications. CRC Press, Boca RatonCrossRefGoogle Scholar
  30. Jacob RJK, Girouard A, Hirshfield LM et al (2008) Reality-based interaction: a framework for post-WIMP interfaces. In: Proceedings of the SIGCHI conference on human factors in computing systems (CHI’08). ACM, pp 201–210Google Scholar
  31. Jain J, Lund A, Wixon D (2011) The future of natural user interfaces. In: Extended abstracts on human factors in computing systems (CHI EA’11), pp 211–214Google Scholar
  32. Killingsworth MA, Gilbert DT (2010) A wandering mind is an unhappy mind. Science 330:932. doi:10.1126/science.1192439 CrossRefGoogle Scholar
  33. Kitchenham B, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering. Technical Report EBSE 2007-01, Version 2.3. Keele University, Keele, UK and Durham University, Durham, UKGoogle Scholar
  34. Laaksolahti J, Isbister K, Höök K (2009) Using the sensual evaluation instrument. Digit Creat 20:165–175. doi:10.1080/14626260903083603 CrossRefGoogle Scholar
  35. Lahey B, Girouard A, Burleson W, Vertegaal R (2011) PaperPhone: understanding the use of bend gestures in mobile devices with flexible electronic paper displays. In: Proceedings of the 2011 annual conference on human factors in computing systems, pp 1303–1312Google Scholar
  36. Lang PJ (1980) Behavioral treatment and bio-behavioral assessment: computer applications. In: Sidowski JB, Johnson JH, Williams TA (eds) Technology in mental health care delivery systems. Ablex, Norwood, NJ, USA, pp 119–137Google Scholar
  37. Laurans G, Desmet P, Hekkert P (2009) The emotion slider: a self-report device for the continuous measurement of emotion. In: Proceedings of the 3rd international conference on affective computing and intelligent interaction and workshops (ACII 2009), pp 1–6Google Scholar
  38. Lee J-H, Park J-Y, Nam T-J (2007) Emotional interaction through physical movement. In: Human–computer interaction. HCI intelligent multimodal interaction environments. Springer, pp 401–410Google Scholar
  39. Lin C-L, Gau P-S, Lai K-J et al (2009) Emotion caster: tangible emotion sharing device and multimedia display platform for intuitive interactions. In: Proceedings of the 13th international symposium on consumer electronics (ISCE’09). IEEE, pp 988–989Google Scholar
  40. Lopatovska I, Arapakis I (2011) Theories, methods and current research on emotions in library and information science, information retrieval and human-computer interaction. Inf Process Manag 47:575–592CrossRefGoogle Scholar
  41. Lottridge D, Chignell M (2009a) Emotional majority agreement a psychometric property of affective self-report instruments. In: Science and technology for humanity (TIC-STH), 2009 IEEE Toronto international conference, pp 795–800Google Scholar
  42. Lottridge D, Chignell M (2009b) Emotrace: tracing emotions through human-system interaction. In: Proceedings of the human factors and ergonomics society annual meeting, vol, 53, pp 1541–1545. doi:10.1177/154193120905301916
  43. Lyytinen K, Yoo Y (2002) Ubiquitous computing. Commun ACM 45:63–96CrossRefGoogle Scholar
  44. Matthews M, Doherty G, Sharry J, Fitzpatrick C (2008) Mobile phone mood charting for adolescents. Br J Guid Couns 36:113–129. doi:10.1080/03069880801926400 CrossRefGoogle Scholar
  45. Mody RN, Willis KS, Kerstein R (2009) WiMo: location-based emotion tagging. In: Proceedings of the 8th international conference on mobile and ubiquitous multimedia. ACM, New York, pp 14:1–14:4Google Scholar
  46. Mora S, Rivera-Pelayo V, Müller L (2011) Supporting mood awareness in collaborative settings. In: 2011 7th International Conference on collaborative computing: networking, applications and worksharing (CollaborateCom), pp 268–277Google Scholar
  47. Morris M, Kathawala Q, Leen T et al (2010) Mobile therapy: case study evaluations of a cell phone application for emotional self-awareness. J Med Internet Res 12:e10CrossRefGoogle Scholar
  48. Nagel F, Kopiez R, Grewe O, Altenmüller E (2007) EMuJoy: software for continuous measurement of perceived emotions in music. Behav Res Methods 39:283–290CrossRefGoogle Scholar
  49. Negru S (2010) A conceptual architecture of an arduino-based social-emotional interactive system. In: Proceedings of the proceedings of the 2010 IEEE 6th international conference on intelligent computer communication and processing. IEEE Computer Society, Washington, pp 93–98Google Scholar
  50. Neyem A, Aracena C, Collazos CA, Alarcón R (2007) Designing emotional awareness devices: what one sees is what one feels. Ingeniare Revista Chilena de Ingeniería 15:227–235CrossRefGoogle Scholar
  51. Niforatos E, Karapanos E (2015) EmoSnaps: a mobile application for emotion recall from facial expressions. Pers Ubiquit Comput 19:425–444CrossRefGoogle Scholar
  52. Papastavrou E, Charalambous A, Tsangari H (2016) Exploring the other side of cancer care: the informal caregiver. Eur J Oncol Nurs 13:128–136. doi:10.1016/j.ejon.2009.02.003 CrossRefGoogle Scholar
  53. Pfister H-R, Wollstädter S, Peter C (2011) Affective responses to system messages in human–computer-interaction: effects of modality and message type. Interact Comput 23:372–383CrossRefGoogle Scholar
  54. Picard RW (2003) Affective computing: challenges. Int J Hum Comput Stud 59:55–64CrossRefGoogle Scholar
  55. Picard RW, Picard R (1997) Affective computing. MIT Press, CambridgeCrossRefGoogle Scholar
  56. Plutchik R (1980) Emotion: a psychoevolutionary synthesis. Harpercollins College Division, New YorkGoogle Scholar
  57. Polzin TS, Waibel A (2000) Emotion-sensitive human–computer interfaces. In: ISCA tutorial and research workshop (ITRW) on speech and emotionGoogle Scholar
  58. Read R, Belpaeme T (2013) Using the AffectButton to measure affect in child and adult–robot interaction. In: 2013 8th ACM/IEEE international conference on human–robot interaction (HRI). IEEE, pp 211–212Google Scholar
  59. Reid SC, Kauer SD, Dudgeon P et al (2008) A mobile phone program to track young people’s experiences of mood, stress and coping. Soc Psychiatry Psychiatr Epidemiol 44:501–507. doi:10.1007/s00127-008-0455-5 CrossRefGoogle Scholar
  60. Russell JA (1980) A circumplex model of affect. J Pers Soc Psychol 39:1161–1178CrossRefGoogle Scholar
  61. Russell JA, Mehrabian A (1977) Evidence for a three-factor theory of emotions. J Res Pers 11:273–294CrossRefGoogle Scholar
  62. Sánchez JA, Kirschning I, Palacio JC, Ostróvskaya Y (2005) Towards mood-oriented interfaces for synchronous interaction. In: Proceedings of the 2005 Latin American conference on human–computer interaction (CLIHC’05). ACM, pp 1–7Google Scholar
  63. Sánchez JA, Hernández NP, Penagos JC, Ostróvskaya Y (2006) Conveying mood and emotion in instant messaging by using a two-dimensional model for affective states. In: Proceedings of VII Brazilian symposium on human factors in computing systems. ACM, New York, pp 66–72Google Scholar
  64. Scherer KR (2005) What are emotions? And how can they be measured? Soc Sci Inform 44:695–729CrossRefGoogle Scholar
  65. Schubert E, Ferguson S, Farrar N et al (2013) The six emotion-face clock as a tool for continuously rating discrete emotional responses to music. In: From sounds to music and emotions. Springer, pp 1–18Google Scholar
  66. Sebe N (2010) Human-centered computing. In: Nakashima H, Aghajan H, Augusto JC (eds) Handbook of ambient intelligence and smart environments. Springer US, Boston, pp 349–370CrossRefGoogle Scholar
  67. Stovold E, Beecher D, Foxlee R, Noel-Storr A (2014) Study flow diagrams in Cochrane systematic review updates: an adapted PRISMA flow diagram. Syst Rev 3:54. doi:10.1186/2046-4053-3-54 CrossRefGoogle Scholar
  68. Torning K, Oinas-Kukkonen H (2009) Persuasive system design: state of the art and future directions. In: Proceedings of the 4th international conference on persuasive technology. ACM, New York, pp 30:1–30:8Google Scholar
  69. Watson D, Tellegen A (1985) Toward a consensual structure of mood. Psychol Bull 98:219CrossRefGoogle Scholar
  70. Weiser M (1995) The computer for the 21st century: specialized elements of hardware and software, connected by wires, radio waves and infrared, will be so ubiquitous that no one will notice their presence. In: Baecker RM, Grudin J, Buxton WAS, Greenberg S (eds) Readings in human-computer interaction, 2nd edn. Morgan Kaufmann Publishers Inc., San Francisco, pp 933–940Google Scholar
  71. Wigdor D, Wixon D (2011) Brave NUI world: designing natural user interfaces for touch and gesture. Morgan Kaufmann, San FranciscoGoogle Scholar
  72. Yu S-H, Wang L-S, Chu H-H et al (2011) A mobile mediation tool for improving interaction between depressed individuals and caregivers. Pers Ubiquit Comput 15:695–706CrossRefGoogle Scholar
  73. Zhao Y, Wang X, Goubran M et al (2012) Human emotion and cognition recognition from body language of the head using soft computing techniques. J Ambient Intell Humaniz Comput 4:121–140. doi:10.1007/s12652-012-0107-1 CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2016

Authors and Affiliations

  1. 1.Department of Computer SciencePontificia Universidad Católica de ChileSantiagoChile
  2. 2.Department of Computer ScienceUniversidad de ChileSantiagoChile
  3. 3.Department of Computer ScienceUniversidad Católica de la Santísima ConcepciónConcepciónChile

Personalised recommendations