Multimedia Tools and Applications

, Volume 76, Issue 4, pp 5051–5071 | Cite as

Stress in interactive applications: analysis of the valence-arousal space based on physiological signals and self-reported data

  • Alexandros Liapis
  • Christos Katsanos
  • Dimitris G. Sotiropoulos
  • Nikos Karousos
  • Michalis Xenos


Measuring users’ emotional reaction to interactive multimedia and hypermedia is important. One particularly popular self-reported method for emotion assessment is the Valence-Arousal (VA) Scale: a 9 × 9 affective grid. This paper aims to identify specific stress region(s) in the VA space by combining self-reported ratings (pairs of VA) and physiological signals (skin conductance). To this end, 31 healthy volunteers participated in an experiment by performing five stressful interaction tasks while their skin conductance was monitored. The selected interaction tasks were most frequently listed as stressful by a separate group of 15 interviewees. After each task, participants expressed their perceived emotional experience using the VA rating space. Our findings show which regions in the VA rating space may reliably indicate self-reported stress that is in alignment with one’s measured skin conductance while using interactive applications. One additional important contribution of this work is the proposed approach for the empirical identification of affect regions in the VA space based on physiological signals.


Human computer interaction Emotional experience evaluation Interactive multimedia environments Galvanic skin response Affect grid Valence Arousal 


  1. 1.
    Anderson NB (1998) Levels of analysis in health science. A framework for integrating sociobehavioral and biomedical research. Ann N Y Acad Sci 840:563–576CrossRefGoogle Scholar
  2. 2.
    Barrett LF (1998) Discrete emotions or dimensions? The role of valence focus and arousal focus. Cogn Emot 12:579–599. doi:10.1080/026999398379574 CrossRefGoogle Scholar
  3. 3.
    Baum A (1990) Stress, intrusive imagery, and chronic distress. Health Psychol Off J Div Health Psychol Am Psychol Assoc 9:653–675Google Scholar
  4. 4.
    Benedek M, Kaernbach C (2010) Decomposition of skin conductance data by means of nonnegative deconvolution. Psychophysiology 47:647–658. doi:10.1111/j.1469-8986.2009.00972.x Google Scholar
  5. 5.
    Benedek M, Kaernbach C (2010) A continuous measure of phasic electrodermal activity. J Neurosci Methods 190:80–91. doi:10.1016/j.jneumeth.2010.04.028 CrossRefGoogle Scholar
  6. 6.
    Boucsein W (1992) Electrodermal activity. Plenum University Press, New YorkCrossRefGoogle Scholar
  7. 7.
    Brantley PJ, Waggoner CD, Jones GN, Rappaport NB (1987) A daily stress inventory: development, reliability, and validity. J Behav Med 10:61–74CrossRefGoogle Scholar
  8. 8.
    Cacioppo JT, Tassinary LG (1990) Inferring psychological significance from physiological signals. Am Psychol 45:16–28CrossRefGoogle Scholar
  9. 9.
    Calhoun BH, Lach J, Stankovic J et al (2012) Body sensor networks: a holistic approach from silicon to users. Proc IEEE 100:91–106. doi:10.1109/JPROC.2011.2161240 CrossRefGoogle Scholar
  10. 10.
    Campbell JD, Chew B, Scratchley LS (1991) Cognitive and emotional reactions to daily events: the effects of self-esteem and self-complexity. J Pers 59:473–505CrossRefGoogle Scholar
  11. 11.
    Chanel G, Rebetez C, Bétrancourt M, Pun T (2011) Emotion assessment from physiological signals for adaptation of game difficulty. IEEE Trans Syst Man Cybern Part Syst Hum 41:1052–1063. doi:10.1109/TSMCA.2011.2116000 CrossRefGoogle Scholar
  12. 12.
    Chauncey A, Azevedo R (2010) Emotions and motivation on performance during multimedia learning: how do i feel and why do i care? In: Aleven V, Kay J, Mostow J (eds) Intell. Tutoring Syst. Springer, Berlin, pp 369–378CrossRefGoogle Scholar
  13. 13.
    Chung S, Cheon J, Lee K-W (2015) Emotion and multimedia learning: an investigation of the effects of valence and arousal on different modalities in an instructional animation. Instr Sci 43:545–559. doi:10.1007/s11251-015-9352-y CrossRefGoogle Scholar
  14. 14.
    de Santos Sierra A, Avila CS, Guerra Casanova J, et al. (2010) Two stress detection schemes based on physiological signals for real-time applications. In: 2010 Sixth Int. Conf. Intell. Inf. Hiding Multimed. Signal Process. IIH-MSP. pp 364–367Google Scholar
  15. 15.
    Deaver CM, Miltenberger RG, Smyth J et al (2003) An evaluation of affect and binge eating. Behav Modif 27:578–599CrossRefGoogle Scholar
  16. 16.
    Diamond DM, Campbell AM, Park CR et al (2007) The temporal dynamics model of emotional memory processing: a synthesis on the neurobiological basis of stress-induced amnesia, flashbulb and traumatic memories, and the Yerkes-Dodson law. Neural Plast 2007:60803. doi:10.1155/2007/60803 CrossRefGoogle Scholar
  17. 17.
    Drachen A, Nacke LE, Yannakakis G, Pedersen AL (2010) Correlation between heart rate, electrodermal activity and player experience in first-person shooter games. Proc. 5th ACM SIGGRAPH Symp. Video Games. ACM, New York, pp 49–54Google Scholar
  18. 18.
    Eich E, Macaulay D, Ryan L (1994) Mood dependent memory for events of the personal past. J Exp Psychol Gen 123:201–215CrossRefGoogle Scholar
  19. 19.
    Endsley MR (1988) Situation awareness global assessment technique (SAGAT). In: Aerosp. Electron. Conf. 1988 NAECON 1988 Proc. IEEE 1988 Natl. pp 789–795 vol 3Google Scholar
  20. 20.
    Ganglbauer E, Schrammel J, Tscheligi M (2009) Possibilities of psychophysiological methods for measuring emotional aspects in mobile contexts. Proc. Mob. HCIGoogle Scholar
  21. 21.
    Hart SG, Staveland LE (1988) Development of NASA-TLX (Task Load Index): results of empirical and theoretical research. In: Meshkati PAH and N (ed) Adv. Psychol. North-Holland, pp 139–183Google Scholar
  22. 22.
    Healey JA, Picard RW (2005) Detecting stress during real-world driving tasks using physiological sensors. IEEE Trans Intell Transp Syst 6:156–166. doi:10.1109/TITS.2005.848368 CrossRefGoogle Scholar
  23. 23.
    Hernandez J, Morris RR, Picard RW (2011) Call center stress recognition with person-specific models. In: D’Mello S, Graesser A, Schuller B, Martin J-C (eds) Affect. Comput. Intell. Interact. Springer, Berlin, pp 125–134CrossRefGoogle Scholar
  24. 24.
    Hernandez J, Paredes P, Roseway A, Czerwinski M (2014) Under pressure: sensing stress of computer users. Proc. SIGCHI Conf. Hum. Factors Comput. Syst. ACM, New York, pp 51–60Google Scholar
  25. 25.
    James W (1994) The physical basis of emotion. Psychol Rev 101:205–210. doi:10.1037/0033-295X.101.2.205 CrossRefGoogle Scholar
  26. 26.
    Jensen AR, Rohwer WD (1966) The Stroop color-word test: a review. Acta Psychol (Amst) 25:36–93CrossRefGoogle Scholar
  27. 27.
    Katsanos C, Tselios N, Goncalves J et al (2014) Multipurpose public displays: can automated grouping of applications and services enhance user experience? Int J Hum-Comput Interact 30:237–249. doi:10.1080/10447318.2013.849547 CrossRefGoogle Scholar
  28. 28.
    Killgore WD (1998) The Affect Grid: a moderately valid, nonspecific measure of pleasure and arousal. Psychol Rep 83:639–642. doi:10.2466/pr0.1998.83.2.639 CrossRefGoogle Scholar
  29. 29.
    Kivikangas JM, Chanel G, Cowley B et al (2011) A review of the use of psychophysiological methods in game research. J Gaming Virtual Worlds 3:181–199. doi:10.1386/jgvw.3.3.181_1 CrossRefGoogle Scholar
  30. 30.
    Koelstra S, Muhl C, Soleymani M et al (2012) DEAP: a database for emotion analysis; using physiological signals. IEEE Trans Affect Comput 3:18–31. doi:10.1109/T-AFFC.2011.15 CrossRefGoogle Scholar
  31. 31.
    Kohavi R (1995) A study of cross-validation and bootstrap for accuracy estimation and model selection. In: Proc. 14th Int. Jt. Conf. Artif. Intell. - Vol. 2. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, pp 1137–1143Google Scholar
  32. 32.
    Law EL-C, Roto V, Hassenzahl M et al (2009) Understanding, scoping and defining user experience: a survey approach. Proc. SIGCHI Conf. Hum. Factors Comput. Syst. ACM, New York, pp 719–728Google Scholar
  33. 33.
    Lazar DJ, Feng DJH, Hochheiser DH (2010) Research methods in human-computer interaction. John Wiley & SonsGoogle Scholar
  34. 34.
    Lee M-F, Chen G-S, Hung JC, et al. (2014) Data mining in emotion color with affective computing. Multimed Tools Appl 1–14. doi: 10.1007/s11042-014-2231-8
  35. 35.
    Liapis A, Katsanos C, Sotiropoulos D, et al (2015) Recognizing emotions in human computer interaction: studying stress using skin conductance. In: Abascal J, Barbosa S, Fetter M, et al (eds) Hum.-Comput. Interact. – INTERACT 2015. Springer International Publishing, pp 255–262Google Scholar
  36. 36.
    Liapis A, Katsanos C, Sotiropoulos D et al (2015) Subjective assessment of stress in HCI: a study of the valence-arousal scale using skin conductance. Proc. 11th Biannu. Conf. Ital. SIGCHI Chapter. ACM, New York, pp 174–177Google Scholar
  37. 37.
    Lin T, Imamiya A, Mao X (2008) Using multiple data sources to get closer insights into user cost and task performance. Interact Comput 20:364–374. doi:10.1016/j.intcom.2007.12.002 CrossRefGoogle Scholar
  38. 38.
    Lin T, Omata M, Hu W, Imamiya A (2005) Do physiological data relate to traditional usability indexes? In: Proc. 17th Aust. Conf. Comput.-Hum. Interact. Citiz. Online Consid. Today future. Computer-Human Interaction Special Interest Group (CHISIG) of Australia, Narrabundah, Australia, Australia, pp 1–10Google Scholar
  39. 39.
    Lopatovska I, Arapakis I (2011) Theories, methods and current research on emotions in library and information science, information retrieval and human–computer interaction. Inf Process Manag 47:575–592. doi:10.1016/j.ipm.2010.09.001 CrossRefGoogle Scholar
  40. 40.
    Lunn D, Harper S (2010) Using galvanic skin response measures to identify areas of frustration for older Web 2.0 users. In: Proc. 2010 Int. Cross Discip. Conf. Web Access. W4A. ACM, New York, NY, USA, p 34:1–34:10Google Scholar
  41. 41.
    Lv H-R, Lin Z-L, Yin W-J, Dong J (2008) Emotion recognition based on pressure sensor keyboards. In: 2008 I.E. Int Conf Multimed Expo. pp 1089–1092Google Scholar
  42. 42.
    Mahlke S, Minge M (2008) Consideration of multiple components of emotions in human-technology interaction. In: Peter C, Beale R (eds) Affect Emot. Hum.-Comput. Interact. Springer, Berlin, pp 51–62CrossRefGoogle Scholar
  43. 43.
    Mandryk RL, Atkins MS (2007) A fuzzy physiological approach for continuously modeling emotion during interaction with play technologies. Int J Hum-Comput Stud 65:329–347. doi:10.1016/j.ijhcs.2006.11.011 CrossRefGoogle Scholar
  44. 44.
    Mandryk RL, Atkins MS, Inkpen KM (2006) A continuous and objective evaluation of emotional experience with interactive play environments. Proc. SIGCHI Conf. Hum. Factors Comput. Syst. ACM, New York, pp 1027–1036Google Scholar
  45. 45.
    Mead KML, Ball LJ (2007) Music tonality and context-dependent recall: the influence of key change and mood mediation. Eur J Cogn Psychol 19:59–79. doi:10.1080/09541440600591999 CrossRefGoogle Scholar
  46. 46.
    Peter C, Herbon A (2006) Emotion representation and physiology assignments in digital systems. Interact Comput 18:139–170. doi:10.1016/j.intcom.2005.10.006 CrossRefGoogle Scholar
  47. 47.
    Picard RW (2000) Affective computing, 1st edition. The MIT PressGoogle Scholar
  48. 48.
    Ritz T, Thöns M, Fahrenkrug S, Dahme B (2005) Airways, respiration, and respiratory sinus arrhythmia during picture viewing. Psychophysiology 42:568–578. doi:10.1111/j.1469-8986.2005.00312.x Google Scholar
  49. 49.
    Russell JA (1980) A circumplex model of affect. J Pers Soc Psychol 39:1161–1178. doi:10.1037/h0077714 CrossRefGoogle Scholar
  50. 50.
    Russell YI, Gobet F (2012) Sinuosity and the affect grid: a method for adjusting repeated mood scores. Percept Mot Skills 114:125–136. doi:10.2466/03.28.PMS.114.1.125-136 CrossRefGoogle Scholar
  51. 51.
    Russell JA, Weiss A, Mendelsohn GA (1989) Affect grid: a single-item scale of pleasure and arousal. J Pers Soc Psychol 57:493–502. doi:10.1037/0022-3514.57.3.493 CrossRefGoogle Scholar
  52. 52.
    Schachter S, Singer J (1962) Cognitive, social, and physiological determinants of emotional state. Psychol Rev 69:379–399. doi:10.1037/h0046234 CrossRefGoogle Scholar
  53. 53.
    Scheirer J, Fernandez R, Klein J, Picard RW (2001) Frustrating the user on purpose: a step toward building an affective computerGoogle Scholar
  54. 54.
    Strain AC, Azevedo R, D’Mello S (2012) Exploring relationships between learners’ affective states, metacognitive processes, and learning outcomes. In: Cerri SA, Clancey WJ, Papadourakis G, Panourgia K (eds) Intell. Tutoring Syst. Springer, Berlin, pp 59–64CrossRefGoogle Scholar
  55. 55.
    Tsui W-H, Lee P, Hsiao T-C (2013) The effect of emotion on keystroke: an experimental study using facial feedback hypothesis. Conf Proc Annu Int Conf IEEE Eng Med Biol Soc IEEE Eng Med Biol Soc Annu Conf 2013:2870–2873. doi:10.1109/EMBC.2013.6610139 Google Scholar
  56. 56.
    Vermeeren APOS, Law EL-C, Roto V et al (2010) User experience evaluation methods: current state and development needs. Proc. 6th Nord. Conf. Hum.-Comput. Interact. Extending Boundaries. ACM, New York, pp 521–530Google Scholar
  57. 57.
    Wang S, Liu Z, Zhu Y et al (2014) Implicit video emotion tagging from audiences’ facial expression. Multimed Tools Appl 74:4679–4706. doi:10.1007/s11042-013-1830-0 CrossRefGoogle Scholar
  58. 58.
    Wang S, Zhu Y, Wu G, Ji Q (2013) Hybrid video emotional tagging using users’ EEG and video content. Multimed Tools Appl 72:1257–1283. doi:10.1007/s11042-013-1450-8 CrossRefGoogle Scholar
  59. 59.
    Ward RD, Marsden PH (2004) Affective computing: problems, reactions and intentions. Interact Comput 16:707–713. doi:10.1016/j.intcom.2004.06.002 CrossRefGoogle Scholar
  60. 60.
    Wilson GM, Sasse MA (2000) Do users always know what’s good for them? Utilising physiological responses to assess media quality. In: CPsychol SMB (Hons) MSc, Waern Y, FRSA GCM (Cantab) PGCE (eds) People Comput. XIV — Usability Else. Springer London, pp 327–339Google Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  • Alexandros Liapis
    • 1
  • Christos Katsanos
    • 1
    • 2
  • Dimitris G. Sotiropoulos
    • 1
  • Nikos Karousos
    • 1
    • 2
  • Michalis Xenos
    • 1
  1. 1.School of Science and TechnologyHellenic Open UniversityPatrasGreece
  2. 2.Technological Educational Institute of Western GreecePatrasGreece

Personalised recommendations