Advertisement

Using the Startle Eye-Blink to Measure Affect in Players

  • Keith Nesbitt
  • Karen Blackmore
  • Geoffrey Hookham
  • Frances Kay-Lambkin
  • Peter Walla
Part of the Advances in Game-Based Learning book series (AGBL)

Abstract

The startle eye-blink is part of a non-voluntary response that typically occurs when an individual encounters a sudden and unexpected stimulus, such as a loud noise or increase in light. Modulations of the startle reflex can be used to infer affective processing in players. The response can be elicited using simple auditory, visual, electric, or mechanical stimuli. The magnitude of the startle eye-blink is used to infer the unconscious positive (pleasant) or negative (unpleasant) emotional state of the player. It is frequently used in psychology where variations in the magnitude, latency, and duration of the startle response are used to understand attention, workload, affective processing, and psychopathologies such as schizophrenia. By comparison, there has been limited use of this objective measure for studying games. As such, there are opportunities to adapt this measure to studies of player affect in the context of game design. We provide a review of the concepts of “affect” and “affective computing” as they relate to game design and also explain in detail the use of the startle eye-blink for objectively measuring player affect. Finally, the use of the approach is illustrated in a case study for evaluating a serious game design.

Keywords

Affective processing Emotion Startle reflex Startle eye-blink 

References

  1. Allen, N. B., Trinder, J., & Brennan, C. (1999). Affective startle modulation in clinical depression: Preliminary findings. Biological Psychiatry, 46(4), 542–550.CrossRefGoogle Scholar
  2. Alzoubi, O., Calvo, R. A., & Stevens, R. H. (2009). Classification of EEG for emotion recognition: An adaptive approach. In Proceedings of 22nd Australasian Joint Conference on Artificial Intelligence (pp. 52–61), Melbourne.Google Scholar
  3. Andreassi, J. J. (2007). Human behaviour and physiological response. New York: Taylor & Francis.Google Scholar
  4. Arroyo, I., Cooper, D. G., Burleson, W., Woolf, B. P., Muldner, K., & Christopherson, R. (2009). Emotion sensors go to school. Proceedings of Artificial Intelligence in Education, 200, 17–24.Google Scholar
  5. Barrett, L. F., Mesquita, B., Ochsner, K. N., & Gross, J. J. (2007). The experience of emotion. Annual Review of Psychology, 58, 373.CrossRefGoogle Scholar
  6. Bartlett, M. S., Littlewort, G., Frank, M., Lainscsek, C., Fasel, I., & Movellan, J. (2006, April). Fully automatic facial action recognition in spontaneous behavior. In 7th International Conference on Automatic Face and Gesture Recognition, 2006. FGR 2006. (pp. 223–230). IEEE.Google Scholar
  7. Beck, A. T., Rush, A. J., Shaw, B. F., & Emery, G. (1979). Cognitive therapy of depression. New York: Guilford.Google Scholar
  8. Bless, H., Clore, G., Schwarz, N., Golisano, V., Rabe, C., & Wolk, M. (1996). Mood and the use of scripts: Does a happy mood really lead to mindlessness? Journal of Personality and Social Psychology, 71, 665–679.CrossRefGoogle Scholar
  9. Blumenthal, T. D., Cuthbert, B. N., Filion, D. L., Hackley, S., Lipp, O. V., & Van Boxtel, A. (2005). Committee report: Guidelines for human startle eyeblink electromyographic studies. Psychophysiology, 42(1), 1–15.CrossRefGoogle Scholar
  10. Bradley, S. D. (2007). Examining the eyeblink startle reflex as a measure of emotion and motivation to television programming. Communication Methods and Measures, 1(1), 7–30.CrossRefGoogle Scholar
  11. Bradley, M. M., & Lang, P. J. (1999). Affective norms for English words (ANEW): Technical manual and affective ratings. Gainesville, FL: The Center for Research in Psychophysiology, University of Florida.Google Scholar
  12. Bradley, M. M., & Lang, P. J. (2007). The international affective picture system (IAPS) in the study of emotion and attention. In J. A. Coan & J. J. B. Allen (Eds.), Handbook of emotion elicitation and assessment (pp. 29–46). New York: Oxford University Press.Google Scholar
  13. Calvo, R. A., Brown, I., & S. Scheding, S. (2009). Effect of Experimental factors on the recognition of affective mental states through physiological measures. In Proceedings of 22nd Australasian Joint Conference on Artificial Intelligence, Melbourne.Google Scholar
  14. Calvo, R. A., & D’Mello, S. (2010). Affect detection: An interdisciplinary review of models. methods, and their applications. IEEE Transactions on Affective Computing, 1(1), 18–37.CrossRefGoogle Scholar
  15. Coan, J. (2010). Emergent ghosts of the emotion machine. Emotion Review, 2, 274.CrossRefGoogle Scholar
  16. Coppins, W. (2014). Measuring the effect of sound on the emotional and immersive experience of players in a video game: A case study in the horror genre. Honours thesis, The University of Newcastle, Australia.Google Scholar
  17. D’Mello, S., & Graesser, A. (2009). Automatic detection of learner’s affect from gross body language. Applied Artificial Intelligence, 23(2), 123–150.CrossRefGoogle Scholar
  18. Dalgleish, T., Dunn, B., & Mobbs, D. (2009). Affective neuroscience: Past present, and future. Emotion Review, 1, 355–368.CrossRefGoogle Scholar
  19. Dan-Glauser, E. S., & Scherer, K. R. (2011). The Geneva affective picture database (GAPED): A new 730-picture database focusing on valence and normative significance. Behavior Research Methods, 43(2), 468–477.CrossRefGoogle Scholar
  20. Darwin, C. (2002). Expression of the emotions in man and animals. New York: Oxford University Press.Google Scholar
  21. Dawson, M. E., Schell, A. M., & Böhmelt, A. H. (1999). Startle modification: Implications for neuroscience, cognitive science, and clinical science. New York: Cambridge University Press.CrossRefGoogle Scholar
  22. Desmet, P., & Hekkert, P. (2007). Framework of product experience. International Journal of Design, 1(1), 2007.Google Scholar
  23. Dimberg, U., Thunberg, M., & Elmehed, K. (2000). Unconscious facial reactions to emotional facial expressions. Psychological Science, 11(1), 86–89.CrossRefGoogle Scholar
  24. Ekman, P. (1992). An argument for basic emotions. Cognition and Emotion, 6, 169–200.CrossRefGoogle Scholar
  25. Ekman, P., & Friesen, W. (1969a). Nonverbal leakage and clues to deception. Psychiatry, 32, 88–106.Google Scholar
  26. Ekman, P., & Friesen, W. V. (1969b). The repertoire of nonverbal behavior: Categories, origins, usage, and coding. Semiotica, 1, 49–98.CrossRefGoogle Scholar
  27. Ekman, P., & Friesen, W. V. (1978). Facial action coding system: A technique for the measurement of facial movement. Palo Alto, CA: Consulting Psychologists.Google Scholar
  28. Ekman, P., Levenson, R., & Friesen, W. (1983). Autonomic nervous system activity distinguishes among emotions. Science, 221, 1208–1210.CrossRefGoogle Scholar
  29. Elmore, W. R. (2012). The effect of violent video game play on emotion modulation of startle. Doctoral dissertation, University of Missouri–Kansas City, Kansas City, MO.Google Scholar
  30. Evinger, C., & Manning, K. A. (1993). Pattern of extraocular muscle activation during reflex blinking. Experimental Brain Research, 92(3), 502–506.Google Scholar
  31. Filion, D. L., Dawson, M. E., & Schell, A. M. (1993). Modification of the acoustic startle-reflex eyeblink: A tool for investigating early and late attentional processes. Biological Psychology, 35, 185–200.CrossRefGoogle Scholar
  32. Filion, D. L., Dawson, M. E., & Schell, A. M. (1998). The psychological significance of human startle eyeblink modification: A review. Biological Psychology, 47, 1–43.CrossRefGoogle Scholar
  33. Fokkinga, S. F., Desmet, P. M. A., & Hoonhout, J. (2010). The dark side of enjoyment: Using negative emotions to design for rich user experiences. In Proceedings of the 7th International Conference of Design and Emotion Society. Chicago, IL: Spertus Institute.Google Scholar
  34. Fontaine, J. R., Scherer, K. R., Roesch, E. B., & Ellsworth, P. C. (2007). The world of emotions is not two-dimensional. Psychological Science, 18(12), 1050–1057.Google Scholar
  35. Geiser, M., & Walla, P. (2011). Objective measures of emotion during virtual walks through urban environments. Applied Sciences, 1, 1–11.CrossRefGoogle Scholar
  36. Graham, F. K. (1975). The more or less startling effects of weak prestimulation. Psychophysiology, 12, 238–248.CrossRefGoogle Scholar
  37. Grahl, A., Greiner, U., & Walla, P. (2012). Bottle shape elicits gender-specific emotion: A startle reflex modulation study. Psychology, 7, 548–554.CrossRefGoogle Scholar
  38. Grimshaw, M., Lindley, C. A., & Nacke, L. (2008, October). Sound and immersion in the first-person shooter: mixed measurement of the player’s sonic experience. In Proceedings of Audio Mostly Conference, pp. 1–7.Google Scholar
  39. Gunes, H., & Piccardi, M. (2007). Bi-modal emotion recognition from expressive face and body gestures. Journal of Network and Computer Applications, 30, 1334–1345.CrossRefGoogle Scholar
  40. Hadley, M. J. (2012). Slender: The Eight Pages [PC game]. Parsec Productions.Google Scholar
  41. Hebb, D. O. (1949). The organization of behavior: A neuropsychological theory. New York: Wiley.Google Scholar
  42. Hoffman, H. S., & Fleshier, M. (1963). Startle reaction: Modification by background stimulation. Science, 141, 928–930.CrossRefGoogle Scholar
  43. Hookham, G., Deady, M., Kay-Lambkin, F., & Nesbitt, K. (2013) Training for life: Designing a game to engage younger people in a psychological counselling program. Australian Journal of Intelligent Information Processing Systems, 13(3): Special issue on Edutainment 2013.Google Scholar
  44. International Society of Electrophysiological Kinesiology. (1980). Units, terms and standards in the reporting of EMG research. Carbondale, IL: Southern Illinois University School of Medicine.Google Scholar
  45. Izard, C. (1994). Innate and universal facial expressions: Evidence from developmental and cross-cultural research. Psychological Bulletin, 115, 288–299.CrossRefGoogle Scholar
  46. James, W. (1884). What is an emotion? Mind, 9, 188–205.CrossRefGoogle Scholar
  47. Juslin, P. N., & Scherer, K. R. (2005). Vocal expression of affect. In J. A. Harrigan, R. Rosenthal, & K. R. Scherer (Eds.), The new handbook of methods in nonverbal behavior research (pp. 65–135). Oxford, MA: Oxford University Press.Google Scholar
  48. Kanfer, R., & Ackerman, P. L. (1989). Motivation and cognitive abilities: An integrative/aptitude-treatment interaction approach to skill acquisition. Journal of Applied Psychology, 74, 657–690.CrossRefGoogle Scholar
  49. Kaviani, H., Gray, J. A., Checkley, S. A., Kumari, V., & Wilson, G. D. (1999). Modulation of the acoustic startle reflex by emotionally toned film-clips. International Journal of Psychophysiology, 32(1), 47–54.CrossRefGoogle Scholar
  50. Kaviani, H., Wilson, G. D., Checkley, S. A., Kumari, V., & Gray, J. A. (1998). Modulation of the human acoustic startle reflex by pleasant and unpleasant odors. Journal of Psychophysiology, 12, 352–361.Google Scholar
  51. Kay-Lambkin, F., Baker, A. L., Kelly, B., & Lewin, T. J. (2011). Clinician-assisted computerised versus therapist-delivered treatment for depressive and addictive disorders: A randomised controlled trial. Medical Journal of Australia, 195(3), S44–S50.Google Scholar
  52. Kemper, T. D. (1991). Predicting emotions from social-relations. Social Psychology Quarterly, 54, 330–342.CrossRefGoogle Scholar
  53. Lang, P. J. (1995). The emotion probe: Studies of motivation and attention. American Psychologist, 50(5), 372–385.CrossRefGoogle Scholar
  54. Lang, P. J., Bradley, M. M., & Cuthbert, B. N. (1990). Emotion, attention, and the startle reflex. Psychological Review, 97, 377–395.CrossRefGoogle Scholar
  55. Lang, P. J., Bradley, M. M., & Cuthbert, B. N. (1998). Emotion, motivation, and anxiety: Brain mechanisms and psychophysiology. Biological Psychiatry, 44, 1248–1263.CrossRefGoogle Scholar
  56. Lang, P. J., Greenwald, M. K., Bradley, M. M., & Hamm, A. O. (1993). Looking at pictures: Affective, facial, visceral, and behavioral reactions. Psychophysiology, 30, 261–273.CrossRefGoogle Scholar
  57. MATLAB and Statistics Toolbox Release R2014b. (2014). Natick, MA: The MathWorks, Inc.Google Scholar
  58. Mavratzakis, A., Molloy, E., & Walla, P. (2013). Modulation of the startle reflex during brief and sustained exposure to emotional pictures. Psychology, 4, 389–395.CrossRefGoogle Scholar
  59. McDaniel, B. T., D’Mello, S., King, B. G., Chipman, P., Tapp, K., & Graesser, A. C. (2007). Facial features for affective state detection in learning environments. In Proceedings of the 29th Annual Cognitive Science Society (pp. 467–472). Austin, TX: Cognitive Science Society.Google Scholar
  60. Meyer, D. K., & Turner, J. C. (2006). Re-conceptualizing emotion and motivation to learn in classroom contexts. Educational Psychology Review, 18(4), 377–390.CrossRefGoogle Scholar
  61. Mota, S., & Picard, R. W. (2003). Automated posture analysis for detecting learner’s interest level. In Computer Vision and Pattern Recognition Workshop, 2003 (Vol. 5, p. 49). IEEE.Google Scholar
  62. Muehlberger, A., Wieser, M. J., & Pauli, P. (2008). Darkness-enhanced startle responses in ecologically valid environments: A virtual tunnel driving experiment. Biological Psychology, 77, 47–52. doi: 10.1016/j.biopsycho.2007.09.004.CrossRefGoogle Scholar
  63. Nacke L. (2009). Affective Ludology: Scientific measurement of user experience in interactive entertainment. Blekinge Institute of Technology, Game Systems and Interaction Research Laboratory, School of Computing, Blekinge Institute of Technology, Doctoral Dissertation Series No 2009:04.Google Scholar
  64. Nasoz, F., Alvarez, K., Lisetti, L., & Finkelstein, N. (2004). Emotion recognition from physiological signals using wireless sensors for presence technologies. Cognition, Technology and Work, 6(1), 4–14.CrossRefGoogle Scholar
  65. Neumann, D. L. (2002). Effect of varying levels of mental workload on startle eyeblink modulation. Ergonomics, 45(8), 583–602.CrossRefGoogle Scholar
  66. Osgood, C. E., May, W. H., & Miron, M. S. (1975). Cross-cultural universals of affective meaning. Urbana, IL: University of Illinois Press.Google Scholar
  67. Pang, B., & Lee, L. (2008). Opinion mining and sentiment analysis. Foundations and Trends in Information Retrieval, 2, 1–135.CrossRefGoogle Scholar
  68. Pantic, M., & Rothkrantz, L. (2003). Toward an affect-sensitive multi-modal human-computer interaction. Proceedings of the IEEE, 91(9), 1370–1390.CrossRefGoogle Scholar
  69. Parsons, T. D., Rizzo, A. A., Courtney, C. G., & Dawson, M. E. (2012). Psychophysiology to assess impact of varying levels of simulation fidelity in a threat environment. Advances in Human-Computer Interaction, 5, 1–9.CrossRefGoogle Scholar
  70. Patrick, C. J., Bradley, M. M., & Lang, P. J. (1993). Emotion in the criminal psychopath: Startle reflex modulation. Journal of Abnormal Psychology, 102(1), 82–92.CrossRefGoogle Scholar
  71. Pekrun, R., Goetz, T., Titz, W., & Perry, R. (2002). Academic emotions in students’ self-regulated learning and achievement: A program of qualitative and quantitative research. Educational Psychologist, 37(2), 91–105.CrossRefGoogle Scholar
  72. Picard, R. W. (1997). Affective computing. Cambridge, MA: MIT Press.CrossRefGoogle Scholar
  73. Picard, R. W., Vyzas, E., & Healey, J. (2001). Toward machine emotional intelligence: Analysis of affective physiological state. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(10), 1175–1191.CrossRefGoogle Scholar
  74. Raghunathan, R., & Trope, Y. (2002). Walking the tightrope between feeling good and being accurate: Mood as a resource in processing persuasive messages. Journal of Personality and Social Psychology, 83, 510–525.CrossRefGoogle Scholar
  75. Ramirez, O. M., & Dockweiler, C. J. (1987). Mathematics anxiety: A systematic review. In R. Schwarzer, H. M. Ploeg, & C. D. Spielberger (Eds.), Advances in test anxiety research (pp. 157–175). Hillsdale, NJ: Erlbaum.Google Scholar
  76. Ravaja, N., & Kivikangas, J. M. (2008, August). Psychophysiology of digital game playing: The relationship of self-reported emotions with phasic physiological responses. In Proceedings of Measuring Behavior (pp. 26–29), Maastricht, The Netherland.Google Scholar
  77. Ravaja, N., Saari, T., Salminen, M., Laarni, J., & Kallinen, K. (2006). Phasic emotional reactions to video game events: A psychophysiological investigation. Media Psychology, 8(4), 343–367.CrossRefGoogle Scholar
  78. Ravaja, N., Turpeinen, M., Saari, T., Puttonen, S., & Keltikangas-Järvinen, L. (2008). The psychophysiology of James Bond: Phasic emotional responses to violent video game events. Emotion, 8(1), 114.CrossRefGoogle Scholar
  79. Reisenzein, R., Hudicka, E., Dastani, M., Gratch, J., Hindriks, K. L., & Meyer, J. (2013). Computational modeling of emotion: Toward improving the inter- and interdisciplinary exchange. IEEE Transactions on Affective Computing, 4(3), 246–266.CrossRefGoogle Scholar
  80. Roy, M., Mailhot, J. P., Gosselin, N., Paquette, S., & Peretz, I. (2009). Modulation of the startle reflex by pleasant and unpleasant music. International Journal of Psychophysiology, 71(1), 37–42.CrossRefGoogle Scholar
  81. Russell, J. A. (1980). A circumplex model of affect. Journal of Personality and Social Psychology, 39(6), 1161–1178.CrossRefGoogle Scholar
  82. Russell, J. A. (2003). Core affect and the psychological construction of emotion. Psychological Review, 110, 145–172.CrossRefGoogle Scholar
  83. Russell, J. A., Bachorowski, J. A., & Fernandez-Dols, J. M. (2003). Facial and vocal expressions of emotion. Annual Review of Psychology, 54, 329–349.CrossRefGoogle Scholar
  84. Sabourin, J. L., & Lester, J. C. (2014). Affect and engagement in game-based learning environments. IEEE Transactions on Affective Computing, 5(1), 45–56.CrossRefGoogle Scholar
  85. Sabourin, J., Rowe, J. P., Mott, B. W., & Lester, J. C. (2011). When off-task is on-task: The affective role of off-task behavior in narrative-centered learning environments. In Proceedings of 15th International Conference on Artificial Intelligence in Education (pp. 523–536).Google Scholar
  86. Salovey, P. (2003). Introduction: Emotion and social processes. In R. J. Davidson, K. R. Scherer, & H. H. Goldsmith (Eds.), Handbook of affective sciences (pp. 747–751). Oxford, UK: Oxford University Press.Google Scholar
  87. Sasse, D. (2008). A framework for psychophysiological data acquisition in digital games. Master’s thesis, Otto-von-Guericke-University Magdeburg, Magdeburg, Germany.Google Scholar
  88. Scherer, K. R. (1993). Neuroscience projections to current debates in emotion psychology. Cognition and Emotion, 7(1), 1–41.CrossRefGoogle Scholar
  89. Scherer, K. R. (2005). What are emotions? And how can they be measured? Social Science Information, 44(4), 695–729.CrossRefGoogle Scholar
  90. Scherer, K. R., Schorr, A. E., & Johnstone, T. E. (2001). Appraisal processes in emotion: Theory, methods, research. Oxford, UK: Oxford University Press.Google Scholar
  91. Schneirla, T. C. (1959). An evolutionary and developmental theory of biphasic processes underlying approach and withdrawal. In M. R. Jones (Ed.), Nebraska symposium on motivation (Vol. 7, pp. 1–43). Lincoln, NE: University of Nebraska Press.Google Scholar
  92. Sebe, N., Cohen, I., & Huang, T. S. (2005). Multimodal emotion recognition. Handbook of Pattern Recognition and Computer Vision, 4, 387–419.Google Scholar
  93. Segal, Z., Williams, J. M. G., & Teasdale, J. D. (2002). Mindfulness-based cognitive therapy for depression: A new approach to preventing relapse. New York: Guilford.Google Scholar
  94. Strapparava, C., & Valitutti, A. (2004). WordNet affect: An affective extension of WordNet. LREC, 4, 1083–1086.Google Scholar
  95. Swerdlow, N. R., Weber, M., Qu, Y., Light, G. A., & Braff, D. L. (2008). Realistic expectations of prepulse inhibition in translational models for schizophrenia research. Psychopharmacology, 199(3), 331–388.CrossRefGoogle Scholar
  96. Valsamis, B., & Schmid, S. (2011). Habituation and prepulse inhibition of acoustic startle in rodents. Journal of Visualized Experiments, 55, 3446. doi: 10.3791/3446.Google Scholar
  97. van Bedaf, L. R., Heesink, L., & Geuze, E. (2014, August 27–29). Pre-processing of electromyography startle data: A novel semi-automatic method. In: Proceedings of Measuring Behavior, Wageningen, The Netherlands.Google Scholar
  98. Villon, O., & Lisetti, C. (2006). A user-modeling approach to build user’s psycho-physiological maps of emotions using bio-sensors. In Proceedings of IEEE RO-MAN 2006, 15th IEEE International Symposium on Robot and Human Interactive Communication, Session Emotional Cues in Human-Robot Interaction (pp. 269–276).Google Scholar
  99. Vrana, S. R., Spence, E. L., & Lang, P. J. (1988). The startle probe response: A new measure of emotion? Journal of Abnormal Psychology, 97, 487–491.CrossRefGoogle Scholar
  100. Vyzas E., & Picard, R. W. (1998). Affective pattern classification. In: Proceedings of AAAI Fall Symposium Series: Emotional and Intelligent: The Tangled Knot of Cognition (pp. 176–182), Orlando, FL.Google Scholar
  101. Walla, P., Brenner, G., & Koller, M. (2011). Objective measures of emotion related to brand attitude: A new way to quantify emotion-related aspects relevant to marketing. PLoS One, 6(11), e26782. doi: 10.1371/journal.pone.0026782.CrossRefGoogle Scholar
  102. Walla, P., & Panksepp, J. (2013). Neuroimaging helps to clarify brain affective processing without necessarily clarifying emotions. In K. N. Fountas (Ed.), Novel Frontiers of Advanced Neuroimaging. InTech. ISBN: 978-953-51-0923-5, doi: 10.5772/51761.
  103. Walla, P., Richter, M., Färber, S., Leodolter, U., & Bauer, H. (2010). Food evoked changes in humans: Startle response modulation and event-related brain potentials (ERPs). Journal of Psychophysiology, 24, 25–32.CrossRefGoogle Scholar
  104. Witvliet, C. V., & Vrana, S. R. (1995). Psychophysiological responses as indices of affective dimensions. Psychophysiology, 32(5), 436–443.CrossRefGoogle Scholar
  105. Wood, R. T., Griffiths, M. D., Chappell, D., & Davies, M. N. (2004). The structural characteristics of video games: A psycho-structural analysis. CyberPsychology & Behavior, 7(1), 1–10.CrossRefGoogle Scholar
  106. Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2009). A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(1), 39–58.CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Keith Nesbitt
    • 1
  • Karen Blackmore
    • 1
  • Geoffrey Hookham
    • 1
  • Frances Kay-Lambkin
    • 2
  • Peter Walla
    • 3
  1. 1.University of NewcastleCallaghanAustralia
  2. 2.University of New South WalesRandwickAustralia
  3. 3.Webster Vienna Private University, Palais WenkeimViennaAustria

Personalised recommendations