Applied Psychophysiology and Biofeedback

, Volume 42, Issue 4, pp 269–282 | Cite as

Analysis of Subjects’ Vulnerability in a Touch Screen Game Using Behavioral Metrics

  • Payam Parsinejad
  • Rifat SipahiEmail author


In this article, we report results on an experimental study conducted with volunteer subjects playing a touch-screen game with two unique difficulty levels. Subjects have knowledge about the rules of both game levels, but only sufficient playing experience with the easy level of the game, making them vulnerable with the difficult level. Several behavioral metrics associated with subjects’ playing the game are studied in order to assess subjects’ mental-workload changes induced by their vulnerability. Specifically, these metrics are calculated based on subjects’ finger kinematics and decision making times, which are then compared with baseline metrics, namely, performance metrics pertaining to how well the game is played and a physiological metric called pnn50 extracted from heart rate measurements. In balanced experiments and supported by comparisons with baseline metrics, it is found that some of the studied behavioral metrics have the potential to be used to infer subjects’ mental workload changes through different levels of the game. These metrics, which are decoupled from task specifics, relate to subjects’ ability to develop strategies to play the game, and hence have the advantage of offering insight into subjects’ task-load and vulnerability assessment across various experimental settings.


Behavioral metrics Performance metrics Heart rate variability (HRV) Vulnerability Touch-screen game 



The human subjects experiments in this study were conducted under an approved IRB protocol #11-19-11. Authors thank Naiqian Zhi in her assistance with the literature review. This work is supported in part by a DARPA Young Faculty Award #N66001-11-1-4161. The content of this research does not necessarily reflect the viewpoints of the funding agency, and no official endorsement of the US Government should be inferred. RS acknowledges fruitful discussions on the topic with Professor Maurizio Porfiri (NYU Tandon School of Engineering), Paul de la Houssaye (SPAWAR), and Gill Pratt (formerly Program Manager at DARPA, currently at Toyota Research Institute). De-identified data pertaining to this study can be accessed at

Compliance with Ethical Standards

Conflict of interest

Authors declare that they do not have any competing interests with the conducted research.


  1. Acharya, U. R., Joseph, K. P., Kannathal, N., Lim, C. M., & Suri, J. S. (2006). Heart rate variability: A review. Medical and Biological Engineering and Computing, 44(12), 1031–1051.CrossRefGoogle Scholar
  2. Allen, J. (2007). Photoplethysmography and its application in clinical physiological measurement. Physiological Measurement, 28(3), 1.CrossRefGoogle Scholar
  3. Antelmi, I, De Paula, R. S., Shinzato, A. R., Peres, C. A., Mansur, A. J., & Grupi, C. J. (2004). Influence of age, gender, body mass index, and functional capacity on heart rate variability in a cohort of subjects without heart disease. The American Journal of Cardiology, 93(3), 381–385.CrossRefPubMedGoogle Scholar
  4. Borghini, G, Astolfi, L, Vecchiato, G, Mattia, D, & Babiloni, F. (2014). Measuring neurophysiological signals in aircraft pilots and car drivers for the assessment of mental workload, fatigue and drowsiness. Neuroscience & Biobehavioral Reviews, 44, 58–75.CrossRefGoogle Scholar
  5. Brainard, D. H. (1997). The psychophysics toolbox. Spatial Vision, 10, 433–436.CrossRefPubMedGoogle Scholar
  6. Burges, C. J. C. (1998). A tutorial on support vector machines for pattern recognition. Data Mining and Knowledge Discovery, 2(2), 121–167.CrossRefGoogle Scholar
  7. Camm, A. J., Malik, M., Bigger, J. T., Breithardt, G., Cerutti, S., Cohen, R. J., Coumel, P., et al. (1996). Heart rate variability: Standards of measurement, physiological interpretation and clinical use. Task force of the European Society of Cardiology and the North American Society of Pacing and Electrophysiology. Circulation, 93(5), 1043–1065.CrossRefGoogle Scholar
  8. Cinaz, B., Arnrich, B., La Marca, R., & Tröster, G. (2013). Monitoring of mental workload levels during an everyday life office-work scenario. Personal and Ubiquitous Computing, 17(2), 229–239.CrossRefGoogle Scholar
  9. Danaher, J. W. (1980). Human error in atc system operations. Human Factors: The Journal of the Human Factors and Ergonomics Society, 22(5), 535–545.CrossRefGoogle Scholar
  10. Dawson, M. E., Anne, M. S., & Diane, L. F. (2007). The electrodermal system. Handbook of Psychophysiology, 2, 200–223.Google Scholar
  11. El-Nasr, M. S., Drachen, A., & Canossa, A. (2013). Game analytics: Maximizing the value of player data. New York: Springer.CrossRefGoogle Scholar
  12. Epp, C., Lippold, M., & Mandryk, R. L. (2011). Identifying emotional states using keystroke dynamics. In Proceedings of the sigchi conference on human factors in computing systems, pp. 715–724. New York: ACM.Google Scholar
  13. Fitts, P. M. (1954). The information capacity of the human motor system in controlling the amplitude of movement. Journal of Experimental Psychology, 47(6), 381.CrossRefPubMedGoogle Scholar
  14. Gao, Y., Bianchi-Berthouze, N., & Meng, H. (2012). What does touch tell us about emotions in touchscreen-based gameplay? ACM Transactions on Computer-Human Interaction (TOCHI), 19(4), 31.CrossRefGoogle Scholar
  15. Hancock, P., Chignell, M. H., et al. (1988). Mental workload dynamics in adaptive interface design. IEEE Transactions on Systems, Man and Cybernetics, 18(4), 647–658.CrossRefGoogle Scholar
  16. Henelius, A., Hirvonen, K., Holm, A., Korpela, J., & Müller, K. (2009). Mental workload classification using heart rate metrics. In Engineering in medicine and biology society, 2009. embc. 2009. Annual International Conference of the IEEE, 1836–1839. IEEE.Google Scholar
  17. Hick, W. E. (1952). On the rate of gain of information. Quarterly Journal of Experimental Psychology, 4(1), 11–26.CrossRefGoogle Scholar
  18. Hyman, R. (1953). Stimulus information as a determinant of reaction time. Journal of Experimental Psychology, 45(3), 188.CrossRefPubMedGoogle Scholar
  19. Kim, S., & Smith-Spark, L. (2014). Cnn: Asiana says pilot error partly to blame for san francisco plane crash. Accessed March 5, 2015.Google Scholar
  20. Kivikangas, J. M., Ekman, I., Chanel, G., Järvelä, S., Salminen, M., Cowley, B., Henttonen, P., & Ravaja, N. (2010). Review on psychophysiological methods in game research. Proceedings of 1st Nordic DiGRA.Google Scholar
  21. Malik, M., Bigger, J. T., Camm, A. J., Kleiger, R. E., Malliani, A., Moss, A. J., et al. (1996). Heart rate variability standards of measurement, physiological interpretation, and clinical use. European Heart Journal, 17(3), 354–381.CrossRefGoogle Scholar
  22. Mandryk, R. L. (2008). Physiological measures for game evaluation. In Game usability: Advice from the experts for advancing the player experience. Amsterdam: ElsevierGoogle Scholar
  23. Mandryk, R. L., & Atkins, M. S. (2007). A fuzzy physiological approach for continuously modeling emotion during interaction with play technologies. International Journal of Human-Computer Studies, 65(4), 329–347.CrossRefGoogle Scholar
  24. Mandryk, R. L., & Inkpen, K. M. (2004). Physiological indicators for the evaluation of co-located collaborative play. In Proceedings of the 2004 ACM conference on computer supported cooperative work, pp. 102–111. New York: ACM.Google Scholar
  25. Massin, M. M., Derkenne, B., & von Bernuth, G. (1999). Correlations between indices of heart rate variability in healthy children and children with congenital heart disease. Cardiology, 91(2), 109–113.CrossRefPubMedGoogle Scholar
  26. Mehler, B., Reimer, B., & Coughlin, J. F. (2012). Sensitivity of physiological measures for detecting systematic variations in cognitive demand from a working memory task an on-road study across three age groups. Human Factors: The Journal of the Human Factors and Ergonomics Society, 54(3), 396–412.CrossRefGoogle Scholar
  27. Murray, N. P., & Russoniello, C. (2012). Acute physical activity on cognitive function: A heart rate variability examination. Applied Psychophysiology and Biofeedback, 37(4), 219–227.CrossRefPubMedGoogle Scholar
  28. Parsinejad, P. (2016). Inferring mental workload changes of subjects unfamiliar with a touch screen game through physiological and behavioral measurements. PhD diss, Northeastern University.Google Scholar
  29. Parsinejad, P., & Sipahi, R. (2014). A touchscreen game to induce mental workload on human subjects. In 40th Annual Northeast Bioengineering Conference (NEBEC), pp. 1–2. IEEE.Google Scholar
  30. Parsinejad, P., & Sipahi, R. (2015). Assessment of human vulnerability in a touch-screen game; metrics and analysis. In ASME 2015 Dynamic Systems and Control Conference (pp. V001T09A004–V001T09A004). American Society of Mechanical Engineers.Google Scholar
  31. Parsinejad, P., Rodriguez-Vaqueiro, Y., Martinez-Lorenzo, J. A., & Sipahi, R. (2014). Combined time-frequency calculation of pnn50 metric from noisy heart rate measurements. In ASME 2014 Dynamic Systems and Control Conference (pp. V001T09A004–V001T09A004). American Society of Mechanical Engineers.Google Scholar
  32. Pelli, D. G. (1997). The videotoolbox software for visual psychophysics: Transforming numbers into movies. Spatial Vision, 10(4), 437–442.CrossRefPubMedGoogle Scholar
  33. Picard, R. W. (2000). Affective computing. Cambridge: MIT press.Google Scholar
  34. Picard, R. W., Vyzas, E., & Healey, J. (2001). Toward machine emotional intelligence: Analysis of affective physiological state. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(10), 1175–1191.CrossRefGoogle Scholar
  35. Quintana, D.S, & Heathers, J. A. (2014). Considerations in the assessment of heart rate variability in biobehavioral research. Frontiers in Psychology, 5.Google Scholar
  36. Reimer, B., & Mehler, B. (2011). The impact of cognitive workload on physiological arousal in young adult drivers: A field study and simulation validation. Ergonomics, 54(10), 932–942.CrossRefPubMedGoogle Scholar
  37. Rowe, D. W., Sibert, J., & Irwin, D. (1998). Heart rate variability: Indicator of user state as an aid to human-computer interaction. In Proceedings of the sigchi conference on human factors in computing systems, pp. 480–487. New York: ACM Press/Addison-Wesley Publishing Co.Google Scholar
  38. Ryu, K., & Myung, R. (2005). Evaluation of mental workload with a combined measure based on physiological indices during a dual task of tracking and mental arithmetic. International Journal of Industrial Ergonomics, 35(11), 991–1009.CrossRefGoogle Scholar
  39. Stroop, J. R. (1935). Interference in serial verbal reactions. Journal of Experimental Psychology, 18, 643–661.CrossRefGoogle Scholar
  40. Sun, F. T., Kuo, C., Cheng, H. T., Buthpitiya, S., Collins, P., & Griss, M. (2012). Activity-aware mental stress detection using physiological sensors. In Mobile computing, applications, and services, pp. 211–230. Berlin: Springer.CrossRefGoogle Scholar
  41. Taelman, J., Vandeput, S., Spaepen, A., & Van Huffel, S. (2009). Influence of mental stress on heart rate and heart rate variability. In 4th european conference of the international federation for medical and biological engineering, pp. 1366–1369.Google Scholar
  42. Wilson, G. F. (2002). An analysis of mental workload in pilots during flight using multiple psychophysiological measures. The International Journal of Aviation Psychology, 12(1), 3–18.CrossRefGoogle Scholar
  43. Wilson, G. F., & Russell, C. A. (2003). Real-time assessment of mental workload using psychophysiological measures and artificial neural networks. Human Factors: The Journal of the Human Factors and Ergonomics Society, 45(4), 635–644.CrossRefGoogle Scholar
  44. Wilson, G. F., & Russell, C. A. (2007). Performance enhancement in an uninhabited air vehicle task using psychophysiologically determined adaptive aiding. Human Fctors: The Journal of the Human Factors and Ergonomics Society, 49(6), 1005–1018.CrossRefGoogle Scholar
  45. Yao, Y. J., Chang, Y. M., Xie, X. P., Cao, X. S., Sun, X. Q., & Wu, Y. H. (2008). Heart rate and respiration responses to real traffic pattern flight. Applied Psychophysiology and Biofeedback, 33(4), 203–209.CrossRefPubMedGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2017

Authors and Affiliations

  1. 1.Department of Mechanical and Industrial EngineeringNortheastern UniversityBostonUSA

Personalised recommendations