Abstract
In this article, we report results on an experimental study conducted with volunteer subjects playing a touch-screen game with two unique difficulty levels. Subjects have knowledge about the rules of both game levels, but only sufficient playing experience with the easy level of the game, making them vulnerable with the difficult level. Several behavioral metrics associated with subjects’ playing the game are studied in order to assess subjects’ mental-workload changes induced by their vulnerability. Specifically, these metrics are calculated based on subjects’ finger kinematics and decision making times, which are then compared with baseline metrics, namely, performance metrics pertaining to how well the game is played and a physiological metric called pnn50 extracted from heart rate measurements. In balanced experiments and supported by comparisons with baseline metrics, it is found that some of the studied behavioral metrics have the potential to be used to infer subjects’ mental workload changes through different levels of the game. These metrics, which are decoupled from task specifics, relate to subjects’ ability to develop strategies to play the game, and hence have the advantage of offering insight into subjects’ task-load and vulnerability assessment across various experimental settings.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
We call this data as Data Set 2 in Parsinejad et al. (2014), but in there the research focus is different and behavioral metrics are not studied. Prior to Data Set 2, the same subjects participated in the same experiments (Data Set 1, (Parsinejad et al. 2014)) 6 weeks earlier on average. However in Data Set 1, subjects’ finger strokes were not recorded and hence are not available for analysis.
These metrics were formulated and studied only on EXP-A data in Parsinejad and Sipahi (2015). Here we present them in details for completeness and comparison with EXP-B.
References
Acharya, U. R., Joseph, K. P., Kannathal, N., Lim, C. M., & Suri, J. S. (2006). Heart rate variability: A review. Medical and Biological Engineering and Computing, 44(12), 1031–1051.
Allen, J. (2007). Photoplethysmography and its application in clinical physiological measurement. Physiological Measurement, 28(3), 1.
Antelmi, I, De Paula, R. S., Shinzato, A. R., Peres, C. A., Mansur, A. J., & Grupi, C. J. (2004). Influence of age, gender, body mass index, and functional capacity on heart rate variability in a cohort of subjects without heart disease. The American Journal of Cardiology, 93(3), 381–385.
Borghini, G, Astolfi, L, Vecchiato, G, Mattia, D, & Babiloni, F. (2014). Measuring neurophysiological signals in aircraft pilots and car drivers for the assessment of mental workload, fatigue and drowsiness. Neuroscience & Biobehavioral Reviews, 44, 58–75.
Brainard, D. H. (1997). The psychophysics toolbox. Spatial Vision, 10, 433–436.
Burges, C. J. C. (1998). A tutorial on support vector machines for pattern recognition. Data Mining and Knowledge Discovery, 2(2), 121–167.
Camm, A. J., Malik, M., Bigger, J. T., Breithardt, G., Cerutti, S., Cohen, R. J., Coumel, P., et al. (1996). Heart rate variability: Standards of measurement, physiological interpretation and clinical use. Task force of the European Society of Cardiology and the North American Society of Pacing and Electrophysiology. Circulation, 93(5), 1043–1065.
Cinaz, B., Arnrich, B., La Marca, R., & Tröster, G. (2013). Monitoring of mental workload levels during an everyday life office-work scenario. Personal and Ubiquitous Computing, 17(2), 229–239.
Danaher, J. W. (1980). Human error in atc system operations. Human Factors: The Journal of the Human Factors and Ergonomics Society, 22(5), 535–545.
Dawson, M. E., Anne, M. S., & Diane, L. F. (2007). The electrodermal system. Handbook of Psychophysiology, 2, 200–223.
El-Nasr, M. S., Drachen, A., & Canossa, A. (2013). Game analytics: Maximizing the value of player data. New York: Springer.
Epp, C., Lippold, M., & Mandryk, R. L. (2011). Identifying emotional states using keystroke dynamics. In Proceedings of the sigchi conference on human factors in computing systems, pp. 715–724. New York: ACM.
Fitts, P. M. (1954). The information capacity of the human motor system in controlling the amplitude of movement. Journal of Experimental Psychology, 47(6), 381.
Gao, Y., Bianchi-Berthouze, N., & Meng, H. (2012). What does touch tell us about emotions in touchscreen-based gameplay? ACM Transactions on Computer-Human Interaction (TOCHI), 19(4), 31.
Hancock, P., Chignell, M. H., et al. (1988). Mental workload dynamics in adaptive interface design. IEEE Transactions on Systems, Man and Cybernetics, 18(4), 647–658.
Henelius, A., Hirvonen, K., Holm, A., Korpela, J., & Müller, K. (2009). Mental workload classification using heart rate metrics. In Engineering in medicine and biology society, 2009. embc. 2009. Annual International Conference of the IEEE, 1836–1839. IEEE.
Hick, W. E. (1952). On the rate of gain of information. Quarterly Journal of Experimental Psychology, 4(1), 11–26.
Hyman, R. (1953). Stimulus information as a determinant of reaction time. Journal of Experimental Psychology, 45(3), 188.
Kim, S., & Smith-Spark, L. (2014). Cnn: Asiana says pilot error partly to blame for san francisco plane crash. Accessed March 5, 2015.
Kivikangas, J. M., Ekman, I., Chanel, G., Järvelä, S., Salminen, M., Cowley, B., Henttonen, P., & Ravaja, N. (2010). Review on psychophysiological methods in game research. Proceedings of 1st Nordic DiGRA.
Malik, M., Bigger, J. T., Camm, A. J., Kleiger, R. E., Malliani, A., Moss, A. J., et al. (1996). Heart rate variability standards of measurement, physiological interpretation, and clinical use. European Heart Journal, 17(3), 354–381.
Mandryk, R. L. (2008). Physiological measures for game evaluation. In Game usability: Advice from the experts for advancing the player experience. Amsterdam: Elsevier
Mandryk, R. L., & Atkins, M. S. (2007). A fuzzy physiological approach for continuously modeling emotion during interaction with play technologies. International Journal of Human-Computer Studies, 65(4), 329–347.
Mandryk, R. L., & Inkpen, K. M. (2004). Physiological indicators for the evaluation of co-located collaborative play. In Proceedings of the 2004 ACM conference on computer supported cooperative work, pp. 102–111. New York: ACM.
Massin, M. M., Derkenne, B., & von Bernuth, G. (1999). Correlations between indices of heart rate variability in healthy children and children with congenital heart disease. Cardiology, 91(2), 109–113.
Mehler, B., Reimer, B., & Coughlin, J. F. (2012). Sensitivity of physiological measures for detecting systematic variations in cognitive demand from a working memory task an on-road study across three age groups. Human Factors: The Journal of the Human Factors and Ergonomics Society, 54(3), 396–412.
Murray, N. P., & Russoniello, C. (2012). Acute physical activity on cognitive function: A heart rate variability examination. Applied Psychophysiology and Biofeedback, 37(4), 219–227.
Parsinejad, P. (2016). Inferring mental workload changes of subjects unfamiliar with a touch screen game through physiological and behavioral measurements. PhD diss, Northeastern University.
Parsinejad, P., & Sipahi, R. (2014). A touchscreen game to induce mental workload on human subjects. In 40th Annual Northeast Bioengineering Conference (NEBEC), pp. 1–2. IEEE.
Parsinejad, P., & Sipahi, R. (2015). Assessment of human vulnerability in a touch-screen game; metrics and analysis. In ASME 2015 Dynamic Systems and Control Conference (pp. V001T09A004–V001T09A004). American Society of Mechanical Engineers.
Parsinejad, P., Rodriguez-Vaqueiro, Y., Martinez-Lorenzo, J. A., & Sipahi, R. (2014). Combined time-frequency calculation of pnn50 metric from noisy heart rate measurements. In ASME 2014 Dynamic Systems and Control Conference (pp. V001T09A004–V001T09A004). American Society of Mechanical Engineers.
Pelli, D. G. (1997). The videotoolbox software for visual psychophysics: Transforming numbers into movies. Spatial Vision, 10(4), 437–442.
Picard, R. W. (2000). Affective computing. Cambridge: MIT press.
Picard, R. W., Vyzas, E., & Healey, J. (2001). Toward machine emotional intelligence: Analysis of affective physiological state. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(10), 1175–1191.
Quintana, D.S, & Heathers, J. A. (2014). Considerations in the assessment of heart rate variability in biobehavioral research. Frontiers in Psychology, 5.
Reimer, B., & Mehler, B. (2011). The impact of cognitive workload on physiological arousal in young adult drivers: A field study and simulation validation. Ergonomics, 54(10), 932–942.
Rowe, D. W., Sibert, J., & Irwin, D. (1998). Heart rate variability: Indicator of user state as an aid to human-computer interaction. In Proceedings of the sigchi conference on human factors in computing systems, pp. 480–487. New York: ACM Press/Addison-Wesley Publishing Co.
Ryu, K., & Myung, R. (2005). Evaluation of mental workload with a combined measure based on physiological indices during a dual task of tracking and mental arithmetic. International Journal of Industrial Ergonomics, 35(11), 991–1009.
Stroop, J. R. (1935). Interference in serial verbal reactions. Journal of Experimental Psychology, 18, 643–661.
Sun, F. T., Kuo, C., Cheng, H. T., Buthpitiya, S., Collins, P., & Griss, M. (2012). Activity-aware mental stress detection using physiological sensors. In Mobile computing, applications, and services, pp. 211–230. Berlin: Springer.
Taelman, J., Vandeput, S., Spaepen, A., & Van Huffel, S. (2009). Influence of mental stress on heart rate and heart rate variability. In 4th european conference of the international federation for medical and biological engineering, pp. 1366–1369.
Wilson, G. F. (2002). An analysis of mental workload in pilots during flight using multiple psychophysiological measures. The International Journal of Aviation Psychology, 12(1), 3–18.
Wilson, G. F., & Russell, C. A. (2003). Real-time assessment of mental workload using psychophysiological measures and artificial neural networks. Human Factors: The Journal of the Human Factors and Ergonomics Society, 45(4), 635–644.
Wilson, G. F., & Russell, C. A. (2007). Performance enhancement in an uninhabited air vehicle task using psychophysiologically determined adaptive aiding. Human Fctors: The Journal of the Human Factors and Ergonomics Society, 49(6), 1005–1018.
Yao, Y. J., Chang, Y. M., Xie, X. P., Cao, X. S., Sun, X. Q., & Wu, Y. H. (2008). Heart rate and respiration responses to real traffic pattern flight. Applied Psychophysiology and Biofeedback, 33(4), 203–209.
Acknowledgements
The human subjects experiments in this study were conducted under an approved IRB protocol #11-19-11. Authors thank Naiqian Zhi in her assistance with the literature review. This work is supported in part by a DARPA Young Faculty Award #N66001-11-1-4161. The content of this research does not necessarily reflect the viewpoints of the funding agency, and no official endorsement of the US Government should be inferred. RS acknowledges fruitful discussions on the topic with Professor Maurizio Porfiri (NYU Tandon School of Engineering), Paul de la Houssaye (SPAWAR), and Gill Pratt (formerly Program Manager at DARPA, currently at Toyota Research Institute). De-identified data pertaining to this study can be accessed at https://figshare.com/s/f99e1e47d65cbdc17338.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
Authors declare that they do not have any competing interests with the conducted research.
Rights and permissions
About this article
Cite this article
Parsinejad, P., Sipahi, R. Analysis of Subjects’ Vulnerability in a Touch Screen Game Using Behavioral Metrics. Appl Psychophysiol Biofeedback 42, 269–282 (2017). https://doi.org/10.1007/s10484-017-9374-0
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10484-017-9374-0