Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

Multimodal PTSD characterization via the StartleMart game


Computer games have recently shown promise as a diagnostic and treatment tool for psychiatric rehabilitation. This paper examines the potential of combining multiple modalities for detecting affective responses of patients interacting with a simulation built on game technology, aimed at the treatment of mental diagnoses such as post traumatic stress disorder (PTSD). For that purpose, we couple game design and game technology to create a game-based tool for exposure therapy and stress inoculation training that utilizes stress detection for the automatic profiling and potential personalization of PTSD treatments. The PTSD treatment game we designed forces the player to go through various stressful experiences while a stress detection mechanism profiles the severity and type of PTSD by analyzing the physiological responses to those in-game stress elicitors in two separate modalities: skin conductance (SC) and blood volume pulse (BVP). SC is often used to monitor stress as it is connected to the activation of the sympathetic nervous system (SNS). By including BVP into the model we introduce information about para-sympathetic activation, which offers a more complete view of the psycho-physiological experience of the player; in addition, as BVP is also modulated by SNS, a multimodal model should be more robust to changes in each modality due to particular drugs or day-to-day bodily changes. Overall, the study and analysis of 14 PTSD-diagnosed veteran soldiers presented in this paper reveals correspondence between diagnostic standard measures of PTSD severity and SC and BVP responsiveness and feature combinations thereof. The study also reveals that these features are significantly correlated with subjective evaluations of the stressfulness of experiences, represented as pairwise preferences. More importantly, the results presented here demonstrate that using the modalities of SC and BVP captures a more nuanced representation of player stress responses than using SC alone. We conclude that the results support the use of the simulation as a relevant treatment tool for stress inoculation training, and suggest the feasibility of using such a tool to profile PTSD patients. The use of multiple modalities appears to be key for an accurate profiling, although further research and analysis are required to identify the most relevant physiological features for capturing user stress.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3


  1. 1.



  1. 1.

    Acharya UR, Joseph KP, Kannathal N, Lim CM, Suri JS (2006) Heart rate variability: a review. Med Biol Eng Comput 44(12):1031–1051

  2. 2.

    Andreassi JL (2000) Psychophysiology: human behavior and physiological response. Routledge, London

  3. 3.

    Benedek M, Kaernbach C (2010) Decomposition of skin conductance data by means of nonnegative deconvolution. Psychophysiology 47(4):647–658

  4. 4.

    Blanchard EB, Jones-Alexander J, Buckley TC, Forneris CA (1996) Psychometric properties of the PTSD checklist (PCL). Behav Res Ther 34(8):669–673

  5. 5.

    Blechert J et al (2007) Autonomic and respiratory characteristics of posttraumatic stress disorder and panic disorder. Psychosom Med 69(9):935–943

  6. 6.

    Boucsein W (2011) Electrodermal activity. Springer, New York

  7. 7.

    Calvo RA, D’Mello S (2010) Affect detection: an interdisciplinary review of models, methods, and their applications. IEEE Trans Affect Comput 1(1):18–37

  8. 8.

    First MB et al (2002) Structured clinical interview for DSM-IV-TR axis I disorders, research version, patient edition (SCID-I/P). Biometrics Research Institute, New York

  9. 9.

    Foa EB et al (2009) Effective treatments for PTSD, practice guidelines from the International Society for Traumatic Stress Studies, 2nd edn. Guilford Press, New York

  10. 10.

    Goldberger J, Challapalli S, Tung R, Parker M, Kadish A (2001) Relationship of heart rate variability to parasympathetic effect. Circulation 103(15):1977

  11. 11.

    Healey JA, Picard RW (2005) Detecting stress during real-world driving tasks using physiological sensors. IEEE ITS 6(2):156–166

  12. 12.

    Hernandez J, Morris RR, Picard RW (2011) Call center stress recognition with person-specific models. In: ACII. Springer, New York, pp 125–134

  13. 13.

    Hoge CW et al (2004) Combat duty in Iraq and Afghanistan, mental health problems, and barriers to care. N Engl J Med 351(1):13–22

  14. 14.

    Holmgård C, Yannakakis GN, Karstoft KI, Andersen HS (2013) Stress detection for ptsd via the startlemart game. In: IEEE ACII. Springer, New York, pp 523–528

  15. 15.

    Kashdan TB et al (2010) Everyday strivings in war veterans with posttraumatic stress disorder: suffering from a hyper-focus on avoidance and emotion regulation. Behav Ther 41(3):350–363

  16. 16.

    Kemp AH, Quintana DS, Gray MA, Felmingham KL, Brown K, Gatt JM (2010) Impact of depression and antidepressant treatment on heart rate variability: a review and meta-analysis. Biol Psychiatry 67(11):1067–1074

  17. 17.

    Kendall M (1970) Rank correlation methods, 4th edn. Griffin, London

  18. 18.

    Malik M et al (1996) Heart rate variability: standards of measurement, physiological interpretation and clinical use. Task Force of the European Society of Cardiology and the North American Society of Pacing and Electrophysiology. Circulation 93(5):1043–65

  19. 19.

    Magruder KM, Frueh BC, Knapp RG, Johnson MR, Vaughan JA, Carson TC, Powell DA, Hebert R (2004) PTSD symptoms, demographic characteristics, and functional status among veterans treated in VA primary care clinics. J Trauma Stress 17(4):293–301

  20. 20.

    Martínez HP, Garbarino M, Yannakakis GN (2011) Generic physiological features as predictors of player experience. In: ACII. Springer, New York, pp 267–276

  21. 21.

    Martinez HP, Bengio Y, Yannakakis GN (2013) Learning deep physiological models of affect. IEEE Comput Intell Mag 8(2):20–33

  22. 22.

    Moody G (1993) Spectral analysis of heart rate without resampling. In: IEEE proceedings of the computers in cardiology, pp 715–718

  23. 23.

    Perala C (2007) Galvanic skin response as a measure of soldier stress. Tech. rep, DTIC document

  24. 24.

    Picard R, Vyzas E, Healey J (2001) Toward machine emotional intelligence: analysis of affective physiological state. IEEE Trans Pattern Anal Mach Intell 23(10):1175–1191

  25. 25.

    Picard RW, Vyzas E, Healey J (2001) Toward machine emotional intelligence: analysis of affective physiological state. IEEE PAMI 23(10):1175–1191

  26. 26.

    Pole N (2007) The psychophysiology of posttraumatic stress disorder: a meta-analysis. Psychol Bull 133(5):725

  27. 27.

    Popović S, Horvat M, Kukolja D, Dropuljić B, Cosić K (2009) Stress inoculation training supported by physiology-driven adaptive virtual reality stimulation. Stud Health Technol Inform 144:50–54. doi:10.3233/978-1-60750-017-9-50

  28. 28.

    Robinson MD, Clore GL (2002) Belief and feeling: ervidence for an accessibility model of emotional self-report. Psychol Bull 128(6):934

  29. 29.

    Siepmann M et al (2003) Effects of sertraline on autonomic and cognitive functions in healthy volunteers. Psychopharmacology 168(3):293–298

  30. 30.

    Tognetti S et al (2011) The affective triad: stimuli, questionnaires, and measurements. In: ACII. Springer, New York, pp 101–110

  31. 31.

    Wolpe J (1973) The practice of behavior therapy. Pergamon Press, New York

  32. 32.

    Wood DP et al (2011) Reality graded exposure therapy with physiological monitoring for the treatment of combat related post traumatic stress disorder: a pilot study. Stud Health Technol Inform 163:696

  33. 33.

    Yannakakis G, Hallam J (2008) Entertainment modeling through physiology in physical play. Int J Hum Comput Stud 66(10):741–755

  34. 34.

    Yannakakis GN, Hallam J (2008) Entertainment modeling through physiology in physical play. Int J Hum Comput Stud 66(10):741–755

  35. 35.

    Yannakakis GN, Hallam J (2008) Entertainment modeling through physiology in physical play. Int J Hum Comput Stud 66(10):741–755

  36. 36.

    Yannakakis GN, Hallam J (2011) Ranking vs. preference: a comparative study of self-reporting. In: ACII. Springer, New York, pp 437–446

  37. 37.

    Zhai J et al (2005) Realization of stress detection using psychophysiological signals for improvement of human–computer interactions. In: Proceedings of the IEEE SoutheastCon. IEEE, pp 415–420

Download references


This research was supported by the Danish Council for Technology and Innovation under the Games for Health project and by the FP7 ICT project ILearnRW (Project No: 318803).

Author information

Correspondence to Christoffer Holmgård.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Holmgård, C., Yannakakis, G.N., Martínez, H.P. et al. Multimodal PTSD characterization via the StartleMart game. J Multimodal User Interfaces 9, 3–15 (2015). https://doi.org/10.1007/s12193-014-0160-5

Download citation


  • Stress detection
  • Post traumatic stress disorder
  • Games for health
  • User profiling