Measuring Stress in an Augmented Training Environment: Approaches and Applications
- 1.7k Downloads
Augmented reality (AR) and virtual reality (VR) training systems provide an opportunity to place learners in high stress conditions that are impossible in real life due to safety risks or the associated costs. Using physiological classifiers it is possible to continually measure the stress levels of learners within AR and VR training environments to adapt training based on their responses. This paper reviews stress measurement approaches, outlines an adaptive stress training model that can be applied to augment training and describes key characteristics and future research that is critical to realizing adaptive VR and AR training platforms that take into account learner stress levels.
KeywordsAdaptive training Objective stress measurement Training fidelity evaluation High-stress training Augmented Reality
One goal of Augmented Reality (AR) training is to inject virtual objects and events into a live environment to allow trainees to acquire the targeted knowledge, skills, and abilities in a highly realistic environment. Two primary reasons to apply AR within a training program are to reduce the cost associated with live training entities and to create stressful conditions that could cause a significant level of risk if completed live. The goals and benefits of Virtual Reality (VR) training are a similar balance of creating highly realistic and potentially stressful conditions while controlling the risk and cost of training. The process of inducing stress within a controlled training environment is important when preparing learners to perform for high-stress conditions. Alternatively, there is utility in increasing levels of perceived stress and associated arousal within AR and VR environments in order to engage learners.
A variety of methods of inducing stress during training have been developed and evaluated [1, 2, 3, 4] and resilience training and stress inoculation programs rely on those applications to elicit target states during training. For example, the U.S. Air Force’s Stress Inoculation Training (SIT) program trains battlefield airmen to first understand the negative effects of stress, learn to detect and control stress responses, and finally practice skills taught under realistic high stress conditions . Guidance followed during this program suggests that stressor intensity should be incrementally increased as task proficiency is demonstrated . This approach ensures that trainees are not overwhelmed early which could interfere with skill acquisition while continuing to push learners operate under more and more realistic high-stress conditions . The greatest challenge with creating training that balances learner stress levels lies in the ability to track learner stress levels over time.
This paper presents an approach to objectively measure stress levels during AR and VR training and reviews applications of the approach to optimize and evaluate training programs. The presented technology leverages a physiology-based stress classification algorithm that has the capability to objectively and unobtrusively capture real-time individualized stress data in a mobile environment with over 95 % accuracy. While originally designed to support Veteran mental health therapy, the algorithm, which uses a wrist-worn device to collect human physiology state provides a continuous realtime stress measure that can be used to optimize AR and VR training. The approach provides a more effective option to alternatives, including subjective stress ratings or cortisol analysis post-training.
Two key applications of the stress classifier are presented in this paper. The first application is the creation of adaptive AR and VR training systems that adjust stressors that are presented to learners based on their state and performance. The goal of this application is to balance stress with training progression during AR and VR training exercises. The second application included is the objective evaluation of AR and VR training event fidelity based on objective trainee responses, with the goal of optimizing future training events. In addition to reviewing frameworks to support the application of stress evaluation in AR and VR environments, a comparative evaluation of stress evaluation techniques is presented.
2 Measuring Stress
The most important and difficult step in creating a training platform that can adapt based on trainee stress is the effective evaluation of stress. Early measurements of psychological stress have relied upon written scales that leverage a battery of validated questions to elicit a rating of stress for each individual. These scales were initially developed to support clinical psychologists and fall within two general categories of measures: Measures of stressful life events that have occurred; and measures of the subjective evaluation of perceived stress and/or ability to cope with stress.
The use of life event scales date back to Holmes and Rahe’s 1967 Schedule of Recent Experiences (SRE) which includes a checklist of 43 stressful life events such as the death of a spouse, divorce, or being fired from work . Each of the life events that were experienced on that scale are counted and the total is used to provide a relative stress level for the person being evaluated. Since the inception of the SRE, longer batteries and a variety of specialized batteries have been developed for special groups such as children  or combatants within the Gulf War . Because the scales that are used require a level of customization to the events that occur within the targeted population, this approach to measuring stress has the potential to lack consistency or sensitivity.
A second approach to measuring stress involves the use of perceived stress measures. The most commonly used perceived stress measure is the Perceived Stress Scale (PSS) . The PSS includes questions that measure of how unpredictable, uncontrollable, and overloaded respondents lives are in combination with direct evaluations of experienced stress levels and is available in a four, ten, and fourteen item batteries. Although perceived stress measures are more generalizable than life event scales and can be gathered more efficiently using the four item form, they still require some level of self-evaluation and cannot be continuously gathered without breaking the flow of training.
2.1 Physiological Stress Monitoring
Physiological states, features, and sources of noise for stress classification
Causes of noise
Skin conductance level, phasic skin conductance response
Environmental and skin temperature, humidity
Heart rate, heart rate variability
Physical movement and performance
Breathing rate, breathing depth, respiratory sinus arrhythmia
Physical movement and performance
3 Leveraging Stress Measures to Improve Adaptive Training
Two of the key characteristics that are evaluated in AR and VR training applications are immersion and presence. Although throughout the history of VR and AR development there have been various interpretations of the terms, immersion generally refers to the level of fidelity of the sensory cues used within the environment while presence is a subjective response of the person experiencing the environment . By this definition, it is apparent that fidelity can be objectively evaluated by comparing the visual, auditory, haptic, and olfactory cues present in a VR or AR environment to those in the live environment. Presence is a more difficult to objectively measure construct as it relies on the evaluation of the person experiencing the environment. To target this evaluation and move away from subjective ratings or presence, physiological data, including those that are described above to measure stress (e.g. change in heart rate and skin conductance) have been studies and demonstrated a correlation with presence measures over 15 years ago [21, 22].
In addition to measuring the sense of realism that VR and AR systems can instill, the use of physiological stress measures in combination with real-time performance can be used to drive adaptive training. Adaptive training has the potential to significantly optimize training efficiency  and based on the varied response of learners to stress exposure training, the process of adapting training based on individual responses to stressors holds even more potential. Originally designed as a clinical intervention for patients that required coping mechanisms for conditions like anger and phobic reactions, the goal of stress exposure training is to alleviate the negative effects of stress on performance by preparing personnel to perform tasks effectively under high-demand, high-stress conditions .
Virtual training scenarios must continually be adapted to add various types of stress within the environment in order for trainees to develop coping mechanisms.
In order ensure effective transfer of the coping strategies to the live environment, stress conditions must be similar to those found in the live environment.
Because the goal of stress exposure training is to ensure effective task performance under stress, training must take into account the real-time performance of trainees to ensure that use of the system does not lead to negative training.
Performance Evaluation- The ultimate goal of training is to improve learner performance. Although stress exposure training has the secondary goal of training performance under high stress conditions, optimizing performance must remain as the core goal of training. In order to ensure that negative training is avoided when inducing stress, it is critical to take into account trainee performance on targeted training objectives prior to evaluating learner stress states. The goal of the evaluation is for learner performance to remain high (or bounce back when performance drops) when new stressors are presented within the training scenario. For this to effectively work in a closed adaptive system, the performance evaluation metrics should be coded within the AR/VR training environment.
Stress State Evaluation- Physiological stress monitoring provides a real-time and objective method to continuously track stress and the response to stressors within the AR/VR training environment. A wrist-worn stress monitoring approach similar to the one described above has the potential to evaluate stress when learners are in an environment that allows them to move around, such as an AR training space.
Trainee State Classification- Once trainee performance and stress state is calculated, it’s important to merge the data to support the determination of overall trainee state and the appropriate response of the training system to meet the targeted training goals. For example, the combination of good performance and a low stress state should be used to trigger a new stressor within the scenario while a combination of high stress and low performance should be used to reduce/remove the stressors present to allow trainee performance to normalize and avoid negative training.
Stressor Activation/Deactivation Methods- In order for scenario adaptations to be applied based on performance and stress states, it is necessary to include hooks into the code to activate and deactivate stressors in real time. To meet this need, modular stressors that can be activated in real-time (e.g. reducing visual acuity of the scene, integrating additional enemies, etc.) must be designed and scripts developed to trigger them based on the evaluated trainee state.
By applying a closed adaptive training cycle that can modify AR and VR training scenarios in real-time it is possible to create an ever-changing environment that pushes learners to create coping mechanisms for environmental stressors that they are expected to operate during the presentation of in a live transfer environment. The approach allows a variety of potential stressors to be presented in close succession and only continuing the focus on those that require additional coping strategies. If this approach is scaled to groups of learners, it is possible to garner knowledge regarding the effectiveness of each stressor to trigger targeted stress states on a more generalizable scale. This capability to evaluate stressor effectiveness is a second key benefit of objectively measuring stress within AR and VR training environments. By measuring the response to cues across trainees, it is possible to classify or order stressors based on their effectiveness of created targeted stress states. By evaluating additional characteristics of each trainee during this evaluation, it is possible to further classify the effectiveness of various stressor cues based on trainee characteristics such as expertise level or past experience with particular cues/conditions in a live environment. This subclassification of training stressor effectiveness has the potential to optimize the order and presentation of scenario stressors based on each trainee’s specific experiences.
Cue activation tracking- Within an AR environment stress induction cues can be triggered live or as an augmented component of the scenario. In order to evaluate the effects of cues and combination of cues accurately, a tracking system must be instantiated to track when live and augmented cues are triggered.
Trainee characteristics- By tracking a database of trainee characteristics in addition to the stress response to various stressors as they are presented, it is possible to create a more precise prediction of the cues that will be responded to in similar ways during future training.
The approach of leveraging physiological data in combination with performance data to drive adaptive training augmentation has demonstrated significant value in previous research. For example, research has suggested that skin conductance response (SCR) as a measure of arousal can be used in combination with a contextual understanding of the training tasks that learners are completing to predict learning gains and modify adaptive tutoring systems . Further research conducted at the Army Research Lab led to the development of an architecture, similar to that presented in Fig. 3, of personalized adaptive training that leverages a combination of trainee physiological state and performance to drive training adaptations .
4 Future Research
The research into adaptive training platforms that leverage physiological stress states to drive training augmentation shows potential to optimize training. In order to meet the potential of the approach, additional research is needed to improve stress classification and the application of stress evaluations in the adaptive training domain. Two particular areas that require additional research include the application of deep understanding of the effects of stress on learning and memories and the development of personalized measures of stress and ruggedized sensor arrays that further improve stress classification accuracy.
A second avenue of research that is critical to improve the capabilities of VE and AR training systems to support adaptive stress training is the enhancement of classifiers of stress that are used to drive adaptations. Specifically, there is a need to develop approaches and systems to automate the development of personalized classifiers of stress and systems to evaluate the effects of stress training across studies and as learners move to a transfer environment. Finally, to meet this need, there is a need to develop new noninvasive hardware platforms that can measure the core features that can be used to classify stress while accounting for the noise that is associated with measuring stress in unpredictable environments. By continuing research in these domains, it will be possible to merge the power of adaptive training within AR and VR environments and objective stress measurement to better prepare people for high risk and high stress jobs and potentially reduce the impact of negative stress on people in those positions.
- 2.Carroll, M., Hale, K., Stanney, K., Woodman, M., DeVore, L., Squire, P., Sciarini, L.: Framework for training adaptable and stress-resilient decision making. In: Interservice Training, Simulation, and Education Conference (I/ITSEC) (2012)Google Scholar
- 3.Jones, D., Hale, K., Dechmerowski, S., Fouad, H.: Creating adaptive emotional experience during VE training. In: Interservice Training, Simulation, and Education Conference (I/ITSEC) (2012)Google Scholar
- 4.Saunders, T., Driskell, J., Hall, J., Salas, E.: The effect of stress inoculation training on anxiety and performance. ARI Research Note, pp. 96–27 (1996)Google Scholar
- 5.Robson, S., Manacapilli, T.: Enhancing Performance Under Stress: Stress Inoculation Training for Battlefield Airmen. RAND Corporation, Santa Monica (2014)Google Scholar
- 6.Keinen, G., Friedland, N.: Training effective performance under stress: queries, dilemmas, and possible solutions. In: Driskell, J.E., Salas, E. (eds.) Stress and Human Performance, pp. 257–277. Erlbaum, Mahwah (1996)Google Scholar
- 12.Sun, F.T., Kuo, C., Cheng, H.T., Buthpitiya, S., Collins, P., Griss, M.: Activity-aware mental stress detection using physiological sensors. In: Gris, M., Yang, G. (eds.) Mobile Computing, Applications, and Services. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol. 76, pp. 211–230. Springer, Heidelberg (2012)CrossRefGoogle Scholar
- 13.Everly, G.S., Lating, J.M.: The anatomy and physiology of the human stress response. In: A Clinical Guide to the Treatment of the Human Stress Response, pp. 17–51. Springer, Heidelberg (2013)Google Scholar
- 14.Plarre, K., Raij, A., Hossain, S.M., Ali, A.A., Nakajima, M., al’ Absi, M., Ertin, E., Kamarck, T., Kumar, S., Scott, M., Siewiorek, D., Smailagic, A., Wittmers, Jr., L.E.: Continuous inference of psychological stress from sensory measurements collected in the natural environment. In: 2011 10th International Conference on Information Processing in Sensor Networks (IPSN), pp. 97–108. IEEE (2011)Google Scholar
- 15.de Santos Sierra, A., Ávila, C.S., Casanova, J.G., del Pozo, G.B.: Real-time stress detection by means of physiological signals. Group of Biometrics, Biosignals and Security Universidad Politécnica de Madrid, Spain, pp. 24–44 (2011)Google Scholar
- 16.Alamudun, F., Choi, J., Gutierrez-Osuna, R., Khan, H., Ahmed, B.: Removal of subject-dependent and activity-dependent variation in physiological measures of stress. In: 2012 6th International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth), pp. 115–122. IEEE (2012)Google Scholar
- 17.Bakker, J., Pechenizkiy, M., Sidorova, N.: What’s your current stress level? Detection of stress patterns from GSR sensor data. In: 2011 IEEE 11th International Conference on Data Mining Workshops (ICDMW), pp. 573–580. IEEE (2011)Google Scholar
- 19.Dechmeroswski, S., Winslow, B., Chadderdon, G., Schmidt-Daly, T.N., Jones, D.: Classifying stress in a mobile environment. In: Interservice Training, Simulation, and Education Conference (I/ITSEC) (2014)Google Scholar
- 20.Slater, M.: A note on presence terminology. Presence-Connect 3, 1–5 (2003)Google Scholar
- 21.Meehan, M.: Physiological reaction as an objective measure of presence in virtual environments. Doctoral Dissertation, University of North Carolina at Chapel Hill (2001)Google Scholar
- 22.Wiederhold, B.K., Jang, D.P., Kaneda, M., Cabral, I., Lurie, Y., May, T., Kim, I.Y., Wiederhold, M.D., Kim, S.I.: An investigation into physiological responses in virtual environments: an objective measurement of presence. In: Riva, G., Galimberti, C. (eds.) Towards CyberPsychology: Mind, Cognitions and Society in the Internet Age. IOS Press, Amsterdam (2001)Google Scholar
- 25.Scherer, K., Shorr, A., Johnstone, T.: Appraisal Process in Emotion: Theory, Methods, Research. Oxford University Press, Canary (2001)Google Scholar
- 26.Hardy, M., Wiebe, E.N., Grafsgaard, J.F., Boyer, K.E., Lester, J.C.: Physiological responses to events during training use of skin conductance to inform future adaptive learning systems. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, pp. 2101–2105 (2013)Google Scholar
- 27.Sottilare, R.A., Gilbert, S.: Considerations for adaptive tutoring within serious games: authoring cognitive models and game interfaces. Presentation at the authoring simulation and game-based intelligent tutoring workshop at the fifteenth conference on artificial intelligence in education, Auckland (2011)Google Scholar
- 29.Diamond, D.M., Campbell, A.M., Park, C.R., Halonen, J., Zoladz, P.R.: The temporal dynamics model of emotional memory processing: a synthesis on the neurobiological basis of stress-induced amnesia, flashbulb and traumatic memories, and the Yerkes-Dodson law. Neural Plast. 2007, 1–33 (2007)CrossRefGoogle Scholar