Keywords

1 Introduction

Virtual Reality (VR) provides a valuable platform for immersive training experiences, especially for high-stress professions like first responders where mistakes can cost lives [6]. The training of police first responders specifically has been the subject of continuous research (see e.g., [7]). Individual, contextual, societal, and organizational factors influence decision-making and acting in the field, making training conceptualizations a highly sophisticated endeavor. This makes it a particularly interesting interaction context for human-computer interaction (HCI) research.

The benefits of VR training include lower costs and higher safety compared to real-life training (see e.g., [10]). It also enables interactions in simulated contexts with high immersion and a sense of presence [11]. Unlike in real-life simulation training, VR also allows for the inclusion of vulnerable groups (e.g., children, elderly people) or dangerous equipment (e.g., explosive materials) [14].

However, to produce an ecologically valid VR training experience for police officers who have to regularly perform under stress, stress needs to be inducible. This can be achieved by augmenting the VR training scenario with stressors that add complexity to the scenario [12]. This allows for personalized training based on the trainee’s learning goals, pace, needs, and time constraints. Stressors have been used in VR training, e.g., for stress inoculation training [24]. However, to the best of our knowledge, there is little to no research investigating how to operationalize stressors, i.e., how to transform ‘descriptive’ stressors into concrete, measurable, observable, and implementable elements in VR. Furthermore, there is little research on designing interaction concepts for trainers to easily affect the stress levels of trainees in VR training by using stressors.

Thus, our main goal is to investigate how the training experience of both trainer and trainee can be enhanced through the implementation of stressors in the complex context of police VR training. For this, we will address two research questions: (1) How can known real-world stressors be translated into audio-visual stress cues in VR training environments? (2) How can the overall concept for trainers interacting with the implementation of stressors in VR look like?

We followed a co-creation approach that iteratively involved stakeholders in the research and development process as experts of their professions and experiences [23]. This was done to gain insights into the complex world of police work and training and to facilitate a collaborative environment that can alleviate potential power imbalances (see e.g. [5]) that can exist in hierarchical and highly specialized structures like police organizations [19]. As its main contribution, the paper presents the concept of ‘stress cues’ as the operationalization of descriptive stressors into concrete, observable and implementable elements in VR to improve the training experience for both trainers and trainees. We also provide first ideas for how trainers could interact with stress cues during VR training with the help of a research prototype.

2 Related Work

2.1 Stressors in Virtual Reality Training of Police First Responders

First responder personnel is regularly put under significant stress due to threats to their psychological and physical wellbeing [13]. Stress can be defined as “any physical, cognitive or emotional reaction that causes physiological or mental tension and that may result in physical or emotional impairment” [2, p. 55] while stressors can be considered “internal or external demands imposed on or inherent” [16, p. 80] to the police officer or trainee. One way to support first responders is to provide operational training under stress to prepare for stressful situations and increase resilience. To do so, stressors for first responders need to be a) identified and b) operationalized for VR.

So far, police research has focused on identifying stressors that contribute to overall work stress in first responders (e.g., [1, 18]) rather than on acute situational stressors in the specific context of interventions in the field. Additionally, stressors that occur in the field are often formulated broadly, e.g., by the type of intervention (e.g., armed robbery) or refer to more general events or ‘critical incidents’ (e.g., officer-involved shooting). These indicate which types of interventions lead to general work stress, but do not indicate which stressors lead to acute stress during specific police interventions.

In virtual training under stress, stressors are used to achieve higher ecological validity. However, the concept remains underspecified for stressors in VR as well. In one review in the context of military stress training, stressors can take physical (e.g., lack of sleep, dehydration) and psychological forms (e.g., information overload) [17]. Another study on the selection of stressors in VR to train stress management skills, describes stressors only as generally ‘stressful situations’ [3]. Overall, there is little to no research on the transformation of stressors into concrete, observable and implementable elements for VR training. Particularly, there are no concepts on how to best design interactions for trainers for them to easily and quickly adjust stress levels in VR training with stressors to achieve a better training experience for trainees.

2.2 Interaction Design for Virtual Training Experiences

In VR training, trainees must physically and mentally engage with the training simulation. Especially in police training, trainees must demonstrate their ability to act quickly, decisively, and professionally in various high-risk scenarios. To increase the ecological validity of the training experience, real equipment like pepper spray, tasers, and firearms are virtualized as tangible, functional objects which the trainee can handle just as in real training [22]. While the trainees are immersed in the training scenarios, the trainers supervise the training from outside the VR. This context requires a simple interaction and interface design that makes it possible to observe the training and the trainees’ response and to adapt the scenario quickly and with little mental effort.

Monitoring and supervisory intervention can be compared with contexts from flight monitoring, control centers, and decision support systems. But especially the training context provides a special challenge for HCI research because the digitalization and use of software represent a new application domain for trainers. While there is, to the best of our knowledge, no research yet on interaction design specifically for trainers, first approaches can be found for general training experience (e.g., [8, 25]).

2.3 Co-creation

For the development of new technologies, researchers in human-centered design argue that it can be highly beneficial to include the knowledge and opinions of people as experts of their professions, experiences, and lives [21]. As co-creation uses tools and techniques that engender people’s creativity [20], it can aid in obtaining people’s tacit knowledge which cannot be accessed easily through other means. This might be especially valuable in special contexts like law enforcement where successful work relies heavily on procedural and tacit knowledge. Furthermore, co-creative methods might be able to engender an equitable, collaborative environment that could equalize potential power imbalances that exist in police organizations [5] where hierarchical and highly specialized structures exist [19].

3 The Problem Context: The SHOTPROS Project

SHOTPROS investigates the influence of human factors (HFs) on decision-making and acting (DMA) of police officers under high-stress and in high-risk operational situations. The aim is to develop a training framework and a corresponding VR system concerned with improving the performance of DMA in high-stress and high-risk situations.

3.1 Used Methods for Developing the Stress Cue Concept

We organized six co-creation workshops at six law enforcement agencies (LEAs) across Europe. 60 police officers and trainers participated in the 1.5-day workshops. Several co-creation tools (e.g. brainstorming and co-designing with LEGO™-like building blocks) were used to facilitate active and creative participation. As suggested by previous research (see [5]), this helps to create a collaborative atmosphere, mitigate power-related biases in the structure of hierarchical, highly bureaucratized police organizations [19], and to yield participants’ tacit knowledge. The participants generated possible HFs influencing stress of police first responders and developed VR scenarios for training DMA in stressful, high-risk situations. Based on the findings from the workshops and an additional co-creation session with representatives of all partner LEAs in SHOTPROS, a list of 40 stressors was created. The list items were ranked according to priority by one LEA trainer expert from each organization through an online survey. The ranked items were then used to iteratively develop the stress cues.

Further, trainers from all LEAs participated in its development by providing their needs and expertise for the training experience through online co-creation and feedback rounds. These interaction design ideas were then included in the VR prototype.

3.2 The Stress Cue Concept

As described earlier, a research gap currently exists on how to operationalize contextual factors that induce stress (‘stressors’) for training in VR and how to design interaction concepts describing how trainers can use these elements. So far, descriptions of stressors remain general which makes it difficult to implement them in interaction concepts. To close this gap, we introduce the concept of stress cues which operationalize descriptive stressors into concrete, observable and implementable elements in VR to improve the training experience for trainers and trainees. The concept also entails the following components: (a) A stress cue repository, (b) the interaction design for controlling and injecting the selected stress cues in the VE via a live editor, (c) a real-time stress measurement dashboard to evaluate and visualize stress cue effects on the trainee and (d) an after-action review (AAR) dashboard. In the following, each of the four components of the concept of stress cues will be explained.

Creation of Stress Cues in a Stress Cue Repository.

Table 1 shows the ten collected stressors rated most relevant for inducing stress. A selection of these stress cues was implemented as audio-visual stimuli, i.e., as animated 3D objects with sound, into the VR prototype to visualize the interactive stress cue concept (see Fig. 1). The prototype will be used to collect feedback from end users, evaluate the concept of stress cues, and provide a baseline for co-creation with the end users on future design instantiations.

Table 1. Top 10 stressors elaborated and voted by the LEA partners.
Full size table

Interaction Design for the Stress Cues: The Stress Cue Live Editor.

Following the implementation of stressors as stress cues into VR, one of the most important aspects to improve the training experience is to consider how the trainer could interact with and use the stress cues during training. Here, we propose a live editor where stress cues can be selected to influence the scenario in real-time with three interaction possibilities: (1) single selection to add a stress cue to the VR training scenario, (2) selection of multiple stress cues in parallel, and (3) generation of a stress cue sequence. For each stress cue, the characteristics and intensity can be defined (e.g., adjusting music volume or the level of aggressiveness of the dog). After selecting the manifestations of the stress cues, they are directly applied to the actual VR scenario.

Real-time Stress Measurement Dashboard and After-Action Review Dashboard.

In the prototype, the stress level is determined through the trainee’s heart rate variability (HRV), a recognized indicator of stress [9], with a Zephyr™ bio-harness which provides reliable and valid measurements of heart rate [15]. The stress level is displayed in the Stress Cue Live Editor, allowing trainers to get immediate feedback on the effect of the selected stress cues on the trainee’s state. Through the integrated real-time stress measurement, the effect of single stress cues and cascading effects of stress cues combinations can be evaluated. This can help trainers in choosing augmentations for the scenario, improving training effectiveness. The after-action review (AAR) dashboard displays visualizations of the physiological stress level and behavioral trainee data through which the trainee’s performance can be replayed and analyzed after training.

3.3 The Stress Cue Prototype

The question of how the training experience can be improved in the complex context of police work with stress cues guided the development of the prototype. Based on design ideas and feedback in the co-creation process, we prototypically integrated ideas for trainer interactions with the stress cues into the VR training system developed in SHOTPROS (see Fig. 1). The end user requirements revealed that an efficient, effective interaction and easy-to-use user interface design is needed. It allows for observations of the training and trainees’ behavior and adaptations to the scenario by (de)activating stress cues rapidly with little mental effort.

Fig. 1.
figure 1

Stress cue live editor showing stress levels of trainees and selected stress cues and control panel to add (1) single stress cues, (2) stress cues in parallel, or (3) in a sequence. Screens based on existing software by the VR partner RE-liON in SHOTPROS.

Full size image

At the top of the stress cue live editor (Fig. 1), the scenario execution control is shown at the ‘exercise execution’ stage. The main area is the 3D representation of two trainees in the VE, marked with symbols indicating position and stress level. Obstacles like walls are disabled for a better view of the trainees. On the left, the list of selected stress cues, the panel for stress cue control, color-coded stress levels of the trainees, and a legend depicts for the stress levels are depicted. The design meets the consulted trainers’ requirements to keep the scenario display central and only use a border area for the live editor. Stress cues can be activated to induce immediate stress in trainees at any time individually by single play buttons for instant playback or in a sequence through ‘play sequence’ with an additional selection of the start and end on the timeline via sliders. Simultaneous playback of all stress cues is possible by pressing all play buttons in quick succession or by defining and playing a parallel sequence.

Fig. 2.
figure 2

After-action review dashboard for analyzing stress responses during the appearance of stress cues for trainees exposed to stress cues.

Full size image

In the AAR dashboard (Fig. 2), the trainees’ data is displayed. The response to the stress cues can be analyzed via a time diagram where stress cues and their active period are shown to align stress cues to stress responses. All (see Fig. 2, trainee 1) or selected stress cues (see Fig. 2, trainee 2, barking dog only) can be displayed. The level of stress can be viewed on the y-axis and is color-coded: green for normal, yellow for increased, orange for high, and red for very high stress.

4 Discussion and Future Work

This paper has described how a proposed operationalization of stress cues will enhance the training experience in the context of police work where contextual, individual, societal, and organizational factors interact with and influence decision-making and acting in the field, making training highly important but difficult to conceptualize. The co-creation process was described through which the stress cue concept was developed (research question 1). It showed how stressors in operative police work were translated into audio-visual stimuli and implemented in a VR prototype based on several iterations of co-creation and interaction design development. We suggest that the effective implementation of stressors not only includes audio-visual stimuli (i.e., the stress cues themselves) but also a VR interaction concept for the trainer, real-time stress measurements, and an AAR dashboard to fully operationalize stressors by making them objectively measurable (research question 2). To validate the proposed implementation empirically, we will evaluate the VR prototype in upcoming studies.

Ecologically valid stress cues can be of high added value to the training experience of any first responder (e.g. fire rescue, medical emergency services) in VR by giving trainers the possibility to manipulate the VE to achieve a stress level that is effective to the training outcomes. They expand the trainer mode and provide more interaction possibilities to actively intervene in the training and dynamically change the interaction space of the trainee, thus improving the training experience of trainee and trainer.

To substantiate this claim, the ecological validity of single and combinations of stress cues must be tested, as well as the effect of stress cues in VR compared to real-life situations. In addition to HRV, we plan to integrate measurements such as breath rate, eye movement, and pupillometry data (see e.g., [4]). An algorithm will use these inputs to measure and visualize experienced stress and the impact of stress cues in trainees. Moreover, future research could move towards automated stress cue additions through artificial intelligence algorithms based on the trainee’s behavior and physiological stress response(s). Lastly, the developed stress cues are currently audio-visual stimuli only. Beyond that, additional stress inducement through olfactory and haptic feedback will be considered in future work, e.g. through vibrations of hand-held controllers or full-body VR suits.

5 Conclusion

This paper investigated how the training experience of LEA first responders can be improved to be more ecologically valid. Our research has shown that this can be achieved through the integration of stress cues in VR training. Therefore, we propose the translation of real-world stressors in the context of existing LEA first responder training into the concept of ‘stress cues’ in VEs. The use of stress cues could be especially beneficial in VR training because it provides various interaction possibilities for trainers. Stress cues are easily adaptable, manipulable, and controllable and thus allow for a personalization of training based on an individual trainee’s training goals, capabilities, and needs. Training enhanced through ecologically valid stress cues will allow trainees to learn and train how to mitigate the effects of stress responses repeatedly with interchangeable, customizable stress cues. We see this approach as part of a future interaction design approach, as it aims to combine human experiences, interactions, and behavior with several human senses such as sight, hearing, but also smell, and touch.