Background

An essential component of the emergency department (ED) is to respond to disasters, infectious disease threats, and other extreme events. Responses to such events are increasingly hampered by increased visits and crowding in the face of decreasing numbers of EDs, beds and providers [1–3], among other factors. The impact of these global stresses is exacerbated when ED personnel are themselves at risk, as occurs with infectious diseases, especially during patient triage in the ED, before the patient is in isolation and appropriate personal protective equipment has been employed. To reduce this risk, hospitals have implemented rigorous infection control procedures that are followed to varying degrees [4].

In addition to personal risk, when an epidemic, earthquake, or other disaster threatens the homes and families of ED staff, it can affect their ability to cope with increased patient loads, their adherence to infectious disease protocols, and even their willingness to come to work [5, 6]. However, information about staff functioning during such events comes only from uncontrolled experience at the few sites at which the events have occurred. In order to determine the likely impact of unusual but potentially disastrous circumstances in order to to modify ED protocols accordingly, it would be helpful to develop simulated models of the ED that can be manipulated experimentally.

Computer simulations provide a tool for enhancing emergency preparedness by creating realistic visual representations of the various patient care challenges faced by emergency providers [7, 8]. Computer simulation is preferable to tabletop, mannequin and simulated live patient protocols because of decreased expense, lack of need to commit physical resources, ability to participate from off-site locations, and ease of reconfiguring a virtual ED to match the circumstance studied. In addition, virtual simulations can model the likely impact of different interventions without disrupting ongoing ED patient care [2, 9–11].

The most frequently used computerized ED model of emergency department patient flow is discrete event simulation (DES) [10], which is used to predict the effects of operational changes on patient throughput, waiting times, efficiency, length of stay, resource utilization and interaction of processes within a system [10, 12, 13]. An extension of DES is agent based modeling (ABM), which models behavior and its outcomes at the individual level [10]. A model using novel software to create a hierarchy of heterogeneous pseudo-agents has been used to represent patients moving through the emergency department during triage, evaluation by a physician, diagnostics, and treatment [10]. The main use of this model has been to develop optimal staffing models for different patient populations.

These computer simulations often focus on a specific factor, but addressing multiple systems that are impacted at the same time may be more realistic [14]. Virtual reality is a computerized model that expands the ability to model multiple influences on interactions of healthcare workers with each other, with patients, and with their environment. In a virtual reality simulation, virtual representations for patients, healthcare workers and other individuals may be automated (robots or “bots”), or they may be actively directed by the actual person they represent, in which case they are avatars. Avatars may then interact with each other and with robots. Second Life is an open-access, multi-user, virtual environment that has been used to train students in various fields [9] and to model multiple casualties in the field and in an emergency department for training [15].

GaMeTT, which has been used for training a military emergency response group, is a 3D, interactive, avatar-based simulation designed to train on an internet platform, that increases a sense of involvement (presence) by participants [16]. Arrow keys and the mouse control avatar movements. Using this model, an online virtual reality model of an emergency room was populated with 10 virtual patients exposed to radiation and 10 exposed to a toxin [17]. Of 10 physicians and 12 nurses participating in the training, 2/3 felt immersed in the virtual model all or most of the time. After the training, the percentage of subjects who felt confident or very confident in managing these events increased from 18 to 86%, with the majority attributing improved confidence to the training.

Since computer simulations have largely been used for training, the degree to which they can be used in a research setting remains to be determined. Other than a single simulation used to test the effect of different numbers of staff on patient flow [1], studies of the effectiveness of computer simulation in predicting outcomes such as the impact on the ED and its staff of epidemics and other disasters that alter patient flow and composition are lacking. Using photographs of our primary emergency department and actual patient scenarios from our practice, we adapted CliniSpace, a novel virtual reality platform used primarily for training for emergency management of trauma, that has a larger range of interactive bots and avatars than have been used previously [18], to develop a model of an ED that could be used to empirically study the possible impacts of such events. Because performance on this (or any other) simulation has not been compared with the actual situations it represents, it was necessary to demonstrate that it could be used as a valid model of an important component of ED activity before we could investigate the effect of varying parameters that impact it. We chose the discrete task of patient triage because it could be readily compared to performance at the actual site, and because most nursing staff who perform triage also work in other ED activities.

Methods

Ethics approval and consent to participate

This study was approved by the University at Buffalo Health Sciences Institutional Review Board. Written informed consent was obtained from 10 Caucasian female ED nurses with a mean age of 51.1 years (range: 34–63). Subjects were recruited through fliers in two local hospitals, announcements at meetings of the local Emergency Nurses Association, and word-of-mouth. All subjects were currently working full- or part-time performing ED triage. Demographic data, nursing experience, and experience with video gaming and virtual reality, are summarized in Table 1.

Table 1 Nursing and gaming experience
Full size table

Questionnaires and scales

Experience of the simulation task was assessed with questions rated on Likert scales using open-ended questions, such as: “What was your experience like?”, “What would you change?”, and “Do you think this virtual world reflects your real world experience?” An analogue scale assessed subjects’ comfort level using the avatar in the simulation task from “0” (not at all comfortable) to “100” (extremely comfortable).

The NASA task load index (NASA TLX) [19–22] was used to obtain information about each subject’s subjective workload during both an average day in the ED, and the simulation task. The NASA TLX is a multi-dimensional scale that provides an overall workload score based on a weighted average of ratings on six subscales (mental demands needed to perform a task, physical demands of the task, temporal demands or feeling a time pressure, self-perceived success during performance, amount of effort put forth, and frustration during performance). Each subscale is rated from 0 to 100, with higher scores indicating higher perceived importance. The TLX has been widely used to assess workload in simulations as well as human–machine environments, such as aircraft cockpits and command, control, and communication workstations [21].

Simulation task

We used CliniSpace [18] to create a 3D computer rendering of the ED of a large urban general hospital (602 inpatient beds, 56,000 general ED and 12,000 psychiatric ED visits/year) that included an ambulance bay, waiting room reception desk, two triage rooms, and connecting hallways (Figs. 1, 2, 3). Standard triage equipment was provided within the environment. The simulation was preloaded with 16 virtual bot (automated) patients. Four of the patients were used to train subjects to navigate in the virtual environment, and the other 12 were used for testing. All patient scenarios represented experience in our ED and were developed using the emergency severity index (ESI) version 4, a triage tool that has been used by ED nursing personnel [23]. Table 2 describes the basic demographics of the 16 patients and their presenting medical conditions.

Fig. 1
figure 1

Lobby and reception

Full size image
Fig. 2
figure 2

Triage room

Full size image
Fig. 3
figure 3

Patient examination with examples of menu options and vital signs

Full size image
Table 2 Description of simulated patients and their presenting medical issues
Full size table

Procedure

The 3-h study consisted of orientation, testing, and debriefing phases. For the orientation phase, each subject was seated in front of a computer screen equipped with a mouse and keyboard and displaying the virtual triage room in order to learn navigating, interacting, and using objects in the simulation. To avoid potential novelty effects during testing, each task had to be satisfactorily completed before the subject could move on to the next task.

During the testing phase, which followed a 3-min break, subjects seated at the computer manipulated an avatar using arrow keys, beginning at the reception desk (Fig. 1) and navigating to the triage room of the subject’s choice (Fig. 2). The subject’s view was from the avatar’s perspective. Subjects were instructed to triage patients in the simulation just as they would in real life, in the order in which they usually prioritized patients, and to continue the triage process until instructed to stop. The order and timing of new patients presented to subjects remained consistent, but subjects decided which patient was seen next based on their assessment of priority. The simulation ended after each subject had triaged six patients. As is typical of triage in the ED, nurses worked by themselves rather than in groups.

Once in the triage room, the subject directed her avatar to open the triage tracking list and choose the next patient. Two patients appeared in the computer window, and subjects called in the patient they wanted to triage first. With each patient triaged, more patients were added to the tracker. Triage included actions such as hand washing, donning and then disposing of personal protective equipment, obtaining vital signs, and taking a focused history to decide patient disposition (see Additional file 1: Table S1, for a full list of these actions). Subjects could obtain information by selecting questions from a drop-down list and reading the patient’s reply. When a disposition was decided, the subject moved on to the next patient.

Data analysis

The simulation software generated “transactions” (Additional file 1: Table S1) corresponding to an action performed by the subject (e.g., putting on gloves, reading a blood pressure value) or a change in patient status (e.g., appearance or blood pressure). These transactions were then used to derive non-standardized variables that were used for further analyses (see Additional file 1: Table S2, for more information on non-standardized variables). Because many of the variables were likely to be correlated both with factors dependent on the subject (e.g., triage skills, keyboard literacy, clinical experience), as well as on the patient (e.g., urgency of triage, complexity of the case), a standardized list of variables was constructed by calculating first the z-scores for each subject-patient data point for that variable, and then the mean of the z-scores for each subject based on the patients the subject worked with during the simulation task (Additional file 1: Table S3). This process was used to reduce the effect of differences in patient variables, so that the remaining differences were more likely to be explained by differences in performance on the simulation, while allowing us to assess the accuracy of triage in assessing patient priority and time spent in triaging each patient.

For some variables, being on the negative or positive side of the z-score spectrum could be reasonably associated with a desirable versus non-desirable situation (e.g., it is more desirable for patients to be triaged faster, while it is not more desirable to prefer a particular triage room if there are no differences between the rooms). For this reason, the standardized variables studied were also differentiated on the basis of being desirable (D) or not desirable (nD) (Additional file 1: Table S3).

Data from questionnaires and scales were analyzed with descriptive statistics and paired t-tests using SPSS v. 21. Microsoft Excel was used to compute a correlation matrix for all standardized variables and all simulation data. The matrix was studied for strong positive (>0.75) and negative (<−0.75) correlations between variables. We did not control for multiple comparison because of the small sample size, which reduces the risk of a Type I error [24, 25]. As argued by Nakawaga [25], applying the Bonferroni correction to a small sample with already limited power, reduces power even further, increasing risk for a Type II error to an “unacceptable level” (p. 1045).

Results

Subjects’ attitudes and experiences

Responses to the exit questionnaire (Table 3) indicate that the subjects’ attitudes toward the simulation were largely positive. Subjects generally regarded the scenarios as realistic, and when asked specifically whether they thought the virtual world in the simulation task reflected their real world experience, 8 out of 10 subjects answered “yes.” The majority of the subjects noted that the speed of the avatar and of procedures should be increased, but some thought that this was a matter of not having become fully acclimated to the simulation. The mean rating (±SD) of how comfortable subjects felt using the avatar was 46.9 (±19.3), suggesting that participants overall felt moderately comfortable using and maneuvering the avatar, with some subjects feeling distinctly more comfortable than others. The most consistent factor that moderated comfort with the simulation was that interactions with patients and other staff members were via typed questions and answers rather than direct verbal interactions, although the questions were felt to be formulated appropriately.

Table 3 Exit questionnaire: attitudes toward the virtual simulation task
Full size table

Paired t-tests on raw NASA TLX subscale scores comparing perceived workload on an average work day with perceived workload on the simulation revealed a significant difference only for physical demand (46.5 ± 24.8 versus 14.5 ± 20.2; p = 0.02), suggesting, as would be expected, that subjects perceived their actual triage work to be more physically strenuous than the simulation task. However, after weighting the scores according to standard procedures (adjusted rating) there were no significant differences across this scale, any of the other scales (mental demand, temporal demand, performance, effort, frustration), or total workload score, indicating that the subjects’ subjective workload demands during the simulation task were equivalent to their subjective workload during a regular work day (Table 4).

Table 4 Comparison of mean (and standard deviation) NASA results for an average day at work and for the simulation task
Full size table

Simulation task

The average time to triage a simulated patient was 7:44 ± 2:18 min (range 1:45–13:48 min). Paralleling experience in most settings, there was inter-subject variability on most measures: z-scores for the average time each subject worked on a simulated patient, to control for complexity of patients, showed that subject 2 was fastest at triaging patients in the simulation, while subjects 1 and 8 were slowest (Table 5). However, removing from the analyses patients who were only triaged once eliminated significant differences in triage time between patients. As seen in Table 6, subjects 2 and 10 had more negative z-scores, and subject 1 had more positive z-scores, than the rest of the group, but the differences were not statistically significant. On non-standardized variables (Additional file 1: Tables S4, S5), subjects 2 and 10 had more desirable results, while desirable results were less frequent for subjects 1, 7, and 8. Tables 5 and 6 indicate that subjects were consistent in assigning priority to simulated patients.

Table 5 Durations of the time periods subjects (S) worked actively with patients (P) and z-scores
Full size table
Table 6 Subject specific z-scores for standardized variables
Full size table

Correlations

The correlation matrix conducted for all standardized variables and all simulation data revealed several strong correlations (r ≥ 0.75 or r ≤ −0.75). Subjects who found the overall workload (NASA total weighted rating) of the simulation task to be low had more previous experience using gaming and/or virtual reality systems (r = −0.80), and more hours playing virtual worlds (r = −0.83). Subjects with more current ED experience reported requiring less mental and physical effort (NASA Effort, raw rating work; r = −0.82) and feeling less frustrated/more secure (NASA Frustration, raw rating work; r = −0.81) at work. Currently working in the ED was associated with feeling more successful in and having higher satisfaction with one’s work performance (NASA Performance, raw rating work; r = −0.76). The more confidence subjects felt in their ability to respond to ED patients (Tables 3, 4), the more successful they thought they would be in accomplishing other work tasks (NASA Performance, adjusted rating work; r = −0.79). Although overall confidence in subjects’ ability to respond to ED patients did not change significantly after performing the simulation task (Tables 3, 4 versus 5; r = 0.92), probably because the level of confidence was already high prior to the simulation, the correlation between confidence and performance became stronger (Tables 3, 5 and NASA Performance, adjusted rating work; r = −0.90), suggesting a positive effect of having completed the simulation task.

With respect to performance during the simulation task, simulated patients of subjects with more real-life triage experience spent less time in the waiting room (r = −0.77). Subjects who reported feeling secure and gratified, and less stressed and irritated at their daily job on the NASA Frustration subscale were found to be more likely to enter correct data (e.g., vital signs) during the simulation task (r = 0.78). Greater confidence of subjects in their ability to respond to ED patients (Tables 3, 4) was associated with a higher likelihood of adhering to hand washing and personal protective equipment protocols prior to interacting with the simulated patient (r = 0.81).

A further parallel with actual work flow was that subjects who reported more confidence responding to ED patients (Tables 3, 4) reported less time pressure while doing the stimulation (NASA temporal demand, raw rating simulation; r = −0.81). The less time pressure subjects felt during the simulation (NASA Temporal Demand), the more time elapsed between calling a patient to the exam room and obtaining vital signs (r = −0.90). Vital signs were entered into the chart more accurately by subjects who perceived the simulation to require more mental and perceptual activity (NASA Mental Demand, raw rating simulation; r = −0.75).

Discussion

The purpose of this preliminary study was to address the validity and feasibility of a newer multi-user virtual reality platform as a proxy for staff behavior in the ED. As can be seen in Tables 5 and 6, in addition to measuring the process of triage (e.g., PPE, interactions with patients), the order in which patients were called and the time spent with each patient was assessed. We were therefore able to evaluate subjects’ ability to prioritize triage patients according to standard principles and procedures. These data, along with a degree of intersubject variability in performance within an expectable range, suggest that virtual reality triage can serve as a valid model of actual ED triage that could facilitate the study of the impact of stresses such as disasters on staff functioning before these events actually occur. As would be expected of a realistic model, more real-life experience working in an ED triage setting was associated with feeling a lower level of workload (e.g., less frustration and less temporal demand) and with better outcomes during the simulation task (e.g., less waiting time for simulation patients). Additionally, feeling less stressed in their daily work and more confident in responding to ED patients was associated with better outcomes during the simulation task (e.g., entering exact data, washing hands, and using personal protective equipment).

In the present study, subjects perceived similar workloads (assessed with the NASA TLX) during their daily work as they did during the computer simulation task. Even though the physical effort of the simulation was, as expected, less than that required in the workplace, subjects reported similar mental demands, and the relationship between time spent in a simulated task and the sense of time pressure while performing it, was similar to the perceived relationship in the workplace. Although familiarity with virtual reality predicted more comfort with the simulation, as has been reported with other platforms [16], self-perceived success and satisfaction with the task, amount of effort put forth, and frustration during the simulated task, were correlated with similar experiences in real-world triage. The impression of a valid relationship between simulated and actual ED experiences is strengthened by 8 of 10 subjects indicating that the virtual world in the simulation task reflected their real world experience.

Some elements of the simulation model should be modified in future work. Subjects felt that the speed and maneuverability of the avatar could be faster. This could be accomplished with greater computing power and enhancing parameters of avatar movement. More extensive training prior to starting the simulation task might address the concern of some subjects that they did not feel fully acclimated to the simulation when testing began. The primary shortcoming of the model involved obtaining patient data through written rather than spoken interactions. Similar concerns have been noted by others when using virtual reality models [18]. Having an experimenter in another room read scripted patient responses to produce a “virtualized” verbal interaction is one cost-effective approach to improving patient-subject interactions. Future models should also address the diversity of clinicians in the ED, the hierarchy of their skills, delegation to other providers, prioritization of tasks, and provider tasks such as teaching and administrative work, or the presence of trainees, who generally slow patient throughput [10]. We are currently modifying the platform to allow us to study interactions of groups of subjects as well as more robust graphics, in a manner that might be useful to institutions that lack computers with sufficient graphics capability or that have firewalls that make accessing servers and downloading more robust programs difficult.

Training with a simulator can improve patient throughput by medical students during simulated triage of a disaster [26], but the impact on actual triage has not been addressed. Computerized simulations have been used to provide information about matching staffing with individual responsibilities and patient flow to reduce length of stay [27], and to improve ED throughput and system performance [1]. One study showed that an actual ED intervention that had been modeled in a simulation resulted in decreased length of stay and decreased the number of patients who stayed longer than 6 h [28]. With only a few exceptions, however, extensive use of physical simulations in medical settings have not resulted in evidence of substantially improved outcomes [9, 29], and the validity of ED responses to disaster that were adapted based on an initial simulation has not been studied.

The challenges of recruiting full-time nursing staff to a time-consuming study limited our sample size in this preliminary study as well as our ability to enroll a random sample of triage nurses, but we thought that the results would be more informative than if the subjects were retired nursing staff or students. Many other studies have utilized small samples, and we were able to collect and analyze extensive data on efficiency and workload management, with statistically and clinically meaningful results. The majority of subjects thought that the exercise was a realistic model of their actual work despite the reality of the scenarios being reduced by dialogue boxes rather than live speech, and performance on the scenarios paralleled reports of actual workplace experience. The validity of this model of ED triage is strengthened by obtaining extensive data on performance on this discrete aspect of emergency work, but generalization of the results to all ED activities is limited, although most nursing staff members who perform triage also perform other ED functions. All subjects were female, which could limit application of the findings to male triage nurses. However, we are not aware of any data indicating gender-specific (or ethnicity-specific) approaches to triage. Obviously, replication of the principles addressed in our study with a larger sample of staff performing more diverse functions would be desirable. The potential usefulness of a research platform does not necessarily translate to effectiveness for training, which requires different functional parameters.

In order to use a practical and cost-effective virtual reality model to study the impact of stresses on ED personnel on ED operations, it is necessary to demonstrate its reliability and validity in representing everyday ED operations that would be expected to be impacted by those stresses. These preliminary results suggest that one essential ED function can be adequately represented in a virtual reality model. If the applicability of the model to triage is confirmed by a larger study of additional ED functions, the model could be used to investigate how ED staff reacts to a variety of stresses and interventions that might optimize their adaptation.

Conclusions

The prospect of an increasing number of natural and man-made disasters that stress the ED, and particularly its staff, makes it imperative that protocols be developed that will facilitate safe and efficient functioning in the face of events that have not been encountered by many institutions. Computerized simulations can be cost-effective approaches to testing new methods of addressing mass events such as epidemics, if they prove to be valid proxies of actual behavior in the ED. Our results suggest that a virtual reality simulation of an actual ED setting can serve as such a platform.