Keywords

1 Introduction

The United States Army makes use of a wide array of data that may be visualized geospatially. Soldiers and Army leadership may subsequently use these data in a variety of ways, including for training and mission planning. In the context of the study presented here, geospatial data are used within a structured decision making process (called the Military Decision Making Process, MDMP) to understand a situation, develop a course of action, and produce an operation plan. Specifically, the study investigated two groups of students (all officers in the U.S. Army or U.S. Marine Corps) that made use of two different sets of tools to complete a course exercise. The conditions were: (1) a set of traditional tools used in the course, including computerized maps, worksheets, and other resources presented via Microsoft PowerPoint; and (2) a new suite of tools provided as part of the Augmented REality Sandtable (ARES), a research and development testbed under development by the U.S. Army Research Laboratory (ARL). The purposes of the study were two-fold: (1) collect subjective and quantitative measures of student performance using the two sets of tools, and (2) evaluate use of a new computer simulation modeling Chemical, Biological, Radiological, and Nuclear (CBRN) effects on ARES. This section introduces the ARES platform, the course, and the CBRN simulation, respectively. Subsequent sections present study methodology, results, and a brief discussion of findings.

1.1 ARES, the Augmented REality Sandtable

The Augmented REality Sandtable (ARES) is a research and development testbed with the aim to improve tools for visualizing and interacting with battlespace information, providing a user-defined common operating picture at the point of need [1]. The ARES platform supports visualization through several modalities, including but not limited to a traditional sand table (7’ × 4’) with projected, visual representations of an area of operations and related terrain and tactical data. The data are fed from a computer through a commercial projector above the sand table. However, ARES is device agnostic and does not necessarily require the physical sand table. The platform provides various interaction modalities that adapt to the use case (e.g., a mobile software application for tactical planning and mixed-reality headsets like the Microsoft HoloLens and HTC Vive). Figure 1 shows the sand table implementation of ARES in use during the study.

Fig. 1.
figure 1

CBRNC3 students utilizing ARES during the TTX.

Full size image

1.2 Chemical, Biological, Radiological, and Nuclear Captain’s Career Course (CBRNC3)

The CBRNC3 is an extensive course that “provides company grade CBRN Officers the technical skills and knowledge to perform the duties and responsibilities required of company commanders and brigade level battle staff CBRN Officers [2].” A Table Top Exercise (TTX) is held at the end of the course which emphasizes the implementation of the military decision making process (MDMP) and rapid decision-making and synchronization process (RDSP) during various CBRN defensive and offensive scenarios. The MDMP is a seven-step “iterative planning methodology to understand the situation and mission, develop a course of action, and produce an operation plan or order [3].” The goal is to provide commanders with information to support their understanding and visualization of the tactical environment [4]. RDSP is a five-step process that “lets leaders avoid the time-consuming requirements of developing decision criteria and comparing courses of action (COAs) [5].” While the MDMP is done in the planning phase which usually takes a few days and results in the optimal solution, the RDSP is conducted during the execution phase and requires timely and effective solutions [5]. The CBRNC3 TTXs are designed to evaluate the students’ ability to apply both processes to operational scenarios. Students are expected to generate multiple map overlays and hazard prediction plots and brief them to the instructor (taking on the role of a commander) in accordance with MDMP/RDSP doctrine. In the past, this course lasted 22 weeks with multiple TTXs conducted throughout the course. Modifications are being made to enhance the effectiveness of the course, therefore the multiple TTXs that were conducted throughout the course are being replaced with a single week-long TTX that incorporates various CBRN scenarios. The evaluation of ARES was conducted during this single week-long TTX.

1.3 CBRN Simulation

The ARES-integrated simulations for CBRNC3 utilize a version of the Weather Research and Forecasting (WRF) model [6, 7] that has been augmented for plume transport and dispersion modeling. The WRF model was adjusted to target a domain size of 50 to 200 km on a side, translating to a simulation capability of 1200 km2 to 20,000 km2. The mesoscale simulation domain size chosen will automatically determine a horizontal resolution between 0.5 km and 2 km. This size of domain can produce 6 simulation hours with the first frame arriving roughly after a minute. This first minute is part of the automated simulation initialization, boundary, and plume release conditions.

This model is not part of the Joint Effects Model (JEM)/Joint Warning and Reporting Network (JWARN) system. The WRF-based CBRN simulation is fully integrated within ARES such that topography of a mapped area (as measured by ARES camera using the user-shaped sand as a proxy to map terrain) is ingested into the WRF-based simulations, directly affecting simulated plume transport and dispersion.

2 Methodology

2.1 Participants

Twenty-eight Soldiers from Fort Leonard Wood’s CBRNC3, 21 males and 7 females (M age  = 29 y.o., SD = 3) participated in the evaluation. Participation was voluntary and no compensation was awarded. Participants were mostly from the U.S. Army, but two were Marines. The number of years in the services ranged from 4 to 16.5 years (M = 6.9, SD = 4). The number of CBRN related training courses they have taken prior to the current one ranged from 1 to 13 (M = 2.6, SD = 2.9). Additionally, only six reported being deployed to a CBRN unit (M = 8.7 months, SD = 8.6).

Information reported on the demographics questionnaire also stated that no one, except for a single participant, had previous experience with an augmented reality sand table. Further, the majority (21 participants) have had experience interacting or using a tablet in general (M = 4.6 years, SD = 3.3), but only a small portion reported using a tablet to edit images (7 participants).

2.2 Questionnaires

Table 1 lists and describes the five subjective and objective questionnaires employed in the study to elicit subjective feedback from participants. Additionally, a grading rubric for the briefs (CGSC Form 1009s) was used to elicit a measure of team performance from the course instructor. This grade provided an assessment of the quality of the students’ presentations when using ARES compared to the traditional method. The instructor focused on grading teams on two main factors: accuracy and support. Grades are provided using 5-point Likert items (1 = Unsatisfactory; 5 = Exceptional).

Table 1. Questionnaires eliciting subjective and objective feedback from participants
Full size table

2.3 Experiment Design

A between-subjects design was used to compare the effects of utilizing ARES on CBRNC3 TTX course deliverables, knowledge acquisition, team collaboration, self-efficacy, and technology acceptance. The class was split into two teams (14 in ARES and 14 in Traditional) for the TTX. The instructor assigned battle staff roles to each person on the teams. Each team was also comprised of sub-teams and coordination among these sub-teams was required to generate the appropriate MDMP/RDSP outputs. One team used ARES to conduct the TTX and the other conducted it as they traditionally would in the course (Fig. 2). ARES was in a separate location from the traditional team, so the teams were not aware of each other’s outputs or briefs. The traditional approach involved sub-teams developing MDMP/RDSP outputs using Microsoft Word and PowerPoint on either desktop or laptop computers. This group would use PowerPoint only to brief the instructor on each output. The ARES team had the same resources available to them as the traditional group, but they also had ARES to create and brief their outputs. Plume model simulations were available to both teams if they wanted to utilize them, but the traditional team had to use an additional system (Joint Effects Model (JEM)/Joint Warning and Reporting Network (JWARN)) located in another room while the ARES team had the benefit of the plume simulation integrated within ARES. Prior to beginning work on the exercise, a researcher led a face-to-face training provided to the entire ARES team at the same time.

Fig. 2.
figure 2

ARES team (left) and traditional team (right) developing MDMP outputs.

Full size image

3 Results

Two-sample t-tests were used to assess the effects of utilizing ARES on CBRN MDMP/RDSP outputs, knowledge acquisition, team collaboration, and self-efficacy compared to a traditional approach. Bonferroni corrections were made to determine significant differences between the two groups (α = .0125). Sample size for each analysis varied as some participants were not present on the first and last day of data collection, but the minimum was n = 27.

3.1 CBRNC3 Briefs Grading Rubric

Two-sample t-test found no significant difference between the traditional (M = 4.33, SD = 0.68) and ARES (M = 4.50, SD = 0.75) teams on accuracy scores, t(60) = −0.094, p = .17. Similarly, two-sample t-test found no significant difference between the traditional (M = 4.57, SD = 0.69) and ARES (M = 4.63, SD = 0.77) teams on support scores, t(62) = −0.300, p = 0.17.

3.2 Knowledge Assessment

Percent difference scores for pre- and post- knowledge assessments were calculated for each team and then a two-sample t-test was run. No significant differences were found between the traditional (M = 21.6, SD = 6.62) and ARES (M = 20.6, SD = 8.71) on percent difference knowledge scores, t(23) = 0.407, p = 0.34.

3.3 Team Diagnostic Survey (TDS)

Two-sample t-tests were run for all 13 subscales of the TDS. A significant difference was found between the traditional (M = 2.52, SD = 0.65) and ARES (M = 1.88, SD = 0.66) teams on ratings for Knowledge and Skill Related Process Criteria, t(26) = 2.59, p = 0.008. No other significant differences were found for all other subscales (p > 0.0125).

3.4 Self-efficacy Questionnaire

Two-sample t-tests were run on all items of the self-efficacy questionnaire. A significant difference was found between the traditional (M = 86.8, SD = 7.8) and ARES (M = 94.3, SD = 8.5) teams on ratings for how confident they felt their team could develop a course of action output, t(26) = −2.43, p = 0.011.

3.5 Technology Acceptance Measure (TAM)

This measure provides descriptive data and was focused specifically on ARES technology so only the ARES team completed this measure. The lowest rating was for elicited anxiety while using ARES (M = 3.4, SD = 0.7) and ranged to the highest rating which was for perceived enjoyment while interacting with ARES (M = 4.9, SD = 1.1).

4 Discussion of Results

The goal for this evaluation was to explore the effects of utilizing ARES on CBRNC3 outputs, CBRN MDMP & RDSP knowledge acquisition, team collaboration, enhancing a sense of CBRN MDMP & RDSP skill competency, and perception of accepting ARES technology as a training and application tool. Findings are briefly discussed in this section.

An interesting significant finding is that the ARES team rated themselves as more confident in developing an MDMP output as a team, specifically the COA deliverable. The COA is an output that is comprised of many parts and requires extensive team communication and collaboration which results in multiple map overlays that contain information from all sub-teams. The results indicate that after interacting with ARES, they felt they had a better sense as a team that they could generate a COA in the future compared to the traditional group.

The ARES team and Traditional team performed similarly in terms of instructor assessments of CBRNC3 MDMP & RDSP outputs generated and briefed during the TTX and there were not statistically significant performance differences. Two limitations due to logistical and time constraints lead to difficulty in accurately assessing group performance. The first is that the grading scale (i.e., 1 to 5) used on the CGSC form 1009s may not be sensitive enough to capture true differences that exist between groups. The second is that the instructor of the course was the same instructor that evaluated both teams. Generally, student assessments in a research experiment should be performed by an independent team of evaluators to avoid the potential for bias. Future evaluations may want to consider using multiple instructors not involved in training the students to grade the course outputs.

Based on feedback gathered during a debriefing session, the ARES team much preferred using ARES over the traditional tools (PowerPoint) to create the MDMP & RDSP outputs. They said they could create these outputs quickly and could make less tangible products but still provide the same, if not better, level of detail during their briefs compared to how they created them previously.

5 Conclusion

This evaluation provided the opportunity to investigate ARES as a training tool for the CBRNC3 TTX. The initial findings of this evaluation hold great promise for future applications of ARES. A key recommendation is that future work conduct detailed user-needs analysis of the potential end-users and focus on developing the functionality to meet their needs while ensuring continuous usability evaluations are conducted to support intuitive and efficient interaction with the system.