1 Introduction

Natural disasters such as floods, hurricane, fire, and earthquakes cause serious problems in the community and infrastructure. For this, community members must be prepared and trained for this kind of emergencies and hazards which might occur as consequences of the disasters. Since these disasters can occur at any time, practice drills must be done periodically not to be in a panic in these circumstances.

Some disasters are more common in crowded cities and large countries in terms of area. For instance, 25,073 fires broke out in Istanbul last year (Brigade 2018) and 38,287 earthquakes occurred in Turkey in 2017 (Disaster 2018). These numbers are not surprising because 20% of residents in Turkey live in Istanbul and Turkey sits atop seismically active spots in the world. Due to this huge number of events happening throughout the year, training is inevitable for community members. However, in case of a fire or earthquake, most of the employees in companies do not have a concrete idea on how to leave the building effectively to survive. While practice drill is the best way to train employees, due to the inefficiency and difficulty in the process, employees either do not participate in these activities or give enough importance.

The main objective of this research was to design, implement, evaluate, and observe a novel evacuation training system which applies AR technology in the context of a university. The young generation is more motivated to use new technologies as part of any training activity, and AR increases engagement and interaction in any educational system. Therefore, this novel training system was implemented based on AR technology.

The research question of this study is shown as follows:

  • RQ: What are the effects of the AR technology in terms of the perceived usefulness and easy of use in the context of evacuation training systems?

To respond to this research question, we aim to apply the technology acceptance model (TAM) and the related questionnaire used in Chandrasekera’s study (Chandrasekera 2015). The mobile application which works on the iOS platform was built with ARKit framework. ARKit is a brand new AR library which is markerless and allows users to place objects on the surface, and also, it can scan and detect surfaces. With the help of this mobile application, we wish to make sure that people in the building leave properly and quickly in the event of an emergency case. AR lets us combine digital data with the user’s environment in real time.

We can divide the system into two parts to investigate ARKit from an engineering perspective. The first part is indeed the hardware. To detect plane and measure the distance between ground and phone, it uses the accelerometer, gyroscope, pedometer, magnetometer, and barometer. The second part is the software which analyzes the corresponding environment. ARKit analyzes the environment and creates a map via visual inertial odometry (VIO). This method mixes data coming from the camera sensor with Core Motion data. To pick up the Core Motion data, it uses hardware components (Pallotta 2017). Based on this infrastructure, objects can be detected and placed on the surface.

Our case study was performed on a university campus (Istanbul Kultur University). The implementation process is represented in Fig. 1.

Fig. 1
figure 1

Implementation process

Full size image

As shown in Fig. 1, first emergency evacuation plans of the university were retrieved from the head of the emergency support team with the approval of Secretary General. Three programs, namely 3ds Max, AutoCAD, and SketchUp, were used to create 3D models of the building. Top-down views of floors (base, second, and third floors) were drawn in AutoCAD software which let us have a 2D view of the building. While the interior of the building was modeled with SketchUp, the outside was visualized with 3ds Max program.

Top-down views produced with AutoCAD were imported into the SketchUp program. Walls, windows, seats, tables, and the other seating places were drawn in detail, and ceramic flooring was attached to this model in SketchUp software. Colors of the exterior walls and windows of the building were made similar to the original colors of the building in 3ds Max program to provide a similar appearance. After all these steps were performed, models were transferred to the Unity3D development platform which was also applied for generating animations. With the help of these programs, we made our training system more realistic.

The following sections are organized as follows: Sect. 2 provides the background and related work. Section 3 explains the process, design elements, and technologies which were applied during this project. Section 4 provides the experimental results. Section 5 discusses the approach and outcome of evacuation training systems. Finally, Sect. 6 explains the conclusions and future work.

2 Background and related work

Virtual reality technology builds an artificial environment for several purposes such as improving skills (Faroque et al. 2018), but AR technology lets us combine the existing environment with the new digital information. Therefore, the platform created with AR is not totally artificial. Nowadays, many interesting applications are being developed with AR technology. For instance, some researchers apply AR for vein visualization (Khor et al. 2016). There are also very useful applications for supporting teachers to enable e-learning (Mota et al. 2018), anatomy education (Ma et al. 2016), image-guided surgery (Lovo et al. 2007), remote surgical assistance (Shenai et al. 2011), laparoscopic surgery (Volonté et al. 2011), surgery of cerebral aneurysms (Cabrilo et al. 2014), and bypass surgery (Cabrilo et al. 2015).

Recently, several review papers have been published on AR technology. Van Krevelen and Poelman (2010) presented numerous applications and several limitations of AR technology. Billinghurst et al. (2015) explained several definitions of AR and the milestones in its history. Bacca et al. (2014) performed a systematic literature review on the application of AR technology in education and provided the state of the art for the education domain. Li et al. (2018) reviewed AR- and virtual reality-related articles. They concluded that the top three domains are safety training, safety inspection, and hazard identification.

For helping the AR researchers on the usability aspects of AR technology, Dey et al. (2016) performed a systematic literature review (SLR), analyzed usability-based AR papers, and classified them based on application areas. After this conference paper, they also published a journal article, which presented what has been done between 2005 and 2014 on the usability aspects of AR technology (Dey et al. 2018). Prior to Dey et al.’s study (Dey et al. 2016), a few review papers were also published on the usability research of AR (Edward et al. 2005; Dünser et al. 2008; Bai and Blackwell 2012). Swan and Gabbard (Edward et al. 2005) investigated four important AR publication venues, but showed that only a small group of papers reported the user studies. Dünser et al. (2008) evaluated AR papers reporting user studies and classified them. Bai and Blackwell (2012) reviewed AR papers reporting user studies, but only focused on ISMAR conference proceedings. Because the article by Dey et al. (2018) is the most recent one on the usability research of AR technology, in this part, we explain their observations and suggestions.

They reported that less than 10% of investigated papers were user study papers among all AR papers and while the number of these papers increased, the percentage of this kind of papers is still low and did not increase dramatically. They stated that there is a shift to use the handheld devices since 2011, there is little field testing, the number of user studies in the educational context increased, and there are fewer pilot studies than expected. The preferred data collection approach was filling out questionnaire forms as in our article. They categorized the papers into nine application areas: (1) perception (18%), (2) medical (15%), (3) education (14%), (4) entertainment and gaming (5%), (5) industrial (10%), (6) navigation and driving (9%), (7) tourism and exploration (2%), (8) collaboration (4%), and (9) interaction (23%). Based on this categorization, our article is considered in the education application domain. In the medical application area, training and simulation studies were dominant and laparoscopy, rehabilitation, and phobia were the widely studied topics. In the education application area, they reported that the median number of studied users was 28, which was the highest number among other application areas. The most common keyword was learning in this set of AR papers. They stated that numerous evaluation methods can be applied in this application area. Also, they suggested integrating intelligent tutoring systems to these applications, which were already implemented in our application.

In addition to these review articles and observations, in this part, we present the similar training and evacuation applications published so far. Iguchi et al. (2016) aimed to train teachers effectively by using AR and smartphone-based head-mounted display (HMD) so that they will be able to provide evacuation instructions to the students properly in an emergency situation. Their system generates 3D virtual students with different reactions, and the teacher is expected to provide proper instructions to these virtual students. They reported that this system was accepted by the teachers. Our system is different than their system because participants are not expected to use HMD devices which might sometimes cause dizziness for some users. In another study (Mitsuhara et al. 2016), these researchers introduced a system called game-based evacuation drill (GBED) that can improve the capabilities of participants in the case of an emergency by using AR and HMD. The scenario-based strategy was implemented in the system supported by animations and audiovisual features. For instance, the crashed car and injured person simulations are the interesting scenarios in this system. Kawai et al. (2016) combined the AR and head-mounted display (HMD) for game-based evacuation drill system. Tatić and Tešić (2017) implemented an AR-based occupational safety system which was applied in the Thermal Power Plant Ugljevik. Due to the inadequate use of electromechanical machines in plants, work injuries happen frequently. For this reason, their new system provides step-by-step procedures for the workers to follow and decreases the risk of injuries. They reported that there was no injury in the short experimental period, it took just a few days to adapt to the new mobile system, and the system was accepted by all the workers. AR application was implemented as a mobile application so that it is easily used by workers. Their study is different than our project because they focused on the use of electromechanical devices and did not focus on evacuation training. Ahn and Han (2011) developed an indoor AR-based evacuation system called RescueMe because this kind of indoor systems is very challenging. They performed two simulations to observe how long it takes to evacuate a building in an emergency situation. To identify the room number, the system uses a commercial image labeling Web service and the server component finds the best exit path for the user. Their system mainly focused on the best path, did not include additional educational animations, and did not model the building as we performed in this project. Tsai and Yau (2013) developed an AR-based mobile application including 3D graphics to provide escape routes for radioactive accidents. The spatial relationship between the location of the user and the anticipated shelter is shown in real time on the mobile application. They demonstrated that this system provides better escape routes compared to their previous study (Tsai et al. 2012). As seen in this example, this kind of applications is quite important for emergency situations such as the radioactive accident. Mitsuhara et al. (2015) developed a Web-based evacuation drill system to allow a number of users to participate in the evacuation drill system process and compose evacuation scenarios. Their system did not use AR technology, but the Web-based system was very useful to simplify the scenario design task. Kawai et al. (2015) developed tsunami evacuation drill system based on smart glasses and reported that the system is very effective as a disaster education tool. Leelawat et al. (2018) implemented a mobile application to be used as a drill exercise for tsunami situation. Teachers can easily apply this tool to improve their drill strategies with the help of a mobile device. Mitsuhara et al. (2018) presented an AR-based evacuee visualization to show how previous participants evacuated.

As we see in these papers, researchers develop new applications based on AR technology for effective evacuation drills. Our project focused on the education of the employees and students in a university and aimed to provide additional educational animations to improve the situation awareness of people working in the building. We also performed user studies to observe and measure the perceived usability of this system.

3 Methodology

This section defines the system architecture and key components of the training system. We implemented an evacuation training software, and the pilot study was performed in a university. We used Unity 3D (Technologies 2018) to comprehend digital objects and ARKit (A. Inc. 2018) plug-in to blend these digital objects with the environment for AR experience. The general overview of the system architecture is shown in Fig. 2. Our training software is designed to work on Apple mobile devices running with iOS version 11 and more recent ones. ARKit Framework allows us to use the mobile device’s motion tracking, camera scene capture, and processing features to build interactive AR environments. Apart from this reality composer, Unity platform houses our game engine and related software components such as views, scenes, and artifacts which provides an immersive AR experience with ARKit. Details of the system elements are given in the following subsections.

Fig. 2
figure 2

Infrastructure of the evacuation system

Full size image

Elements of the evacuation training system are game, scenes, views, artifacts, activities, and visual makers (Bach et al. 2017).

3.1 Game

Evacuation game is designed to make feel the disaster scenes for educational purposes. Our pilot application includes three scenarios regarding the evacuation in the event of an earthquake, fire, and chemical attack. The general task of this software is to provide appropriate views based on the underlying scenario and use artifacts by matching measured values from a mobile platform with the corresponding actions.

3.2 Scenes

The scenes are essentially the background of our pilot application. Basically, our scenes allow the user to move around and change artifacts to accomplish several missions. The first scene is related to the fire evacuation scenario. For evacuation cases, models were navigated with Unity 3D NavMeshAgent tool and referenced scripts.

A game was created and added to the scene to teach the use of fire extinguisher and common areas. In the second scene, the scenario was created to teach where the shelter is located at. Models were navigated as in the fire evacuation scene and added a cinematic scene at the beginning of the scenario. Components part was added to improve the positive feedback of the application which gives information about how to use a fire extinguisher and get a position in an earthquake case.

We spent a considerable amount of time to implement an indoor GPS system to guide the user based on his/her coordinate. For a realistic implementation, we had to know where the user location is exactly. We utilized the services of Unity3D for the GPS part of our system. For each floor, longitude and latitude have been stored for mainly the corridors and the other distinguishing points needed to be represented in the model. When the application is initiated, the system checks at which location the use is according to this stored information. Since we used 3D modeling programs for our models, which can be considered as the most accessible method compared to the other alternatives such as using a series of photographs, we ended up with a fully optimized model. Models were built at a scale of 1:1000, and our system is accurate to 10 m2 in tracking accuracy. After our solution was implemented, we achieved to determine the corridors and the special places where the user is located at. As a first approach to deal with this problem and increase the accuracy, we decided to make a 3D model of the university. Then, we used SketchUp tool to create the floors. Every floor was represented as in the real environment. Walls and columns were drawn by using AutoCAD software. Floors were also represented as in the real condition. Some of the sketched scenes are shown in Fig. 3.

Fig. 3
figure 3

a SketchUp draw of a single floor. b Autodesk 3ds Max draw of the university

Full size image

3.3 Views

Views refer to the visual representation of user interface in the application. We developed views to enable users to interact with the game at the most realistic level. In Fig. 4, some screens of the application are shown. Finding the fire extinguisher, pulling its pin-out, and turning it to the correct direction for the fire are some samples from the fire scenario. To make it more realistic, we add smoke, fireworks effect, and light in the environment. This scenario has virtual 3D models (human characters) which go to the nearest fire exit when the animation is triggered. In the same figure, the right reactions which should be made in the case of an earthquake are shown, and the ideal defense position is depicted.

Fig. 4
figure 4

Various screens from evacuation game

Full size image

3.4 Artifacts

The artifact is a physical representation of virtual entities in the real world that is linked to the data (Willett et al. 2017). Artifacts used by our application are fire extinguishers, flames, and earthquake-related items. Animations were used to simulate the use of objects in the real world. In this way, we make sure that the artifacts are placed in the real positions in the real environment and AR does not adversely affect the real experience.

3.5 Activities

Realization of the game with views through the scenes is made possible by the activities presented by the user through the interfaces. The management of the activities is orchestrated by the game, and artifacts and real-world interaction are the most important parts which should be focused on.

3.6 Visual makers

Visual markers are the objects that are used to encode information for the application. In order to solve the localization problem, we applied the signboards in the building. Corridor signboards and notices with the lecturer names were marked on the 3D model to help the identification of the user location.

4 Experimental results

To evaluate and measure the perception of the users, after users performed the training by using our application, we performed a survey based on the Technology Acceptance Questionnaire form. Users actually had to leave the building following virtual instructions on the tablet. Our survey includes 15 questions in total, and answers are on a 7-point Likert-type scale (strongly disagree/disagree/somewhat disagree/neither agree or disagree/somewhat agree/agree/strongly agree). We excluded the question 15 of the questionnaire form used by Chandrasekera (2015) because tests were performed on tablets and the effect of nausea, dizziness, or discomfort was not significant in this context. Questions retrieved from the study of Chandrasekera (2015) are shown as follows:

  • Q1: It was easy to understand the instructions

  • Q2: It took a long time to learn to use the system

  • Q3: This system is difficult to use

  • Q4: I felt in control over the system

  • Q5: My interaction with the system is clear and understandable

  • Q6: Interacting with the system does not require a lot of my mental effort

  • Q7: I find the system to be easy for us

  • Q8: This system is fun to use

  • Q9: I find it easy to get the system to do what I want it to do

  • Q10: I would like to use this type of system to receive instructions in my profession

  • Q11: I would like to use this system in other contexts than my profession

  • Q12: I feel confident that the system is giving me correct instructions

  • Q13: I would prefer to receive instructions from a person (teacher/tutor)

  • Q14: The system does not have any apparent shortcomings

  • Q15: I would have difficulty explaining why using the system may or may not be beneficial.

36 voluntary participants attended to this evaluation. Demographic information about the participants is presented in Table 1.

Table 1 Demographic information of participants
Full size table

We asked participants to use our system and evacuate from the building based on the default play mode. Before they used the system, they were given a didactic lesson which took between 30 and 60 min depending on the participant’s prior experience with AR. Statistical results of the survey are shown in Table 2. We also measured the time parameter, and the statistical results of the time parameter for 36 participants are shown in Table 3. Complex scenarios were not considered while measuring the time parameter. The mean column indicates that the average evacuation time from the building is around 5 min, but there were some participants who left after 3 min (minimum) and 6.5 min (max).

The survey includes several statements. Some of them are positive in the sense that they support at least one positive aspect of the system, and some of them are negative in the sense that they point to the drawback of the system. For instance, ‘It took a long time to learn to use the system’ is an example for negative statement in survey, but ‘It was easy to understand the instructions’ is an example for positive statement in survey.

Based on Table 2’s mean column, we draw Fig. 5. Questions which are only positive statements (Q1, Q5, Q6, Q7, Q8, Q9, Q10, Q11, Q12, Q14) are represented in this figure. For instance, Q1’s mean value is 6.39 based on Table 2, and therefore, we transformed this value into the percentage by dividing by 7 (7-point Likert-type scale) and multiplying with 100 (6.39/7 × 100 = 91.28%). As seen in this figure, the percentage value for each question is over 80.

Q1 was about the easy to understand the instructions, and the average value was calculated as 6.39 (max. 7). This indicates that the instructions were easily understood by the participants. Q5 was about the understandability of the interaction with the system, and the average value was calculated as 6.17. Q6 was about the requirement of the mental effort during the interaction, and the result was 6.36. Q7 was about the easiness of the system, and the result was calculated as 6.33. The result of Q8 was 6.67. The average result of Q9 was 5.81. Q10 was about the use of this type of system in their own profession and the result was 6.28, which means that participants would like to use this kind of these systems in their own jobs. Q11 was about the use of this system in other contexts than their own jobs, and the result was 5.78; this shows that they want to use this system in different fields. Q12 was about the confidence they feel for the correct instructions, and the result was 6.61. The result of Q14 was 6.19 which indicates that the system does not have shortcomings from the participant perspective.

In Fig. 6, results are presented for the negative statement questions (Q2, Q3, Q4, Q13, Q15). For this figure, we did not transform results into percentage values. The best value for these questions is 1, and the average results are quite near to this value. The highest value was for the question Q4 (feeling control over the system), but the value 2.25 means that participants’ opinions are between disagreeing and somewhat disagreeing. This result will be taken into account while the new version of our system is being implemented. In this evacuation system, feeling control over the system should be minimized, and therefore, the overall system will be checked to improve this drawback.

Results of the other negative statements are low (less than 2). In summary, the system was evaluated positively by our participants. These experimental results support the applicability of this evacuation training system in the university context.

Fig. 5
figure 5

Survey results regarding the positive statements in survey—part A

Full size image
Fig. 6
figure 6

Survey results regarding the negative statements in survey—part B

Full size image
Table 2 Statistical results of the questionnaire analysis
Full size table
Table 3 Statistical results of the evacuation time tests
Full size table

5 Discussion

When the reactions and feedbacks of the users who used the application were evaluated, it was observed that we developed a fun, but a very useful solution for a critical issue, evacuation drills. The purpose of the evacuation training software is not to replace the traditional evacuation education, but to support it with a state-of-the-art technological solution. In the first pilot study, we had localization problems because we used GPS to locate users (Ahad and Hossain 2004). We solved most of the localization problems by using static visual markers in the building and by pinning them. We accepted this solution because the overhead does not hinder the use of the application. The ideal solution to this problem is the use of Beacon-based indoor positioning systems. The availability of such kind of devices increased as their costs recently decreased.

Another challenge was about the integration of different toolsets to build this training system. It also took some time to design the 3D environment, and for the new developers, this might be another dimension to take into account because 3D modeling tools are not generally applied by software engineers.

We performed our experiments in one university with 36 participants. The number of participants is acceptable compared to the other studies. Although our case was performed in one university, the platform is flexible to be customized to the other universities as well.

For the implementation of this system, free character models have been used and their rigid bodies were not perfect. However, all the characters stay on the ground, and therefore, our system gives a realistic impression. When the participants experienced in this system, our screens were considered realistic. For the new version of this system, we will address the registration and spatial position detection for a better user experience.

Regarding the visualization of this system, we utilized the Unity3D visual packages for the fire visualization. However, for the new version of our system, we plan to develop a better fire visualization component by using different tools that we intend to buy.

For the fire part of the training, we also plan to implement an animation for a better realism, but due to the time and budget constraints of the first prototype, we decided to implement the animation in the new version.

We also got feedback from the emergency support team, which consists of three experts, in the university while developing the game. Some iterations have been performed to fulfill their expectations, and at the end of the process, the emergency support team and the general secretary of the university were satisfied. Since this emergency support team defines and follows all the policies in the university during an emergency case, their feedback had to be satisfied properly and at the end of the process, this team confirmed that the game does not teach any wrong things and provide a positive training for the participants.

The main motivation of this study was to make this kind of trainings more fun and useful. From this point of view, people who work or study in the university are not interested in regular lessons including an expert lecturing with supporting materials about what they need to do. This was also expressed during our conversations by the emergency support team in the university. As such, we compared the average time calculated for the evacuation with the time provided us based on regular lessons. Although there is no explicit control group due to the motivation of this study, we carefully considered the effect of our system with respect to the evacuation time.

Regarding the real-time performance, we did not notice any major problem which can cause lag or serious frame rate drops. To avoid the lag or performance problem, we utilized Unity profiler application during our beta tests. We monitored all assets using multiple run time tests with the help of Unity profiler and identified exactly which function is consuming too much resource or taking too much time. Then, we performed refactoring and followed the quality assurance activities to optimize our codebase. However, if there is insufficient RAM on the mobile device due to the multiple applications running in the background, that might impact the overall time of the analysis, but during the experiments, this was not the case.

Another issue is about the average evacuation time. Although this average time might change from one building to another depending on the size and complexity of the building, we aimed to calculate this time because the emergency support team has provided us a baseline, and therefore, the difference between the traditional average evacuation time and the time due to this new system should not be very large. Therefore, it was necessary to include the average time. Otherwise, they would not consider this system for operational purposes.

6 Conclusion and future work

Design and implementation of evacuation training systems which provide adequate evacuation drills to the community members contain several challenges. In this study, we developed a mobile application based on AR technology and demonstrated its effectiveness in the emergency situations. To evacuate quickly and correctly, the system includes several scenarios and features which guide and train the user in different contexts. In this study, the Unity3D platform was used as the main development platform and ARKit plug-in was imported into this platform. Top-down view of the floors was designed as 2D map with AutoCAD software. SketchUp software was applied to create windows, ground, seat, table, chairs, and walls as 3D models. These artifacts were imported into the 3ds Max program as input. It was demonstrated that the AR-based evacuation system is very effective for evacuation drills and this kind of systems can be preferred instead of theoretical training sessions. As part of future work, this application can be customized for specific domains. One of the possible customization areas can be plants which require occupational safety systems. There are also some plans to realize the same system in different universities in the same city because we observed that there is a great interest from many universities.

In the gamification part of this system, there is no first-person view, but this feature is considered as a future work. In the other parts of the system, we provide a first-person view feature. To add this feature to the gamification part, the back-end part of our system must be updated.