1 Introduction

People follow a wide range of different strategies, from traditional (e.g., asking other people about a route) to novel ones (e.g., use of new technologies), to navigate. To support users in navigation, several tools have been built, based on positioning and location-aware technologies. However, they suffer from limitations that are mainly due to the users’ disengagement of the environment [7], such as orientation and navigability problems and inconsistencies. To overcome such limitations, novel technologies, such as augmented reality (AR), have been introduced. Such technologies aim to increase the level of accuracy and detail when demonstrating the navigation paths, and thus, improve the user performance and enhance the user experience [2, 15, 16, 18].

Despite the fact that AR technology can be used to provide a more natural way of interaction when navigating, it introduces limitations, which are mainly related to spatial mapping and rendering, especially in outdoor conditions. To overcome such issues, persistent spatial points can be used in AR applications. Persistent spatial points represent important points, which “force” the virtual objects that augment the physical space to stay in specific real world positions across instances of the AR application. The use of such persistent points could enhance the user performance and experience, however, it can significantly complicate the development process and be a burden for the developers.

Hence, in this paper, we evaluate the use of persistent spatial points for navigation in an AR environment and we investigate the trade-off between the development effort, the user performance, and the user experience. We start with a discussion on the related work upon which we build our motivation. Then, we present a two-fold evaluation study; from the designer and the end-user perspective. Finally, we discuss the findings, the implications, and the limitations of the present work and conclude the paper.

2 Related Work

Use of Technology in Spatial Navigation. Many technologies and tools (e.g., Google Maps) have been used by people for spatial navigation. Such tools recommend a path to the user by considering varying factors such as identification of obstacles [5], accessibility issues [6], points of interest [8], emergent events [9], and diverse means and routes [17]. Novel technologies, such as AR, have been used to improve the user experience that was offered by conventional technologies, such as web and mobile applications. Such improvements can be achieved with various ways such as real-time detection of objects (e.g., obstacles) in the surroundings [2], use of virtual objects that guide the user through the recommended route [15] which are displayed in various formats and points (e.g., different height) on the display [16], real-time generation of the recommended path on the display [18], use of dynamic and real-time notification messages about not expected situations in the surroundings of the recommended path [13], and keeping aware the user of important events and situations [1], such as lights, estimated time, distance of alternative paths, etc.

Use of Persistent Spatial Points in AR Applications. Persistent spatial points have been used to improve accuracy by identifying the target objects clearly without ambiguity [10], improve the stability of the augmented objects [3], and avoid the misplacement (e.g., shifting away) of the augmented objects which can lead to the disconnection of the virtual experience with reality over time [12]. However, while the use of persistent spatial points improves the experience, it is not an automatic process, but a process that requires the developer to define the persistent points and anchor them to the real world, through spatial scanning and mapping, which may require time and repetitive adjustments to provide the desired outcome.

2.1 Motivation

From the discussion on the related work, it is evident that while AR has been used to enhance the user experience in navigation tasks and that while persistent spatial points have been used to overcome positioning problems of the augmented virtual objects, it has not been investigated, to the authors’ knowledge, the trade-off between development effort, user performance, and user experience for navigation tasks. Therefore, in this paper, our research question is whether and how the use of persistent spatial points influences the development effort, user performance, and user experience in AR navigation.

3 User Study

3.1 Method

To answer our research question, we followed a two-step study approach. In step I, we investigated the overall effort of a software engineer for developing \(HoloNav\), which is a spatial-navigation AR tool. In step II, we investigated the performance and experience of 28 individuals who used \(HoloNav\) to navigate through various paths in the city of Patras, Greece.

Tool. \(HoloNav\) is an AR navigation tool that guides the users to their destinations through auto-generated paths. It augments the physical space by presenting virtual arrows that guide the users through the path. \(HoloNav\) scans the area around the user and dynamically places the arrows about 50 cm above the ground without any action required from the user (Fig. 1). \(HoloNav\) supports both versions implied in the research question (i.e., with and without persistent spatial points).

Fig. 1.
figure 1

Virtual arrows augmented the physical space and guided the users to their destination.

Full size image

Apparatus. \(HoloNav\) was developed for Microsoft HoloLens, which is a head-mounted display (HMD) device developed and manufactured by Microsoft. All participants used the same device, which was adjusted for each of them to best fit their head and not affect their experience. To implement the functionality of the persistent spatial points, the software engineer used the spatial anchors feature of Microsoft HoloLens.

Participants. We recruited 28 participants (12 females, 16 males) of varying age (\(MIN=19\), \(MAX=36\), \(M=26\), \(SD=5\)). All participants had the same level of experience with navigation tools (they were familiar with geolocation and positioning technologies, such as GPS and Google Maps), HoloLens (none of them had ever used HoloLens or other HMD device), and use of AR systems to navigate (none of them had ever navigated with the assistance of an AR tool). Each participant used only one version of \(HoloNav\), hence, they were allocated in either the \(W-PSP\) or the \(WO-PSP\) group. The participants of the \(W-PSP\) group used the \(HoloNav\) version that was built with persistent spatial points (i.e., spatial anchors), while the participants of the \(WO-PSP\) group used the \(HoloNav\) version that was not based on persistent spatial points. Aiming to create balanced groups (e.g., same size, equal distribution of age and gender), the allocation was based on the participants’ demographic characteristics.

Paths. We designed three paths of varying length (short: 250 m, medium: 500 m, long: 1000 m) in a city area that all participants were familiar with. However, they were not aware of the final destination or the path prior to the study, as \(HoloNav\) generated the paths and demonstrated them (one path each time) to the participants during the study. When the participant reached the destination of the first path, the second one was activated, and so on.

Metrics. To measure the development effort, we focused on (a) the time needed to create each version, (b) the actions and resources required to create each version, and (c) the application footprint. To measure the user performance, we used (a) the success rate and (b) the completion time. To measure the user experience, we used the NASA-TLX [4] and User Engagement Scale (UES) [11] tools. We also conducted an interview with each participant after the completion of the task to uncover hidden aspects regarding their experience.

Procedure. In step I, a software engineer (SE), who had intermediate experience in developing AR applications, created two versions of \(HoloNav\): one with the persistent spatial points (\(W-PSP\)) and one without (\(WO-PSP\)). The SE developed \(HoloNav\) in his own working conditions (i.e., ecologically valid conditions). He was asked to keep track of the activities, resources, and the time needed to perform the necessary tasks, adopting an activity diary-log approach. Moreover, several semi-structured interviews were conducted during and after the development phase.

In step II, (a) we recruited the participants via posting flyers on bulletin boards at various places on the campus, and directly by contacting acquaintances of the research team; (b) each participant was given an overview of the study, and provided their consent; (c) each participant completed a demographics questionnaire and was allocated to a group (d) each participant undertook a tutorial in HoloLens to familiarize themselves with the technology; (e) each participant performed the activity using the allocated version of \(HoloNav\); (f) each participant answered the questionnaires and had an interview.

3.2 Results

Development Effort. The SE spent approximately the same time to create each version for coding, as the only additional step for developing the \(W-PSP\) version was to include the spatial anchors feature of Microsoft HoloLens. This was a quick procedure with minimum integration effort. In particular, the SE spent 48 man-hours to develop the \(WO-PSP\) version and 56 man-hours to develop the \(W-PSP\) version (8 of which were allocated for using the spatial anchors feature). However, considering that each persistent spatial point represents an important point in the world that the system should keep track of over time, the SE needed to perform spatial rendering and mapping tasks to ensure that anchored holograms will stay precisely in place. Hence, the SE navigated through the city and performed the aforementioned tasks by scanning the various important points, and thus, the total time spent to create the \(W-PSP\) version increased. The SE spent 20 additional man-hours to develop the \(W-PSP\) version. Overall, the SE spent 48 man-hours to create the \(WO-PSP\) version and 76 man-hours to create the \(W-PSP\) version. Hence, the use of persistent spatial points resulted in an increase of the total development time by more than \(50\%\).

Likewise, the actions and resources required to develop each version were similar regarding the coding part, as the only additional step for the \(W-PSP\) version was the use of the spatial anchors feature. However, the overall procedure of creating persistent spatial points included the commuting of the SE to the city, the navigation through the various paths, the scanning, mapping, and rendering of the important points, etc. Therefore, the additional actions and resources required for the creation of the \(W-PSP\) version had a direct impact on the total creation time and increased the development effort, which highly depend on the size of the area that the navigation tool supports. The resulting assets can be re-used once the actions performed (e.g., the persistent spatial points remain stored in the device), saving time and development effort for future projects.

Regarding the application footprint, the total size depends on the number of the meshes, which in turn, depends on the desired accuracy and the covered area. The larger the area is, the more meshes are required, and thus, the bigger the filesize is. In our scenario, the filesize of \(W-PSP\) version was 350 MB bigger than the filesize of \(WO-PSP\) version, which was expected, as the \(W-PSP\) stores spatial mapping meshes. Nonetheless, increasing the length and/or the number of the paths could lead to unrealistically large filesize, which might also affect the application performance, and thus, user experience and/or performance. Regarding the loading-time, it depends both on the application and on the technical characteristics of the device used. In our case, no differences between the two \(HoloNav\) versions when using Microsoft HoloLens were revealed.

User Performance. All participants reached their destinations; hence, no difference was found on the success rate (i.e., \(success\_rate=1.00\)). Regarding the completion time, the independent-samples t-test (independent variable (IV): \(HoloNav\) version; dependent variable (DV): completion time) revealed statistically marginal and significant differences for the medium and long paths respectively (Fig. 2). \(WO-PSP\) users needed more time to reach their destination than \(W-PSP\) users (medium path: \(t=2.023\), \(p=.065\), \(d=1.012\); long path: \(t=3.016\), \(p=.009\), \(d=1.151\)). In the \(WO-PSP\) version, the paths are not “anchored” with any spatial point of the real world, and there is an increased likelihood of “drifting”, meaning that the arrows may shift away and the path will be misplaced. In short-distance paths, a light drifting was observed which did not influence the users, as they were aware of the path during the navigation and reached their destination without any problems. However, in middle- and in long-distance paths, the absence of stationary spatial points, which would serve as reference points, negatively affected the user performance in terms of completion time. The virtual arrows were often misplaced, and the users followed misleading paths which ended up to dead ends. Thus, they had to start over the process, and they spent more time for reaching the end point of the paths. The longer a path without persistent points was, the more likely it was for a virtual arrow to be misplaced and a misleading path to be generated. On the other hand, in the \(W-PSP\) version, the paths were precisely located as they were based on stable and stationary reference points, and thus, the users performed well in all conditions (short-, middle-, and long-distance paths).

User Experience. According to the results of the NASA-TLX questionnaire, the \(WO-PSP\) users put more effort than the \(W-PSP\) users in accomplishing the navigation tasks (Fig. 2), which was a statistically significant difference (t-test with \(HoloNav\) version as IV and effort dimension as DV: \(t=3.076\), \(p=.005\), \(d=1.158\)). No difference was found for any of the other NASA-TLX dimensions. The discussion with the users revealed that the difference derived from the effort made by the \(WO-PSP\) users to re-adjust their route by starting the navigation over when the arrows were misplaced, which was frequent in long-distance paths where there was an increased likelihood of poor accuracy or drifting.

Regarding the UES questionnaire, the analysis revealed that the \(WO-PSP\) users had higher focused attention than the \(W-PSP\) users (Fig. 2), which was a statistically significant difference (t-test with \(HoloNav\) version as IV and focused-attention dimension as DV: \(t=4.173\), \(p<.001\), \(d=1.568\)). In particular, the \(WO-PSP\) users mentioned that they lost track of time, lost track of the world around them, and they were absorbed in following the arrows. Considering that the path was not precisely located (e.g., it could be traversed through shops) the \(WO-PSP\) users could not follow it “arrow-by-arrow”, and thus, they had to pay attention to the direction of the arrows, and often, they had to guess the path when the arrows were not visible (e.g., inside buildings) or indistinguishable. The aforementioned condition was frequent in long-distance paths. Finally, an issue raised by the \(W-PSP\) users was the “jittering” effect of the arrows, which was intense in long-distance paths. The users observed the arrows to be shaking in high frequency, which was a result of the high demand of spatial mapping in scenarios with increased amount of persistent spatial points.

Fig. 2.
figure 2

Left: \(WO-PSP\) users spent more time navigating in middle- and long-distance paths; middle: \(W-PSP\) users put more effort in accomplishing the navigation tasks; right: \(WO-PSP\) users had higher focused attention than the \(W-PSP\) users.

Full size image

4 Discussion

Our study results revealed that a trade-off between the development effort, the end-user performance, and the end-user experience when considering (or not) persistent spatial points in AR navigation. This trade-off depends on the length of the navigation paths. In short-distance paths, there is no need of including persisting spatial points that increase the development effort, because the user performance and user experience are comparably similar to the condition where persisting spatial points are used. However, as the length of the navigation path increases, it is critical for the developer to use persisting spatial points in order to ensure a high user performance and an enhanced user experience. But, as the length of the navigation path increases, the development effort also increases, and thus, smart and efficient solutions should be considered to overcome such limitations.

As a direct implication of our work, the developers of AR navigation tools should consider the length of the supported paths to decide whether to use persistent spatial points. A more sophisticated way of handling the observed trade-off would be the consideration of adaptive dual-mode AR navigation tools that activate the use of persistent points after determining the length of the path. Regarding the long-distance paths, the developers should consider releasing spatial points that are not needed, through dynamic allocation and prioritization techniques, to minimize the application footprint. For example, when a persistent point is active, the system will keep the points around it activated, while the others will remain deactivated. To mitigate the jittering and drifting effects, the augmented objects should form clusters that are dynamically rendered based on stationary reference points. To handle the trade-off, we could also investigate the minimum distance between the successive persistent spatial points, aiming to identify the minimum required number based on the length of the paths. Moreover, considering that the density of the points affected the rendering, as a future step we will experiment with density levels to provide additional insights about maximum area size for navigation and maximum density within the path.

To further decrease the development effort, we envisage the re-use of created persistent spatial points through an open and cloud-based repository. Through such a repository, the developers could upload and download spatial points, and thus, they contribute to collective generation of large and interconnected navigation paths and worlds, which can be used for the design of multi-user and collaborative navigation experiences. Cloud infrastructures could host persistent spatial points that would be dynamically available to the users, contributing to a realistic application footprint, an improved user performance, and an enhanced user experience while keeping the development effort minimum (e.g., a developer is not required to scan an area than had been scanned by another developer in the past) in varying distance conditions.

Regarding the limitations of our work, our sample size was relatively small, but, the statistical tests performed met the required assumptions. Another limitation is the fact that the end-users were familiar with the city, and as a future work, we will investigate scenarios with users unfamiliar with the city. Moreover, we will investigate the effect of other factors that influence user experience in AR settings, such as cognitive characteristics [14].

5 Conclusion

In this paper we evaluated the use of persistent spatial points in an AR navigation tool from a developer and an end-user perspective. We performed a two-fold empirical study with one software engineer, who created two versions of an AR navigation application (one with persistent spatial points and one without), and twenty-eight individuals who used the application to navigate. The analysis of the results revealed a trade-off between development effort, user performance, and user experience, which depends on the length of the navigation path. The shorter the path is, the less the need for persistent spatial points is, whilst, the longer the path is, the more critical it is to use persisting spatial points. The present study provides evidence on the importance of the trade-off between development effort and user experience, and is a starting point towards incorporating the new emerging technologies in everyday tasks.