Augmented Reality (AR) and Spatial Cognition: Effects of Holographic Grids on Distance Estimation and Location Memory in a 3D Indoor Scenario

Abstract

Recent advances in augmented reality (AR) technology enable the projection of holograms to a fixed location in 3D space. This renders new possibilities for influencing peoples’ spatial perception and to address cognitive limitations as structural distortions in cognitive representations of space. The study presented in this paper investigated whether these structural distortions can be reduced by projecting a holographic grid into 3D space. Accuracy of the cognitive representation of space was assessed based on distance estimations and an object location memory task. The findings revealed that distance estimations were indeed more accurate when a holographic grid was available. Location memory performance, on the other hand, was worse when a holographic grid was available. Based on feedback from the participants, it can be assumed that design characteristics of the used AR headset are at least partly responsible for this result. These characteristics include a reduced field of view and visual distortions in the peripheral areas of the field of view. Overall, the findings show that AR can be used to influence and, when applied correctly, improve peoples’ spatial perception. However, more research is needed to specify requirements, strengths, and weaknesses of geographic AR applications.

Zusammenfassung

Augmented Reality (AR) und Raumwahrnehmung: Effekte holographischer Gitter auf die Distanzschätzung und das Positionsgedächtnis in einem 3D-Innenraum- Szenario. Jüngste Fortschritte in der Augmented-Reality (AR)-Technologie ermöglichen die Projektion von Hologrammen auf eine bestimmte Stelle im 3D-Raum. Dadurch ergeben sich neue Möglichkeiten, die menschliche Raumwahrnehmung zu beeinflussen und kognitive Einschränkungen, wie strukturelle Verzerrungen von mentalen räumlichen Repräsentationen, zu korrigieren. Die vorgestellte Studie untersuchte, ob solche Verzerrungen durch die Projektion eines holografischen Gitters in den 3D-Raum reduziert werden können. Die Genauigkeit des mentalen Raummodells wurde auf der Grundlage von Distanzschätzungen und einer Positionsgedächtnisaufgabe bewertet. Die Ergebnisse zeigten, dass die Distanzschätzungen tatsächlich genauer waren, wenn ein holographisches Gitter zur Verfügung stand. Die Erinnerungsleistung der Objektpositionen war dagegen mit einem holographischen Gitter schlechter. Aufgrund der Einschätzungen durch die Probanden kann davon ausgegangen werden, dass die Designmerkmale des verwendeten AR-Headsets zumindest teilweise für dieses Ergebnis verantwortlich sind. Zu diesen Merkmalen gehören ein reduziertes Sichtfeld und visuelle Verzerrungen in dessen Randbereichen. Insgesamt zeigen die Ergebnisse, dass AR die räumliche Wahrnehmung des Menschen beeinflusst und bei richtiger Anwendung verbessern kann. Es sind jedoch weitere Untersuchungen erforderlich, um die Anforderungen, Stärken und Schwächen von geografischen AR-Anwendungen zu spezifizieren.

Introduction

The technology of augmented reality (AR) has left the stage of development and found its way into a large field of activities (Brito and Stoyanova 2018; Werner 2019). These activities include applications in business-driven fields, such as gaming and entertainment, marketing as well as geo-marketing, mechanical engineering, real estate, warehouse logistics, and architecture. In these application scenarios, the technical opportunities of AR systems are used to project additional elements into the physical world. These virtual holographic elements (incl. text) enrich the perception of the real environment (Schart and Tschanz 2015). The virtually projected information is available in real time in the field of view. In other words, AR technology provides a mix of virtual (illusory) elements and real world elements (Broll 2013; Milgram and Kishino 1994; Noor 2016). Therefore, AR can be classified as a type of mixed reality, which describes the continuum between unmodified reality and purely virtual environments (Milgram et al. 1995). Most AR applications focus on the relation of these virtual elements to the real environment. In contrast to purely virtual environments, which usually do not interact with objects in the real world, (e.g., Edler et al. 2018a; Hruby et al. 2019; Kersten et al. 2018), AR environments contain virtual information units that support the interaction with real spatial objects.

From a GIScience perspective, AR systems seem to be valuable tools that could improve performance in spatial orientation, spatial navigation, search processes, and decision-making (Carbonell Carrera and Bermejo Asensio 2017; Caudell and Mizell 1992; Çöltekin et al. 2019). They close a gap between abstract approaches to the communication and analysis of spatial information and society (Halik 2012; Liu et al. 2020; Lochhead and Hedley 2019). AR systems can also help to share (additional) spatial information and scientific understanding across networks of stakeholder groups (Londergan and Hedley 2014).

Some applications of AR have already been established as mass media elements, such as additional virtual cues in applications for mobile devices (Clarke et al. 2019). For example, animated arrows in (smartphone) navigation apps (see “Live View” app from Google Maps) or holographic projections onto the windshield of cars and other vehicles (head-up displays) are about to become widely used tools of everyday life. These examples show that people already intuitively rely on additional virtual elements to solve spatial tasks. Some empirical findings already indicate that AR interfaces provide advantages over common desktop interfaces. For example, Hedley (2003) showed in an early study that a 3D representation mediated by AR interface can improve perceptual and memory performance. Other empirical research points to improvements in mental rotation tasks if AR-based 3D holograms are used as training devices (Lee et al. 2016).

However, it remains an ongoing research topic to quantify the effects caused by virtual objects in larger spaces used for navigation and orientation scenarios. In this study, we investigate the effects of augmented virtual grids on the formation of mental representations of space.

Background

When people navigate through geographic space, they rely either on external navigation aids or on internally stored information about the geographic structure (Field et al. 2011). Based on their interactions with the geographical environment, they gradually shape out and improve their mental representations of this space, often called “cognitive maps” (Millonig and Schechtner 2007). This widely used term—originally introduced by Tolman (1948)—should, however, not be understood as a cartographic concept (Kitchin and Blades 2002). The organization of spatial information in memory could also be described as a “cognitive collage” (Tversky 1993). These cognitive representations are not flawless reproductions of geographic space, as they contain gaps and spatial inaccuracies (Mark et al. 1999). These inaccuracies are related to complex distortion errors caused by, for instance, hierarchical and perception-based grouping effects (Edler et al. 2015; Klippel et al. 2005; McNamara et al. 1989; Tversky 1993). Cartographic researchers have an interest to understand cognitive processes and to reduce these distortion effects in a systematic and superordinate manner. They develop cartographic construction rules that improve cartographic communication, following the direction of “cognitive map-design research” (Montello 2002).

Some decades ago, Eastman (1985a, b) already discussed the quality of linear map elements as suitable reference elements in 2D maps. Linear symbols are not only associated with geographical contents which they graphically represent. They may also structure maps and divide them into spatial chunks. These chunks can be interpreted as a set of regions and are proposed to be beneficial for the memory of objects, their locations, and configurations (Clements-Stephens et al. 2011; Edler et al. 2014; Hommel et al. 2000; Sargent et al. 2010). Evidently, the learning of map objects is linked to a superordinate reference frame determined by the positions of specific map features (Kuchinke et al. 2016). Thus, linear map elements can create a functional reference frame that can support the perception and memory of map objects. In an earlier study, Kulik and Klippel (1999) point to potentials of grid lines in maps for cognitive processes, such as the division of a map field to better organize and reference spatial information.

In the last years, different experimental investigations in the fields of cognitive cartography and spatial cognition identified specific linear map elements that support the reduction of distortion errors in spatial memory. It was shown that (content-related) linear map symbols, such as salient lines, representing an urban road network improve the object location memory performance (Edler et al. 2015). These lines do not appear as isolated lines, but as a linear system that creates boundaries of regions in the map. Furthermore, non-topographical geometric map elements, such as map grids, were identified as a robust tool to reduce distortions in object location memory. If a square or hexagonal grid pattern is overlaid on a topographic base map, study participants recall the locations of objects more accurately (Edler et al. 2014, 2019). The grid effect prevailed, even if grids are presented in a visually reduced way, such as grid crosses or illusory contours (Dickmann et al. 2017; Edler et al. 2014).

Grid-based effects also occur in experimental investigations on distance estimations in 2D maps. It was reported that grids in 2D maps lead to more accurate distance estimations and object location memory (Dickmann et al. 2019; Edler et al. 2014, 2015, 2018b). As both, object location memory and distances between objects, are essential aspects in the formation of mental representations of space, grids may serve as superordinate graphical reference frame reducing distortions in the mental representations of geographic space.

The HoloLens, an AR headset introduced by Microsoft (see Fig. 1), projects spatially fixed holographic objects directly into real world space. Integrated cameras track changes of the position and rotation of the headset and the displayed holograms are modified accordingly. Thus, the user is able to walk around the holograms and to perceive them from different perspectives. Even though some technical restrictions are still limiting wearing comfort and fields of view, the AR headset could be used to display a wide span grid in 3D space. This allows us to investigate whether the positive effects of grids in 2D maps on distance estimations and object location memory mentioned above are transferrable to 3D space—despite the obvious change of perspective and reduced field of view when wearing such an AR headset. To answer this question, we designed an experiment to examine the following two hypotheses:

H1

Holographic grids improve distance estimations.

H2

Holographic grids improve object location memory.

Fig. 1

The left side shows a photo of the HoloLens. The right side demonstrates how a green cube is projected onto the glasses of the HoloLens. Projecting different perspectives of the cube on each of the glasses creates the illusion that a 3D cube is floating in front of the user

If such positive effects on spatial perception can be found, AR creators could exploit the (design) opportunities of holographic spatial support structures and thereby improve peoples’ mental representations of space.

Methods

The following study has been controlled and approved by the ethics committee of the Faculty of Geosciences at the Ruhr-University Bochum.

Participants

The study sample consisted of 60 participants (28 females, 32 males) with an age range of 19–33 years (M = 24.7 years, SD = 3.2). All participants had normal or corrected to normal vision.

Research Design

The experiment consisted of three parts. The first two parts investigated the ability to estimate distances and the third part investigated object location memory. To assess potential effects of holographic grids on distance estimation and location memory, a between-subjects crossover design was used. Participants either wore the Microsoft HoloLens during the first two parts or during the third part. This AR headset was used to project a holographic grid into real world space. Male and female participants were evenly distributed across both conditions. In the following sections, the conditions will be compared separately for each part of the experiment (no grid/grid).

Procedure

In all three parts of the experiment, green cylinders (real world objects, height = 14 cm, and diameter = 7 cm) were placed on the floor of a large empty room (11.1 m × 6.4 m). In each part of the experiment, a fixed set of cylinder positions was used. These positions were similar for all participants from both experimental conditions (no grid/grid). Participants had to stand in a fixed position of the room. Additionally, participants from the grid condition wore the Microsoft HoloLens, which projected a holographic grid (see Fig. 2, spacing = 1 m, line width = 1 cm) onto the floor. A holographic interface developed by Keil et al. (2019) was used to adjust the grid to the floor height, the position, and rotation of the room.

Fig. 2

Holographic grid projected onto the floor with the Microsoft HoloLens

As Li et al. (2011) found structural differences between egocentric and frontal distance estimations, we decided to investigate these two measures separately (see Fig. 3). In the first part, participants were asked to estimate egocentric distances between their own position and the center points of nine cylinders located in front of them (distance P–O) in cm. The positions of the cylinders formed no specific geometric pattern and were identical for all participants. In the second part, participants had to estimate frontal distances between the center points of nine cylinder pairs located around them (distance O–O). Again, the positions of the cylinders were identical for all participants. Both cylinders of each object pair were located at the same distance from the participant.

Fig. 3

Demonstration of the two distance estimation tasks. First, participants estimated egocentric distances between their own position and objects in the room (P–O). Afterwards, they estimated frontal distances between object pairs (O–O)

The third part required participants to memorize and recall the positions of six cylinders positioned around them with no specific geometric pattern (object location memory). To allow differentiation between cylinders, each one was marked with a unique letter. First, participants had 45 s to memorize the positions of all cylinders (encoding phase). The duration was based on a pretest used to find a duration that was sufficiently long to capture the cylinder positions, but short enough to induce mild time pressure to stimulate a wider variation of recall performance. After the encoding phase, participants had to leave the room to perform a distractor task (search tasks in a crowded image). Meanwhile, all cylinders were removed. After 3 min, participants were brought back into the experiment room and were asked to place the cylinders at their original positions (recall phase). This required them to move through the room. Thus, the visual perspective differed from the perspective during the study phase. The available time to place the cylinders was limited to 2 min. Participants assigned to the grid condition wore the HoloLens and saw the holographic grid during both the encoding and the recall phase.

Measures

Based on the distances reported in the first two parts of the experiment, distance estimation errors were calculated as the absolute difference between the reported and the correct distances in cm. Additionally, directional distance estimation errors were calculated by subtracting the correct distances from the estimated distances (rounded to cm). This allowed us to investigate whether participants underestimated or overestimated distances under specific conditions. The extent of object location memory errors in the third task was calculated by measuring the distance between the center point of the original position of a cylinder and the center point of its recalled position in cm.

Statistics

The data of all three measures were aggregated per participant and task. This created two values per participant (mean distance estimation error and mean directional distance estimation error) in each of the two distance estimation tasks and one value per participant (mean location memory error) in the object location memory task. As the variances of the three measures differed between the experimental conditions and some measures were not normally distributed, the nonparametric Mann–Whitney U test was used to compare mean and mean directional distance estimation errors as well as object location memory errors between the two conditions (no grid/grid).

Results

The examination of distance estimations between the participants’ location and object locations in the room (P–O) revealed that mean distance estimation errors were significantly lower in the grid condition compared to the no grid condition (see Fig. 4, M_{no grid} = 0.597 m, M_grid = 0.474 m, U = 310, p = .039). In both conditions (no grid/grid) combined, distances were on average underestimated (M = − 0.371 m). Underestimations were more pronounced in the no grid condition, but the difference between the two conditions was not statistically significant (M_{no grid} = − 0.450 m, M_grid = − 0.291 m, U = 322.5, p = .060). Similar results were found for distance estimations between object pairs in the room (O–O). Again, mean distance estimation errors were lower in the grid condition (see Fig. 5, M_{no grid} = 0.349 m, M_grid = 0.217 m, U = 145, p < .001), distances were on average underestimated in both conditions (M = − 0.098 m), and underestimations were more pronounced when no holographic grid was available. However, in contrast to the P–O distance estimations, the difference of underestimations in the O–O task was statistically significant (M_{no grid} = − 0.151 m, M_grid = − 0.045 m, U = 303, p = .030).

Fig. 4

Estimations of distances between participants and objects in the room (P–O) were significantly more accurate when a holographic grid was available (p = .039)

Fig. 5

Estimations of distances between object pairs in the room (O–O) were significantly more accurate when a holographic grid was available (p < .001)

Analysis of the location memory errors revealed that the recalled object locations were closer to the original object locations when no holographic grid was available during the task (see Fig. 6, M_{no grid} = 0.476 m, M_grid = 0.713 m, U = 202, p = .001).

Fig. 6

Object locations were recalled significantly more accurate in the no grid condition (p = .001)

Discussion

The results of this experiment show that modern AR interfaces have the potential to improve the performance of basic spatial tasks, which corresponds to similar results reported by Hedley (2003). A holographic grid system mediated by a Microsoft HoloLens surface and projected onto an indoor floor supports distance estimations. The results also indicate that this advantage for distance estimations is not valid for the recall of object locations.

The first hypothesis addressed whether oblique holographic grids can improve distance estimations in real world space. With our findings from the first two experiment parts, we were able to demonstrate that the availability of a holographic grid indeed led to more accurate egocentric (P–O) and frontal (O–O) distance estimations. Therefore, we can confirm our first hypothesis. The results support and extend previous findings concerning effects of grids on distance estimations in 2D maps (Dickmann et al. 2019). Similar to grids in 2D maps, subdividing a 3D environment with oblique holographic grids seems to support distance estimations. Additionally, in agreement with previous findings, distances in both conditions were generally underestimated (Viguier et al. 2001), but underestimations were more pronounced for egocentric than for frontal distance estimations (Li et al. 2011). We assume that the underestimations are caused by the perspective view. With increasing distance from the observer, changes of the viewing angle between two objects (O–O) or one object and the illusionary horizontal line on eye level (P–O) decrease and are less likely to be perceived. Based on the significantly higher distance estimations in the grid condition, we conclude that the availability of a holographic grid system might counterbalance underestimations of frontal distances (O–O).

A question, which was not addressed with this experiment, is whether the implied use of grids for distance estimations is deliberate or implicit. As participants did not receive any information about the grid line spacing, the selected scale of 1 m may have implicitly stimulated the metric mindset of participants. However, they may also have consciously assumed that grid line spacing follows the metric scale and, based on this assumption, may have counted the amount of grid lines between two positions. Future experiments using for the metric scale uncommon grid line spacing conditions (e.g., 93 cm) or disclosure concerning grid line spacing may shed more light on this question.

The second hypothesis implied that holographic grids might improve object location memory. However, contrary to our expectations, the object recall performance was actually worse when a grid was shown during the task. Thus, we cannot confirm our second hypothesis. These first findings on the impact of grids in an ego-perspective 3D usage scenario contradict previous findings concerning grids in 2D maps (Dickmann et al. 2017; Edler et al. 2014, 2015, 2019; Kuchinke et al. 2016). This result may be based on other perceptual differences between the two conditions rather than the availability of holographic grids. Many participants reported that, during both the encoding and the recall phase, they felt restricted by the limited field of view provided by the Microsoft HoloLens. Indeed, the nontransparent headband (see Fig. 1) limits the vertical field of view and overlapping transparent surfaces deform and blur peripheral areas. Apparently, the reduced field of view impairs the building of a mental spatial representation, stronger than expected.

Whereas distance estimations only require the perception of two objects simultaneously, memorizing and recalling object locations requires comparison of an objects’ location relative to multiple other object locations (Loomis et al. 1999; Xu and Chun 2009). Thus, ideally, all to-be-learned objects should be visible simultaneously. The limited field of view and subsequent distortions in the peripheral field of view may, therefore, have counteracted a potential positive (holographic) grid effect on the performance of object location memory. For further investigations, we suggest to examine effects of holographic grids with a similar field of view in both conditions (no grid/grid). Participants could, for example, also wear the HoloLens in the no grid condition. Additionally, in contrast to location memory tasks using 2D maps, walking through the room and placing objects in the recall (memory retrieval) phase led to constant changes of spatial perspective. As this also affects the perceived shape of the holographic grid, availability of the grid may have made the object location recall task more difficult. Therefore, additional research comparing the effects of field of view widths and varying spatial perspectives on object location memory is required.

Summary and Outlook

Overall, the findings from our study yield mixed conclusions concerning the usefulness of augmented holographic grids for correcting distortions of cognitive representations of space. Distance estimations seem to benefit from the subdivision provided by holographic grids. However, the effects of holographic grids on location memory are still uncertain. Difficulties seem to arise when the used AR headset limits the field of view and distorts peripheral areas, or when the spatial perspective is unstable. To isolate grid effects from field of view effects, we suggest to replicate the experiment with participants from both conditions (no grid/grid) wearing the headset. Further potentials of holographic grids in real world space could be investigated by expanding the displayed grid into the third dimension. Such truly three-dimensional volumetric grids might support vertical distance estimations. Taken together, additional research concerning holographic grids, effects of a modified field of view, and effects based on changes of perspective in 3D space could broaden our understanding of how people perceive space and generate cognitive representations.