Basic Study on Connecting AR and VR for Digital Exhibition with Mobile Devices

Abstract

In this paper, we propose a prototype of an AR–VR (Augmented Reality-Virtual Reality) connected system in which users can enter a space in which they cannot go physically while emphasizing spatial consistency between the real and VR space. Museums have important collections that cannot be displayed, such as crumbling buildings, lost sceneries, and fragile trains. To display them, museums have used AR or VR systems. In AR systems, the visitors can appreciate the collections with their context because AR systems emphasize spatial consistency, and users can compare the current and past scenery around the collections. In VR systems, the visitors can interact with the collections with a high degree of freedom because VR systems have spatial extensibility and the users can ignore the physical limitations. Therefore, we connect the AR and VR systems to take advantage of the technology in appreciating collections with high comprehension of their context and degree of freedom. We implemented a prototype AR–VR connected system in which users can transition from the real space to the VR space seamlessly. In our system, they first superimpose mobile tablets onto the museums’ collections, such as a crumbling building. Then, the current scenery in the tablet will gradually change to a VR scenery where they cannot enter physically. Finally, they can move around the VR space using their own body. Through our experiments, we found that the concept of connecting the AR and VR systems could be acceptable, and the system could work well in actual exhibitions.

1 Introduction

Museums have attempted to convey their exhibits effectively in many ways. The exhibits, as well as its background information, are important to be conveyed to their visitors [1]. However, museums cannot convey the background information of their collections, such as crumbling buildings, lost sceneries or too fragile trains because of their great value and their physical constraints. To solve this problem, we focused on AR and VR exhibition systems with respect to its spatial consistency and spatial extensibility.

The AR system can emphasize a spatial consistency and have an effect of conveying a correspondent positional relationship between the real scenery and the superimposed scenery [2]. As a result, users can easily notice what is changed and unchanged in the place, which leads users to understand the background information of the place more deeply [3]. On the other hand, the VR system can provide experiences beyond the real space constraints by showing elaborate CG images. When users wear an HMD and sit on the chair, they can go anywhere they want to go. In addition, given the rising the technique to distort users’ perception of space, such as Redirected Walking [4], the VR system can emphasize a spatial extensibility. For example, when users wear an HMD and walk around the room, they can feel as if they could go to a prohibited or even a lost place, and move a lot larger than it actually is. Therefore, if the AR and the VR systems are integrated into an AR–VR connected system with respect to its spatial consistency and spatial extensibility, the AR–VR connected system can lead users to understand the background information beyond the real space constraints. For example, when users visit the site of a building where another building is already standing, by using the system, it is possible for visitors to enter inside and appreciate the past building where they cannot physically enter.

Therefore, we implemented a prototype AR–VR connected system that can connect the real space and VR by offering both AR and VR system’s advantages, namely, spatial consistency and spatial extensibility.

2 Related Work

2.1 Conveying Background Information by Using AR Exhibition System

On-site virtual time machine [2] is an on-site AR exhibition system that superimposes past-scene photographs onto a current scene through a PC display using Bundler and PTAM. This study suggests that compared to simply viewing printed images on-site, seeing superimposed images on a PC display can allow users to understand the background information of the contents more deeply. Reliving Past Scene Experience System by Inducing a Video-camera Operator’s Motion with Overlaying a Video-sequence onto Real Environment [5] is also an on-site AR exhibition system. In this system, users search for the photographed position. Upon reaching the ideal location to view an AR content, the system starts to overlay a past scene in video materials onto the real environment and gives users the experience on how the camera operator captured the scene by inducing them to move similar to the operator. The Westwood Experience [3] is also a mobile on-site AR system in which users can view some contents associated with the real landscape and appreciate a 360-degree spherical image. Users can look at a 360-degree spherical image, while the system can detect the gyro sensor value in their tablet.

These studies show that the on-site AR system can convey the background information of the content, and active appreciation methods, such as searching for the photographed position, can induce the interest of the users in the content itself. However, in the AR exhibition system, instead of its spatial consistency, users have to be in the exact location as the photographed location and cannot appreciate the content from various view points.

2.2 Providing Expanded Experience in the Constrained Real Space by Using VR Exhibition System

The digital display case [6] enables users to interact with an intangible object in real space, such as clay figures and train trucks, through the exhibition case. In this system, users can use a controller and control the intangible object in the VR space and understand the mechanism that cannot usually be seen. In addition, VR applications, which can enable a walkthrough in the image-based VR space, are becoming popular [7, 8]. These applications can enable users to experience image-based VR spaces by allowing them to explore larger spaces despite not moving around the real space. These system and applications can enable users to experience larger and not constrained spaces even in small real spaces, but they actively restrict the users’ activity to appreciate the contents themselves because these systems can be used without moving their own body. However, Redirected Walking [4] can be mentioned as a method of walking in a large VR space by using the users’ body. By showing a distorted motion of the users in the real space to users in the VR space, the system can make users change their real walking direction without noticing it, and the system can allow the users to move around a larger virtual environment. The important point is that using this system, the users can move around a larger VR space despite being in a small real space.

To take advantage of the AR system, the AR–VR connected system should run on a mobile tablet, and the system should detect the users’ movement, such as translation and rotation, to apply the concept of Redirected Walking. To detect the movement of users, the Mobility Change-of-State Detection Using a Smartphone-based Approach [9] can be applied. The system can detect the users walking motion by using mobile devices, such as smartphone and other tablets. In this system, the user wears the smartphone on the waist belt and presume the user’s state, namely, walk, stop, stand, and sit, according to the value of its acceleration sensor. Especially, to detect the walking state, the system monitors a sudden change in acceleration in the vertical direction.

Therefore, by adding appropriate gain for the movement of users and using the acceleration sensor in mobile tablets, the system can allow the users to move around a VR space despite being under the real space constraints, and the system can take advantage of both the AR and VR systems.

3 Design of the Prototype AR–VR Connected System

3.1 Concept of AR–VR Connected System

In this chapter, we introduce our implementation of the prototype AR–VR connected system, which can emphasize a spatial consistency as an AR system, as well as emphasize a spatial extensibility and provide experiences beyond the real space as VR system. To take advantage of the AR–VR system, the proposed system should run on a mobile device because users have to move around the real space. As a result, the main required functions of our proposed system are as follows:

• 1. AR mode for emphasizing a spatial consistency.

• 2. Seamless connection between AR–VR mode.

• 3. VR mode for providing experiences beyond the real world constraints.

3.2 AR Mode for Emphasizing a Spatial Consistency

The main function of AR mode is to convey an accurate correspondent relation between the real scenery and the scenery that appears later by superimposing the user’s mobile tablet onto the target object.

First, the users are instructed by the proposed system to superimpose their own tablet onto the real object that has some correspondence with the place, such as a sign of demolished building or a building itself, or an ornament involved in the place. Second, the users walk around to find the scenery shown on the screen. Third, the users superimpose their tablet onto the real object (Fig. 1). As a result, the system can detect the relative position and angle of the real object and users using the image recognition function.

Fig. 1.

Concept of the AR mode in the system

Through these procedures, the users can instinctively understand that what they will see next on their tablet is related to the superimposed real object because of its characteristics as on-site AR system. For example, when they superimpose their tablet onto a sign of demolished building, they will see a past image of the building with the same sign as they are superimposed. Finally, they can easily understand the sign of the building.

3.3 Seamless Connection Between AR–VR Mode

The main function of Seamless connection between AR–VR mode is to transit to the VR mode from the AR mode while taking advantage of the AR system.

To transition to the VR mode while taking advantage of the AR system, the users must understand the positional relationship between the real space and the VR space. Therefore, the system gradually transits the appearance on the tablet from the scenery of the real space to the landscape of the VR space. The system recognizes three-dimensional relative positional relationship and relative angle with the gyro sensor and the camera on their tablet to reflect appropriately the movement of the user in the real space to the movement of the user in the VR space. For example, when they superimpose their tablet onto a site of a building, the users’ view will gradually invade the inside of the building (Fig. 2), and the system keeps the relative position and angle between the users and the building to reflect them in the VR space.

Fig. 2.

Concept of the seamless connection between the AR–VR mode in the system

3.4 VR Mode for Providing Expanded Experience in the Constrained Real Space

The main function of the VR mode is to allow users to move around the VR space according to the movement of the users’ body. Specifically, when users move in the real space, the viewpoint in the VR space moves according to the two types of users’ movements (Fig. 3). To expand the range that the users can move in the VR space, the system can apply a certain degree of gain to the movement of users in the real space. For example, when the users in the real space walk 1 m, the viewpoint in the VR space can move 3 m.

Fig. 3.

Concept of the VR mode and expanding the VR space

Translational Movement in VR space:

The system can detect the users’ walking movement by monitoring the changes in acceleration of their tablet and value of the gyro sensor. When users walk, the waveform of the acceleration becomes a specific shape, but the value of the gyro sensor does not change significantly. Therefore, while the system monitors the values of the acceleration and gyro sensors, the system can determine whether they are walking or not.

However, given the users’ rotational movement or other unexpected motion, this walking detection function sometimes fails to detect the walking motion. To correct this misrecognition, we use iBeacon [10], which can broadcast BLE (Bluetooth Low Energy) that the system can catch from each iBeacon. As a result, according to the value of RSSI from each iBeacon, the system can roughly recognize the users’ position in real space. Therefore, we set some beacons in the real space and when the users’ position in the VR space is greatly different from the users’ position in the real space, the system corrects the users’ position in the VR space.

Rotational Motion in VR space:

The system can detect the users’ rotational motion by using the gyro sensor on their tablet.

Specifically, when users look around in the real space, the value of the gyro sensor changes and the system detects the degree of the user’s rotation. Therefore, when users look around in the real space to look around in the VR space, the system can rotate the line of sight of the users in the VR space.

However, given that the gyro sensor drifts after using the system for a long time, the direction of the users’ line of sight will turn to the wrong direction in the VR space compared to the users’ line of sight in the real space. To correct this drifted value, we use a compass and a camera on the tablet. Using a compass, the system can roughly detect which direction the users are facing and the system monitors the changes in the value of the compass and gyro sensor for every five seconds. Then, the system can correct the value of the drifted gyro sensor by comparing the values of the compass and gyro sensor. In addition, by using a camera to apply an image recognition function, the system can calculate the relative angle between the superimposed object and the users. Therefore, when the system detects the superimposed object on the screen, the system corrects the angle in the VR space according to the relative angle.

4 Experiments at Actual Exhibitions

4.1 Experimental Purposes

These experiments are aimed to evaluate whether the concept of connecting the AR system and the VR system can be acceptable for the users, and evaluate the system’s possibility of providing the background information of historical objects. We conducted experiments in actual exhibitions, which were evaluated by many users.

In the first experiment, using the prototype system, we measured the relationship between the users’ activity and users’ evaluation to the AR–VR connected system itself. In the second experiment, we exhibited an application with the system, which can show users a VR space twice as large as the real space. We compared the AR–VR system and a conventional VR system in which users can appreciate the VR content without any spatial consistency and with virtual joystick with respect to the users’ interest in the content.

4.2 Experiment 1: Evaluation on the Indicators to Measure the Performance of the System

Detailed Procedure:

We exhibited an iPad application with a system under prototype “Walking in the Memories” to evaluate the indicators to measure the performance of the system. Using the application, users can move around the exhibition room in the real space (about 10 m × 4 m). We archived a past 360° scenery of the exhibition and made VR space in which users could move around similar to ordinary image-based VR contents. Therefore, when users move around the room, they can see and compare both the past scenery and the current scenery. In addition, we had two VR spaces, namely, the exhibition one year ago and the exhibition half a year ago (Fig. 4, left). Then, the users could see the current exhibition and two previous exhibitions on the iPad.

Fig. 4.

Concept of the VR space in the experimental application. Left: similar sizes of multiple VR spaces. Right: incompatible size of the VR space.

When users touch the iPad and start the application, they can see a tutorial of the system. After the tutorial, the users superimpose the iPad onto a small sign of the exhibition as AR mode. In a few seconds, the scenery on the screen transitions to next exhibition, and in a next few more seconds through the screen, the users can see the current scenery of the exhibition. Then, on the iPad, the users can see the previous exhibition and move around. After the screen transitions to the VR screen, the users in the VR space can move around according to their movement. Since we install some iBeacons in the exhibition, when the system catches the value of iBeacons, it can correct the position of the users. Finally, when the users tap the quit button on the application or return the iPad to its original location on an exhibition stand, the screen transitions to a questionnaire view, and users can answer the questionnaires. We had 359 users who participated in this experiment.

Results and Discussion:

Figure 5(a) shows the relationship between the average play time and the answer users provided after finishing the application. For the question “Did the movement between you in real space and in VR space match?” the users answered with a Likert scale. For example, if a user thought his movement matched greatly, he answered as 7 points. The correlation coefficient is 0.97, and the degree of coincidence of the users’ action has a strong correlation with the play time. Figure 5(b) also shows the relationship between the average play time and the answer. For the question “Did you feel interesting in experiencing application by moving your body?”, the users responded with Likert scale. The correlation coefficient is 0.99, and the degree of interest in playing the application by moving the body has a strong correlation with the play time.

Fig. 5.

Relationship between play time and the answer values with Likert scale

These results suggest that when users feel that their movement and the movement in the VR space are matched, their play time will increase. In addition, when they feel interesting in playing the application by using their own body, their play time will also increase. Therefore, we thought that to evaluate the system’s accuracy of the movement detection capability and the users’ interest in the content with the AR–VR connected system, the play time of an application with the system can be one of the main indicators.

4.3 Experiment 2: The Evaluation of the System for Spatial Consistency and Spatial Extensibility

Following the previous experimental results, the concept of the AR–VR connected system could be substantially acceptable and be practically used by ordinary users. Therefore, in this experiment at an actual exhibition, we evaluate whether the spatial consistency could lead to the increase in the users’ interest in the content, and whether the spatial extensibility could be acceptable by using the users’ own body.

In the exhibition, similar to Experiment 1, the users can touch the iPad with the system and see the tutorial. After the tutorial, the users superimpose a small sign of the application near the exhibition stand. Then, the system transitions the users to the VR space where there is a similar sign as on the exhibition (Fig. 6). As a result, when the users move to the VR space, they can see and compare the sign on the exhibition and on the VR space. The VR space is about 1.5 times as large as the real space (the real space: 3.0 m × 3.0 m; the VR space: 5.5 m × 2.3 m). The system increases the amount of walking movement in the VR space relative to the user’s real step (Fig. 4, right). In the VR space, users can see some contents they can interact with, such as gazing at some content for a while and starting a movie that explains the VR space. We exhibited the application with the system under different conditions as follows.

Fig. 6.

The screen of the application with the system and the exhibition space

The system with/without a virtual joystick:

In this system, we compared which method could attract the users’ interest: the proposed VR input method, walking and rotation by using the users’ body, and an ordinary VR input method such as virtual joypad. In the VR mode, the system can randomly choose which input method should be applied for each user. After the AR mode, the users can move around the VR space with/without a joystick. We had 494 users (253 users with Joystick condition, 241 users without Joystick condition) who participated under the condition with/without Joystick.

The system with/without AR mode:

In this system, we evaluated whether the AR mode could provide the spatial consistency and help users to have an interest in the content. Then, the system can randomly choose with/without the AR mode. With the AR mode, after the tutorial, the users have to superimpose the iPad onto the target, and the view of the iPad gradually changes into the VR view. In addition, the relative angle between the users and the target is detected and reflected to the VR view. However, without the AR mode, after the tutorial, the screen transitions to the VR mode immediately. We had 344 users (169 users with AR condition, 175 users without AR condition) who participated under the condition with/without the AR mode.

Results and Discussion:

Figure 7 shows the average playtime under the two conditions: with/without Joystick and with/without the AR mode. In Fig. 7 (left), the red bar shows the average time with a joystick and the blue bar shows the time without a joystick. In Fig. 7 (right), the red bar shows the average time with the AR mode and the blue bar shows the time without the AR mode. The playtime with a joystick is 48.4 s and the playtime without a joystick is 51.1 s. The Mann–Whitney U test shows that there is no significant difference (p = 0.12). The playtime with the AR is 52.3 s and the playtime without the AR is 49.7 s. The Mann–Whitney U test shows that there is a marginal difference (p = 0.02). Figure 8 shows the rate of the users who made interaction in the VR content at least once. In Fig. 8 (left), the red bar shows the rate of users with a joystick and the blue bar shows the rate without a joystick. In Fig. 8 (right), the red bar shows the rate with the AR mode and the blue bar shows the rate without the AR mode. The rate with a joystick is 0.29 and the rate without a joystick is 0.32. The Chi-square test shows that there is no significant difference (p = 0.60). The rate with AR is 0.32 and the rate without AR is 0.22. The Chi-square test shows that there is no significant difference (p = 0.13).

Fig. 7.

Play time of the application under the two conditions. Left: with/without Joystick as input method. Right: with/without the AR mode.

Fig. 8.

Rate of users who interacted with the item in the content at least once. Left: with/without Joystick as an input method. Right: with/without the AR mode.

In terms of spatial consistency, we thought that since the playtime was longer and the rate was higher than not applying the AR mode, the application of the AR mode could be effective for providing the relationship between the actual location of the users and the VR space, and for increasing the users’ interest in the content. This scenario could be attributed to the view of the current scenery that could transition to the view of the VR scenery with warp-like effect under the condition with the AR mode. Therefore, the users could have a sense of spatial continuity: a sense that the object in the real space and the object in the VR space are the same object. The sense of spatial continuity could help the users in comparing the real space and the VR space. Therefore, the users could easily have an interest in the VR space itself. These results suggest that the users’ interest in the content could be increased by the spatial consistency. In terms of the input method, using users’ own body could be acceptable for users similar to the use of a virtual joystick. This finding could be caused by the accurate system detection of the user’s motion and the user’s natural understanding of the system. In addition, despite the larger VR space than the real space, the users could play the application naturally. Therefore, these results suggest that using their own body and expanding the users’ experience space have good compatibility.

5 Conclusion

In this paper, we proposed a prototype of an AR–VR connected system, which runs on personal mobile devices. The system can enable users enter a space in which they cannot go physically while emphasizing spatial consistency between the real space and the VR space. The AR system can emphasize spatial consistency and have an effect of conveying a correspondent positional relationship between the real scenery and the superimposed scenery. On the other hand, the VR system can emphasize a spatial extensibility and provide experiences beyond the real space constraints. Therefore, integrating the AR system and the VR system into an AR–VR connected system while respecting its spatial consistency and its spatial extensibility can lead users to understand the background information beyond the real space constraints.

We implemented a prototype AR–VR connected system, which can connect the real and VR space by offering both AR and VR system’s advantages, namely, spatially consistency and spatial extensibility. After the users superimpose their tablet onto the real object, the system gradually transits the appearance on the tablet from the scenery of the real space to the scenery of the VR space. Then, users can move around the VR space according to the movement of the users’ body. For example, when they superimpose the tablet onto a site of a building, the view on the tablet will gradually enter the inside of the building where they cannot physically enter. Next, the users in the VR space can move around and appreciate the inside of the building by using their body.

Through our experiments, we found that the AR mode in the system could induce users to have interest in the content. We also found that using the users’ own body and expanding the VR space could have good compatibility. These results suggest that the concept of connecting the AR system and the VR system could be acceptable, and the system could work well in actual exhibitions to provide background information of their collections.