1 Introduction

The museum is generally a non-profit institution in the service of a community that exhibits the tangible and intangible heritage of humanity environment which include art pieces for which the public use for the purposes of education and enjoyment. The atmosphere in a museum is wide and of an open space. Visitors freely walk around and see different collections of exhibit items, while keeping themselves quiet to not bother other guests. Museum visitors often question themselves about the origin of pieces on display and the stories behind them. Some questions can be about the inspiration, tools used, and or goal of the piece. All the answers to these questions are assumed by the visitors’; unless there is supplementary information available or an attendant to the piece. The attendant could be the artist themselves.

This paper shows an exploratory work of visualizing the story behind museum exhibits by using augmented reality (AR). We proposed another way of gaining insight behind the pieces, by the way of augmenting the “behind the scenes” of museum exhibits. Instead of asking visitors to download a specific application to their smartphones, the implementation had a more physical component. The visitor had to use a cube box and position it in front a camera installed on a mobile cart with a television. Once the box was positioned correctly in front of the camera, a side specific augmentation was displayed on the television. The experience was in an open but dedicated space, so that both the user of the AR experience and fellow museum goers were able to see all the different augmentations on the television. We worked with the University of Nevada Las Vegas (UNLV) Marjorie Barrick Museum to create an augmented reality experience that showcased how AR bridges museum visitors to the exhibits. The museum had recently installed a new exhibition which was done by the artist Andrew Schoultz and behind the stories of the artist’s exhibits were visualized by AR.

2 Background

Augmented Reality in short AR, is an immerging technology that superimposes computer-generated images on a user’s view of the real world generally using a mobile device that is equipped with a camera on it [1]. AR provides the user with a composite view of real world and computer-generated virtual world which is called augmentation [2,3,4].

There have been several different approaches in AR for museums, art galleries and historical sites [5, 6]. The paradigm of AR is that it is getting more end users based on the dramatic technological advance in AR [7]. One of the most recent examples of using AR in the museum is the Smithsonian Museum’s Skin and Bones exhibit that adopted AR for providing visitors with the supplemental information of the skeletons and bones being displayed [8]. The concept behind the project was to augment the bones displayed with the physical body of the animal when it was alive. For example, the users hold their tablet up to the bones and fish are augmented on to the bones.

There are other museums that adopted AR to provide visitors with a unique augmented experience for museum visitors to get a better visualization of the exhibits [9]. The National Museum of Singapore used AR in augmenting various plants and animals as graphics called, Story of the Forest [10]. In the experience, museum visitors use a mobile app equipped with a camera on their phones to view virtual plants and animals in real environment as well as to get supplemental information of them within the drawings. Museum visitors can take photos of or with the augmented scene and then the app provides more information about the plant or animal captured. The Kennedy Space Center in Florida applied AR to provide an immersive experience to visitors called, Heroes and Legends where AR brings holograms of astronaut royalty to life and tells their stories in front of visitors interactively [11]. Jinsha Site Museum in Chengdu, China created 3D models of relics [12, 13]. Visitors to the museum can use an AR app to explore the relics and see them in 3D. It allows visitors to learn more about how the relics look like and what they were used for. Google launched a new feature called Pocket Gallery as art of the Google Arts and Culture app [14]. It uses AR to allow people step inside of an art gallery through their mobile phone. This helps people see every art piece around world without the need to travel, just needed a mobile phone.

All the previous cases are based on an individual’s smartphone camera as an AR museum that each user uses his or her own phone to get the augmentation of an object shown in Fig. 1(a). This approach would effective for a large-scale environment where many people visit the site, but it may not be effective in a small-scale environment where the walk flow of the exhibit is less dynamic with the small number of people. What if we trigger the augmentation by a tangible object, that is recognized by a camera from a remote distance? The concept of camera to target is still intact, but the mediums to do so are different, such as displayed in Fig. 1(b). The paper goes over the implementations and concept of object to camera augmented reality; which includes triggering the target and augmented the specific visuals.

Fig. 1.
figure 1

Two different ways of augmenting objects

Full size image

3 The OTC Design and Prototype

The OTC (Object To Camera) AR system was implemented with three separate components – an OTC (Object To Camera) Mobile Display, an OTC Tangible Trigger and an OTC application. More descriptions of the three components are described below.

3.1 OTC Mobile Display

A structural embodiment of the OTC was based on constructing a portable mobile cart consisting of a 32-in. LED display, an Android phone equipped with a camera, a mobile cart and a battery. These items were chosen so the OTC mobile display can be easily transported and put together for use. The highest of the display was critical, the display should be based on the height on the museum goers. But, also be not too tall to prevent discomfort when trying to position the target object in front of the camera. A battery was used to make the OTC Mobile Display as a standalone mobile AR display and not to rely on positioning near power outlets. Figure 2 shows an overall appearance of the mobile display.

Fig. 2.
figure 2

OTC mobile display

Full size image

3.2 OTC Tangible Trigger

The OTC Tangible Triger was a cube box as an instance in the project. It was a cube shape cardboard box size around 20″ × 20″ × 20″ shown in Fig. 3(a) donated by shipping services at UNLV. We intentionally made the box with the size in order to give two constraints in terms of weight and size. The prototype was fitted with a total of six target images shown in Fig. 3(b) and they were affixed on each side face of the box shown in Fig. 3(c). Testing occurred thereafter and the testing system was an Android phone with the software downloaded and the cube with the images.

Fig. 3.
figure 3

OTC Tangible Trigger

Full size image

3.3 OTC App

The OTC application app was developed in Unity and Vuforia [15, 16]. Each side of the cube features an image that triggered an augmentation associated with the picture of the side. Six images corresponding to each surface of the OTC Tangible Trigger were uploaded onto a Vuforia database which Unity uses for image recognition [17]. The overall algorithm implemented in the app is shown in Fig. 4. As shown in the figure, OTC App first detects and recognizes a target image then locate and bring up its corresponding augmentation on the 3D cube model while aligning its size and position. Once triggered correctly, the augmentations would stay with its corresponding side. If the object was rotated with the same side and augmentation being shown, it would follow the objects juxtaposition.

Fig. 4.
figure 4

An overall three steps of augmenting in the OTC app

Full size image

Augmentation

Each side of the cube has one of Andrew’s art pieces on it. We wanted to share some more insight behind his pieces, so we augmented features such as images and video to the art. Some of the augmentations included a time-lapse of Andrew painting one of the murals at the museum and another including a video of his completed artwork done at a local skate park in Las Vegas. Other sides of the cube were augmented with images that had a connection with the artwork. For example, one of Andrew’s pieces was a painting of a knight’s helmet, so the team augmented an image of the Vegas Golden Knights’ logo onto it to connect the piece to the city. The team also included a credits section on the cube in which the DEX and museum team members were augmented.

Distance to recognize

We measured the minimum and maximum distance to recognize an object with the camera. The minimum distance was 5 feet and the maximum distance was 20 feet. However, the 20 feet distance was measured once the image target had already been detected at a closer distance. The initial distance needed for the target to be detected varied but seemed to be more consistent at a distance of less than 10 feet.

3.4 Integration and Testing

Once all the AR effects were being triggered correctly, we started using our event day equipment; which was the OTC object, OTC mobile display and OTC application all used together. The OTC application used the androids phone camera; which was the engine behind rendering the augmentations and displaying them. The android phone running the OTC application then was mounted on the television, with the camera facing objects in front of the television screen, as displayed in Fig. 2. The phone was connected to the television via HDMI port, so the interactions being shown on the cell phone could be displayed on the television. Figure 5 shows the two snapshots of the OTC App showing augmenting a story associated with one side of the OTC Tangible Trigger box displaying on the side of the box. All of the augmentations were positioned to the side of the box, not on top of the target. This was chosen so the connection between the target artwork and the “behind the scenes” augmentation. Figure 5(a) shows when the OTC App does not detect the target and Fig. 5(b) shows when the OTC App does detect the target and align the augmentation on the surface of the physical box.

Fig. 5.
figure 5

Snapshots of the OTC Setup and Testing

Full size image

4 Museum Installation

We worked with the UNLV Marjorie Barrick Museum to create an augmented reality experience that would showcase the arts and STEM/STEAM to the community. Figure 6 shows an overall view of the UNLV Marjorie Barrick Museum.

Fig. 6.
figure 6

UNLV Marjorie Barrick Museum

Full size image

The museum had recently installed a new exhibition which was done by the artist Andrew Schoultz. We wanted to use his art pieces in our OTC approach so we could keep our project relevant to the museum. We decided to create an OTC experience in which the community could learn more about Andrew’s art through augmented reality as part of the event’s sub-program called, STEAM AT THE MUSEUM and installed the OTC system shown in Fig. 7. We were given a large empty space for this AR attraction. The AR experience required users to hold the OTC Box, trigger an image target detection using the android phone camera running the OTC application, and overlay augmentations which are associated with his art pieces on the monitor of the OTC Display.

Fig. 7.
figure 7

Install the OTC system in the museum

Full size image

5 Results and Discussions

The OTC system was successfully installed in the museum as part of the STEAM at the museum sub program. Figure 8 shows children are experiencing the OTC in the museum.

Fig. 8.
figure 8

Children visitors experiencing the OTC experience

Full size image

The OTC experience was a success and received a lot of positive reactions. Most of the visitors were teachers from the community and other were children with their families. Most of the visitors that came to learn more about our project had very little knowledge of AR, but they were excited to learn more about it and where it could go in the future. The OTC Box has two constraints that were effective for adult attendees giving them two different effects - one in physical manipulation of the tangible cube-based user interface and the other in cognitive awareness allowing them to check each surface of the art piece with curiously and found it out with its augmentation. The same constraints were effective with our children participants.

A unique interaction that was made during the event, was the juxtaposition between the augmentations and the target images of the artwork. Once an augmentation was triggered, it would stay juxtaposed to the object; until a new target was triggered. What made this interaction special for the event was participants could rotate both the target and augmentation. Since the target and augmentation correlated to art, participants perspectives caught out information that could not be seen with holding the target in the same place once the augmentation was triggered.

One concern that we observed during the event was that the size of the OTC Box a bit too large to be handled by a child. We invited children attendees to try out the OTC Box for themselves, but when they held the box to the camera; they had a hard time viewing the augmentations because of the size of the box. However, it turned out to be a positive effect that a few other children joined the child on the manipulation of the box and worked together. The box enabled them to collaborate on handling of the box. Another concern that our team captured was that our phone running the AR application was quickly becoming discharged. Thus, it would be better to have the application downloaded on multiple devices to allow other phones to charge while the other is in use and would have demonstrated the portability of the AR application on different devices.

An observation conducted at the community event showed that interactive AR has potential applications in the museum to enhance the learning process of art related STEM/STEAM knowledge. From the media art perspectives, the OTC experience presented in the project was based on displaying art, in which we used both cases listed above. We superimposed digital images/video on a physical object for immersion and to see how much interest would be garnered throughout the event. Some attendees came back for a second trial, after see all of the artist exhibitions in person. During this trial they did not ask questions but made further connections between the augmentations showing the “behind the scenes” information of the pieces displayed at the museum. This was our initial goal, but the uniqueness and separation of our event drew people in before seeing all the exhibitions at the museum. It may have been better to have placed our event at the end of a set path that showcased a majority of the exhibitions, before arriving at ours.

6 Conclusions and Future Works

This paper explored the use of AR in a public museum in collaboration with the UNLV Marjorie Barrick Museum. We introduced the OTC (Object To Camera) allowing users to manipulate a tangible object as a user interface to trigger graphical augmentations. The goal was to make this unique interaction something that a user would not forget; due to its rareness and likeness. Also, to use this experiment as a learning experience for creating new and unconventional ways of using AR as a medium.

As an instance of the OTC experience, we showcased one’s artwork through the system that linked the behind scenes of six art pieces. The OTC experience program was a success and we gained more insight on future AR projects. The visitors and collaborators of the project were both satisfied with the new approach showed in the OTC experience and mediums of AR interactions. The OTC experience program was created to cross the boundaries of traditional AR programs that would require just a device to host the AR features. We had a physical object and a larger user interface display, which led to people who have already used AR experience something that felt new. The data we received through feedback and observation will help our team in future experiments and developments. A contribution made in the project was to have a more physical approach in creating AR experiences, to distinguish itself from other augmented reality experiences, in which objects can be more used in the context of AR experience as means of interacting with different augmentations.

As an extension of the work, we planned to extend the installation of the OTC system and invite more people to experience it.