Keywords

1 Introduction

The application of augmented reality (AR) technologies in recent years opened up many new possibilities to incorporate AR into our daily lives. In [3], by conducting a live user study, it has been shown that augmenting various points of interest (POIs) with images and facts about each POI has a potential to enhance user experience in tourism. In this work, we aim towards this goal of exploiting the AR technologies for tourism and transform the famous buildings into an unique experience by bringing iconic characters onto the buildings wall to increase the engagement of users. With a goal to create a unique experience using the virtual characters and the iconic buildings of a touristic city, we developed a mobile application that projects a virtual character when a user visits the building. By conducting a live user study, our system demonstrated has a high usability and users using our system felt a pleasant and enjoyable experience.

Fig. 1.
figure 1

On the left we present some of the selected buildings with their corresponding augmented characters to the right.

Full size image

2 Talking House: Transforming Buildings into Characters

Augmented reality has the potential to enhances a touristic experience by inserting virtual elements onto the buildings in a city. In general, famous touristic buildings and architectures attract a large number people. In this work, we propose a technology which is based on AR to transform buildings into intelligent characters to increase the engagement of users when visiting new places. Our techniques transforms a building or part of a building into a visible character with which the user can interact in order to learn useful information (or listen to stories) about the building. In this section, we describe the details of our prototype and various steps we adopted in creating the AR experience.

We have developed an AR mobile application called ‘Talking House’ for a small part of Zurich city’s old town called Niederdorf. This area offers a variety of buildings with visually distinct features, which enabled us to craft unique virtual characters. Furthermore, many buildings in Niederdorf offer a rich history of Zurich’s past. Users can then visit buildings in the city of Zurich that are supported by our application and experience our talking house functionalities.

As a first step, we manually selected the buildings by visiting the areas in Niederdorf and researching the history of the place. For our final application, we selected 11 buildings and one monument to augment with virtual characters. In Fig. 1, the left images gives an overview of some of the selected buildings. Once the buildings are identified, the next step is writing engaging character dialogues using the history of the buildings and their surrounding area. These dialogues vary in content ranging from a conversation between characters and historical facts to jokes and poems. Afterwards, we recorded the dialogues with the help of two professional voice actors. We then created a visual look and animation for each character by considering the geometry of the buildings and the spoken dialogues. Our characters are designed using simple shapes like squares and circles to better blend the virtual elements with the real world buildings. For example, the window of a building is used to create animated eyes for a character. Finally, we enhanced the dialogues by adding background music and sound effects to make the AR experience more immersive. We also share the videosFootnote 1 of our virtual characters talking to a user about the building.

Fig. 2.
figure 2

Virtual reconstruction with immersal

Full size image

3 AR Technology and Localization

To achieve a seamless experience, the AR virtual content objects must be properly rendered and aligned with the buildings (onto the walls) in the real world [2]. Augmenting buildings and monuments with virtual characters requires precise techniques to estimate the position and rotation of the user’s device. This is one the main aspects to ensure an authentic AR experience since otherwise the virtual content would be misaligned with the real world. For our application, we use the Immersal SDKFootnote 2 which allowed us to spatially map real-world locations and then augment them with digital content. As an initial step, we went to our selected locations of the buildings to create a spatial map of the area. This is done by taking numerous pictures of the streets and buildings from multiple viewpoints. Then, each set of images is uploaded to the Immersal Cloud Service which generates a 3D point cloud and a textured mesh (see Fig. 2). This textured mesh can then be used inside the Unity editor as a point of reference to align the virtual characters with their corresponding buildings. Aligning the character manually by drag and drop with the mouse is too imprecise. We therefore used the Kabsch algorithmFootnote 3 to automate the alignment process by selecting reference points between the character assets and the textured mesh. When a user visits one of the augmented buildings, they first have to move the mobile device across the surrounding buildings. Immersal then captures the current camera frame, computes its point cloud and tries to find a match with one of the stored point clouds. If there is a match, the digital character is shown correctly aligned with the building.

Implementation Details: We developed our AR application using the Unity engine which is a popular development platform to create video games and 3D applications. In order to optimize the memory usage and performance of our mobile application, we split the animation for each character into smaller parts (face, mouth, eyes etc.) and imported each of them as a sprite sheet into Unity. Each sprite sheet contains a sequence of images which are then combined by our app to create the final animation. Our source codeFootnote 4 is publicly available to extend our application for other cities. We also released our app as a beta version in the Google Play StoreFootnote 5 supporting the experience for the Niederdorf area of Zurich.

4 The AR Experience: The Square and the Hunt

In this work, we present two unique experiences that we designed to interact with the users. The first one is called as ‘The Square’ and in this experience the character will tell the story about the building’s history and other anecdotes about the building’s surroundings. The user can remain in one location and interact with different virtual characters appearing in front of surrounding buildings. The second experience is called as ‘The Hunt’ which is similar to a scavenger hunt where the user has to find buildings using the hints given by virtual characters. The experience will start with a virtual ghost named as ‘Stussi’ who appears to provides the hint to find the buildings. The user has to find these buildings based on the given hints and collect rewards like ‘rings’ at each building.

5 Experimental Evaluation

We conducted a user study to test and evaluate the usability of our application. We asked the participants to experience both the Square and the Hunt. In total, 9 people participated in our experiments. They were aged between 18 to 35 and the majority were either undergraduates, research students or working employees. In our experiments all our participants had to complete both the square and the hunt experiences. At the end, to evaluate the perceived usefulness of our application users filled out a questionnaire. We used System Usability Scale (SUS) [1], a simple and well established tool which consists of 10 statements for measuring usability. Our user study demonstrated that our prototype has a high usability and was generally perceived as pleasant and enjoyable experience. For perceived user enjoyment, we asked users about how much they enjoyed both the experiences. We found that 70\(\%\) of users strongly enjoyed the hunt and 100\(\%\) enjoyed the square. All participants of the user study agreed that our application was enjoyable to use when visiting the city buildings. 70\(\%\) users strongly enjoyed the hunt and 100\(\%\) enjoyed the square. Users were also interested in using such an application for other attractions in Zurich or other cities. Finally when asked about which experience to enhance further 72\(\%\) users wanted to see more advancements towards the hunt experience.

6 Conclusion and Future Works

In this paper, we have presented our AR system, a mobile AR application which provides users an unique experience by transforming building into a visible character. Our application shows a live character talking to a user when he visits a famous building in a city. Such techniques will have a great potential to enhance a user experience and in our future work, we will explore towards the direction of adding personalized advertisements on the buildings. We aim to also design machine learning models for automatic content creation such as stories, news or jokes where the character can converse or engage with users with up to date information.