Encyclopedia of Computer Graphics and Games

Living Edition
| Editors: Newton Lee

Mixed Reality, Gamified Presence, and Storytelling for Virtual Museums

  • George PapagiannakisEmail author
  • Efstratios Geronikolakis
  • Maria Pateraki
  • Victor M. López-Menchero
  • Michael Tsioumas
  • Stella Sylaiou
  • Fotis Liarokapis
  • Athina Grammatikopoulou
  • Kosmas Dimitropoulos
  • Nikolaos Grammalidis
  • Nikolaos Partarakis
  • George Margetis
  • Giannis Drossis
  • Martha Vassiliadi
  • Alan Chalmers
  • Constantine Stephanidis
  • Nadia Magnenat-Thalmann
Living reference work entry

Later version available View entry history

DOI: https://doi.org/10.1007/978-3-319-08234-9_249-1

Synonyms

Definition

Mixed reality as display technology, gamification as motivational element, and storytelling as interaction metaphor while maintaining the feeling of presence are identified as key elements for defining the next generation of virtual museums.

Introduction

Storytelling, presence, and gamification are three basic fields that need to be taken into account when developing novel mixed reality applications for cultural heritage, based on the recent renaissance of commercial VR and AR hardware. This survey aims to cover a gap in the bibliography and the last relevant surveys of Papagiannakis et al. (2008), Jung et al. (2011), Foni et al. (2010), and Anderson et al. (2009) which are more than 7 years old, whereas some of these topics are mentioned in several different chapters within Ioannides et al. (2017) but not in single, comparative study. Section “The Role of Storytelling in Cultural Heritage” of this survey covers the state of the art in storytelling for cultural heritage and virtual museums, section “The Role of Presence in Cultural Heritage” the role of presence, and section “The Role of Gamification in Cultural Heritage” the usage of gamification principles. In the last section “Survey of Recent MR Methods for Virtual Museums,” a comparison among latest methods in the above areas is presented.

The Role of Storytelling in Cultural Heritage

Storytelling in Museums

Museums have realized the value of storytelling the second half of the twentieth century. Nowadays museums find themselves competing with the large offer of cultural products coming not only from the cultural sector but also from the entertainment industry. Therefore, museums need to differentiate and make their products more appealing and attractive to a variety of audiences. Influenced by the “new museology” concept and the transformation of the museological practice, which refers to a shift in the social roles of museums encouraging new styles of communication and expression, in contrast to the “cultural authority” of museums based on classic, collections-centered models, museums have changed.

Museums can act as primus inter pares (first among equals) and create a direct communication with the visitors, in which museum is the communicator and the visitor is the receiver and vice versa, providing the visitor the opportunity to actively participate in the story. Museums can have a conversation with the visitor, which can share her/his experiences and personal views. To this end, recent AR/VR commercial h/w technological advances enable the use of sophisticated tools to deliver virtual museum stories and information in a number of ways for experience enhancement, knowledge construction, and meaning making (Sylaiou et al. 2009).

Storytelling Using AR and MR (Merging AMI Installations with Mobile Devices and Physical Artifacts Through Stories)

Storytelling in Mixed Reality

Static visualizations have been traditionally employed to support storytelling in the form of text, diagrams, and images. The adoption of dynamic approaches utilizing state-of-the-art 2D and 3D graphics is emerging in an effort to explore the full potential of interactive narration. Rather than simply constituting an additional layer to exhibitions, the enhancement of CH institutions through interactive MR exhibits added value to the overall user experience (Marshall et al. 2016), especially if combined with personalization to each user’s interests (Partarakis et al. 2016). Interactive storytelling in MR environments bridges digital and physical information, augmenting the real world and offering interaction which corresponds to the user’s actions in the physical space. Storytelling can be employed for associating tangible and intangible information; such an example is Huang and Huang (2013), where the authors compound information for the promotion of indigenous cultural heritage.

Storytelling Authoring

Authoring refers to the process of creating narrations that form a digital story. In terms of authoring, the most common story types are character-based stories (Cavazza et al. 2002), linear timeline-based stories (Bimber et al. 2003; Drossis et al. 2013b), and ontology-based stories (Casillo et al. 2016). In terms of authoring virtual worlds, Lu et al. (2008) present an editing environment for facilitating the construction of 3D museums. Additionally, narratives description and structuring is also performed using authoring tools. Ardito et al. (2017) create a similar tool for storytelling creation and customization, focusing on the aspects of smart objects integrated in CH installations. Modern cameras are equipped with depth sensing capabilities allowing the real-time scanning of surroundings with sufficient detail. Such cameras are increasingly being adopted by consumer smartphones, such as the Google Tango compatible devices (Klingensmith et al. 2015). Therefore, such approaches can facilitate augmented reality environment rapid prototyping and authoring in a straightforward manner without requiring user expertise on computer science.

The Role of Presence in Cultural Heritage

“Presence” refers to the phenomenon of people behaving and feeling as if they “are there” in the virtual world created by computer displays (Ioannides et al. 2017). It is an incredibly powerful sensation, which is unique to MR, as it is not possible to recreate it in any other medium.

The Role of Illumination in MR Presence for Cultural Heritage

Introduction

There are two key components that are necessary in order to achieve the right illumination and thereby a high sense of presence in a MR cultural heritage environment: authentic appearance of the light and the correct dynamic range. Prior to the introduction of electricity, past societies relied entirely on daylight and lighting from flames for illumination. Any MR cultural heritage environment which is lit by modern lighting would thus not be authentic. In addition, the real world contains a wide range of lighting conditions, from dark shadows to bright sunshine. The MR environment needs to recreate this dynamic range of a scene. Where this is not possible using existing technology, techniques such as tone mapping need to be used.

High-Dynamic-Range (HDR) Environments

The dynamic range in a scene is defined as the ratio between the darkest part of the scene and the brightest. While the human visual system is able to adapt to the full range of light in a scene, traditional (also known as low-dynamic-range (LDR)) imaging is not able to capture or display a dynamic range of more than 256 to 1 (8 stops). High-dynamic-range (HDR) imaging, on the other hand, by using 32 bit IEEE floating point values to represent each color channel, can capture and display all the visual data in a scene. Failure to capture the full range of visual data can lead to important information being missed due to under- or overexposed regions in the image. Even if a scene has been captured, or created in HDR, detail can still be lost when the image is subsequently displayed on a device which is not HDR. In such a case, the HDR images need to be tone mapped in order to attempt to preserve the perception of the real scene on the LDR device. Many tone mapping operators (TMOs) have been presented over the years. More recently new TMOs have needed to be developed specifically for displaying 360o HDR images on head-mounted displays (HMDs) which are not yet HDR display devices. This is because the lighting in a full environment can vary significantly depending where the person is looking. HDR imaging is especially important in MR environments to ensure the virtual objects are relit with the same levels of lighting as in the real scene. As such, HDR is increasingly being been used within CH applications to improve their authenticity.

The Impact of Virtual Narrators to Presence in Virtual Museums

One of the important aspects which enhance the feeling of presence in CH (Papaefthymiou and Papagiannakis 2017) is the interaction with and behavior of the virtual narrators. These narrators should have humanlike behaviors, so that the visitors will feel like the virtual narrator that is speaking to them is real and that he/she is in the same room with them. An issue that rises at this point is how a virtual narrator can be as realistic as possible, with humanlike behavior and characteristics. For a virtual character to look as real as possible, it is not enough just to create him/her programmatically with code. It would be more realistic if that character could be reconstructed out of a real human. That way, the appearance of the character would resemble the appearance of the real person based on whom this character was reconstructed. This can be realized by scanning a real person with the Occipital™ depth sensor, specially designed for this task. The photogrammetric method is one of the best methods yet to reconstruct the texture of a real person. By scanning the person with the special sensor mentioned before and applying the texture, a 3D model of that person can be created. That way, a 3D model of a real person will be able to be used in computer graphics and mixed reality applications, in order to tell a story and communicate with the people that will use the specific mixed reality application, as shown in Figs. 1 and 2 (Papaefthymiou and Papagiannakis 2017).
Fig. 1

The hologram of the priest of the Asinou Church and the viewer interacting via gestures through Microsoft HoloLens, in the ITN-DCH project (Papaefthymiou and Papagiannakis 2017)

Fig. 2

Digitization of the priest of Asinou Church (to reconstruct his 3D model as a virtual narrator) using the Structure Sensor (Papaefthymiou and Papagiannakis 2017)

Mobile AR Interactive Applications for Virtual Museums

Mobile augmented reality systems (MARs) are increasingly currently being tested in rich content heritage environments by both creative professionals and laymen. Recent mobile hardware such as GPU-enhanced, multicore smartphones and novel untethered AR headsets (e.g., HoloLens by Microsoft) pave the way for a new breed of AR services and applications; however there are a number of issues to consider which regard the aspects of information presentation and physical interaction (Papaefthymiou and Papagiannakis 2017; Papagiannakis 2017; Ioannides et al. 2017; Li and Duh 2013). With respect to direct hand interaction, there are limitations in handheld devices where the free hand is used for interaction with the 3D objects (e.g., small screen size, unsuitability of point and click gestures for manipulation, finger occlusion), whereas in HMDs hands are an intuitive input channel although bimanual interaction cannot ensure better performance than single-hand interaction, and it is important to optimally assign functions to two hands. For example, in recently introduced ARKit by Apple, which represents a commercial tipping point in terms of mass-market adoption of AR technology, interactions are kept simple with object’s movements restricted on a plane, and the possibility of recognition of conflicting gestures. On the HMD side Microsoft HoloLens, although it has promised a hands-free experience, interaction can be frustrating given the limited field of view and the misreading of gestures. Although the promise of MARs is turning into reality, the technical challenges with respect to computational efficiency, information retrieval from different data sources, markerless detection, and hand gesture recognition performance still affect the overall user experience when interacting with MARs.

Immersive Experiences for Interaction with Cultural Heritage

Mixed Reality Installations

Mixed reality has the potential not only to increase motivation to learn but also to raise interest on CH. Thus, CH institutions can increase their appeal and enhance visitor engagement through interactive installations that include public displays (Partarakis et al. 2017). In addition to improving the aesthetic experience, mixed reality environments have influence on visitor experience, thus favoring the probability of revisiting a specific attraction (Jung et al. 2016). Apart from providing an enjoyable experience, mixed reality installations can facilitate cultural awareness, historical reconstruction, and heritage awareness. State-of-the-art approaches are not limited to installations in indoor spaces (Grammenos et al. 2012), but can also involve vehicles that act as portable kiosks (Zidianakis et al. 2016).

Setups and Interaction

Interactive installations in public spaces such as in CH institutions have certain requirements in terms of interaction and setup. The installations need to adapt to fit to the space available, provide content which interests both domain experts and non-expert users, and also present thorough information on demand (Mortara et al. 2014). At the same time, the system design should provide information immediately and support straightforward interaction techniques. Multiuser interaction with public displays is an open issue and constitutes an active area of research. Once people approach the interactive display, they decide their actions with regard to the system. Especially in the context of MR applications, the establishment of interaction with a public display involves transitioning from implicit to explicit interaction (Vogel and Balakrishnan 2004) as the users become engaged to the pervasive display.

Tangible interaction is a form of interaction with mixed reality installations in which physical items act as mediators between the users and the environment. One aspect of tangible interaction refers to physical objects, which are simply the medium of communicating information. For instance, physical paper is employed as a portable display, augmenting maps or glass-protected models (Grammenos et al. 2011). Interactive maps (Margetis et al. 2017) are a mixed reality application, where printed maps are augmented with additional multimedia information. Furthermore, tangible interaction can involve objects which have meaningful substance with a semantic meaning, such as smart objects (i.e., physical items equipped with sensors such as RFID tags). Smart objects can facilitate related information in its context of use (Petrelli et al. 2016).

Bridging Worlds: Combining AR, MR, and VR

Digital cultural heritage content is combined with physical 3D replicas (Bugalia et al. 2016) in order to provide a virtual tour in architectural sites using MR. This approach has the advantage of moving in a wall-projected 3D virtual environment while placing a physical model in front of the users to act as a minimap assisting navigation. Head-mounted displays are employed in the context of CH foundations for holographic AR and additionally support VR visualizations (Pedersen et al. 2017). VR can act as a medium to visualize CH exhibits otherwise unavailable to users as well as reenactments of historical events offering hands-on experience regarding unattainable content. Such indicative examples of VR usage are elements which are under conservation or even stolen (The Museum of Stolen Art 2017) and such as the battle of Thermopylae (Christopoulos et al. 2011), respectively. A different approach for mixing realities involves using tangible objects as souvenirs which summarize narratives as they were presented to users (Petrelli et al. 2017), acting as personalized keepsakes of their visit in CH institutions. Finally, an interesting approach at Geevor Tin Mine Museum (Jung et al. 2017) combines AR, VR, and 3D printed objects with the common goal of adding value to the overall user experience. The technologies are employed for enriching visits (AR), providing access to inaccessible areas (VR), and souvenirs which the users can keep, respectively (3D printing).

The Role of Gamification in Cultural Heritage

Gamification has been applied to a number of application domains including CH, and it is a popular approach to increase the entertainment and thus the motivation factor of users (Vassileva 2012). It is different from computer games even if it shares a lot of common theories and practices of development. It is considered to be one of the more recent developments of computer game design principles that have been researched since their appearance (Liarokapis et al. 2017a). A major objective of gamification is the improvement of users’ engagement that can provide positive effects, but the effects are heavily dependent on personalization (Hamari et al. 2014). Recently, an interactive tool called PhotoTrip was presented that can autonomously recommend CH locations along travel itineraries (Bujari et al. 2017). The application can identify these points of interest by gathering pictures and related information from social media and then provide suggestions and recommendations.

Gamification elements also exist in a number of simulations ranging from serious games to virtual and augmented reality. Similarly, edutainment and gamification are argued to be common recurring themes in the fields of education and cultural communication (Mancuso et al. 2017). In terms of virtual reality, gamification elements were recently shown in a kinesthetic application of sculpturing Cycladic figurines (Koutsabasis and Vosinakis 2017). The user takes the role of an ancient craftsman who creates a figurine with bare hand in a virtual reality environment. In another study, an immersive virtual underwater environment was developed to explore Mazotos shipwreck site and get relevant information about it and its contents (Liarokapis et al. 2017b).

In the following sections, key recent MR projects with rich gamification elements are presented.

Playful Interaction with Cultural Heritage Exhibits (Improved UX)

Immersion is based on covering physically a person’s stimuli, namely, vision, spatialized sound, and haptic feedback (Bowman and McMahan 2007). Haptic feedback (Ott et al. 2007) is examined in literature as an additional means of enhancing immersion. Kosmalla et al. (2017) combine tactile feedback from physical worlds with a virtual rock climbing environment in combination with full body movement and exertion. Immersion is also strongly related to the interaction process: in addition to perceiving a mixed reality (MR) application with human senses, the interaction modality employed constitutes a decisive factor in feeling of immersion and the overall user experience. Contrary to mission-critical domains such as a working environment, CH applications belong to a field in which users are more open to novel alternate interaction modalities. Requirements such as precision and efficiency are not fundamental in such a context, as users may be willing to sacrifice precision and overrule tiredness for entertainment and playful user experience (Drossis et al. 2013a). Playfulness encourages exploration, fosters creativity, and stimulates social interaction by entertaining users and allowing them to escape from the reality (Tsekleves and Darby 2017). Playful interaction is employed for attracting users toward public installations and therefore facilitating user engagement (Williamson and Sundén 2015).

Re-Play: A Cultural Heritage Project that Allows to Replay Ancient Games

In Europe only, there are over 3000 Traditional Sports and Games (TSG). The EU project Re-Play has focused on two families of TSG (Gaelic and Basque) that are integral to the fabric of their communities of origin and have successfully staved off this trend of convergence. The group at MIRALab, University of Geneva, has focused on the development of a realistic real-time animation and rendering platform that enabled the visualization of the virtual national sports heroes as well as the visualization of avatars representing the local heroes (Tisserand et al. 2017). This state-of-the-art animation-rendering system includes dynamic muscle effects, which are modeled over the skin using a novel GPU approach. Based on existing MRI datasets and accurate reconstructed 3D models, the Re-Play platform computes the displacement map and assigns weights to each coordinate in the displacement map based on EMG measurements. To increase the feeling of immersion, photo-realistic 3D models acquired using a 3D scanner have been employed to animate and render the virtual sports heroes.

Gamification Applications for Folklore Dance

Intangible cultural heritage (ICH) includes fragile expressions mainly transmitted orally or by gestures from one generation to the next. In order to support the learning and transmission of ICH expressions, gamification can also be used. Based on the theory of “experiential learning” (Kolb, D.A., 1984, Experiential learning: Experience as the source of learning and development, Englewood.), one of the main pillars of game-based learning, the acquisition of learning is achieved by observing, reflecting, mentally representing, and enacting movements. In accordance with this theory, within the i-Treasures (Dimitropoulos et al. 2018) project, a number of game-like applications for sensorimotor learning of specific dance types and other activities involving full body gestures were designed and developed to support the learning and transmission of a number of ICH expressions. Specifically, a serious game application for transmitting ICH knowledge concerning the Greek traditional dance “Tsamiko” was developed (Kitsikidis et al. 2015) and shown in the figure below. The game was structured as a set of activities, aiming to teach different variations of the dance allowing users to learn by either observing experts’ prerecorded movements in the Observe mode or by starting practicing the dance step sequences respectively in the Practice mode (Fig. 3).
Fig. 3

Observe and Practice screen of Tsamiko’s gamified application within the i-Treasures EU project (Kitsikidis et al. 2015)

During the Practice mode the learner is expected to imitate the moves of the expert avatar displayed on the screen. The addition of a virtual tutor to correct/manipulate/guide the user by providing visual and audio feedback was also supported in order to encourage the learners’ “participation” and engagement.

Survey of Recent MR Methods for Virtual Museums

In Table 1, a summary of key papers in the last 7 years, after the last relevant survey paper from Anderson et al. (2009), is presented. Although there is no specific MR method that features gamified storytelling with heightened interaction that still maintains full immersion and the feeling of presence, several conclusions and recommendations for next lines of research can be drawn and summarized.
Table 1

Comparison of recent MR methods for virtual museums

Mixed reality method

MR installation

Gamification

Storytelling

Interaction

Tracking

Immersion

Intangible heritage

Anderson et al. (2009)

Rome Reborn serious game

x

x

  

No

Ancient Pompeii application

x

x

  

No

Parthenon Project

x

x

  

No

Virtual Egyptian Temple

x

x

Walking/movement

Cave Automatic Virtual Environment (CAVE)

Full

No

The Ancient Olympic Games

x

x

Navigation wand of the VR system

 

No

Virtual Priory Undercroft

x

x

  

No

Commercial historical games

x

x

  

No

Bugalia et al. (2016)

3D printed models, projector, camera, laser pointer

x

x

Laser pointer

IR camera, project camera, view camera

Partial

No

Dong et al. (2017)

VR application using Focus 3DX 330 Faro scanner, Spheron PanoCam, two GoPro Hero 4 cameras for scanning, and utility for rendering

x

 

Depending on VR platform

Depending on VR platform

Full

No

Drossis et al. (2015)

3D model prototype (depth sensor, touch screen, interactive cube, projection)

x

 

Touch screen, interactive cube, walking/movement

RGB-D sensor, Kinect or Asus Xtion camera

Partial

No

Gimeno et al. (2011)

Region of Valencia map, seven pointers, and Hostess

x

 

Walking/movement

Map, pointers, Kinect camera

Partial

No

Grammenos et al. (2011)

Tabletop augmented reality system

 

x

Finger-based input

Projector and pieces of white paper

Partial

Yes/no

Javornik et al. (2017)

Apple iPad Pro

x

 

Touch screen

Inner iPad camera

Partial

Yes

Kitsikidis et al. (2015)

Microsoft Kinect Sensor

x

 

Motion capture

Skeleton tracking with Kinect Sensor

Partial

Yes

Kosmalla et al. (2017)

HTC Vive with Leap Motion

x

 

Controllers

Positional and hand tracking

Full

No

Koutsabasis and Vosinakis 2017

Leap Motion

x

 

Hand gestures

Hand and finger tracking

Partial

No

Liarokapis et al. (2017b)

HTC Vive

x

 

Controllers

Motion and laser based

Full

No

Margetis et al. (2017)

Interactive map tabletop system

 

x

Handwriting, gestures, touch/click

Projector, high-resolution camera, depth sensor

Partial

Yes/no

Nakevska et al. (2017)

CAVE

x

x

Back projection on walls

Pressure sensors

Full

Yes/no

Papaefthymiou and Papagiannakis 2017

Oculus Rift

x

x

Controllers

Rotational and positional (sensors)

Full

No

Papaefthymiou and Papagiannakis 2017

Apple iPad Pro

x

x

Touch screen

Apple’s ARKit for camera tracking

Partial

No

Papaefthymiou and Papagiannakis 2017

Microsoft HoloLens

x

x

Voice commands, gestures

Body motion

Partial

No

Papagiannakis et al. (2004)

DELL P4 M50 Mobile Workstation

x

  

Real-time markerless camera tracking

Full

No

Pedersen et al. (2017)

Meta developer Kit

x

x

Gestures

Markerless surface tracking, head movement

Partial

No

SpatialStories (2018)

Toolkit for VR/AR platform

x

x

Depending on VR/AR platform

Depending on VR/AR platform

Full/partial depending on the platform

No

Tisserand et al. (2017)

Computer with Kinect Sensor for traditional sports preservation

x

 

Movement

Kinect Sensor

Partial

Yes

The MR technologies that are used in the key papers (located in Table 1) contribute to the preservation of cultural heritage, each one with its own level of storytelling, presence, gamification, interaction, and tracking methods. Since all of the installations below are MR applications, many of them take into account the gamification field, in order for those applications to be interesting and fun for the viewers. Some of them include storytelling elements (e.g., Papaefthymiou and Papagiannakis 2017; Pedersen et al. 2017), which are used to inform the viewers about the story of a monument, for instance, thus contributing in cultural heritage curation. Furthermore, it can be noted that most of the MR methods below support partial immersion and few of them support full immersion. The term “immersion” is included in purpose as it can be easily quantified based on the display, whereas “presence” is elusive and depends on many parameters and thus difficult to provide that holds true throughout a simulation. VR HMDs thus support full immersion, whereas AR and holographic AR support partial one. Moreover, some applications that run tethered with a computer and do not support VR or AR provided no immersion at all. The majority of these key papers use mobile AR or holographic AR, and that explains the majority of the partial immersion entries on Table 1. A recommendation at this point would be that there should be more MR applications that support desktop or mobile VR with full immersion, as it creates a more realistic experience for the viewers because they do not have access to the real world and the feeling of presence is respectably higher than it is with partial immersion. These technologies can also be categorized in two additional categories, tethered and untethered, according to if an installation needs to be connected to a PC or not (standalone device), respectively. An example of an untethered MR technology, which can be found in the table below, is the Papaefthymiou and Papagiannakis (2017), which uses the Apple iPad Pro device in order to run. It uses Apple’s ARKit for camera tracking. The viewers’ movements are not limited by cables (since no cables are used), which enhances the feeling of presence as their movements would not be limited if they were exploring the real monument. Also, gamification and storytelling elements are supported, and along with the feeling of presence and freedom of movements, all those elements create the perfect experience for the viewers. Another version of this work runs on the Microsoft HoloLens holographic AR HMD, which is also an example of untethered AR.

On the other hand, there are some great MR applications (e.g., Pedersen et al. 2017; Drossis et al. 2015; Tisserand et al. 2017), which support gamification and/or storytelling elements, but they limit the movements of the viewers due to the existence of cables or need of a connection with a desktop PC. Although these applications are impressive and greatly contribute to cultural heritage preservation, they restrict the freedom of movement to the users, which in some cases may disrupt the feeling of presence. Based on this discussion, it is recommended for MR applications to be able to run in MR installations that do not restrain the movements of the viewers in any way. Without restriction of movements and with full immersion, the experience of a viewer will reach very high levels. Finally, SpatialStories (Foni et al. 2010), a very recent commercial effort, is a toolset for real-time interactive VR/AR experiences featuring storytelling for non-programmers. Although it has not yet been fully tested, from information gathered from their website, its manual, and its videos, it poses a promising commercial solution in contributing to MR cultural heritage applications.

Conclusions

In conclusion, the ability to innovate in the field of MR gamified storytelling while maintaining presence in the virtual museums of the future will be directly related to the ability to find synergies between the above fields of knowledge, without forgetting that new technologies can play a transcendental role in this equation. And as the quintessential reductive machine, MR technologies are employed as the device that will restore its unity through multiplicity and fragmentation. Hence this work needs to be considered as a further step toward reconciliation and a renewed mutual beneficial relationship between humanities, social sciences, and computer science.

Cross-References

References

  1. Anderson, E.F., McLoughlin, L., Liarokapis, F., Peters, C., Petridis, P., de Freitas, S.: Serious games in cultural heritage. In: Ashley, M., Liarokapis, F. (eds.) The 10th International Symposium on Virtual Reality, Archaeology and Cultural Heritage VAST – State of the Art Reports, St Julians, Malta, 22–25 Sept 2009, pp. 29–48Google Scholar
  2. Ardito, C., Buono, P., Desolda, G., Matera, M.: Empowering CH experts to produce IoT-enhanced visits. In: UMAP ‘17 25th Conference on User Modeling, Adaptation and Personalization, Bratislava, 9–12 July 2017, pp. 327–328. ACM, New York (2017)Google Scholar
  3. Bimber, O., Encarnação, L.M., Schmalstieg, D.: The virtual showcase as a new platform for augmented reality digital storytelling. In: IPT/EGVE03 Imersive Progection Technologies/Eurographics Virtual Environments, Zurich, 22–23 May 2003, pp. 87–95. ACM, New York (2003)Google Scholar
  4. Bowman, D.A., McMahan, R.P.: Virtual reality: how much immersion is enough? IEEE Comput. 40(7), 36–43 (2007)CrossRefGoogle Scholar
  5. Bugalia, N., Kumar, S., Kalra, P., Choudhary, S.: Mixed reality based interaction system for digital heritage. In: VRCAI ‘16 The 15th International Conference on Virtual-Reality Continuum and its Applications in Industry, Zhuhai, 3–4 December 2016, pp. 31–37. ACM, New York (2016)Google Scholar
  6. Bujari, A., Ciman, M., Gaggi, O., Palazzi, C.E.: Using gamification to discover cultural heritage locations from geo-tagged photos. Pers. Ubiquit. Comput. 21(2), 235–252 (2017)CrossRefGoogle Scholar
  7. Casillo, M., Colace, F., De Santo, M., Lemma, S., Lombardi, M., Pietrosanto, A.: An ontological approach to digital storytelling. In: MISNC, SI, DS 2016 The 3rd Multidisciplinary International Social Networks Conference, SocialInformatics 2016, Data Science 2016, Union, 15–17 August 2016, p. 27. ACM, New York (2016)Google Scholar
  8. Cavazza, M., Charles, F., Mead, S.J.: Character-based interactive storytelling. IEEE Intell. Syst. 17(4), 17–24 (2002)CrossRefzbMATHGoogle Scholar
  9. Christopoulos, D., Mavridis, P., Andreadis, A., Karigiannis, J. N.: Using virtual environments to tell the story: the battle of Thermopylae. In: 2011 Third International Conference on Games and Virtual Worlds for Serious Applications (VS-GAMES), pp. 84–91. National Technical University Great Amphitheater, Athens (2011)Google Scholar
  10. Dimitropoulos, K., Manitsaris, S., Tsalakanidou, F., Denby, B., Buchman, L., Dupont, S., Nikolopoulos, S., Kompatsiaris, Y., Charisis, V., Hadjileontiadis, L., Pozzi, F., Cotescu, M., Ciftci, S., Katos, A., Manitsaris, A. and Grammalidis, N.: A Multimodal Approach for the Safeguarding and Transmission of Intangible Cultural Heritage: The Case of i-Treasures. IEEE Intell Syst, pp. 1–1 (2018)Google Scholar
  11. Dong, Y., Webb, M., Harvey, C., Debattista, K., Chalmers, A.: Multisensory virtual experience of tanning in medieval Coventry. In: EUROGRAPHICS Workshop on Graphics and Cultural Heritage, Graz, 27–29 Sept 2017Google Scholar
  12. Drossis, G., Grammenos, D., Bouhli, M., Adami, I., Stephanidis, C.: Comparative evaluation among diverse interaction techniques in three dimensional environments. In: International Conference on Distributed, Ambient, and Pervasive Interactions, pp. 3–12. Springer, Berlin/Heidelberg (2013a)Google Scholar
  13. Drossis, G., Grammenos, D., Adami, I., Stephanidis, C.: 3D visualization and multimodal interaction with temporal information using timelines. In: IFIP Conference on Human–Computer Interaction, pp. 214–231. Springer, Berlin/Heidelberg (2013b)Google Scholar
  14. Drossis, G., Ntelidakis, A., Grammenos, D., Zabulis, X., Stephanidis, C.: Immersing users in landscapes using large scale displays in public spaces. In: International Conference on Distributed, Ambient, and Pervasive Interactions, pp. 152–162. Springer, Cham (2015)Google Scholar
  15. Foni, A., Papagiannakis, G., Magnenat-Thalmann, N.: A taxonomy of visualization technologies for cultural heritage applications. ACM J. Comput. Cult. Herit. 3(1), 1–21 (2010)CrossRefGoogle Scholar
  16. Gimeno, J., Olanda, R., Martinez, B., Sanchez, F.M.: Multiuser augmented reality system for indoor exhibitions. In: IFIP Conference on Human–Computer Interaction, pp. 576–579. Springer, Berlin/Heidelberg (2011)Google Scholar
  17. Grammenos, D., Michel, D., Zabulis, X., Argyros, A.A.: PaperView: augmenting physical surfaces with location-aware digital information. In: TEI ‘11 Fifth International Conference on Tangible, Embedded, and Embodied Interaction, Funchal, 22–26 January 2011, pp. 57–60. ACM, New York (2011)Google Scholar
  18. Grammenos, D., Zabulis, X., Michel, D., Padeleris, P., Sarmis, T., Georgalis, G., …, Adam-Veleni, P.: Macedonia from fragments to pixels: a permanent exhibition of interactive systems at the archaeological museum of Thessaloniki. In: Euro-Mediterranean Conference, pp. 602–609. Springer, Berlin/Heidelberg (2012)Google Scholar
  19. Hamari, J., Koivisto, J., Sarsa, H.: Does gamification work? – a literature review of empirical studies on gamification. In: Proceedings of the 47th Hawaii International Conference on System Sciences (HICSS), IEEE Computer Society, Waikoloa, pp. 3025–3034 (2014)Google Scholar
  20. Huang, C.H., Huang, Y.T.: An annales school-based serious game creation framework for Taiwanese indigenous cultural heritage. J. Comput. Cult. Herit. 6(2), 9 (2013)CrossRefGoogle Scholar
  21. Ioannides, M., Magnenat-Thalmann, N., Papagiannakis, G. (eds.): Mixed Reality and Gamification for Cultural Heritage. Springer, Heidelberg (2017).  https://doi.org/10.1007/978-3-319-49607-8Google Scholar
  22. Javornik, A., Rogers, Y., Gander, D., Moutinho, A.: MagicFace: stepping into character through an augmented reality mirror. In: CHI ‘17 CHI Conference on Human Factors in Computing Systems, Denver, 6–11 May 2017, pp. 4838–4849. ACM, New York (2017)Google Scholar
  23. Jung, Y., Kuijper, A., Fellner, D.W., Kipp, M., Miksatko, J., Gratch, J., Thalmann, D.: Believable virtual characters in human–computer dialogs. Eurographics (STARs). 2011, 75–100 (2011)Google Scholar
  24. Jung, T., tom Dieck, M.C., Lee, H., Chung, N.: Effects of virtual reality and augmented reality on visitor experiences in museum. In: Information and Communication Technologies in Tourism 2016, pp. 621–635. Springer, Cham (2016)Google Scholar
  25. Jung, T.H., Jung, T.H., tom Dieck, M.C., tom Dieck, M.C.: Augmented reality, virtual reality and 3D printing for the co-creation of value for the visitor experience at cultural heritage places. J. Place Manag. Dev. 10(2), 140–151 (2017)CrossRefGoogle Scholar
  26. Kitsikidis, A., Kitsikidis, A., Dimitropoulos, K., Uğurca, D., Bayçay, C., Yilmaz, E., Tsalakanidou, F., …, Grammalidis, N.: A game-like application for dance learning using a natural human computer interface. In: International Conference on Universal Access in Human–Computer Interaction, pp. 472–482. Springer, Cham (2015)Google Scholar
  27. Klingensmith, M., Dryanovski, I., Srinivasa, S., Xiao, J.: Chisel: real time large scale 3D reconstruction onboard a mobile device using spatially hashed signed distance fields. In: Robotics: Science and Systems, vol. 4 (2015).  http://dx.doi.org/10.15607/rss.2015.xi.040
  28. Kosmalla, F., Zenner, A., Speicher, M., Daiber, F., Herbig, N., Krüger, A.: Exploring rock climbing in mixed reality environments. In: CHI ‘17 CHI Conference on Human Factors in Computing Systems, Denver, 6–11 May 2017, pp. 1787–1793. ACM, New York (2017)Google Scholar
  29. Koutsabasis, P., Vosinakis, S.: Kinesthetic interactions in museums: conveying cultural heritage by making use of ancient tools and (re-) constructing artworks. Virtual Reality. pp 1–16. (2017).  https://doi.org/10.1007/s10055-017-0325-0
  30. Li, N., Duh, H.: Cognitive issues in mobile augmented reality: an embodied perspective. In: Huang, W., et al. (eds.) Human Factors in Augmented Reality Environments. Springer, New York (2013).  https://doi.org/10.1007/978-1-4614-4205-9_5Google Scholar
  31. Liarokapis, F., Petridis, P., Andrews, D., de Freitas, S.: Multimodal Serious Games Technologies for Cultural Heritage, Mixed Reality and Gamification for Cultural Heritage, Part V, pp. 371–392. Springer International Publishing (2017a).  http://dx.doi.org/10.1007/978-3-319-49607-8_15
  32. Liarokapis, F., Kouřil, P., Agrafiotis, P., Demesticha, S., Chmelík, J., Skarlatos, D.: 3D modelling and mapping for virtual exploration of underwater archaeology assets. In: Proceedings of the 7th International Workshop on 3D Virtual Reconstruction and Visualization of Complex Architectures and Scenarios, ISPRS, Napflio, pp. 425–431 (2017b)Google Scholar
  33. Lu, W., Zeng, D., Pan, J.: An XML-based scene description language for 3D virtual museum. In: 30th International Conference on Information Technology Interfaces, 2008, ITI 2008, Croatia Hotel, Cavtat/Dubrovnik, pp. 445–450. IEEE (2008)Google Scholar
  34. Mancuso, S., Muzzupappa, M., Bruno, F.: Edutainment and gamification: a novel communication strategy for cultural heritage. J Pub Archaeology. 1, 79–89 (2017)Google Scholar
  35. Margetis, G., Ntoa, S., Antona, M., Stephanidis, C.: Interacting with augmented paper maps: a user experience study. In: Proceedings of the 12th Biannual Conference of the Italian SIGCHI Chapter (CHITALY 2017), Cagliari, 18–20 Sept 2017Google Scholar
  36. Marshall, M.T., Dulake, N., Ciolfi, L., Duranti, D., Kockelkorn, H., Petrelli, D.: Using tangible smart replicas as controls for an interactive museum exhibition. In: TEI ‘16 Tenth International Conference on Tangible, Embedded, and Embodied Interaction, Eindhoven, 14–17 February 2016, pp. 159–167. ACM, New York (2016)Google Scholar
  37. Mortara, M., Catalano, C.E., Bellotti, F., Fiucci, G., Houry-Panchetti, M., Petridis, P.: Learning cultural heritage by serious games. J. Cult. Herit. 15(3), 318–325 (2014)CrossRefGoogle Scholar
  38. Nakevska, M., van der Sanden, A., Funk, M., Hu, J., Rauterberg, M.: Interactive storytelling in a mixed reality environment: the effects of interactivity on user experiences. Entertain. Comput. 21, 97–104 (2017)CrossRefGoogle Scholar
  39. Ott, R., Thalmann, D., Vexo, F.: Haptic feedback in mixed-reality environment. Vis. Comput. 23(9), 843–849 (2007)CrossRefGoogle Scholar
  40. Papaefthymiou, M., Papagiannakis, G.: Gamified augmented and virtual reality character rendering and animation enabling technologies. In: Ioannides, M., Magnenat-Thalmann, N., Papagiannakis, G. (eds.) Mixed Reality and Gamification for Cultural Heritage, pp. 333–357. Springer (2017).  https://doi.org/10.1007/978-3-319-49607-8
  41. Papagiannakis, G.: Gamification and serious games. In: Lee, N. (ed.) Encyclopedia of Computer Graphics and Games. Springer, Cham.,  https://doi.org/10.1007/978-3-319-08234-9_90-1, ISBN 978-3-319-08234-9 (2017)Google Scholar
  42. Papagiannakis, G., Schertenleib, S., Ponder, M., Arevalo-Poizat, M., Magnenat-Thalmann, N., Thalmann, D.: Real-time virtual humans in AR sites. In: IEE Visual Media Production (CVMP04), London, pp 273–276 (2004)Google Scholar
  43. Papagiannakis, G., Singh, G., Magnenat-Thalmann, N.: A survey of mobile and wireless technologies for augmented reality systems. J Comput. Anim. Virtual Worlds. 19(1), 3–22 (2008)CrossRefGoogle Scholar
  44. Partarakis, N., Antona, M., Zidianakis, E., & Stephanidis, C.: Adaptation and content personalization in the context of multi user museum exhibits. In B.N. De Carolis, C. Gena, T. Kuflik, & F. Nunnari (Eds.), Proceedings of the 1st Workshop on Advanced Visual Interfaces for Cultural Heritage (AVI*CH 2016), Bari, Italy, 7-10 June. Published by: CEUR-WS (Vol. 1621) (2016)Google Scholar
  45. Partarakis, N., Grammenos, D., Margetis, G., Zidianakis, E., Drossis, G., Leonidis, A., Stephanidis, C.: Digital cultural heritage experience in ambient intelligence. In: Mixed Reality and Gamification for Cultural Heritage, pp. 473–505. Springer, Cham (2017)Google Scholar
  46. Pedersen, I., Gale, N., Mirza-Babaei, P., Reid, S.: More than meets the eye: the benefits of augmented reality and holographic displays for digital cultural heritage. J Comput. Cult. Herit. 10(2), 11 (2017)CrossRefGoogle Scholar
  47. Petrelli, D., Dulake, N., Marshall, M.T., Pisetti, A., Not, E.: Voices from the war: design as a means of understanding the experience of visiting heritage. In: CHI ‘16 CHI Conference on Human Factors in Computing Systems, San Jose, 7–12 May 2016, pp. 1033–1044. ACM, New York (2016)Google Scholar
  48. Petrelli, D., Marshall, M.T., O’Brien, S., McEntaggart, P., Gwilt, I.: Tangible data souvenirs as a bridge between a physical museum visit and online digital experience. Pers. Ubiquit. Comput. 21(2), 281–295 (2017)CrossRefGoogle Scholar
  49. SpatialStories: A toolset for real-time interactive VR/AR experiences. http://apelab.ch/spatialstories (2018). Accessed 6 Jan 2018
  50. Sylaiou, S., Liarokapis, F., Kotsakis, K., Patias, P.: Virtual museums, a survey and some issues for consideration. J. Cult. Herit. 10(4), 520–528 (2009)CrossRefGoogle Scholar
  51. The Museum of Stolen Art.: http://mosa.ziv.bz/ (2017). Accessed 22 Dec 2017
  52. Tisserand, Y., Magnenat-Thalmann, N., Unzueta, L., Linaza, M.T., Ahmadi, A., O’Connor, N.E., Zioulis, N., Zarpalas, D., Daras, P.: Preservation and gamification of traditional sports. In: Ioannides, M., Magnenat-Thalmann, N., Papagiannakis, G. (eds.) Mixed Reality and Gamification for Cultural Heritage, pp. 421–446. Springer, Cham (2017)CrossRefGoogle Scholar
  53. Tsekleves, E., Darby, A.: The role of playfulness and sensory experiences in design for public health and for ageing well. In: Sensory Arts and Design. Bloomsbury p. 49–66. 18 p (2017)Google Scholar
  54. Vassileva, J.: Motivating participation in social computing applications: a user modeling perspective. User Model. User-Adap. Inter. 22(1–2), 177–201 (2012)MathSciNetCrossRefGoogle Scholar
  55. Vogel, D., Balakrishnan, R.: Interactive public ambient displays: transitioning from implicit to explicit, public to personal, interaction with multiple users. In: UIST ‘04 17th Annual ACM Symposium on User Interface Software and Technology, Santa Fe, 24–27 October 2004, pp. 137–146. ACM, New York (2004)Google Scholar
  56. Williamson, J.R., Sundén, D.: Enter the circle: blending spherical displays and playful embedded interaction in public spaces. In: Proceedings of the 4th International Symposium on Pervasive Displays (PerDis ‘15), Saarbruecken, 10–12 June 2015, pp. 195–200. ACM, New York (2015)Google Scholar
  57. Zidianakis, E., Margetis, G., Paparoulis, S., Toutountzis, T., Stratigi, K., Paparoulis, G., Stephanidis, C.: Turning an electric cargo vehicle into a portable interactive information kiosk. In: International Conference on Human–Computer Interaction, pp. 463–469. Springer, Cham (2016)Google Scholar

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  • George Papagiannakis
    • 1
    • 2
    Email author
  • Efstratios Geronikolakis
    • 1
    • 2
  • Maria Pateraki
    • 2
  • Victor M. López-Menchero
    • 3
  • Michael Tsioumas
    • 4
  • Stella Sylaiou
    • 5
  • Fotis Liarokapis
    • 6
  • Athina Grammatikopoulou
    • 7
  • Kosmas Dimitropoulos
    • 7
  • Nikolaos Grammalidis
    • 7
  • Nikolaos Partarakis
    • 2
  • George Margetis
    • 2
  • Giannis Drossis
    • 1
    • 2
  • Martha Vassiliadi
    • 8
  • Alan Chalmers
    • 9
  • Constantine Stephanidis
    • 1
    • 2
  • Nadia Magnenat-Thalmann
    • 10
  1. 1.Computer Science DepartmentUniversity of CreteHeraklionGreece
  2. 2.Foundation for Research and Technology HellasHeraklionGreece
  3. 3.Spanish Society of Virtual ArchaeologySevilleSpain
  4. 4.Hellenic Ministry of Culture and Sports, Service of Modern Monuments and Technical Works of Central MacedoniaThessalonikiGreece
  5. 5.Hellenic Open UniversityPatrasGreece
  6. 6.Masaryk UniversityPonava, BrnoCzech Republic
  7. 7.Information Technologies Institute, CERTHThessalonikiGreece
  8. 8.School of Philology, Aristotle University of ThessalonikiThessalonikiGreece
  9. 9.WMG, University of WarwickCoventryUK
  10. 10.MIRALabUniversity of GenevaGeneva 4Switzerland