1 Introduction

The world of information and communication technologies (ICT) has been supporting the enhancement of cultural heritage for years. Various tools are now available to improve the use and preservation of historical memory. In cases where sites, monuments, historical and archaeological contexts are no longer, in whole or in part, usable, the use of supporting technological tools appears even more peculiar and necessary. Very often, the absence of tangible evidence surviving over time in historical contexts precludes the possibility of fully understanding the potential and history that the site itself originally represented.

Extended Reality (XR) technology, through the juxtaposition of information or virtual elements to the real context, contributes to redesigning the way of using the site itself, weaving together what is still visible and supporting virtual content. It allows users to engage directly through game-oriented environments and multichannel and multisensory platforms adopting an approach known as edutainment (De Paolis 2013). To date, the majority of XR applications realised are usable in indoor contexts, while outdoor use presents technical problems that are still only partially resolved. The application context of this study concerns the use of XR technologies (specifically Augmented and Mixed Reality) in outdoor contexts of cultural heritage enhancement. The idea is to give the user the perception of crossing different space–time dimensions, thus being able to fully live an experience that integrates a virtual scenario within the real world. In this context, the metaphor of virtual portals allows the virtual visit of monuments that are inaccessible or no longer exist in their original form, integrating them into the real environment. They allow the user to take a walk in the past, passing from reality (present) to virtuality (past) and vice versa.

In the present study, a mobile Mixed Reality application aimed at exploring outdoor cultural heritage using virtual portals is proposed and evaluated for user experience in a real-world (in situ) context. The developed application (CumeRA) is able to revive an old cult building, which no longer exists, by recovering its memory and enhancing its context of use. The innovative contribution compared to the state of the art was to analyse how the use of this type of application leads to significant improvements in user experience, especially regarding the sense of presence experienced. In this regard, the intrinsic features of virtual portals will be discussed in terms of interactivity, realism and sense of presence. In order to achieve this, the ARI questionnaire (Georgiou and Kyza 2017), which was originally developed for location-based Augmented Reality, was adapted and employed to assess the user experience. The collected data were then used to identify the key factors that define the user experience in a Mixed Reality environment and to establish the main correlations between them. Other data that emerged from the analysis of the results concern the type of experience administered, and the degree of realism expressed in the reconstruction of the virtual scenario.

The paper is developed as follows: Sect. 2 presents a brief survey of the state of the art; Sect. 3 introduces the church of Saint Elijah the prophet, i.e. the case study on which the developed application focuses; Sect. 4 discusses the problems encountered and the approaches that were adopted for the choice of all the materials useful for the reconstruction of the 3D model of the church; Sect. 5 describes the design and development phases of the application; Sect. 6 describes the experimental setup adopted in terms of the characteristics of the chosen hardware and the experimental sample selected for testing; Sect. 7 presents the analysis of the results obtained from the administration of the test to evaluate the usability of the application; Sect. 8 reports a discussion on the results obtained, also comparing them with those of other Augmented and Mixed Reality applications; Sect. 9 concludes the paper and hints at possible future developments.

2 Related work

The efficient use of XR applications in outdoor contexts, especially in the mobile environment, is still a topic of intense research and presents a number of challenges.

2.1 Virtual Reality applications

Traditionally, the use of Virtual Reality (VR), enjoyed through HMDs such as Oculus Quest or HTC Vive, has allowed visitors to museums, archaeological parks or, more simply, school students, a virtual but realistic journey through history (Schofield et al. 2018). In this context, De Paolis et al. 2022a proposes a VR application that allows the exploration of an ancient castle in Corsano, a small Italian village, and provides the user with historical information about it. Based on archaeobotanical studies, another VR application was developed that enables the exploration of the Upper Paleolithic settlement in the Museum of Preclassical Civilizations of the southern Murgia in Ostuni and allows the user to assist in a burial ritual (De Paolis et al. 2022b).

2.2 Mobile Augmented Reality applications

The need to move freely and unconstrained in the surrounding environment, to explore buildings and archaeological remains directly in situ in their historical context, has led to a slow shift towards mobile use via smartphones. In this way, several Augmented Reality (AR) applications were developed in recent years, typically designed for outdoor scenarios, which aim at an enrichment of what is still visible by adding virtual reconstructions superimposed on the real scene (Gleue and Dähne 2001; Duguleana and Voinea 2018; Panou et al. 2018). Nowadays, more and more art galleries, cultural parks and historical sites are using itineraries that allow the user, through the simple use of a mobile device, to enrich the experience with information, images and reconstructions embedded in the real context (Galatis et al. 2016). However, this introduces a number of critical issues, requiring the use of accurate tracking techniques even in a dynamic and uncontrolled environment such as the outdoors (where the use of classic marker-based tracking loses its practicality) and an accurate registration of virtual models, all while ensuring the user’s mobility requirement. Such requirements call for the development of ad hoc solutions (e.g. based on hybrid tracking) (Zendjebil et al. 2008). Two representative examples of mobile apps are ARCHEOGUIDE (Gleue and Dähne 2001) and Cenacolo Vinciano Official App (Magnelli et al. 2021). The former, often cited in the literature as one of the first outdoor AR apps, aims to develop interactive methods for accessing outdoor cultural heritage, offering the opportunity to display the damaged site reconstructed in 3D. The latter (although designed for an indoor context) allows access to a series of contents related to Leonardo’s work and the Basilica of Santa Maria delle Grazie. Once on site, when the fresco is framed, the app gives access to a series of interactive AR contents that guide the user to discover the work. In a similar way, another mobile AR application allows the user to frame portions of the frescoes located across 11 points of interest inside the Basilica of Saint Catherine in Galatina and access to multimedia content that helps understand the story and meaning behind the images. This mobile application was accompanied by a Spatial Augmented Reality performance to create a unique pathway that starts from the façade of the Basilica, to narrate the history of the monument through images and sounds, and continues through the frescoes inside (Cisternino et al. 2021). A follow-up to this work studied the usability, user experience and mental workload factors that characterise the use of the mobile AR application (De Paolis et al. 2022c).

The mobile application described in Cisternino et al. 2018 displays 3D models and other contextual information when framing an aerial photograph of the archaeological areas of the Museo Diffuso ‘Castello di Alceste’ in San Vito dei Normanni (Brindisi) and the ‘Fondo Giuliano’ site in Vaste (Lecce). In this context, virtual portals represent an interesting possibility of in situ archaeological fruition (Liu and Fuhrmann 2018). Although, at present, this use of Mixed Reality (MR) does not aim for a precise one-to-one correspondence between the virtual reconstruction and the real context, it allows the user a literal historical walk, joining the present to the past. In Cisternino et al. 2019, a mobile application on virtual portals is offered as a convenient and intuitive tool for the enhancement and dissemination of sites of cultural interest. The idea is to realise a tool capable of exploiting a network connecting various historical sites and archaeological artefacts in contexts open to the public, linked together by georeferenced cultural itineraries. The goal is to guide the user along the conceived cultural route and, once he/she arrives in the proximity of the site of interest, geo-localised by GPS technology, let him/her enjoy the virtual experience through a virtual portal.

2.2.1 Virtual portals

In De Paolis et al. 2021, an AR application based on virtual portals was implemented to support the fruition of a 16th–17th century underground oil mill, located in the province of Lecce (Italy). In this study, the model of the hypogeal oil mill was realised using a photogrammetric survey, with the aim of obtaining a reconstruction to scale, faithful to the original, of the asset of interest. The use of virtual portals also allows the fruition of a cultural site, usually closed to the public, simply by using a mobile device. A leap between the present and the past was possible here, where the past is restored thanks to the introduction of animated models reproducing the machinery used for oil extraction.

The Map Story application analysed in Raeburn et al. 2021 invited the users to visit six locations indicated on a map, guiding them to a green area where they had the possibility of passing through an AR portal to view a virtual world on their smartphones.

As far as we know, the use of virtual portals in archaeological and cultural contexts does not seem to have produced relevant case studies in the literature despite its versatility. The small number of applications concerning virtual portals on the various stores is an indication of this. Instead, they are more frequently used for promotional purposes.

2.2.2 User experience evaluation in Augmented Reality

In the study by Georgiou and Kyza 2017, an Augmented Reality Immersion (ARI) questionnaire was developed for the evaluation of the user experience in location-aware AR environments. The ARI questionnaire comprises 21 questions based on a Likert scale with scores between 1 and 7. The test used for the evaluation of the application presented in this paper takes its cue from the article by Geourgiou and colleagues, and its results will be presented in Sect. 7.

Another study based on the ARI questionnaire can be found in Raeburn et al. 2022, where several variants of mobile AR storytelling applications were examined, evaluating in particular the effects of a site-specific AR experience. First of all, the walking version of a mobile application was proved to offer a higher level of immersion due to the possibility to place and visualise virtual objects nearby physical locations. Moreover, after a new design of the application to make it usable in a single predetermined location (the Queen Mary University of London campus), the results revealed that involvement, presence and attention increase when the story is related to the features of a known location.

The tabled-based AR application presented in Partala et al. 2023 displays a 3D reconstruction of Mannerheim’s Saloon Car, a historical train carriage in Finland, in close proximity to the real saloon car, for which visits are restricted. The questionnaire used for the post-experience evaluation combined items for the evaluation of user experience (Partala and Salminen 2012), spatial situation model (Vorderer et al. 2004) and spatial presence (Hartmann et al. 2016).

The study in Yang 2023 compared an AR book and an AR storytelling application developed for the Shanghai Museum of Glass. According to the results of a questionnaire, the AR storytelling application was able to provide higher levels of presence, flow, education, information utility, enjoyment and engagement.

A systematic review on the relevant dimensions of presence in AR games can be found in Marto and Gonçalves 2022.

3 The case study: the church of Saint Elijah the prophet

As the setting for the present study, a no longer existing place of worship was chosen, the ancient church dedicated to Saint Elijah the prophet, which stood on the main square of Ruggiano, a hamlet of a small village on the Salento peninsula (in Salve, Lecce, Italy). The village lies in a valley inland from the Ionian coast and is 13 km north of Leuca. It covers an area of 80 hectares and has about 500 inhabitants. The small hamlet has a number of religious buildings from different eras, the most famous and well known being the sanctuary of Santa Marina which, in its present state, dates back to the 18th century; however, recent excavations have shown that the first phase of the building may date back to the 10th–11th centuries. The ancient church dedicated to St. Elias the prophet, the patron saint of the town, was built in 1586 (Caloro 2000) and stood on the present-day St. Elias Square, right in front of the noble palace. It played the role assigned to it for about four centuries, at least until 1948, when it was closed to worship because it was seriously damaged and finally demolished in 1953 to make way for a larger church about 220 ms to the west. The building had a typical commissa Latin cross shape and a south–west/north–east orientation and bordered on the east with an underground oil mill while the back wall and south–east side bordered with private structures and gardens. The rest of the perimeter was unobstructed and adjacent to the main road leading to Montesardo. The reconstructive hypothesis put forward in this study is inspired by extensive archival research and interviews with local inhabitants. During this investigation of documents and oral testimonies, we often came across conflicting accounts of the demolition of the church. In particular, the documents tell a somewhat different story than the collected testimonies.

4 Working assumptions

This section presents a series of assumptions and design and methodological choices adopted as a starting point for the development of the Mixed Reality application underlying this study. First of all, the present study, which can be outlined as an analysis of user experience (UX) in a mobile Mixed Reality use scenario for cultural heritage, is part of a broader framework of discussions pertaining to the so-called digital humanities. In this context, the study, reconstruction, recontextualisation and final restitution of historical sites that are no longer accessible to the public are carried out using technologies from the ICT world and, in particular, Mixed Reality (MR). The case study identified in this paper is a peculiar example of this line of investigation; thus, a first set of hypotheses were adopted in the content production phase. Since the site is no longer present, a preliminary analysis leading to a reconstructive hypothesis of the building used was required. Without foregoing historical and methodological rigour in the study of the site’s relevance, an initial reconnaissance of the area was conducted, collecting the oral testimonies reported; a further search for structures similar in age and type was carried out. This was crucial for the reconstruction of the interior furnishings and decorations. The model obtained was finally corroborated by documents kept in a local historical archive.Footnote 1

The application was designed and developed to be used on site, in the original building location; it belongs to the outdoor mobile application category. A virtual portal, which allows the building to be placed in the same place, re-contextualised in the current urban context, was used (see Fig. 1). This is a simpler version of outdoor Augmented Reality, which does not aim for a one-to-one correspondence between the virtual reconstruction and the real context. Tracking is typical for a Mixed Reality app, taking over what is provided by the ARFoundation modules (i.e. simultaneous localisation and mapping (SLAM)) (Durrant-Whyte and Bailey 2006). A Mixed Reality application exploiting virtual portals involves the positioning of a three-dimensional model of an access threshold in the real world. Crossing this threshold, the user enters the virtual world and is catapulted into a different reality by seeing as it was, what no longer exists, just as in the case of the old parish church. Blender and Metashape were used for the modelling and photogrammetric survey: the former because it is a complete, opensource and easy-to-use tool, and the latter because it is extremely convenient and immediate in the various work phases. The need to create a cross-platform application, which could be used on both Android and iOS systems, justified the development in Unity. The use of the ARFoundation 3.1.3 library offers support, integrated into Unity, for AR development on both systems, replacing the old ARCore (for Android) and ARKit (for iOS).

Fig. 1
figure 1

The 3D model of the original building is placed in the same location, re-contextualised in the current urban context

Full size image

5 Application development

CumeRA (literally ‘How was it’ in English) is a mobile MR application aimed at supporting the use of a cultural heritage that no longer exists. It was conceived and developed with user-friendliness in mind, with simple and intuitive graphics to guide the user to its correct use.

At start-up, the app presents a simple menu, which allows users to navigate through different information sections. From this menu, it is possible to access the central part of the app, which allows the insertion of a virtual portal as an access point to the digital reconstruction of the building, right where it once existed (see Fig. 2). The Unity project consists of two scenes: the first handles the home screens and the navigation menu and the second the entire MR experience. Pre-development phases included research into the most suitable positioning methodology for the portal in the augmented scenario. A placeholder, a three-dimensional visual element rendered on the screen and projected on the floor, which slides along the scanned surface following the movements of the device’s camera, was used to this end. The best positioning point, in terms of distance from the scanning area, is indicated by a coloured circle alternating red and green. The use of the ARFoundation library has made it possible to inherit a whole series of objects, components and scripts useful for managing the MR experience. Among these, AR Session Origin is the gameobject to which is associated all the interaction logic with the virtual elements of the scenario. The AR Session object has the purpose of controlling the life cycle and the configuration options of the MR session. Among the various scripts and shaders developed, there are four fundamental ones: Placeholder, PortalController, PortalMask and SpecularStencilFilter. The Placeholder script manages the positioning of the placeholder and, consequently, of the virtual portal and the 3D model beyond it. If the user looks through the portal, a partial visualisation of the virtual content is activated; when the portal is crossed, the user is fully immersed in the virtual scenario. This is implemented by superimposing a fictitious interaction plane behind the model. The management of the materials of this plane and of each sub-part of the model is done thanks to the shaders PortalMask and SpecularStencilFilter. Finally, the PortalController script manages the interaction events that affect the plane, exploiting the use of a collider applied to the camera, and appropriately setting the properties of the shaders.

Fig. 2
figure 2

Virtual portal overimposed on the real scenario

Full size image

6 Experimental setup

The device used to execute the tests was the Samsung Galaxy S8, which is provided with the following features: 5.8-inch Super Amoled display with a resolution of 1440 x 2960 pixels. The smartphone was also equipped with the following hardware features: 4-GB RAM, a 64-GB storage and, at the time of testing, Android 9.0 as the operating system.

The testing phase of the application involved the participation of primarily students who were recruited from a high school and the University of Salento (Lecce, Italy). These users volunteered to take part in the testing phase of the application. Additionally, a diverse range of participants of various ages, professions and genders was also included in the study. Thus, based on the collected data, the sample can be described as gender-balanced and consisting mainly of students aged between 15 and 24 years old.

7 User experience evaluation

Among the various tools developed to assess the usability of an application (Assila et al. 2016), (Lewis 2018), (Hajesmaeel-Gohari and Bahaadinbeigy 2021), the HARUS questionnaire was specifically introduced for mobile Augmented Reality applications (Santos et al. 2014), (Santos et al. 2015): it takes into account perceptual and ergonomic principles, which are related to the dimensions of comprehensibility, i.e. the ease of understanding the content, and manipulability, i.e. the ease of handling the device during a task. User experience is a wider concept than usability, as it deals with ‘perceptions and responses that result from the use or anticipated use of a product, system or service’ (International Organisation for Standardisation 2019). A key concept characterising the user experience in virtual environments is that of presence, commonly known as ‘the feeling of being in a certain place’. Various definitions and models describing presence are reviewed in Skarbez et al. 2017, alongside various related constructs such as social presence, copresence, immersion, agency, transportation, reality judgement and embodiment. The questionnaire employed to test the application described in this paper was structured on a scale between 1 and 7, where 1 represents ‘I do not agree at all’ and 7 ‘I agree completely’, and was inspired by the ARI questinnaire developed by Georgiou and Kyza 2017. The items of the ARI questionnaire are grouped into the factors Flow, Presence, Emotional Investment, Focus of attention, Usability and Interest, which, in turn, are grouped two by two into the macro-factors Total immersion, Engrossment and Engagement. The application was tested by a total of 60 users. For the user experience evaluation of the Mixed Reality scenario described in this paper, the ARI items deemed most relevant were selected and adapted. A factor analysis was then performed on them to identify the most representative factors of the user experience in a Mixed Reality scenario. Moreover, two questions were added to identify users who visited and remember the original St. Elias church and ask them to rate the accuracy of the reconstruction provided by the application.

The initial form of the questionnaire, derived from the ARI questionnaire, consisted of the 15 items listed in Table 1. Then, when factor analysis was performed, some items were progressively removed until the models described in the following subsection were obtained, which appear reliable in terms of Composite Reliability, Convergent Validity and Discriminant Validity.

Table 1 Items of the questionnaire designed for the evaluation of user experience with their Kaiser–Meyer–Olkin factor adequacy values 
Full size table

To test whether factor analysis was possible, a Bartlett’s test of sphericity (Bartlett 1951) was performed, which verifies whether variables intercorrelate or not. The p-values shown in Table 2 reveal that the test is statistically significant, thus suggesting that all the considered sets of items were suitable for factor analysis.

Factorability was assessed also by means of Kaiser-Meyer-Olkin measure of sampling adequacy (MSA) (Kaiser 1970; Kaiser and Rice 1974), which verifies whether the partial correlations within data are close enough to zero to suggest the existence of at least one latent factor underlying variables. The overall MSA for each item set, shown in the last column of Table 2, is above the recommended threshold of 0.60.

Table 2 Bartlett’s test of sphericity and Kaiser–Meyer–Olkin measure of sampling adequacy (MSA) for each set of questionnaire items
Full size table

Table 1 shows the MSA of each individual item within the four different configurations considered for the factor analysis: almost all values are above the minimum threshold of 0.50 except the two highlighted in bold.

Besides factor analysis, hierarchical cluster analysis was carried out by means of the ICLUST R package (Revelle 1978), which aims at maximising internal consistency and homogeneity. ICLUST considers Cronbach’s alpha (Cronbach 1951), defined as the mean of all the possible split-half reliabilities of a scale, as a measure of internal consistency and Revelle’s beta coefficient (Revelle 1979), defined as the minimum value among all the possible split-half reliabilities, to assess the scale homogeneity.

7.1 Factor and cluster analysis

Although the parallel analysis suggested a two-factor structure for the questionnaire item scores, this model was discarded because it did not meet the criteria based on factor loadings and average variance extracted (AVE) for Convergent Validity (Hair et al. 2022). Therefore, a restricted set of items was considered after removing three items with low loadings from the initial questionnaire, namely: ‘I was glad to take my time for the visit’, ‘If interrupted, I looked forward to returning to the activity’ and ‘The application was not overly complex’. From this new set, for which the parallel analysis suggested a four-factor structure, item ‘I found the application well structured’, which had similar loadings on two factors (namely Usability and Interest), was also removed.

The Composite Reliability values of the factors shown in Table 3 can be considered acceptable, as they are greater than the 0.7 recommended threshold for Internal Consistency (Hair et al. 2011).

The criteria for Convergent Validity can be considered satisfied, as the AVE values of all factors are greater than 0.5, and most factor loadings are greater than 0.7 (the few loadings below the recommended threshold are still acceptable as greater than 0.5) (Hair et al. 2022).

Fig. 3
figure 3

Four-factor analysis based on oblimin rotation after removal of items 3, 5, 6 and 8 from the original questionnaire

Full size image
Fig. 4
figure 4

Four-factor analysis based on promax rotation after removal of items 3, 5, 6 and 8 from the original questionnaire

Full size image
Table 3 Composite reliability and average variance extracted of a four-factor analysis based on oblimin and promax rotations after removal of some items from the questionnaire
Full size table

The Fornell–Larcker criterion for Discriminant Validity is satisfied: the AVE square root of each factor, reported in the corresponding diagonal cell of Table 4, is higher than the correlations with other factors, reported in the cells below the diagonal (Fornell and Larcker 1981).

The heterotrait–monotrait ratio (HTMT) was adopted as a further method to assess Discriminant Validity (Henseler et al. 2015). The table cells below the diagonal contain both the values for the more recent HTMT2 variant, based on geometric mean (Roemer et al. 2021), and those for the previous HTMT method, based on arithmetic mean (Henseler et al. 2015). Both HTMT2 and HTMT values, reported in Table 5, fulfil the criterion for Discriminant Validity as they are below the recommended threshold of 0.85 (Kline and St 2022).

The model in Fig. 4 appears more reliable than the one in Fig. 3, as it has fewer items with loadings below 0.6: it shows a correlation between all three factors Interest, Attention and Presence, which do not correlate with Usability.

Table 4 Lower triangular matrix for discriminant validity verification, based on the Fornell-Larcker criterion, in the four-factor analyses based on oblimin and promax rotations
Full size table
Table 5 Heterotrait–monotrait ratio (HTMT) based on geometric and arithmetic mean
Full size table

A cluster analysis conducted on the same set of items produced the diagram depicted in Fig. 5.

Fig. 5
figure 5

Cluster analysis for the items of the four-factor model (cluster fit = 0.9, pattern fit = 0.99 and RMSR = 0.05)

Full size image

A cluster analysis performed on the identified factors is depicted in the diagram in Fig. 6, where its sub-factors Involvement and Realism were considered instead of the Presence factor. Cluster C1 corresponds to the Presence factor made up of Realism and Involvement. The correlation between Usability and the remaining factors is very unreliable due to the high discrepancy between alpha and beta coefficient in cluster C4.

Fig. 6
figure 6

Cluster analysis of the identified factors (cluster fit = 0.75, pattern fit = 0.98 and RMSR = 0.08)

Full size image

7.2 Factor scores

For each user who filled in the questionnaire, Usability, Presence, Interest and Attention scores were calculated as the averages of the factor items according to the model in Fig. 4.

The density ridgeline plot in Fig. 7 compares the distributions of the scores based on the density estimates of the factors calculated for all the 60 users involved in the tests. For the Presence factor, both the overall score and the scores of its two subfactors, Involvement and Realism, were considered.

Interest is the factor with the most pronounced peak in the waveform, indicating substantial agreement among users in expressing high interest in the experience.

Fig. 7
figure 7

Distribution of factor scores calculated for the 60 users involved in the tests

Full size image

Tables 6, 7 and 8 show the average values, standard deviations and coefficients of variation (i.e. the ratios between standard deviations and means) of the factors calculated over all the 60 users and those calculated on various subsets of the dataset. Users were splitted in elderly and young users based on their ages through the k-means method, as the NbClust package for R detected two age clusters. Furthermore, users were divided according to gender and whether or not they had visited the church in person. The mean scores and standard deviations of the factors were represented graphically in the histograms in Figs. 8, 9 and 10, the former as coloured bars, the latter as extensions or ‘whiskers’.

Table 6 shows that Interest has the highest average score for all user categories. Moreover, the very low coefficients of variation confirm the substantial agreement between users’ opinions on this factor. Attention is the factor with the lowest average score for all user categories, with the exception of female users, who showed a much higher level of attention than male users.

The difference in scores between male and female users is even more pronounced for the Presence factor, for which there is also a significantly higher average score for older users than for younger ones. Of the two subfactors that make up Presence, Involvement has higher average scores than Realism for all user groups except for older users, who seem to perceive a much higher sense of realism than younger users.

In general, most of the coefficients of variation have values below 0.2, indicating a good level of agreement between users’ opinions with the exception of the Usability factor, which is characterised by higher variability for older users.

Table 6 Mean values of the scores calculated over different user groups
Full size table
Table 7 Standard deviations of the scores calculated over different user groups
Full size table
Table 8 Coefficients of variation of the scores of different user groups
Full size table
Fig. 8
figure 8

Histogram comparing the scores of elderly and young users

Full size image
Fig. 9
figure 9

Histogram comparing the scores of male users and female users

Full size image
Fig. 10
figure 10

Histogram comparing the scores of users who knew the church and users who did not know the church

Full size image

8 Discussion

8.1 Enhancing presence in Mixed Reality: interactivity and realism

Augmented Reality (AR) and Mixed Reality (MR) are immersive technologies that blend virtual and real elements to create a unique user experience. Understanding how these two realities merge is critical to determining where users feel most present (McCall et al. 2011). Notably, there are two influential elements that significantly contribute to the sense of presence in a Mixed Reality scenario, namely the degree of interactivity and the level of realism (Von der Pütten et al. 2012) or vividness (Kim et al. 2023).

Interactivity refers to the level of user involvement in changing both the form and content of the environment in real time (Steuer et al. 1995). Visual realism is dependent on two components: the virtual object’s ability to appear real compared to its geometry and texture and the accuracy of the lighting (Bruno et al. 2010; Comes et al. 2014). Similarly, the term ‘vividness’ refers to how richly an object is represented in a mediated environment, as perceived by the user (David et al. 2021). This perception is influenced by the visual display’s resolution and affects how realistically the app generates a product image (Wedel et al. 2020).

8.2 Virtual portals in outdoor Mixed Reality

For an MR app to provide a seamless experience, it is crucial to maintain a sense of realism that allows digital objects to look as if they belong to the real world. Therefore, 3D content must visually match the real world as much as possible.

Outdoor MR experiences face several limitations, such as fluctuations in lighting conditions due to environmental changes, including object height and orientation, obstructions and occlusions (Blanco-Pons et al. 2019). Other constraints include weather conditions, time of day and unpredictable changes in the surroundings resulting from weather conditions, pollution or physical alterations. These limitations can hamper the tracking process, leading to reduced interactivity, realism and perceived sense of presence, thereby impacting user experience.

In the present study, the purpose of virtual portals is not to solve the inherent issues of outdoor MR by providing precise one-to-one alignment between virtual and real objects. Instead, portals provide a means to switch between reality and virtuality. Once the user passes through the virtual portal, he/she can explore the environment and enjoy a semi-immersive experience (not fully immersive as they are not wearing a visor). By moving the perspective of his/her mobile device, the user can observe the entire reconstruction of the historical context around them. As there are no issues with alignment, stability or verisimilitude of the virtual contents on the real, the user’s sense of presence increases.

Firstly, the user experiences an increase in the quality of the experience in terms of realism and vividness. In the case of virtual portals, the environment is entirely virtual and reconstructed based on assumptions relevant to the site’s state in the past. Since there is no real-virtual alignment, the environment is completely homogeneous, which enhances the perceived levels of realism. Secondly, the user experiences improved levels of interactivity between him/her and the augmented scenario. In the virtual space, the user is immersed through the smartphone display, which acts as a mediator, in a context he/she can continuously interact with while remaining aware of the surrounding real context. This allows for continuous comparison between the reconstructed historical scenario (the virtual world) and the actual appearance of the site (the surrounding real world).

8.3 User experience factors

The average values of Realism and Presence resulting from the conducted analysis are high (close to 5), so—while it is beyond the scope of the goals of this study to compare different design features—we can assert that the design features of the proposed solution represent a promising improvement over solutions found in the literature.

In Table 9, the mean scores of presence were compared with those obtained for other AR applications. In order to make a comparison, scores based on questionnaires other than ARI were also reported on a scale of 0–6.

Table 9 Comparison with the sense of presence measured in other AR applications
Full size table

In general, the walking version of a mobile application was proved to offer a higher level of immersion due to the possibility to place and visualise virtual objects nearby physical locations. However, the scores in the first two rows, which refer to two applications compared in Raeburn et al. 2022, show that presence increases significantly for an application developed to be used in a single predetermined location, i.e. when the story is related to site-specific features. The presence score of the application discussed in this paper (reported in the last row of the table) is very close to this score. This result outperformed also the one obtained for Map Story, an application analysed in an earlier work (Raeburn et al. 2021), in which the user, after visiting six locations indicated on a map, was led to a green area where he/she had the possibility of passing through an AR portal to view a virtual world on his smartphone. In such a context, the authors had highlighted how excessive walking can decrease the level of immersion for some users and hypothesise that an increase in the number of points on the map may help overcome the problem. This scenario is similar to that of the application analysed in the present work, in which, however, the user can focus directly on the virtual portal without first having to follow a route through hotspots.

Two studies based on other questionnaires can be found in Yang 2023 and Partala et al. 2023. The comparative study in Yang 2023 divided the visitors of the Shangai Museum of Glass in two groups: the former was involved in the use of an AR book for 3D visualisation of the exhibits, while the latter was involved in the use of an AR storytelling application that invites participants to visit six AR presentation panels in the museum area. Data collected through a questionnaire showed higher scores for the group involved in the AR storytelling application in terms of presence, flow, education, information utility, enjoyment and engagement. The study in Partala et al. 2023 exploits the Spatial Presence Experience Scale (SPES) (Hartmann et al. 2016) to measure the Spatial Presence during the use of an AR application showing a 3D model of Mannerheim’s Saloon Car. The self-location component of Spatial Presence describes the sense of being located in a virtual environment, while the possible actions component refers to assumptions on the actions that the users could perform.

Moreover, tests with various users revealed that the CumeRA application is intuitive and relatively easy to use. It appears to be a useful tool for reliving or experiencing for the first time sensations related to immersion in a place out of time that few people now remember. Experiencing these memory-related sensations has turned into a new and exciting experience, perhaps comparable to a real journey through history.

In particular, four factors were identified that describe the user experience with a Mixed Reality application based on a virtual portal. Three of them, namely Interest, Presence and Attention, although distinct factors, influence each other. The calculated scores revealed a very high level of Interest, with no significant difference between user categories.

The level of attention, on the other hand, is not too high, probably because virtual portal applications do not have the same immersiveness as applications usable through VR headsets. The lower level of immersiveness could also be one of the reasons for a lower level of realism compared to a typical VR application. This is, however, balanced by a good level of Involvement, which is the other sub-factor of Presence along with Realism: this is probably due to the surprise or novelty effect of the virtual portal, which may have aroused a certain curiosity in the users underlying their high Interest.

Surprisingly, the analysed data revealed a higher level of attention towards this type of application by female users, who also perceived a higher sense of Presence.

The analysis revealed no influence of Usability on other factors, even on Attention, which is typically expected to be the most affected by usability issues. The reason could be the absence of particularly complex forms of interaction or control: the user should simply keep the smartphone in front of himself/herself and eventually change its orientation in order to explore a different area of the virtual reconstruction. Even elderly users, who gave lower scores for Usability, expressed positive opinions on the other factors, probably because the surprise effect prevailed over any difficulties. Their sense of surprise was probably also amplified by the fact that they recognised well a place they had already visited or frequented in person: the fairly high Reconstruction Fidelity score suggests that they found the virtual reconstruction very similar to the original appearance of the church.

In summary, the analysis of user experience suggests that virtual portals are an engaging application that offers high levels of realism and sense of presence in outdoor Mixed Reality (MR) scenarios, indicating a promising improvement over existing solutions. While comparing different design features goes beyond the scope of this study, it can be concluded that the design features of the proposed solution represent a promising improvement over existing solutions reported in the literature. Virtual portals serve as a highly evocative and adaptable mobile tool for exploring inaccessible historical environments, thereby enhancing the enjoyment of cultural heritage.

In the future, we could study how the balances and relationships between the factors change after the inclusion of hotspots for insights to be provided in Augmented Reality (De Paolis et al. 2022c) or after the provision of a well-defined storytelling route with treasure hunts or quizzes along the way.

9 Conclusions and future work

The work presented brought together the various stages of research and development that led to the realisation of CumeRA. This is an application of MR that combines the virtual reconstruction of a place of worship, demolished decades ago, with physical reality, in order to reintroduce a cultural asset that has been erased from memory, through recontextualisation in its urban context. In addition, the use of photogrammetry and 3D modelling made it possible to create a historically accurate reconstruction that is faithful to historical records and documents. As envisioned and implemented, and particularly in the light of the conducted tests, this work offers the hypothesis for future implementation to enable the free distribution of CumeRA and its availability for use on any mobile device. The installation of appropriate horizontal signage, in today’s St. Elias Square in Ruggiano, would facilitate the complete and safe use of the app and, therefore, of the reconstructed model in its original location. In addition, a future improvement and increase in content within the app will allow the creation of a historical–cultural itinerary based on the reconstruction of places of interest, both in Ruggiano and elsewhere, that no longer exist or have been greatly modified, which can be visited virtually once one has arrived near the reconstructed property. It is hoped that the creation of such an itinerary will, in the future, be a stimulus for the re-appropriation of the historical memory of all those communities that have lost fragments of their own experience, so that it can once again become the individual and identity heritage of the community.

From an application perspective, the mobile MR tool for outdoor heritage exploration proposed in this study does not introduce any particular innovations. Nevertheless, this contribution emphasises the significant improvements in user experience achieved through the use of virtual portals in this context; the intrinsic features of this tool are analysed here in terms of interactivity, realism and sense of presence. The results of the analysis indicate high average values for realism and sense of presence (close to 5). While the comparison of different design features was not within the scope of this study, it can be concluded that the design features of the proposed solution represent a promising improvement over existing solutions reported in the literature. Virtual portals serve as a highly evocative and adaptable mobile tool for exploring historical settings that are no longer accessible, thereby enriching the appreciation of cultural heritage.

Moreover, experimental tests revealed a very high level of user interest. Usability did not significantly influence the other user experience factors. For the study of user experience factors, it might be of interest in the future to evaluate the impact of introducing a well-defined visit path through various hotspots with gamification elements, quizzes and information sheets.