Introduction

Digital culture has given rise to a fluid, continuously updatable, iterative, and interactive model of communication [1] by enabling access to information through immediate and customizable communication. Knowledge is being produced and disseminated at an increasing rate, driven by new information and communication technologies that foster the digitization of cultural production as a form of change for access to resources and knowledge [2].

In order to meet the needs of contemporary society in terms of knowledge sharing, Burdick et al. [3] identify the inseparability of digital design and production from the physical ones as a necessary action, arguing for the need to give rise to research practices aimed at developing “shared, inclusive rather than exclusive knowledge” [3]. Its purpose is not to replace physical experience, but to facilitate the transfer of knowledge to various user groups by leveraging the opportunities provided by advanced technologies.

In the outlined scenario, it is necessary to analyze the ways in which knowledge is conveyed towards the users in order to design solutions in accordance with the opportunities offered by new technologies.

In this sense, the first part of the paper provides a multi-layered account of user experience by foreshadowing forms and possible applications addressed in subsequent sections. Starting with innovative strategies for communication and narrative, the tools and technologies that enhance interaction and acquisition of the paths, the visitor preferences and behavior, as well as neural tracking of emotions during the experience, are explored.

Digital narrative conveys the adaptive transfer of content with reference to the visiting experience, fostering the personalization of pathways. The experience is amplified at the sensory level in interconnected physical and digital spaces where visitors can be tracked and monitored. Based on this scenario, the research delves into the main regulations, ethical implications, and privacy management issues were introduced and explored in order to understand their limitations and opportunities.

The overview presents the research results in order to discuss and compare the current modes of use and enjoyment to outline new tools and adaptive paths.

The evolution of User Experience among cognitive perceptions and emotional reactions

Even though nowadays users have become accustomed with design and technological innovations, these continue disseminating around them an ingens syla of mysterious presences. Its exotic character is multiplied by the technical perfection hiding unknown universes behind the seemingly ease of functioning [4].

For a valorization project, it is clearly not enough to be digital in order to be defined as innovative. For a useful interaction in the real world to take place, it will be necessary for the machines to understand something from the intentions and the reasoning ways of their human counterpart. Gershenfeld highlights how the needs of wearable devices arise from three factors: people’s desire to increase their abilities, the growing technological capacity, wearing computers within clothes due to nanotechnologies and the industrial demand to shift interaction from computers to people [5].

The main problem of these interactions is sight [6]. Consider how complex human communication is, since in addition to vocal communication, humans also use facial expressions, gestures, and movements that are very difficult for machines to mimic and understand.

The use, the possession, the interaction with an artifact could generate different types of cognitive perceptions and emotional reactions. There is a shift from physical interaction to the perceptual one and a direct connection between user-artifact-context is established since the relationships with the context cannot be understood without implementing forms of active user engagement.

It is necessary to synergically develop solutions able to reproduce the unique connection between the body and the mind by prefiguring a perfectly integrated sensory system, tactile, visual, auditory, to detect stimuli from the environment in real time [7].

First the shape, the colour, the size, the image, the readability, then plasticity, operativity to commands, smoothness, functionality in the gripping points; finally, sound, process signals, silence. These sensory stimuli represent the system of relationships standing between users and the used objects. Artifacts are perceived through their own shape, which is not just how the envelope is configured but the language that makes them understandable. Assessing a dimension, testing hardness, are also language elements used to create a relationship not only in an intimate soliloquy but in a sort of closed dialogue [4]. It is not the “brain” which perceives the world (Brainbound), it is the different interactive relationship with the world which expands, amplifies, the perceptual possibilities of the “mind” (Extended Mind): cognitive, emotional, and creative [8].

For example, blind people acquire necessary information to build a real world mainly through tactile, hearing, and kinaesthetic experiences. Hearing represents a very important aspect in orienting blind persons, allowing them to move safely within a structured space. It provides them with the ability to notice the presence of obstacles. The blind person flanks touching to hearing in order to develop and acquire concepts. Touch is used not only in a static manner but also in a dynamic and exploratory way (active touch). Tactile perceptions are received through any part of the body but particularly through the hand which is the privileged organ for touch. Through attentive and accurate palpation, hands identify the different details of the object to be “understood” and allow the blind user to create a mental image of the object and acquire its spatial position [9].

The distinction between experiential and reflective thought deserves to be taken into consideration at least partly since much of the existing technology seems to force us towards one or another direction. With the proper artifacts, the potential of any cognitive modality could be amplified [10]. Experiential cognition includes mental states where one perceives and reacts to environmental stimuli in an efficient way and without considerable effort. This cognition takes place mainly through unconscious phases of human behavior drawing directly from experiences and knowledge already stored in the user’s mind. Even though experiential processing involves some intellectual work, it works more like a reflex because the relevant information should already be stored in our memory and is only reactivated by experience [11].

To enhance user experience, several ways of interaction are necessary and the combination between gestures and vocal recognition are among the most powerful, efficient, and natural ways of communication. As devices become increasingly performant, they provide an intuitive and effective interaction thus allowing the development of natural user interfaces (NUI) which enhance user experience and enable learnability [12].

In this sense, design shifts in the Experience Design field by integrating perceptual, intellectual and values issues [13]. It impacts the future of social and affective informatics where technology could contribute not only to functioning but also to the expressive and emotional track of people’s lives [14].

In fact, McCarthy and Wright [15] claim that there is a direct correspondence between technology and experience which is possible to conceptualize in four intertwined threads: (a) thread of senses; (b) thread of emotions; (c) compositional thread and (d) spatial–temporal thread [16, 17].

In this way, the synchronization of several sensory channels enables enhanced user satisfaction and pleasure, encouraging a higher use and adoption of technologies.

How the user–artifact interaction takes place depends, in fact, on the system properties, on the users’ characteristics and the contextual parameters. This produces specific emotional and behavioral perceptions related to the features of the product and the context of use [18].

Emotional, functional, and environmental connections in User Experience

Current UX research mainly focuses on three areas: emotional experience, functional experience, and environmental experience. Emotional experience explores user attitudes and behavior, and it analyzes sensory, social needs, the aesthetic perceptions, and the aesthetic imagination of users. Functional experience is based on the artifact usability and on connected applications including content creation, technical processing, and interface design. The environmental experience focuses on usability of the contextual factors of the artifacts and of the application services. Among these, there are the sensory environment, the means of communication and technology. These three User Experience approaches focus on users (emotional experience), artifacts (functional experience) and on the context of use (environmental experience) [19].

If museums were able to provide visitors with more calibrated visiting paths considering their interests, probably the level of enjoyment of the experience would grow accordingly [20].

Aspects such as the user profile and interests are considered fundamental for a constructive Cultural User eXperience (CUX) [12] that might allow for effective visitor profiling through intelligent data analysis and Artificial Intelligence. Such technologies are enabled to process acquired information and provide personalized suggestions and recommendations that would be useful in continuing the visit.

In this way, it would be possible to articulate dynamically the type of message based on the visitor characteristics (a) cognitive (attention level and duration; perception; memory; learning capacity; fears; personality features; previous knowledge); (b) physical (differences in age, body characteristics); (c) emotions/affective (what motivates and engages them); (d) experience and expectations; (e) language; (f) culture; (g) special needs; (h) homogeneous or inhomogeneous users; (i) discretionary vs committed [21].

This type of information would allow not only to focus on visitors but also to transform the cultural site in a multimodal intelligent environment, emphasizing the emotional aspects of interaction [22].

Experiencing a positive or negative emotion can completely transform the experience of use. Several studies in the state-of-the-art identify the affective/emotional factors such as the experiential needs [23], the affective responses [24], the emotional benefits [25], the user enjoyment [26].

Neurosciences and in particular the model developed by James Russell [27], suggest that it is possible to change the actual quality of an experience by manipulating the core affect which corresponds to the mood, a fluctuating affective state without a specific object.

Through technologies (cameras, microphones, movement sensors, temperature and humidity sensors…), the behavior, the facial expressions and the voices of users can be detected and measured through cognitive-emotional processing techniques to identify each user’s emotions. By observing people’s faces, the affective processing algorithms can identify both macro-expressions and micro-expressions to detect emotions [28].

To support this process, virtual reality has been associated with the use of biofeedback: systems that monitor physiological changes and reactions in specific situations and make them visible to users allowing them to know their own physiological response [16].

It is essential to assess the appropriateness to the context of use with reference to the connected physical, organizational and social-cultural features. User Experience is therefore defined as an entirety of the user’s perceptions, feelings, and responses in front of the anticipated effective use of a product, system or service [21]. The importance of emotions in the user experience is relevant. In fact, they allow us to get close to a certain object and that positive “proximity” can impact the global quality of use.

In recent years, research highlighted the importance of positive emotions for well-being. The “Broaden-and-Build” theory developed by the psychologist Fredrickson underlines how positive emotions are fundamental for subjective development. This would allow to (a) extend the thought-action repertoire; (b) reduce the impact of negative emotions; (c) increase resilience (they help to increase concentration, to reconfigure experience, providing a positive significance to events); (d) build new psychological resources (they enable the development of permanent intellectual, psychological and physical resources); (e) activate a positive spiral for psychological development [29].

Therefore, understanding the user behavior highlights priorities, preferences, and implicit intentions to configure the fruition experience within the relationship between cultural place and visitors. This is interpreted through two main questions: (a) how people behave inside the exhibition space; (b) why visitors behave in a certain way. There should be considered how physical disorientation might be associated with “intellectual bewilderment”. This can be generated by poorly comprehensible or unintelligible information apparatuses in the ‘average’ visitor or in all those who approach cultural heritage without a humanistic or scientific background or special interpretative tools [30].

It is therefore necessary to configure interactive solutions able to adapt to the human environment in an intuitive manner and to recognize gestures and expressions, objects, and situations. When technology proposes interaction with people, understanding its functioning is mainly determined by the system usability; by information comprehensibility; by language and by the ease in carrying out input and output procedures [5].

Customizing Cultural User eXperience by synchronizing several sensory channels

From the visitors’ perspective, circulation inside the exhibition space represents the main tool for expressing their system of preferences towards the cultural offer in a tangible and therefore observable way.

Eye tracking, movement tracking, gesture recognition could be integrated to become primary inputs [28]. Current technological devices which simulate real and virtual perception seem easy to use but they provide unnatural sensations of immersive experiences. In the future, the ideal input should be completely natural to guide users–visitors within the exhibition space by making the cultural offer, the spatial layout of the collections, and eventually the paths to be taken, immediately clear. This is useful to induce an extemporaneous fruition behavior or a “blind” exploration [30] of the cultural place.

The customization of the museum offer could be developed through the creation of an advanced guiding system able to consider each visitor’s preferences system expressed through their actual fruition behavior.

The User Experience takes on a multifaceted dimension able to comprehend the user’s “internal” state (behavior, expectations, needs, motivation, etc.), the characteristics of the system (complexity, scope, usability, functionality, etc.) and those of the context (or the environment) where interaction takes place (e.g., organizational/social context, significance of the activity, voluntary use, etc.) [12]. This effective interaction is conceived as a mental process where users are asked to reflect and plan attentively their actions. Both Embodied Cognition and human-artifact interaction reach a vision of effective artifacts as promoters of fluid actions. They are invisible or transparent tools able to obliterate in the users’ consciousness when they engage in mediated action [21].

To better understand the various levels of interactive experiences, it is possible to use the Theory of Situated Action (TSA) that develops on three levels: construction of the context; interpretation of the situation and local interaction with artifacts.

The actors find themselves in a specific situation to which they should react by choosing one of the possible actions. The results of the selected action are to be interpreted and assessed according to the future perspectives (plans). These help reflect and modify the personal interpretation (retroaction) of the situation that has been created (the new situation) [21].

A semantic representation of the user profile and needs may lead to effective and accurate methods for user modeling. Combined with structured data from cultural institutions, these may lead to a customized user experience. Similarly to what happens during a guided visit—when visitors explicitly express the desire to receive information on artifacts that are not included in the standard visiting itinerary—an evolved guiding system may have the “adaptive” capacities to modify in real time the suggested itinerary, based on (implicit) user fruition behavior [20, 31, 32].

The users’ needs to quickly learn the interaction ways, allowing them to perform tasks quickly, maintaining a low level of error, and contributing to enhancing the user satisfaction, are to be taken into consideration. Customization of information on cultural heritage requires systems able to extract the features specific to each user, such as previous knowledge, interests, aim of the visit and personal information, as well as contextual aspects. The available information is to be adapted to these aspects and provide them in the most proper way [12].

In this scenario, the “experiential” knowledge of visitors, dynamically generated through the analysis of their fruition behavior [20], is important because it is the result of the collection and processing of a large quantity of punctual data and of multiple “viewpoints” of each individual. The availability of fruition data in aggregated forms allows us to extrapolate significant correlations between different objects that compose museum collections. On this basis, visiting solutions could be proposed not only with reference to the behavior (and individual choices) of one visitor, but by taking into consideration the synthesis of the choices made by many “equal” individuals, common museum visitors not necessarily endowed with specific specialized skills, as happens in most cases.

Innovative narrative tools and strategies to deliver adaptive content

Studies on the visitor-artifacts and visitor-museum space relationships have resulted in identifying the ways in which users are led to establish a relationship with the cultural space and object and of the most influential factors for the visitor experience and knowledge transfer. Moving away from the model of linear transmission of information [33], visitors are no longer considered passive receptors of content, instead they play an active role in the fruition experience, activated through the functionalities of advanced technologies. Current fruition systems configure a narrative [34] and visitor-centred museum [35], based on the centrality of the user involved in personalized and adaptive narratives.

Interactive digital narration, content personalization and adaptability represent tools available to museums to effectively convey knowledge to the public [36]. Narrative tools erase the boundary between the different phases of the visit, making it more permeable [37]. The visitor-user comes into an environment that is part of a multi-layered narrative mediated by fixed devices (such as touchscreens and interactive stations) and mobile devices (such as smartphones, tablets, and multimedia guides) that respond to different functionalities in relation to space and object and allow the experience to be continuously articulated, preparing and enriching it before, during and after the visit.

It is interesting to analyze the current modes of knowledge transfer in order to trace communication strategies that are able to provide personalized and adaptive narratives in the fruition space.

The AthenaPlus research group [37] classified digital narratives into the following types based on user experience:

  • interactive, where users can influence the narrative and its evolution in real time, and they can interact with the content; this is reassembled according to the users’ actions and preferences;

  • collaborative, based on a participative model where users have the necessary tools to create their own content;

  • mobile, based on the use of digital devices and their components (GPS, compass, accelerometer, data connection, camera) and functionalities for localized activation of multimedia content;

  • transmedia, developed on different devices with specific use and complementary technological capabilities for the creation of a ‘continuous’ fruition;

  • immersive, based on the relationship between physical and digital where the narrative defines the experience in which the user is immersed;

  • generative, based on real-time data collection to generate stories autonomously through the use of Artificial Intelligence to interpret user behavior or a set of information and provide adaptive experiences.

Supporting the interpretation of objects in the exhibition space

The current fruition systems move in the direction of a mixture of the physical and digital information dimensions, intercepting and enhancing the visitor's experience during all phases of the visit.

It is not possible to define a true classification of technological systems as configured solutions often integrate different technologies and functionalities to narrate the collections and deliver personalized content based on specific target users.

Through the analysis of the fruition contexts in cultural spaces, the research has identified tools to (a) support the understanding of objects in the exhibition space by recognizing three functions: (a1) integrate digital narratives; (a2) explore additional information; and (a3) zoom in on the artifact.

With reference to the artifacts, the visual delivery of details and content can be accompanied by narrative systems that contextualize the exhibited object in time and space integrating digital narratives into the physical space of the collection.

In this direction, the Crossroads exhibition of the CEMEC—Connecting Early Medieval European Collections project uses the virtual to narrate archaeological finds, grave goods, and everyday objects through digital content and dramaturgy techniques. As part of the project, a holographic showcase was realized, which through holograms, lights, sound speakers, scenery and projections provides an engaging narrative for the user, augmenting information on the exhibited object. The content visualization consists in an optical illusion realized with Pepper's Ghost principles and the objects in the collection are supported by projections and sound effects to simulate Mixed Reality. This “story box” represents a new way of communicating, in which digital contents are not placed in separate spaces from the objects in the display cases. On the contrary, for the first time, they are projected into the very space of the object by animating it.

The system developed enables readability and recontextualization of the artifact and the emotional narrative activates the transfer of knowledge to visitors [38].

The tools and technologies analyzed do not replace the traditional visit, which is linked to direct contact with the exhibits. On the contrary, these tools make the user experience clearer and more engaging in exploring additional information by amplifying the understanding of meanings.

To support the fruition of the collection with additional content at the National Museums Scotland (a2), two types of digital touch screens have been configured, the Digital Label and the Digital Interactive [39]. In particular, the Digital Interactive is an interactive touch display that allows users to enjoy a personalized experience, exploring information about the collection. Each digital label, through a daily server connection, tracks key interactions, including clicks on each section, video plays and screen time-outs. By accessing the analytical data of the systems installed in the rooms, it is possible to identify the number of users interacting with the displays, to monitor the most activated content and draw insights to be applied to the creation of new content, the position and orientation of the systems in relation to the showcases, and the location according to the user's point of view.

Digital and multimedia systems allow users to zoom in on the information about the artifacts, to gain a better understanding of their history, context and collection, to define connections between other objects and collections, and to shift the visitor's attention to details and aspects that the traditional visit does not allow.

In the exhibition “Il Codice Leicester di Leonardo da Vinci. L’Acqua Microscopio della Natura” realized by the Uffizi Galleries and the Museo Galileo, an innovative multimedia support was used for the fruition of the documents, the Codescope (a3). Through the system, users can digitally scroll through the 72 pages of the Leicester Codex, they can access the transcriptions and information on the themes addressed, exploring Leonardo da Vinci's considerations and annotations through a digital zoom lens.

Narrating collections and providing accessible experiences

In the museum space, in addition to solutions integrated in the exhibition, mobile applications and systems inserted in the visiting paths support the adaptive delivery of content according to the user's position in relation to the space and to the artifacts.

In this context, it is possible to identify solutions to (b) narrate the collections and provide personalized content in the exhibition space by identifying four functionalities: (b1-2) make artifacts accessible; (b3-4) activate geolocalized content; (b5) personalize the narrative and (b6) the user experience.

Advanced systems can structure a differently organized exhibition space with alternative paths that multiply the actual spaces. This will enable overcoming the traditional museographic approach, which has always been set more on “seeing” than on “interacting”, in order to dilate the descriptive material of the artifacts and shape the experience in accordance with the diversity of user needs and expectations [40].

Digital narration can allow content and modes of enjoyment to be shaped based on specific needs, interests, and preferences to make the visitor experience adaptive. Digital systems facilitate the configuration of ways to access information through text, audio, and video to accommodate different learning styles and abilities. Solutions used by museums to enhance the visiting experience of people with sensory impairments include audio guide systems and tactile stations. However, restitution through only one modality of exploration, auditory or tactile, does not support the visitor’s experience, and a trained mediator is often needed to facilitate the fruition. It is therefore necessary to integrate different systems to make the artifact and the museum space in its full extent accessible. Proximity technologies and systems such as Beacons and RFID (Radio-Frequency IDentification) using a digital device and an interactive station activate and expand the reception of content by following the visitor's path. The use of Beacons enables short-distance communication with devices equipped with Bluetooth sensors by sending short text messages through a UUID-Universally Unique Identifier. Such technology makes it possible to identify the user's location in space, activating specific content in the presence of an artifact. In the latter case, the point-of-interest recognition function is interesting, which activates the content automatically, unlike the QR CODE system that requires the user's action (framing the corresponding point) to receive the information [30].

The tactile interactive audio guide OVERTHEVIEW (b1) is dedicated especially to people with sensory impairments, and it implements the tactile experience with the audio experience in a single application. Using Natural Language Question-Answering algorithms, the system allows interrupting the sequence of audio messages by giving users the opportunity to ask questions in natural language (written or spoken) to obtain adaptive responses [41]. The station is equipped with a screen, keyboard, and touch table with the 3D reproduction of the elements of the artifact. In addition, a RFID reader triggers recognition of the boards. Through the tactile sensors embedded in the board, the application recognizes which portion of it is touched and allows the audio guide to customize the content of the message based on the user's exploration. The screen displays descriptive text for users with hearing impairments while the virtual keyboard and graphical interface allow deaf and dumb users easy composition of questions. Interactive tactile audio guides are configured for individual artifacts and for the description of models representing, for example, archaeological areas or places to be explored.

The National Museum Aquileia makes the exhibition accessible through the “Percorso tattile” (b2). In detail, the tactile exploration is guided by an audio description that is automatically activated in the different rooms through a system using beacon technology. The application helps with orientation and facilitates the exploration and understanding of the described artifacts, through the "zoom-in" of details and captions with pictograms, contrasting characters and Braille text [41].

The review highlights the need to guide users within the visit path and in the relationship with the artifacts by activating narrative contents related to the exhibition through advanced systems and technologies.

In particular, the MarTa National Archaeological Museum in Taranto (b3) is experimenting with the use of LED light signal to activate Li-Fi technology and return personalized content about the collection. Li-Fi technology uses access points within the exhibition space, creating a network that delivers multimedia content, audio guides and interactive experiences directly to visitors' devices. The user, while looking at the corresponding showcase, marked by a Li-Fi icon and under a cone of LED light, is guided to discover the stories of the artifacts. Audio and video content can be enjoyed through the App downloaded to mobile devices and differentiated on the basis of the Li-Fi beacon and thus based on the museum exhibit associated with it.

To make experience “natural” for the user, MAXXI—Museo Nazionale delle Arti del XXI secolo (Italy) (b4) has started experimenting with Get, an interactive bracelet that uses bone conduction technology and allows visitors to access insights through audio content about the museum by bringing a finger close to the ear. In fact, the innovative device decodes sound waves and converts them into vibrations that can be received directly by the cochlea of the ear without the use of the tympanum.

The digital cultural offer has focused on exploiting the advantages of mobile devices in providing audio/multimedia guides following the Bring Your Own Device-BYOD model [36]. The use of mobile digital guides allows museums to include images, video, and audio of their collections in the exhibition experience to acquaint visitors with them, geolocalizing users in the exhibition space or activating content remotely. Collections from the MOMA Museum of Modern Art, New York, the British Museum, and other international cultural institutions are made accessible through multimedia applications as on-site and/or remote guides to the works, with interactive maps and text and audio narratives targeted by user type and visit time. Narrative delivery can be implemented with accessibility features such as subtitles, audio descriptions, and sign language interpretation supporting visitors with different abilities, as in the case of the MIXT, an app designed by the MAXXI—Museo nazionale delle arti del XXI secolo museum as an accessible and inclusive tool through voice commands and LIS content.

However, the BYOD model used for mobile applications has some limitations, including declining visitor attention to real/physical objects, difficulty in navigation and orientation, and limited user control over the preset experience [36]. In these systems, narrative is used to a limited extent and content is presented in a didactic and descriptive way [42].

The integration of Augmented Reality and Virtual Reality technologies in mobile applications encourages the activation of content layered on top of the real one. The user experience can be augmented for greater engagement with the artifact by activating interactive content based on the user's location through sensors and technologies.

The project CHESS—Cultural Heritage Experiences through Socio-personal interactions and Storytelling (b5) overcomes the criticality of BYOD systems through the design of augmented narrative experiences (AR) on mobile devices [43]. CHESS aims to enrich the museum visit through personalized interaction with content. This is made possible by dynamically adapting information about cultural artifacts to each individual visitor or group of visitors [44] with an iterative User-Centered Approach of design-evaluate-review. The system pioneered at the New Acropolis Museum in Athens (Greece) and the Cité de l'Espace in Toulouse (France) is based on visitor profiling through a mobile app linked to a database. Prior to the visit, a questionnaire is filled that allows the system to match the user to a visitor type among five predefined ones (personas), coinciding as closely as possible with the profiling. The association with the profile allows for the personalization of audio narratives and information within the museum, providing an adaptive path. Content is delivered through additional information in Augmented Reality, such as color reconstruction of statues. The outcome is a nonlinear and dynamic experience, resulting from the continuous updating of the system based on the profiles acquired.

Pre-defining user-profiles encourages the personalization of narratives related to the artifacts, delivering content for specific types of visitors that can be implemented through Artificial Intelligence systems.

The Voice of Art (b6) is an interactive guide that provides users-visitors with personalized information at the Museo Pinacoteca de São Paulo by using the Watson technology. This is an IBM's cognitive computing system that uses Artificial Intelligence to learn content from numerous sources and return information about the artifacts with reference to visitors' interests. Due to a device with a dedicated app and a headset, the user receives notifications during the visit in the proximity of the artworks included in the tour and is invited to ask questions about the paintings and sculptures. The system generates and provides relevant answers to the questions formulated by the user, delivering adaptive audio narratives.

Figure 1 shows the application examples, analyzed based on the identified functionalities and the digital narrative classification proposed by Brouillard et al. [37].

Fig. 1
figure 1

Framework of functionalities and tools, classified according to the model of Brouillard et al. [37]

Full size image

Pervasive models and technologies for interaction and augmented perception between physical and digital dimensions

In the contemporary scenario, numerous interactive devices have been introduced to support the enjoyment of cultural heritage, capable of broadening the perception of the experience.

This is considered by Schnapp [1] to be a change that spills over into society and individuals who, as increasingly interconnected “digital citizens”, solicit knowledge, participation, involvement and interaction.

In the cultural system, in terms of content production, the search for new digital expressive languages has produced a technological and aesthetic evolution: three-dimensional synthetic images, installations and interactive artifacts, immersive sounds and music, multimedia or virtual works [1].

According to Fabris [45], environments in which information and communication processes occur automatically are configured, without the will to activate them. This increasingly removes these processes from user’s control. It means that they can be conceived as a real environment, a kind of “infosphere” in which people live and with which they interact [45].

In this scenario, it is evident how interaction and knowledge dissemination tools and solutions able to create interactive and engaging experiences for the user have changed over time. They are increasingly miniaturized and integrated into the fruition environment.

Innovative digital tools—both stationary and mobile—that are becoming more widespread, integrate fruition experience through sensory and perceptual amplification gimmicks: from wearable systems and devices to digital media such as tablets or individual devices, to robotic assistive systems, to tools that can support the user through indirect interaction gimmicks.

With reference to the interactions between Artificial Intelligence and robotics, Suzuki et al. [46] define four levels of interactivity to human–machine interactive processes:

  • no interaction, output only;

  • implicit interaction, which considers the non-explicitly enunciated movement of the user as input;

  • explicit and indirect manipulation, where interaction can occur, for example, by pointing or selecting objects or by unambiguously determining actions with body movement;

  • explicit and direct manipulation, where the user physically interacts by direct input with hands or body;

In addition, Suzuki et al. [46] identify seven ways in which interaction actions can be performed:

  • tangible, the user can physically alter, deform and modify the object;

  • touch, the user can interact with touch screens and virtual menus by touch;

  • pointer and controller, the user can perform control, manipulation and communication actions through spatial interaction with tools like pointers and controllers;

  • spatial gesture, the user can interact through spatial gestures and manipulate defined points in the virtual environment;

  • gaze, the user can perform a selection and input command by gaze, this mode is often associated with Spatial Gesture;

  • voice, the user can use voice input to execute commands;

  • proximity, based on the user's behavior and location, control input is sent. This mode can be used as an implicit form of interaction.

From this reconnaissance, it is possible to see that today the repeated input of the user to initiate an information process is no longer necessary. It is necessary to interact with machines, with the procedures and activities of those systems that demonstrate thus an increasing degree of autonomy [45].

Each tool provides for a level of interaction that differs depending on the context/space in which it is embedded, on the complexity of information conveyed, and on the models adopted for narrating information. Finally, it depends on the level of technological-functional complexity that is integrated.

The fusion of the physical and digital dimensions is first introduced in the marketing industry with the term “Phygital” to define personalized emotional content capable of sticking in users' minds [47]. The phygital dimension integrated with the pervasive dimension of innovative technologies for the use and enjoyment of spaces and the dissemination of information, define new modes of interaction between the real and the virtual, between the physical and the digital. These require the constant updating of the analysis of human–machine interactive models and modes.

In this regard, a survey of tools and technologies that amplify the user experience was conducted. In particular, tools that can (c) alter and modify the space enjoyed by integrating information transmitted to the user; technologies that become the new (d) experience to be enjoyed; tools that (e) integrate and support the fruition experience.

Therefore, as summarized in the framework of Fig. 2, a classification has been defined from the type of technology that supports the fruition experience, functionality and tools used.

Fig. 2
figure 2

Framework of tools and technologies, classified according to the model of Suzuki et al. [46]

Full size image

Figure 2 shows the application examples, analyzed with reference to the modes and levels of interaction proposed by Suzuki et al., 2020 [46].

Modify, enrich and reconstruct real and virtual space

In recent years, various technological gimmicks have been adopted to amplify perception and support the user's cognitive process while enjoying and using “space”.

Today, the physical space and the space "enriched" by technologies can be considered as an “information continuum” in which users interact with various devices to enhance the experience [37, 48].

Tools can be identified to (c) alter and modify the space by transforming the visited environment, integrating information conveyed to the user and identifying three functions: (c1) modify the space, (c2) enrich the space with information, and (c3) reconstruct the space.

For example, the project "Night Forms: Dreamloop" (c1) by Klip Collective is configured as a multisensory experience that combines art with nature through video projections, lights and sounds, which allow the space to be enjoyed at night. The sculptures within the space are mapped and "covered" by digital projections that modify the view of the artifact and become immersive perceptual experiences. In this case, the work is altered and enriched by technology that generates new multisensory experiences and becomes a solution to be enjoyed and used.

Differently, with the "MuseumEye" application (c2), at the Cairo Museum, Microsoft Hololens HMD visors were used to configure the "MR Museum" experience. The use of visors in the context of fruition allows enriching information conveyed to the user through mixed reality (MR). These devices project interactive images and characters from antiquity into the museum [49] by integrating additional information within the visitor space with digital representations that amplify the cognitive process.

According to Schnapp [1], the main challenge of effective cultural policy in the digital age is to open up to innovation to ensure greater access to information, also by adopting new forms of online use and enjoyment of cultural assets.

An example of tools that define an environment usable from a distance is the immersive experience "Beyond the Walls" (c3), which allows users to move freely through the reconstructed space of the Smithsonian American Art Museum's galleries. Through the use of Oculus or Vive viewers and a controller, a high-fidelity reconstruction of the exhibits in the gallery is provided to the user through photorealistic 3D images and augmented reality elements. This also allows works that no longer exist to be viewed through reconstructions that otherwise would not have been accessible.

The immersive experience allows visitors to approach and explore the space and artifacts. By using the controller in a specific room they wish to visit, it is possible to “teleport” into that specific place. In addition, by integrating headphones to the device, the visit can be accompanied by audio narratives. For the outdoor spaces, a 3D volumetric reconstruction was made, accessible to users through a 360° video in 6 K.

Building and exploring information to enhance the user experience

According to Resmini [50], the postdigital is «a world of information that escapes the restrictions of the two-dimensional and takes its place in physical space, where it becomes a constantly re-produced, reshuffled, and reshaped pervasive layer, whose architectures silently create new behaviors, new opportunities, and new barriers that replace or eliminate existing ones».

With the help of new methods of implementation and communication, technological tools have changed the fruitive modes of spaces that do not need to physically contain the "asset”, as technology becomes the new (d) experience to be enjoyed. For such experience, three functions have been classified: (d1) building new experiences (d2) exploring information and (d3) transforming knowledge.

By associating lights and mirrors, Yayoi Kusama creates an illusory space through the “Infinity Rooms” installation. It defines and builds spaces whose boundaries are not determined, and experiences of fruition diversified for each user based on the point of observation by amplifying the user’s visual perception.

Differently, at the Cleveland Museum of Art, the interactive experience provided by the "ARTLENS Wall" (d2), allows users to virtually explore artifacts through interactive touch displays, creating novel digital compositions from the fruition experience in situ.

In the information age, the big data phenomenon is also revolutionizing the ways of reality reconnaissance and information representation techniques to "draw" intelligence from the vast amount of available data. In this scenario, data depiction strategies make it possible to capture relationships and evidence that would not have manifested themselves without the creation of appropriate signs for their display [51]. An example is the virtual data fruition experience of Refik Anadol's "Machine Memoirs: Space" (d3) project created from the NASA archive.

The space images available in the exhibition are publicly released photographs taken by NASA satellites and spacecraft, processed through learning algorithms and Artificial Intelligence and transformed into data sets. These data create digital sculptures and abstract shapes to convey visual information and show digital connections that are integrated with audio experiences, designed on the basis of data.

User–instrument–space interaction between technology and culture

Currently, information and communication are not only dependent on practicing specific techniques but are developed through the use of particular technologies that operate through the transmission of information.

According to Epifani [52], these technologies have changed the way people inform themselves, have redesigned decision-making processes and have re-established the ways in which they interact with the tool, with other people and with the ecosystem of services, until they become interfaces to the reality around us. Reality in which such tools are used to manage relationships, access information, and enrich interaction with the physical world.

Today, in fact, information processing and transformation are not solely the result of a process, they are the way devices function. They are capable of transmitting data in an automated manner, with extreme speed, demonstrating ever-increasing memory capacity and acting in conjunction with each other. As a result of this and in relation to their operation, people’s capacity of information, communication and learning also changes [45].

These digital tools define assisted fruition experiences and alternative ways of conveying information that (e) integrate and support the fruition experience. It is possible to identify three functions for them: (e1) discover (e2) navigate (e3) deepen.

Among the examples of tools that mediate and integrate (e) the fruition experience and the transformation of information, the project implemented by GURU for the (e1) Dalì Museum in St. Petersburg, Florida, which has always experimented with the synergy between technology and culture, is of interest. This allows visitors to explore the museum and the artifacts using their personal mobile device. In fact, through the installation of an Augmented Reality application, framing the painting allows AR animations to be superimposed on the works and focus on elements and details, as well as animating Dalì's works and bringing the exhibition to life. The use of AR facilitates interactive learning and exploration of the artifacts in a novel way and allows for the discovery of features and symbolism present in the work of the Surrealist painter.

Another model of technology as an aid to fruition is the virtual tour proposed by the (e2) Young Museum in San Francisco. Through a pair of robots, this allows people with severe motor disabilities, who cannot physically visit the museum, to navigate and move around the galleries. In fact, from any location equipped with a Wi-Fi connection, it is possible to drive the robot. Moreover, through a display placed at the end of this robot, users can interact and share the experience with other museum visitors. In addition, through a voice activator and eye movement tracking, they can send command inputs to the robot.

At the Cleveland Museum of Art, there is also another communication tool that enables users to enhance their visit experience. Specifically, (e3) ArtLens is the digital application, which allows visitors to explore in-depth the collections through the use of a device-tablet. The latter, through an interface designed to be intuitive and accessible, allows visitors to view the artifacts in Augmented Reality and orient themselves through space using maps transmitted through iBeacons scattered throughout the museum.

Systems for detecting and tracking the fruition experience

Technological progress in the field of cultural enjoyment has favoured the use of advanced tools to enable tracking and assessing experiences. It is important for visitors to be able to assess the experience by expressing their personal level of satisfaction [53]. Providing customized paths can represent an important factor to enhance user satisfaction [54].

One of the methods primarily used in museums to understand and improve experience is represented by questionnaires or feedback forms to be filled at the end of the visit, to collect information on demographic characteristics, on the motivation and frequency of the visits [55]. Aiming to increase the number of visitors and to build audience loyalty, the results of the questionnaires are used to provide useful information for the design of exhibitions, spaces and programs that would be able to encourage recurring visits [56].

Despite the ease of implementation of this method, it does not allow to obtain information on real-time interaction with artifacts and space during the visit [55], on the quality of contents and on the efficiency of the means for conveying information. In the process of user-artifact-space interaction, the aim is enhancing the emotional impact of the visiting experience and increasing the engagement of an extended range of users by satisfying different requirements and adapting paths to the various audience needs. The visiting experience generates reactions at the cognitive and emotional levels that may be analyzed to enhance museum paths [54].

Movement within space, the visitors' behavior, and the physical proximity with reference to the exhibited artifacts determine the way how the environment is perceived, and the relationship developed by visitors with the context and the artifacts.

It is necessary to foresee the use of tools able to track the position and movement of visitors within the path, the movement of hands, and facial expressions. Systems for voice recognition, eye-tracking, body gestures tracking, tracking emotions through biofeedback (ECG, EEG, galvanic skin response, sensors for monitoring heartbeat and other biometric sensors) are able to provide useful information to increase experience and extend the target users at museums.

To this purpose, the use of tools enabling the registration of profiles at the beginning of the visiting path to track the visitors’ itinerary can be useful to understand the diversity of the fruition process with reference to the target users (f). Even today, this diversity of individuals represents an absolute ‘black box’ for museums; it is a multi-shaped, iridescent universe that remains completely unknown [30]. Knowing the target users, understanding the itinerary taken, the times of fruition and the salient points could contribute to enhancing the experience in real time.

An interesting example is the Sony Wonder Technology Lab in New York where, at the beginning of the visit, users can choose to create a customized profile that accompanies them during the path through the use of Integrated Circuit card Sony FeliCa contactless technology. The recorded data can be manipulated and disseminated within the exhibition through displays that allow shared experiences between visitors soliciting interaction with space and technologies. The visualization of collected data becomes an object of fruition (f1) contributing to enrich the visiting experience but it requires reflections with reference to the users’ privacy and their real willing in sharing data with other visitors.

In the same direction, there can be highlighted the exhibition “Who We Are: Visualizing NYC by the Numbers” at The Museum of the City of New York addressing the topic of the New York census. This exhibition included the “What Counts” installation, which allowed visitors to fill a digital questionnaire to obtain layered “Data Portraits” based on shapes, colors and symbols that identify the visitor. The tracked data have been projected in real time within the exhibition providing the possibility to print them as customized gadgets to increase the engagement of users in creating the experience (f2). It is however necessary to understand the limitations in using such methods as not to overcome barriers related to ethical and users’ privacy issues.

Within these limitations, tracking user data may mean providing customized paths, answering to the different preferences, and increasing visitors’ engagement in the experience. At The New International Spy Museum, users are tracked while performing digital experiences within the museum itineraries during an “undercover mission”. At the end of the visit, the results of the behavior tracking collected during the entire path allow to assign the user with a profile associated with a spy “personality”. Data becomes a tool able to enhance curiosity and increase memory of the experience (f3).

Tracking movements within space

The importance of the visitors' movement within the exhibition space for building the experience has been acknowledged over time. The ability to perceive depends on specific receptors for the external environment (exteroceptors), internal environment (enteroceptors) and in relation to movements and to the position of the body and arms within space (proprioceptors) [57]. Initially, the practice of “timing and tracking” has been used within museum paths through direct observation of the visitors' itinerary, but the development of recent technologies has led to the use of new tools to facilitate the process of tracking movements with reference to the fruition space to enhance the experience (g) [58] and increase the quantity and quality of the revealed data.

In order to track the position in external environments, the commonly used technology is the Global Positioning System (GPS). However, with reference to interior spaces, this technology faces several criticalities mainly due to signal interferences with the building components [58, 59].

In the case of internal fruition paths, tracking systems based on sensors for active or passive localization of visitors may be used. Specifically, active localization requires visitors to use a device during the visit in order to acquire information and transfer them to a system able to determine the position within space [55, 58]. In this category there are technologies such as Bluetooth, infrared, RFID, ultra-wideband, wireless network, ZigBee radio frequencies [58].

Passive localization systems are instead less invasive since they do not require visitors to use specific devices and they can be Computer Vision systems, systems for differential air pressure, or technologies with similar functioning [58].

An example of active localization is represented by the system for tracking visitors’ activity proposed at the Trowulan Museum in Indonesia by Handojo et al. [60] which uses Bluetooth Low Energy (BLE) beacons displaced in specific positions within the path (g1). Usually this technology is used on small distances to ensure wireless connection between fixed and mobile devices [61]. Based on the power of the signals detected through the visitors’ personal devices (e.g., smartphones) it is possible to approximate the users’ position. The collected data is stored in a server to provide information on the main points of interest and with reference to the duration of the stops. Despite the potential of the study in developing new systems for tracking users’ location, there are several criticalities regarding data accuracy. Specifically, there is a difference that varies from 47 to 174 cm between the real position and the one approximated by the system. These could be considered high values with reference to museum paths where the artifacts are placed with a certain density.

In fact, the dimensions of the space able to cover (coverage) and the accuracy as the capacity to identify the real position of the visitor within the space, are among the main parameters for the assessment of tracking systems [58].

The approach of the Museo Nacional de Artes Visuales in Uruguay, which uses an indoor Wi-Fi tracking system combined with an Android application to track the users' position, is of particular interest. The coverage of this technology has been significantly improved over time, reaching from 100 m to more than 1 km due to the use of specific protocols for IoT services [61, 62]. The system used at the Museo Nacional de Artes Visuales is implemented through algorithms of Machine Learning that learn how to map the Received Signal Strength Indicators in order to understand the users' position [63] and to provide data to the museum to enhance the visiting experience (g2). There are however some criticalities regarding the accuracy and interferences in using such technology, requiring the use of complex processing algorithms [61].

Other systems foresee the use of light and sensors for indoor localization such as Light-Emitting Diodes (LED) or Light Detection and Ranging Localization (LIDAR) systems. The systems using LED have several advantages such as the low cost, the resistance to environmental changes, e.g., humidity, and low energy consumption, but visual continuity is needed to ensure accuracy of the positioning [62]. Instead, the LIDAR systems combined with inertial sensors can provide accurate results, but a sensor needs to be positioned in each room, which translates in increased cost for bigger museums [61]. In this direction, Rashed et al. [64] propose methods based on LIDAR technologies to track the position, body orientation and the itinerary followed during experiments at the Art Gallery of the Ohara Museum of Modern Art, Kurashiki, Japan (g3).

Acoustic signals represent a further technology useful to locate users through sound sources with the support of microphones. However, this system has several critical points with reference to the capacity to transfer sound without undesired interferences. The ultrasound systems are instead very accurate even if they might be impacted by variations in temperature and humidity or by the environmental noise [61].

Relating the user with the fruition space, it is interesting to track the body and hands movements to allow increased engagement in the visiting experience [65]. Through the FingerTrak device (g4), it is possible to "translate" in 3D the position of the hand and fingers due to a combination between thermal imaging and machine learning. The use of these technologies together makes the system different from other traditional tools. Four miniaturized low-resolution cameras allow rebuilding the outline of the hand wrist in 3D to acquire data regarding the movements of the hand. This device may be used in translating sign language and in virtual reality applications by tracking the user experience to understand the main points of interest [66]. Despite the fruition opportunities, the tool may still be perceived as incumbent and there is a need to improve wearability features.

Perceptions and emotions within museum paths

To understand users' perception of the artifacts and explore emotions during the fruition process (h), several studies have thoroughly investigated the main aspects observed in images by analyzing how visual exploration takes place. It regards an attentive study of the cognitive aspects, which are fundamental for art perception, and the analysis of the user's interests, expertise and the context of the visit [67].

In this sense, eye tracking is useful to track and analyze the trajectory of the gaze during the fruition of an artifact in order to observe the main points of interest. A study from 2001 [68, 69] used eye tracking and verbal protocols to analyze and understand the fundamental cognitive processes for the perception, conceptualization, and verbal description of images from users (h1). The experimentation allowed to observe the correspondence between the trajectory of the gaze (a) while listening to the verbal description of the artifact and (b) during the image reconstruction from memory. In this second phase, visitors have imagined the scene by rebuilding it through the gaze while situated in front of a white screen. This proved the importance of verbal descriptions in evoking and stimulating the creation of mental images also in the case of visitors with visual impairments. Data acquired through verbal descriptions have also provided information on user perception and experimentation thus contributing to create an overall framework of the cognitive processes during the fruition of the artifact [67]. It was therefore possible to reveal the main points of interest regarding the artifacts based on the distribution of elements. The time of fruition was analyzed to highlight differences between artifact perception and interpretation from different visitors with reference to their background, interests, previous knowledge, expectations, specific knowledge on the topic, expertise, emotions, and behavior. The context where the artifact is exhibited represents a relevant aspect for its perception and interpretation [67].

In fact, the context and the means for knowledge transfer contribute to stimulate emotions in the user enhancing the visiting experience.

Several studies based on Affective Computing models have proposed automatic solutions to reveal the users' engagement based on emotions and interests, providing details on the level of attention during the visit [54]. Most systems for the analysis of emotions starting from facial expressions use Convolutional Neural Networks (CNN). They use images as input allowing to predict, starting from a trained model, which could be the actual emotions of users-visitors. These systems are considered less invasive than tools based on biosensors since they alter less the users’ behaviour during the visiting experience [54].

In this direction, a study made at the Museo d'Arte Moderna “Palazzo Buonaccorsi” in Macerata (h2) has used a system that tracks and measures human emotions through facial recognition estimating the age and gender of users. They are associated with a user profile chosen among the three types framed within the project, which is useful to provide suggestions on the visiting path by selecting five artifacts. These are to be shown to the visitor through a set of photographs and video information with details regarding the artifacts and the authors [54]. The system thus provides the museum with information on the user experience to improve the contents conveyed. Even so, providing predetermined sets of five artifacts and of information to be delivered seems limiting with reference to the different needs of visitors.

In order to shape customized experiences based on interests and preferences of the diversified users and in relation to the detected emotions, Ferrato et al. [70] propose the META4RS system (h3). This is a recommendation system applied to museum paths based on Computer Vision algorithms and RGB cameras. Such systems allow data acquisition with reference to the visitors' position and emotions while observing artifacts. At the beginning of the visit, users choose a badge which, together with the users' face, are identified by the cameras positioned within the museum. This allows tracking the users' position and investigating their reaction in front of artifacts to explore the most relevant features in analyzing facial expressions. At the same time, it allows to understand the duration of the visit or details on the time spent in front of a specific artifact.

Advanced tools for cognitive tracking

Design for cultural heritage evolves towards emotional engagement to open museums to a wide audience and transform them into spaces able to host multilayered experiences.

There is a necessity to take into consideration complex interactions between the visual, auditory, olfactory and spatial aspects that need to combine harmoniously within the visitor experience. It is necessary to understand the strong impact they could have on visitors and the opportunity to “shape” their mind [71]. In this direction, neurosciences investigate on the mechanisms that take place in the brain focusing on cognition both with reference to basic mental processes such as sensations, attention, and perception, and to complex operations such as memory, learning, use of language, problem solving, decisional processes, reasoning, intelligence [72]. This leads to study the interaction with artifacts and architecture also through connected disciplines such as neuroaesthetics and neuroarchitecture. With specific reference to arts and aesthetics, neurosciences deal both with the study of experience components and in understanding the visitors’ aesthetics preferences and their perception of the artifacts. On the one hand, the rational component is studied, and on the other hand, the emotional one is investigated to discover whether there are universal rules for the perception of beauty [73].

The systems employed in tracking the brain activity contribute to prefiguring “digital neuro-experiences” by activating systematic actions useful to design adaptive experiences [74].

Traditional imaging methods such as Functional Magnetic Resonance Imaging (fMRI), Computerized Tomography (CT), Positron emission tomography (PET) limit observed information since it is necessary for users to remain stationary during data acquisition process to accurately measure cerebral activity. To avoid these problems, nowadays Mobile Brain-Body Imaging technologies (MoBI) such as Functional near-infrared spectroscopy (fNIRS), electroencephalogram (EEG) through portable devices are being developed. These are able to simultaneously acquire data referred to body movements and to cerebral activity increasing the comprehension of emotions and that of the cognitive system during interaction with the surrounding environment [75]. Specifically, research on neuroaesthetics allows the analysis of cerebral processes (i) and functions engaged in experiencing arts and the consequent impact on users-visitors’ health and well-being.

In this sense, the approach of the Munch Museum in Oslo within the Sensitive Pictures project is interesting for the use of a portable EEG device integrated in an audio guide. This system is useful to measure in real time the cerebral activity during the visiting experience (i1) in order to customize the path [76]. The audio descriptions with reference to six artworks within the museum have been interconnected with a self-reporting tool useful to detect the participants' emotions. Processes of Affective Interaction are used to engage users with their emotions through digital technologies. The results of the self-reporting have been associated with the analysis of the body reactions as investigated through the portable device in order to provide customized contents. The experimentation shows the growing interest in the field of adaptive experiences able to engage users with reference to their characteristics.

Information acquired through the various tracking systems can contribute to enhancing the visitor experience. At the same time, they can provide further information to the museum allowing a possible reorganization of the exhibition layout, of the space and the institution, such as updating the visiting times considering the number of visitors, further enhancing specific artifacts or museum spaces. Analyzing the visitors' behavior, the visiting time and itinerary, the choice to stop in specific areas of the museum instead of others could provide information on the main points of interest within the museum path and they may be able to support curators in assessing the entire exhibition [55] in order to improve user experience.

Figure 3 shows the framework of tracking technologies associated with the corresponding signals/indicators and the target tracked reported to the cases analyzed in this paragraph.

Fig. 3
figure 3

Framework of tracking technologies

Full size image

Standards, ethical implications and privacy management in the use of advanced digital systems

In the current context, the museum theories and practices show a general tendency to improve actions aimed at accessibility and increasing the participation of the public in its diversity [77]. It is therefore necessary to understand the potential that new technologies, such as augmented reality or user-artifact "interaction" technologies, represent for museum realities, offering diversified responses to the many challenges facing the field.

The connection between scientific knowledge and humanistic culture has constantly pervaded the cognitive journey of humankind. Nowadays it is increasingly evident that these fields can and should progress together, to the point that it is no longer realistically conceivable to have an exhibit that is not an implementation of cutting-edge technological solutions [78].

The digital revolution is generating new connections between cultural heritage and people. It defines new "perspectives" to the field through the creation of a cultural ecosystem capable of increasing potential demand and the volume of information directed to the user of the asset.

As highlighted by Deligiannis et al. [79], current technological devices and, consequently, innovative digital content enable the enrichment of the cultural experience leading to the generation of data that needs to be well managed to become “value”. Data management itself is a focal point for the advancement of the entire cultural heritage field.

According to Poulopoulos and Wallace [80] the paradigm change that is going on in recent years is closely related to the generation of data and to the diffusion of culture shifting from the object-centered approach to the people-centered approach.

There is a trend for museum institutions to invest and include technologies, such as tracking and monitoring technologies, among the resources considered strategic to "innovate" at every territorial level [81], to generate innovative forms of enjoyment of places and cultural accessibility, and to make the narrative of the artworks themselves more coherent, sensory and customizable.

In the current context of cultural heritage, people are confronted with the presence of digital models and systems that enable the detection, tracking and evaluation of the user experience in the fruition and interaction with heritage. Such innovative solutions—closely related to the dissemination, enhancement and communication of artistic-cultural content—aim to enable the concept of personalized exploration of cultural heritage through information, narratives, stories that enrich the vision [82].

From the study by Roussou and Katifori [36], it is easy to deduce how technologies and associated digital solutions—employed for the purpose of "exploiting" cultural content—represent the main current implementations in cultural places, such that they represent a strongly investigated research topic.

The different studies identified in the literature highlight the various challenges in the use of such systems that need to be addressed as early as in the design stages such as, for example, visitor attention between environment and device, navigation, device usability, social aspects, and all content-related aspects in terms of information and data. In particular, there is constant streams of information made not only of pronounced or written words but also of sounds and images. Such flows are capable of engaging us without specific skills and without the possibility of stopping them [45].

Many of these solutions rely on personal data as "knowledge bearers" about bodily parameters and behaviors, actions performed in physical and digital spaces, and interactions with interfaces and users [83]. If it is true that technological evolution has given rise to advanced technological systems, it is equally necessary to ask to what extent it is possible to use collected data while avoiding the violation of users' privacy. Although highly debated, this issue nevertheless needs more attention in terms of ethical implications and data privacy management, through the close relationship between legal and technical standards and technology itself.

In fact, the plurality of data sources available today, which is destined to further expand in the years to come, "forces" us to set up data processing based on data protection-aware strategies.

The qualitative relevance of data to support museum fruition

The use of technological monitoring and tracking devices redefines the progress in the innovation for cultural heritage and results in the production of an amount of information that represents a unique "capital" for museum systems. In fact, through the analysis of the collected data, the strengths and the main critical points for the improvement and optimization of the enjoyment of museum paths and user-artifact interaction are highlighted.

It is therefore necessary, through the identification and recognition of the main reference standards and guidelines, to analyze the factors related to the management of user data processing.

To consolidate the current trend regarding the strong interest in the topic of data in different formats, two of the main standards related first of all to data quality should be mentioned: UNI ISO/IEC 25012:2014 and ISO/IEC 25024:2016 [84, 85].

In particular, one of the main reference standards for the correct use of data is the UNI ISO/IEC 25012:2014 standard [84], which defines a general model for the quality of data stored in a structured format within an information system. This standard also establishes quality requirements by identifying 15 specific features for characterizing data such as accuracy, timeliness, consistency, completeness, credibility, accessibility, understandability, compliance, efficiency, accuracy, confidentiality, traceability, availability, portability, and restorability.

The ISO/IEC 25024:2016 standard [85], on the other hand, defines quantitative data measures in order to identify the level of quality of the characteristics defined in the previously mentioned standard. Specifically, the document defines the data lifecycle, consisting of stages from data design, data collection, external data acquisition, data integration, data processing, presentation, other use, data store to delete.

In the connection between technologies and standards, it is also worth mentioning documents that better frame and outline the path of the issue at hand. In particular, the document "Resolution of the European Parliament of February 12, 2019" [86]Footnote 1 highlights the importance of interoperability and veracity of data in a way that ensures a high level of reliability and security in new technologies, as well as calling for greater clarity in existing regulatory frameworks on data ownership.

The ethical dimension in technology use and data management

Particularly in the field of Artificial Intelligence, data management is a key factor to think about in "safeguarding" personal information and enabling the proper use of data. In this sense, data management should also be considered in an ethical dimension, especially in the presence of highly technological systems. For example, consider the use of Artificial Intelligence to support the enjoyment of cultural heritage or the introduction of integrated technological eye-tracking or facial recognition devices. Ethical issues should never be overlooked in these cases.

New technological tools enable the processing of large amounts of data to facilitate the fruition of information, services, and content narrative based on the visitor's interests. In accordance with Calamai et al. [87], in parallel with the exponential growth of information, it is of paramount importance to decide which data should be preserved and which should be eliminated, how to access data and of its reuse in the correct form, considering consistency, sustainability and, above all, ethics.

In the Italian context, the National Plan for the Digitization of Cultural Heritage [88] has been developed, which is the new strategic vision for promoting the digitization process of cultural content and cultural realities—archives, museums, superintendencies and more generally cultural places—in the direction of an aware, participatory, shared, sustainable and inclusive digital transformation [88]. The document represents the methodological and operational guidance tool that leads the process of sharing a common path by all Italian cultural institutions. The final section of the document contains operational annexes for the execution of the different practices of digitization and transformation of cultural heritage. Regarding data, the "Guidelines for the Drafting of the Data Management Plan" (DMP) are attached, which provide the first references to Data Protection developments. In fact, the DMP pays particular attention to the different security measures applicable to digitization activities, it suggests good practices in the field of open data, and it contains the main Italian and European regulatory references. In compliance with legal and statutory requirements and preventing possible risks and critical issues related to preservation, access policies and sharing, data is summarized according to four main characteristics such as accuracy, consistency, completeness and timeliness.

A distinctive feature of the document is the focus on publishing data in an open format also with reference to the cultural heritage field. This is considered relevant for implementing knowledge and creating greater cultural value and sharing. The document describes the standards adopted for Open Data, the licenses chosen for data publication, and compliance with the "FAIR Guiding Principles for the Management and Maintenance of Scientific Data”. These principles—which are intended to ensure findability, accessibility, interoperability, and reusability of digital resources—appear to be fundamental in the national and European context.

The FAIR principles, considered as a fundamental step in addressing the data management issue, refer to any digital object, metadata, and infrastructure [89]. In detail, these principles aim to improve the quality of data on the Web and to implement the ability of computational systems to find, access, interoperate and reuse data and digital resources, in order to increase research and knowledge as well as promote data sharing and reuse, taking into account ethical requirements.

Data protection in the European context: the new General Data Protection Regulation

Even in the European context, the right to data protection—enshrined starting with the EU Charter of Fundamental RightsFootnote 2 and the Treaty on the Functioning of the European UnionFootnote 3—is evolving in the same direction as in the Italian context.

In particular, there are more references regarding the subject covered, starting with Regulation (EU) 2016/679 [90] or General Data Protection Regulation (GDPR), replacing Directive 95/46/EC of the Parliament and the Council of Europe, by which guidelines are determined in the area of protection of natural persons.

Particular attention is being paid to data processing through the harmonization of privacy regulations across the European Union starting with the principles of fairness, transparency, and lawfulness. With the introduction of GDPR, there is a paradigm shift that, through a proactive approach to assessing the impact of data processing, represents the accurate analysis of risks to people's rights and freedom. The main objective of the new document is the approach and design of data processing that can protect data privacy for the purpose of secure circulation. One of the concepts introduced within the new Regulation is the "accountability principle”, according to which appropriate technical and organizational measures are put in place in order to ensure the correct processing of personal data.

According to this principle, it is necessary to prove that any such measures are taken to prevent loss, uncontrolled disclosure, or falsification of data. This will be done from the analysis of the risk to which data is exposed and for the identification and application of the most appropriate means of protection based on the nature of data, processing operations or used information systems.

With the GDPR, the principles of "privacy by design" and "privacy by default" are also mentioned. Specifically, "privacy by design" defines the approach to establish that data protection takes place from the design of a product/service/process in accordance with the GDPR, while "privacy by default" considers the need to protect users as a default setting through a selective approach and thus avoiding an excessive amount of data.

Another aspect explored in depth in the GDPR, is the concept of profiling, which is the automated processing of personal data and allows users to be divided into different groups, based on interests or preferences, through the acquisition of information and personal data that are used to define an "identity" profile. The new Regulation particularly protects users in the profiling process starting with explicit consent to the processing of personal data. An additional measure found within the GDPR is pseudonymization, made possible by many technologies that allow only part of the user information to be kept visible.

Through this technique, there is a reduction of the risks of a possible attack on the system according to two different ways of use: (I) as a processing mode applied in a preventive manner; and (II) as a security measure used at a later stage when the owner, after having put in place all the necessary "technical-organizational measures," considers it indispensable to apply additional measures to reduce the risks.

There is a complex and articulated issue that certainly highlights the increased awareness of the value of data privacy in the digital society and, specifically, in the cultural heritage field as a whole, the interest in ensuring the accessibility and enjoyment of assets [91]. In particular, the diffusion of current Artificial Intelligence-based systems has allowed access to a considerable amount of data [92], raising quite a few problems related to ethics and regulatory aspects and increasing reflections on the respect of the right to privacy. The GDPR itself currently represents the "bearer" of a new culture of data protection, also increasing the perception of related risks. In this regard, the European Union aims to enhance research while ensuring security and fundamental rights. In fact, the GDPR approach is the basis for the proposed European AI Act regulation, which in the months to come is expected to be the first piece of legislation governing implementations of Artificial Intelligence systems [93].

In particular, the AI Act proposal—in accordance with EU values and rights—defines harmonized rules for the development, marketing, and use of AI systems, introducing specific restrictions starting with the definition of AI applications based on four levels of risk: (I) unacceptable risk, (II) high risk, (III) limited risk, and (IV) minimal or no risk [94].

Recognizing the need to implement the regulatory system aiming to prevent the misuse of technology and to safeguard personal information [95], represents the starting point for moving in the direction of promoting common initiatives, standards, and languages capable of regulating data quality and improving the properties of systems, through thorough investigation and application of new technologies to support the fruition of cultural heritage (Fig. 4).

Fig. 4
figure 4

Framework of main national and international standards and guidelines regarding data protection

Full size image

Results of the overview: user-artifact-space experience in the cultural fruition

The analysis of the advanced systems carried out in this research highlights the main potential and the criticalities of technologies in using and enjoying cultural spaces through the identification of tools and devices best suited to the functionalities intercepted for an effective user experience. There is a need to take into consideration the interconnection between the different aspects analyzed to enable an integrated and personalized experience with reference to the user characteristics.

From the review illustrated through the paper starting from the selected case studies, the insights regarding the narrative, interaction and user tracking are connected to the requirements identified for the User Experience in order to highlight the methods for defining fruition paths in cultural spaces. The different categories identified (a–i) have been further analyzed with reference to aspects of user data in the data acquisition, collection, analysis, and conservation phases. These are useful for the configuration of adaptive paths that would consider the behaviors, actions and needs of users.

In detail, the research results are illustrated in the synthesis overview (Fig. 5) that reports the analyzed case studies, the main technologies and tools and the functionalities “activated” in relation to the different elements of the User Experience: system/product; user; artifact with the corresponding (physical and virtual) content; (museum, exhibition, fruition) space and the experience.

Fig. 5
figure 5

Overview of the technological systems with reference to the selected case studies

Full size image

The analysis of the representative examples showed features that characterize the User Experience such as adaptability, ease of use of the system and wearable issues with reference to the fruition devices. Also, the user’s engagement factors have been taken into consideration, the emotional-sensory, cognitive, and the active interaction and participation. Moreover, the levels of comprehensibility of information mediated by systems and devices and the level of physical and sensory accessibility have been assessed.

The ways how the selected technologies modify, alter the space perception, enrich it with information or increase its physical and sensory accessibility have been highlighted. To analyze the tools that support the reconfiguration of the fruition space based on the users’ behavior, further elements of evaluation have been associated to the User Experience such as sharing the experience in the different visiting phases, the integrated and continuous fruition through interconnected systems and the correspondence with the real world. The impact of technologies has been assessed in the experience “mediated” between the physical and the digital-virtual dimensions.

The analysis highlights that the “main” technology combined with different systems concurs to amplifying the user experience in perceiving the exhibition space at different levels of interaction with the devices and in relation to the artifact.

In fact, technologies and case studies have been investigated with reference to the digital narrative. This analysis shows how technologies support the comprehension and interpretation of artifacts by activating fruition ways that provide users with insights during the visit. The technological systems for the delivery of adaptive narratives transfer content through customizable experiences based on the user’s needs by making the fruition space accessible and the visiting experience continuous. Activating narratives automatically and semi-automatically in proximity of the point of interest represents the objective to reach in order to deliver an engaging experience in the exhibition space. This opportunity has implications in terms of user data collection that are particularly evident when using systems of Artificial Intelligence, as in the case of the project “The Voice of Art”.

In identifying the elements of User Experience, technologies enhance the user-artifact relationship in accessing and understanding artifacts, e.g., systems and interfaces that overlay information in the fruition space, such as through devices for the virtual and augmented reality. Therefore, the review of technological systems showed in this research has allowed to identify the key interactions in the fruition of cultural spaces.

The analyzed tools and technologies for tracking highlight different opportunities for renewing and adapting the museum offer based on data tracked with reference to the users’ characteristics, the duration of the stops and the preferences identified during the visiting path.

It was therefore useful to highlight, for the identified case studies, the different functionalities also with reference to their impact in the User-Data management in the phases of data collection, management, analysis, and use. This impact is represented in the overview (Fig. 5) and it is associated—within the functionalities—to each case study. Specifically, values on a scale from 0 to 4 have been associated, where 0 indicates no impact and/or implications for the user in terms of privacy and data management, while 4 represents the maximum level of attention that needs to be assigned to these indicators. The minimum levels of impact regard indirect ways of data acquisition or those where there are no references to the users’ personal data. The highest level regards instead the examples where data referred to visitors have been used, such as facial tracking, voice registration and/or the association to a profile containing personal data.

The use of systems for the registration of profiles allows to personalize the experience with reference to the revealed information, but some criticalities are highlighted with reference to the need to dedicate part of the visiting time to filling information by the visitors. Data visualization during the experience may contribute to building a shared experience and making the user an active participant.

An in-depth knowledge of the visitors through the use of tools for tracking information on the levels of attention, interaction and emotional engagement will allow to reorganize visiting paths and enhance the artifacts or the areas that register minor or major interest to increase in fact the quality of the experience and encourage recurrent visits. It is therefore a question of increasing knowledge on the diverse target users to enhance the visiting experience by customizing it and adapting it to the different characteristics and needs to satisfy the visitors’ expectations and at the same time provide institutions with the necessary data for the reconfiguration of paths as to increase interest from the extended audience.

Conclusion

The constantly increasing research on tools and methods to enhance User Experience puts knowledge and innovation at the service of cultural heritage, thus strengthening the dialogue between users, artifacts, and technology. In the following years, there will be a massive use of technologies able to revolutionize the means and dynamics of cultural heritage enjoyment and use. This becomes the basic structure to attract visitors and increase their loyalty [44].

The paper reports the overview of the main functional-performance characteristics of systems, tools, and “best-in-class” technologies (Table 1) referring to the analyzed directives, models and approaches, by providing instructions based on data management before, during and after the museum fruition experience.

Starting from the adoption of User Experience in its various forms, the analysis of the systems and tools for knowledge dissemination has highlighted the opportunities and the complexity of the current scenario. In particular, the main tools have been systematized and the innovative narrative strategies have been analyzed based on the existent criticalities in the museum context with reference to the delivery of adaptive contents. The critical-analytical investigation and the functional-performance survey of advanced tools and technologies for the augmented perception and interaction have brought to attention the need to adopt new fruition approaches that consider the various levels of user-artifact-space interaction. The systems must be integrated in the direction of coherent communication strategies that would put users at the center of the fruition process to provide customized experiences. By overcoming the classifications known in literature, it is necessary to shift towards multichannel, hybrid systems. Such systems do not only use several technologies, but they also create interconnections between different narrative modalities and they involve various forms of interaction between physical and digital. In this direction, the need to integrate digital narratives by developing adaptive contents based on the users’ experience has to be highlighted. These are to disseminate knowledge through different communication channels by defining multidimensional interactive ways.

In order to adapt paths to the different users’ needs, it is essential to track and monitor visitors during the different moments of the experience. In fact, the analysis of the state-of-the-art has proved a growing interest in using profiling tools and systems for detecting and tracking users within the exhibition space and during interaction with artifacts.

Both the opportunities and the limitations of advanced technologies have been highlighted through this research. Indeed, the role of Artificial Intelligence in collecting and analyzing data to guide institutions and designers towards the definition of adaptive paths, improving the user experience, should be considered. In this sense, it has been necessary to investigate the main directives and standards concerning ethical consequences and user privacy management with reference to data processing. Such aspects lead to reflection on the enormous amount of data that museums and cultural institutions are supposed to manage, in the direction of “big data” archives. They need to be “regulated” through a systematic study and by updating the standardization framework to support cultural heritage fruition. The spread of mobile connections, the integration between new communication tools and technological systems for augmented interaction and for tracking and monitoring user experiences, will guide the progressive overview of data, systems, and the standardization framework to implement new opportunities for adaptive fruition able to open up museums to an extended range of users. This is facilitated starting with the proposed European AI Act regulation [94, 95], which is the first reference to regulate Artificial Intelligence, through the introduction of restrictions able to ensure the ethical, safe and responsible use of this technology.

Table 1 Web links of the analyzed projects and the selected case studies
Full size table