Empowering cultural heritage professionals with tools for authoring and deploying personalised visitor experiences

Abstract

This paper presents an authoring environment, which supports cultural heritage professionals in the process of creating and deploying a wide range of different personalised interactive experiences that combine the physical (objects, collection and spaces) and the digital (multimedia content). It is based on a novel flexible formalism that represents the content and the context as independent from one another and allows recombining them in multiple ways thus generating many different interactions from the same elements. The authoring environment was developed in a co-design process with heritage stakeholders and addresses the composition of the content, the definition of the personalisation, and the deployment on a physical configuration of bespoke devices. To simplify the editing while maintaining a powerful representation, the complex creation process is deconstructed into a limited number of elements and phases, including aspects to control personalisation both in content and in interaction. The user interface also includes examples of installations for inspiration and as a means for learning what is possible and how to do it. Throughout the paper, installations in public exhibitions are used to illustrate our points and what our authoring environment can produce. The expressiveness of the formalism and the variety of interactive experiences that could be created was assessed via a range of laboratory tests, while a user-centred evaluation with over 40 cultural heritage professionals assessed whether they feel confident in directly controlling personalisation.

1 Introduction

Since the first experiments of the late 90s, personalisation applied to cultural heritage visits has been seen as a way to improve the overall visitor’s experience. This is done via techniques that can adjust various aspects of it: the information selected for presentation and its order; the media used to interact with the visitor; the adaptation to types of individuals versus groups; the interaction modalities (Ardissono et al. 2012). To reach this goal, personalisation techniques rely on gathering users’ preferences, background knowledge, and expectations via explicit [e.g. form filling (Oppermann and Specht 2000)] or implicit means [e.g. artwork selection (Wang et al. 2007)]. Moreover, a dynamic personalisation needs an infrastructure to collect use data to feed the computational models of the current context and the visitor’s interaction history (Not and Petrelli 2018). Personalisation also requires the digital content to be in a format suitable for adaptive instantiation, as well as a set of rules that decide when to present what content and how to interact with the visitor. To deal with so many factors, researchers have developed complex personalisation systems that require technical expertise to operate and therefore end up excluding cultural heritage professionals with the consequence that, despite over 25 years of research in the area (Ardissono et al. 2012), personalisation for cultural heritage has yet to be adopted at a large scale. One of the possible explanations for such promising technology not progressing beyond research-led experiments is the lack of attention to the gatekeeper: the museum’s curatorial team. Indeed, any system that aims to reach the visitors must take into account the curators’ goals, which can be as diverse as: to foster visitors’ learning (Falk 1999; Antoniou and Lepouras 2010); to generate emotional involvement (Marshall et al. 2016b); to extend the connection with the visitor beyond the visit via post-visit recall of the experience (Callaway et al. 2007; Lanir et al. 2013) or to follow-up the engagement with the cultural organization (Kuflik et al. 2015). While the visitors are the “consumers” of the visiting experience, the curatorial team is in charge of creating it, and therefore most likely to use the personalisation system as authors of the visiting experience itself in a real-world scenario.

The investigation presented in this paper revisits personalisation in order to empower those in charge of the design and deployment of interactive experiences of heritage to adopt it by granting them hands-on control over the adaptive structures for both the content and the interaction. In our research, we interpret ‘personalisation’ in its broad sense that encompasses three types of system behaviour (Fink et al. 1998; Gellersen et al. 2002): adaptability (also called customisation) offers end-users a number of options to set up the application/system the way they like it; context-awareness is the ability of the system to sense the current state of the environment and to respond accordingly; adaptivity implies the system maintains a dynamic model of the on-going interaction and dynamically changes its own behaviour to adapt to the changing situation. For the cultural heritage domain, all these three types of personalisation are essential to allow visitors self-tailor some aspects of the visit, to create a physical space that is reactive to individual/group trails and actions, and to adapt the amount and type of interaction as the experience unfolds (Not and Petrelli 2018). Seen in this perspective, the creation of personalised visitor experiences requires those in charge to orchestrate multiple personalisation facets, and, in turn, this calls for dedicated tools.

By fitting with the work of cultural heritage professionals (CHPs), we aim to open the way toward personalisation becoming deployable in real settings: an easy-to-use authoring environment will enable CHPs to use personalisation to adapt both the content delivered to and the interaction with visitors, and to monitor the quality of adaptive experiences from the design to the actual onsite delivery. This approach requires the designing of the collaboration between the system and its human user by: (1) defining when the automatic mechanisms are appropriate and when, instead, the judgement should be left to the author, and (2) to synchronise the activities of the two agents (the computer and the human) to achieve a superior quality of results. A similar approach to human-intelligent system collaboration has been successfully used in other cases [e.g. machine learning (Ciravegna et al. 2003; Petrelli et al. 2005) and end-user development (Ghiani et al. 2009; Desolda et al. 2017)]. To design and implement such a vision of user-system collaboration, CHPs were involved throughout the project in a user-centred design process. Our ambition was to create an authoring environment that:

• is open and flexible to deploy a very wide range of personalised experiences in a variety of heritage settings as diverse as indoor museums and outdoor archaeological sites; to go beyond screen-based interaction towards smart objects and interactive spaces; to handle and connect both onsite and online experiences.

• supports a do-it-yourself approach as to empower CHPs to be part of the definition and experimentation of the intended adaptive interactive experiences, to follow its deployment, and to be able to manage updates when in use.

• enables the reuse of content as well as of technologies owned by the heritage in order to reduce costs and foster use.

This paper discusses in depth both how these challenges were investigated, and the resulting implemented authoring environment. Examples of interactive installations in public exhibitions are used to illustrate the many facets that need representation and coding, and what our authoring development environment can produce. We particularly reflect on the implications of modelling the aspects of tangible and embodied interaction when visitors manipulate physical objects and move in places augmented via the Internet of Things. Our approach is open and more traditional location-aware scenarios that use smartphones or tablets are supported too. Throughout the paper we unpack how different forms of personalisation are controlled in the authoring process and how it empowers CHPs to control some of them at varied levels of complexity. At the end of the design and development process, an evaluation that involved over 40 CHPs allowed us to answer questions such as: Do CHPs feel confident in directly controlling personalisation? Does the authoring environment support a good variety of personalisation forms, both in content and in interaction? What is the potential of such a tool in the hands of its intended users, i.e. the CHPs?

The paper is organised as follows: we first discuss related work on editing tools for involving non-technical users in the creation of visiting experiences; in Sect. 3 we present the co-design process by which the research team gained a deep understanding of the practice around the development of interactive installations, which led us to rethink personalisation for nontechnical users (the CHPs). In Sect. 4 we use one of the exhibitions that was implemented as an example to introduce a formalism that enables the representation of all the ingredients that need to be controlled when creating content-based personalised tangible and embodied interactions for very different settings. In Sect. 5 we present the authoring environment that provides an abstraction over the formalism; we illustrate how the environment works and we report the result of the evaluation before offering our final reflections to conclude the paper.

2 Related work

Our research lays at the intersection between three areas: interaction design—an extended user-centred study on the practice of exhibition design, which fed our effort to create an easy-to-use tool for CHPs; personalisation—the tool had to enable the creation of personalised experiences in cultural contexts; and the Internet of Things—the personalised visiting experiences must have a tangible or embodied component. We review the literature from this standing, therefore focussing on research that addressed at least one of these aspects.

Ways to avoid coding the content within the logic of the system were envisaged in the early context-aware mobile information guides (Long et al. 1996). The subsequent advances in the field of mobile guides in general (Kenteris et al. 2011) and for the cultural heritage sector in particular (Ardissono et al. 2012), prompted the development of specific authoring tools to support the porting of prototypes to new application settings. Some tools facilitate the composition of the digital content for the personalisation of audio (Petrelli et al. 2000) or context-aware video (Pan et al. 2002) on mobile platforms. Other forms of authoring include the creation of a collection of multimedia pages organised in a tree-like structure (Linaza et al. 2008), the filling of visual templates with contents from heterogeneous sources (Ardito et al. 2012), and the authoring of complete mobile applications instead of its content alone (Economou et al. 2008). Editing tools have been developed for outdoor settings (Weal et al. 2006) as well as for indoor ones, possibly integrating mobiles with stationary screens, projections (Ghiani et al. 2009) or tabletops (Sprengart et al. 2009). Research systems have been followed by commercial tools for making mobile multimedia guides¹ enabled by the built-in localisation sensors now embedded in smartphones and tablets that support position awareness. These editors use maps (city maps for the outdoors or floor maps for museums) as anchor points for both the editor of the tour app and the consumer of the tour experience.

Although most editors are indeed based on a location/map approach, with authors required to associate multimedia items to hotspots or to physical objects via menu selection or drag-and-drop (Fidas et al. 2015), a few are based on the organisation of the content alone. The latter provide templates that organise content as a specific network (Petrelli et al. 2000; Hargood et al. 2016), an approach similar to those adopted for the development of adaptive hypermedia (Weber et al. 2001; De Bra et al. 2003; Cristea and Aroyo 2002). An approach centred on the content structure is also common in interactive storytelling: in this field, the authoring tools are based on the content graph with personalisation conditions related to the modelling of the user profiles or of the environment. For example, Vayanou et al. (2014) model the composition of digital experiences to be delivered to single visitors or groups via Augmented Reality on a tablet; branching points in the narrative script correspond to automatic choices to be computed by the system or to explicit requests via user input. More complex scenarios such as quizzes and AR games where visitors’ interaction depends on their position in the real world, other participants or virtual entities have been represented as a graph of activities (Balet et al. 2015; Hansen et al. 2012). This, in a sense, takes the editing of such visiting experiences closer to game-design and development² than to exhibition design. For example, in the nonlinear-story editor Twine,³ the content is split in chunks organised as a graph, each node controlled via a decision-making point. The Twine story editor can be used with professional tools such as Unity to model interactive storytelling as part of a videogame experience. It should be noted that these editing tools are not intended for non-technical users. Quite to the contrary, game design is surely seen as a profession or a demanding hobby for technically-inclined individuals.

To overcome the need for technical expertise, different approaches have been tried. The EU project Chess experimented with multidisciplinary workshops where technical users engaged with curators to create the story as the editing of the content, based on a graph and rules layout, required to understand logical structures, an attitude that most CHPs lack (Roussou et al. 2015). An alternative approach is End-User Programming (also called End-User Development) where specific tools are provided for the end users to design a prototype or automatically create a program while interacting with bespoke tools. Díaz et al. (2015) developed a mobile app paired with an online service: curators use the app for situated resource gathering while brainstorming about the exhibition on the museum floor then compose in a meaningful and systematic way the design thinking outcomes with the online service. Ghiani et al. (2009) experimented with a tool for curators to enable them to create a representation of museum rooms and exhibits, associate information to them and generate games with the aid of templates. Raptis et al. (2019) envisage a framework to implement information experiences tailored to the users’ cognitive characteristics where CHPs would be allowed, during an initialization phase, to select from a repository the types of activities to offer to visitors (e.g. information exploration), the objectives of personalisation (e.g. learning) and the cognitive factors to consider for adaptivity (e.g. user cognitively more attentive to visual vs. textual information).

How to bring the Internet of Things closer to the end-user is another aspect relevant for our research. A first step is to ease the set-up of interconnected heterogeneous smart devices, that is to say to simplify the specification of the system behaviour (e.g. to set the parameters that trigger a sensor by abstracting from the low level), and to offer physical debugging of sensor events during testing (Kubitza and Schmidt 2016, 2017). A graphical approach to the same goal was investigated to develop friendly user interfaces to help synchronize the behaviour of multiple devices in smart environments (Desolda et al. 2017; Ghiani et al. 2017). Building upon Desolda et al. (2017), Ardito et al. (2018) developed a menu-based graphical user interface for the cultural heritage sector to register and configure events and actions for smart objects. The interface abstracts from the low-level technical details and defines custom attributes such as location, semantic categories and resources associated to the objects. Visual abstractions also enable the definition of Event-Condition-Action rules that capture device-generated events and trigger the corresponding actions on the same or different device(s). However, this type of abstraction can only model action-based experiences (e.g. a quiz with a true or false answer), excluding narrative-based experiences, which, we found in our studies, are key for CHPs.

In summary, examples of easy-to-use editing of content for applications in museums have often used maps as centre point. However, this restricts the author in two ways: only experiences based on location⁴ can be modelled, and only simple one-path experiences can be designed. Editors for non-linear stories rely instead on graphs to capture the complexity of multiple paths, but this more complex storytelling comes at the expenses of an easy-to-use tool. Attempts to balance the trade-off between simplicity and flexibility have led to end-user programming approaches, though they do not originate from co-design methods based on users’ work practices to foster adoption. Finally, the potential of smart objects and reactive spaces in the cultural domain enabled by the Internet of Things creates a further level of complexity, which, currently, has been addressed only to streamline the setup rather than to support end-users in prototyping and building tangible and embodied interactive narrative experiences.

The intersection of these challenges was our starting point. What distinguishes our research from recent work with a similar purpose (e.g. Ghiani et al. 2017; Ardito et al. 2018) is the extended study of work practices, ambitions and expectations of potential users (i.e. the CHPs) that grounded our design choices. The advantage of this approach is that the representation adopted and the functionalities implemented are solutions the end-users are comfortable with. The role of content in creating the visiting experience emerged as pivotal in the eyes of the CHPs, therefore our creation process is centred around the narrative and a declarative representation of its context of delivery, as opposed to a map-based approach with its limitation to the physical space (Fidas et al. 2015). The multidisciplinary collaboration was also key to reveal the different skills and roles required at different stages of the exhibition design process. This is a further point in which our research differs: previous work is based on the idea that a single user is responsible for the editing and creates the visiting experience anew, while the reality is that a team is responsible for the exhibition and there is a strong wish to reuse and repurpose one’s own exhibitions or to follow best practice created by others. Finally, instead of choosing between ease of use or powerful representations, we embraced the challenge to create an authoring environment that is, simultaneously, simple and powerful: simple enough for everyone to use while powered by a flexible formalism that enables its application across a wide range of scenarios of tangible interaction. This study also contributes a new approach that is based on principles of reusability, modularity and personalisation control for the hands of CHPs. The complexity of representing digitally augmented experiences is broken down into separate steps of: defining the relevant dimensions of the context; editing the digital and the physical ingredients of the experience and annotating them with aspects of the context; writing the rules that monitor the context and control the delivery of content. Moreover, we report an extensive validation conducted with a large cohort of prospective users in different settings, this one a step rarely reported in the literature with the exception of Roussou et al. (2015).

3 Understanding exhibition design

The authoring environment and its underlining formalism for personalised tangible interactions illustrated in this paper was created and evaluated with a collective effort as part of the meSch project⁵ that addressed the challenges of creating personally meaningful, sensory rich and socially expanded visitor experiences through tangible and embodied interaction with digital content (Petrelli et al. 2013). The project employed an iterative, user-centred design approach and developed the environment through cycles of design, assessment and redesign in continuous consultation with an extended group of about fifty CHPs involved at different stages of the project and with differing degrees of commitment (Risseeuw et al. 2016). The outcome of these extended and articulated user studies is summarised in this section where we first look at the different roles and skills required in designing an exhibition; we then discuss the meaning of personalised augmented experiences as intended by the CHPs; and finally we unpack the ideal design process as it emerged from our collaborative studies.

3.1 Skills and roles

The first step was to understand the process of designing an exhibition, the expertise needed, and the roles involved. This requires a wider investigation than the study of the process of designing multimedia presentations for mobile museum guides (Katz et al. 2006), as we are addressing scenarios where the digital and the physical aspects of exhibition intertwine. Via interviews, observations, creative and hands-on workshops meSch researchers identified roles, skills, needs, expectations and ambitions in introducing interactive technology to museums (Maye et al. 2014; McDermott et al. 2014). The findings clearly show that multiple experts are needed as the creation of an exhibition is a collaborative effort often partially outsourced. Key roles⁶ are that of the curator of the exhibition, who decides what content to include, and that of the communicator, who prepares the content for the audience. Two other key figures are the designer and the maker in charge, respectively, of the layout of the exhibition (both graphical and physical) and its final implementation (both fabrication and installation). These latter roles require expertise in architecture and design, sometimes theatre and video making for audio-visual material, and traditional making such as carpentry for the final setup. Even large museums tend to create mixed teams with the more content-related roles covered by the institution and the more practical side (architecture and graphics) outsourced to contracted partners. For the purpose of this paper, we call these collective roles Cultural Heritage Professionals, irrespective of them belonging to the heritage institution or the contracted exhibition design company.

The CHPs involved in the meSch investigation expressed diverse views on what they felt the purpose of technology was, and gave a variety of reasons and intended outcomes for including interactive digital technologies in exhibitions, such as to attract a younger audience or more complex goals linked to the potential for educational, engagement and participatory purposes (Maye et al. 2014). Concerns about the lack of technical skills and a fear of facing overwhelmingly complex technology were quickly voiced by CHPs when discussing their involvement in the design of novel experiences. At the same time, there was much excitement and a strong willingness to experiment with a platform to create personalised interactive installations. Of particular interest, was the possibility of modifying and updating the content over time, thus allowing the correction of mistakes but also providing the opportunity to add further interpretations and/or to target additional types of visitors. The vision of the continuous editing of the interactive installations opens up new possibilities for collaboration across a group of museums. Indeed, sharing and designing exhibitions collaboratively via a project consortium is an increasingly common practice amongst science and technology museums (see (Ruiz 2012) for some examples). These types of ‘shared’ exhibitions come about in two ways: either an exhibition is designed and developed by a single museum and then repackaged for use at different museums or a specific exhibition is collectively designed and developed by a consortium of museum partners. In this latter case, content specific to the museums will be added when the touring exhibition is hosted.

A survey of the CHPs (eCultValue 2013) highlighted their desire to embrace technology, but at the same time reservations towards overly technical solutions that could result in an expensive, unreliable outcome with little return on investment. To investigate the potential of providing support to CHPs in creating complex narratives, during one of our co-design sessions we put forward for discussion the idea of an intelligent system able to automatically generate stories starting from an annotated knowledge base and visitor’s logs. It was clear that the issue of authenticity is a primary concern for a CHP and this has an impact on the types of narratives they are willing to accept as means of communicating content to visitors. This is why the idea of a completely automatic intelligent storytelling system was strongly opposed by the curators and communicators who want to keep for themselves the control of the content, the construction of the possible narrative threads and how the narrative is personalised. Their stance is that they want to be in charge of the stories told by the interactive system in the same way as they are in charge of the information panels around the exhibition. This strong opposition to an intelligent system should not be interpreted as a rejection of technology. Although the background of curators and communicators is generally in the humanities, they have adapted well to the use of digital library tools, including the update of online catalogues. This suggests that CHPs can embrace new technologies and concepts of interaction when these fit with their working practices and goals.

In summary, the design and development of an exhibition is a team effort carried out by members that have very different roles and skills. What is consistent across all our studies is how the team is organised, with expertise on the content coming from the cultural institution and more professional services from outside it. As they all work on the same goal, a system that supports the design of interactive installations should be designed for a team consisting of members with different expertise and technical skills that will collaborate on the same project from different locations, at different points in time and for different purposes. Moreover, the responsibility of personalisation is split: personalisation of the content stays with the curatorial team while the adaptation to the context stays with the technical team. Taking advantage of the power of digital technology, an interactive exhibition can be continually updated and more content can be added, thus providing the curatorial team with the possibility of creating more personalised experiences over time while reusing the same technology for the visitor’s interaction.

3.2 Imagining personalised tangible and embodied interactions

The meSch authoring environment was designed on the basis of the vision of a group of CHPs. In a “reverse engineering” fashion, we started from the type of installations they wanted to build in order to design and then implement the platform on which to make them. In a series of creative workshops in collaboration with interaction designers and technologists, CHPs were invited to envisage what digitally augmented experiences could look like in their own exhibition spaces (McDermott et al. 2014; Ciolfi et al. 2016). Dozens of interaction scenarios were generated to give more substance to ideas and to help identify requirements and constraints. A few of those concepts were later implemented as exploratory prototypes to have a sense of what tangible and embodied interactions feel like when used, e.g. a digitally augmented book or a belt to deliver narratives in outdoor historical sites (Petrelli et al. 2016a), an augmented plinth to display digital information around exhibit objects (Wolf et al. 2015), a loupe to provide information details about framed exhibits (van der Vaart and Damala 2015). Through this exercise of envisaging and prototyping, we developed an understanding of what makes an interactive experience successful, what are the ambitions of the CHPs, and which criteria hold across heritage settings:

• Complement what is on display instead of competing for the visitors’ attention as it happens with digital devices such as mobile phones (vom Lehn and Heath 2003). Tangible interaction offers new encounters with the heritage.

• Active visiting should be fostered by interactive technology: crafted content and storytelling can shift visitors’ attitude from passive ‘consumers’ of information to ‘seekers’ of meaning via thought provoking and explorative interactions with the space and/or objects.

• A social experience is what visitors expect to have. Digital technology for heritage is often designed for individual use, e.g. apps for the phone (Aoki et al. 2002), and hamper the social experience of a group visiting together, this being the most common scenario (Falk 2009). Particularly when tangible and embodied interactions are used, aspects of social interaction have to be considered (Hornecker 2010).

• Offer choices because visitors have different interests, visit in different ways, and have different aims. Interactive technology should enable the visitors to appropriate the place (e.g. to follow individual visiting paths) and the content (e.g. to choose a theme or follow a character) as they like.

• Exploit the novelty technology brings to create unexpected and memorable experiences. For example, sensors and actuators can create surprise and a sense of magic by automatically starting the delivery of content (Petrelli et al. 2016a).

• Personalisation at different points and in different ways was the vision of curators that put forward ideas that were very different from those explored and implemented by the scientific community thus far. An interaction design that evokes emotions and personal connection is essential to engage visitors in a meaningful way (Not and Petrelli 2018).

The creation of an interactive exhibition requires multiple different decisions: the overall type of visit CHPs want their audience to have (e.g., reflective, self-directed, fed by multiple, contradictory perspectives); the type of content and when it is presented in the visit (e.g., excerpts from historical documents or personal accounts that might engage visitors emotionally); what objects will be handled and what visitors will do with them (e.g., to use them to activate/stop presentations); how visitors can express their preference (e.g., by choosing among different options); how to interpret the visitors’ actions (e.g., which events are meaningful); what forms of automatic personalisation can be offered (e.g., the tailoring of contents according to interest or the generation of personalised summaries of the visit). In addition to achieve such challenging exhibition design objectives, we wanted our editing environment to be simple to use as to induce confidence in CHPs in managing the new technology and sustain use over time. We started the design of such a platform by first unfolding the flow of the authoring task together with potential users.

3.3 Envisaging the authoring process

To model the authoring task on a plausible process, a co-design workshop with 4 curators, 6 interaction designers, and 6 technology experts was organised. Participants were divided into three groups of mixed expertise. Each group received a different task for which they had to imagine the creation process and make it explicit in a walkthrough scenario. The three tasks were complementary: (1) a novice user of the authoring environment who has to create a new digitally augmented exhibition; (2) an experienced user; and (3) a situation in which the process is driven by reusing a technology already in the museum. During this phase, we also discussed what terminology would be more suitable as to avoid technical jargon while conveying the key concepts. The recipe metaphor emerged at this stage to indicate a template for an interactive experience that needs a set of ingredients (e.g., multimedia content and devices) and a specific procedure to assemble them to achieve a given result (the interaction rules that control the system behaviour) (Zancanaro et al. 2015). The analysis of the storyboards created by the participants resulted in 7 steps, described below, that we used to define the authoring environment:

• 1. Inspiration Heritage institutions feel the pressure to offer “something digital” as the public now expect it. While they seek to offer bespoke experiences, only a few institutions are willing to take the risk that comes with groundbreaking installations. Often they take inspiration from what others have done. Therefore a collection of best practice examples to browse through allows CHPs to understand the requirements (e.g. devices and content), the costs and the complexity in implementing a visitor’s experience.

• 2. Set-up It should be possible to copy and modify an inspiring recipe. A recipe is a template of a visiting experience that is made specific for a given heritage setting: in repurposing an existing recipe it should be possible to change—for example—the languages or the target audience (adults vs. children), the points of interests or the objects. If a new recipe is created, the set-up phase is when the system behaviour (i.e. the interaction rules that control how content is delivered in context) is specified.

• 3. Create When the overall experience has been specified via the recipe template, the content is uploaded and tagged to allow referencing by the interaction rules.

• 4. Configure Tangible and embodied interaction use Internet of Things (IoT) technology, such as sensors and actuators that need to be assembled. In the configuration step authors bring in the needed technology tagging it according to the rules in the recipe (specified in the set-up phase) that control the interaction.

• 5. Test “To test” means to deploy the content and the rules on the hardware configuration in order to perform the visitor’s actions and check the response of the final installation is as expected when a visitor interacts with it.

• 6. Logging is enabled by the interconnected elements within the IoT: when deployed, the installation collects statistics that can be exploited by the museum to adapt the online follow-up of the visit (Petrelli et al. 2016b). Log statistics is useful also, both, to monitor how the installation is used, and to perfect it—if needed—by redoing steps 3–5.

• 7. Sharing the knowledge with the community captures the step in which a recipe is shared to enable other institutions to reuse and repurpose and so sharing feeds step 1 (above).

This process allows the reuse and repurpose of both the technology and content a museum may have invested in as well as the expertise of the community. Looking at what others have done doubles as a learning experience for the CHPs: starting by finding inspiration from others and replicating their work, CHPs can then be guided through the process of modifying the existing recipe, thus building new skills that can facilitate the adoption of this sophisticated tool. However, the process above highlights how the CHPs rely on the behaviour of the interactive installation to already be defined: by reusing a recipe CHPs reuse the core mechanisms of the interactivity. How to enable non-technical users to create a new behaviour that implements personalised experiences is a well-known challenge (Ardissono et al. 2012) that our workshop confirmed. Therefore, which tools could be proposed to non-technical CHPs was explored more in depth as part of our user-centred approach.

3.4 Defining the system behaviour: ease-of-use vs expressiveness

The co-design of the authoring process made evident how CHPs are uneasy with tasks that feel like requiring technical knowledge. While they were confident in creating a vision for a new visiting experience and new content to go with it, the behavioural mechanisms of the interaction were perceived as beyond their skills. We were therefore fully aware of the trade-off existing between a simple interaction and powerful personalisation mechanisms: an editing tool to create personalised behaviour that is accessible to not-technical users may need to have reduced functionalities; at the opposite end, an editing tool that exploits a flexible and powerful representation may be handled only by highly skilled developers. The critical point is to find the level of complexity CHPs can handle while offering the highest possible flexibility in defining personalised experiences; this was investigated in a specific comparative study.

At first, we imagined multiple editing views to fit the author’s level of technical confidence. The easiest setting was menu-based to guide the user in the selection of the condition (e.g., “the visitor is close by”) to control the selected content item (“text one”) to be presented by means of a certain device (a “beamer”) on a determined output location (“wall”). A paraphrase in natural language of this menu selection would allow the author to check the result is what intended: “Text one is presented with a beamer when the visitor is close by. The visitor will see the projection on the wall”, with a feedback approach similar to (Ghiani et al. 2017).

Although very intuitive and likely to be accepted by curators and communicators, this hypothesis was soon abandoned as too limiting with respect to our ambition of a flexible and powerful platform: a pre-defined menu-based interface does not scale well to more complex scenarios where the activation conditions depend on multiple visitor preferences, or on the behavior of a group of visitors, or on the state of several objects. To be able to combine multiple conditions was essential for the major objective of the project: to open up opportunities to use personalization in as many contexts as possible.

In an effort to harmonise flexibility and ease-of-use, we investigated the option to represent conditions and actions via visual blocks that can be put in sequence and nested to model complex contextual conditions. A block-based approach had proved successful for novices learning to program (Resnick et al. 2009), thus we considered an extension of the Google Blockly library to abstract an event-based language to model visitor interactions (Stratton et al. 2017). We then sought the feedback of CHPs and asked them to comment on different options: the block interface (Fig. 1, left); a pseudo-algorithm, written for the experiment (Fig. 1, right); and a Javascript editor to code any behaviour imagined (Fig. 17).

Fig. 1

A set of interaction rules expressed with the block formalism (left) and a sample pseudo-algorithm description of the interactive behaviour in textual form (right)

Four CHPs external to the project and knowledgeable in designing interactive exhibitions took part in this comparative study: two curators, a communicator (with experience of HTML coding), and an interaction designer (with experience with Arduino programming). In individual sessions, each of the four CHPs was asked whether they would create new interactive behaviours and, if so, which interface they would favour. Only the participant with HTML experience was positive about using the block interface but expressed concerns on the variety of interactions that a limited number of predefined blocks can generate and the need for a dedicated programmer familiar with the block framework if a new one had to be added. Overall, we found that a block interface would introduce complexity many CHPs would not be willing to face while reducing the expressiveness for those CHPs that can create new behaviours by coding. We concluded there is not enough evidence a simplified graphical representation of the behaviour would be used by curators and communicators and therefore we preferred to design a tool that allowed different expertise to collaborate on the same project.

As part of this study, we also questioned our participants about the possibility of reusing recipes previously used by the same museum or by others. In this way we were able to validate the outcome of the co-design workshop reported in Sect. 3.3 above, with CHPs external to the project. One of the four participants commented that, by reusing a recipe, there would be no need for CHPs to create new behaviours as repurposing someone else’s exhibition gave confidence the solution worked. By changing the content and possibly by adapting and tweaking the interaction, e.g. recombining the same technology in different ways, the museum creates a bespoke solution and therefore achieves innovation as the visitors’ experience is bespoke to the place. These findings once again confirm the usefulness of the authoring functionalities for reusing and repurposing experiences which have already been developed, deployed and tested.

4 Representing personalised and tangible visitor’s experiences

In the previous section, we have explained how our understanding of the exhibition design process shaped the overall functional requirements for an authoring environment supporting CHPs in taking up tangible interactive technology for their museum institutions. The next step is to understand how to formally represent a tangible experience in a way that captures all the aspects related to content, context and interaction effectively, and how to render elements of personalisation that can be controlled by authors. The definition of a formalism that makes systematic the expression of all the experience ingredients is required before designing an authoring environment to support the CHPs in their manipulation.

4.1 Taking personalisation of tangibles within the heritage context

Before illustrating the formalism, we describe here in depth one of the visiting experiences that was implemented in meSch. This installation was deployed as part of the temporary exhibition The Hague and the Atlantic Wall: War in the City of Peace held at MUSEON⁷ (The Hague, The Netherlands) and attracted about 40,000 visitors⁸ over 6 months (May–October 2015) (Marshall et al. 2016a).⁹ This interactive exhibition is used throughout the remaining of the paper to connect the technical aspects of the formalism to the graphical user interface solutions adopted in the authoring environment.

In planning this exhibition, the curators aimed to offer an exploration of the impact of the Atlantic Wall structure on the city of The Hague and the people living there, and what it meant for the identity of the city after the war. A multi-party exhibition design process took place: the curatorial team provided the vision, the objects and the interpretation; an exhibition design firm looked after the overall layout, the information panels and the displays; audio-visual experts created specific content (i.e. a 8-min film, quiz and video-on-demand, wall-size photo display); the meSch research team designed and built eleven interactive stations, ten displayed multimedia content and one printed a personalised postcard. The exhibition was implemented as an open plan with eleven distinct areas mapping the different quarters of the city affected by the Wall. Like on a city map, white lines on the floor represented the streets connecting stations/quarters (Fig. 2a); each station focussed on a topic related to the quarter, i.e. how the beach became a mine field, why the Peace Quarter was built there. Objects on display, labels and panels provided specific content.

Fig. 2

The Atlantic Wall: the exhibition is designed as a map of the city of Den Haag (a); visitors look at and choose the replicas to use during the visit (b); the interactive station reacts to a smart replica (c); the printing of the postcard (d)

The meSch team designed the interactive experience to present different perspectives of the events to the public, as multiple stories could be told without using additional exhibition space. Ten interactive stations offered the personal perspectives of those involved: how the German soldiers believed they were defending the Dutch population from the Allies; how the Dutch civilians were displaced, resisted and resented the occupation; and how the civil servants dealt with doing the bidding of the German occupiers. The German soldier, the Dutch civilian and the Civil servant were the three complementary and contrasting personal voices of the exhibition, each represented by a replica of an original object on display. Two sets of smart replicas were made, three for the Dutch language and three for the English language. The crafted replicas were augmented with an NFC tag and NFC readers were embedded in the interactive stations and controlled the projection of multimedia content.

At the entrance (Fig. 2b), visitors chose a replica to be used during the visit thus determining their preferred language and the perspective they wanted to follow. When at one of the ten interactive cases, visitors placed the replica on a glowing spot (that covers an NFC reader) to play the multimedia for that station in the language and from the perspective represented by the replica (c). Sound played in the earpiece and a carousel of historical photos or videos was projected onto the glass of the case. The system logged the visitor interactions to print a personalised postcard at a final checkout station (d) and to generate a personalised website for online exploration and visitors’ contribution of personal memories.¹⁰

The design and implementation of the Atlantic Wall exhibition was co-designed and led by MUSEON. The team and the followed process maps precisely what discussed in Sect. 3 in terms of: (1) dispersed and heterogeneous expertise; (2) type of interactive experience (non-didactical, challenging and personal); (3) choice and engagement (via smart replicas).

4.2 Defining personalisation forms for tangible and embodied interaction

The Atlantic Wall exhibition shows that, in following a co-design approach with CHPs, we had to rethink personalisation: it becomes a blend of customisation, context-awareness and adaptivity that affects both the digital and material aspects of the experience. A broad personalisation perspective needs to include elements that (1) allow visitors to have a say on what they like, what is engaging for them and what they expect from today’s visit, (2) automatically sense what is going on and activate the system at proper times to accommodate visitors’ behaviour, (3) possibly adjust the presented information and environmental stimuli to a model of the visitor and of the interaction that is dynamically built. The importance of all three forms of experience-tailoring clearly emerged from a specific co-design workshop in which 10 curators, 7 designers and 8 computer scientists investigated what personalization means for CHPs and how this relates to the state of the art in the literature (Not and Petrelli 2018). In particular, the novelty introduced by the tangible aspects of the interaction opens up new opportunities for customisation and context-awareness as visitors are granted new means to shape the individual unfolding of their experience. For example, choosing a smart replica that holds one of the many stories prepared by curators is a way to express a preference on what to hear (customisation); placing a smart replica on an interactive showcase and removing it is a way to tell the system to start and stop a related information presentation (context-awareness). Indeed, customisation and context-awareness enabled by a sensible interaction design allow the system to capture some aspects of the visitors’ motivations (Falk 2009) in a much more straightforward way than by second guessing them solely with automatic inference mechanisms (adaptivity) (Not and Petrelli 2018). In our expanded vision of personalisation, different factors are decided and instantiated at different times (during the design and authoring of the experience, during the runtime execution of the system rules, when visitors make choices) and by different actors (CHPs, the system, the visitor). More specifically:

Customisation defines alternatives (in content, devices etc.) that are available to the visitor to choose from. For content customisation, which options are available is decided in advance by CHPs that prepare a rich information space within which visitors can shape their personal visit on the bases of their motivations and expectations, their personal interests, preferences, background or time constrains. An example of customisation in the Atlantic Wall exhibition is the choice between the German soldier, the Dutch civilian or the Civil servant. Customisation can affect the content received (as for the Atlantic Wall) or the type of device used that delivers the same content (an interactive book or an interactive belt, Petrelli et al. 2016a; a smart replica or a mobile phone, Petrelli and O’Brien 2018). To implement customisation the underlying formalism must allow: (1) the specification of the alternatives and (2) the declaration of how the alternatives combine and what the outcome is. In the Atlantic Wall example, three themes and two languages are available for the visitor to choose from and the language-theme combinations are available at each point of interest: the system recognises the point of interest the visitors are currently at and the smart replica they are using, which maps their preferred theme and language.

Context-awareness is the automatic ability of the system to sense the state of the interaction and to respond accordingly. A system that senses the presence of the visitor or the position of the objects and activates or stops the play of content at appropriate time shows context-awareness that combines the physical and the digital. An example is a reactive space that responds in different ways depending on the visitor’s movements, playing a loud sound to attract the visitor and a story when they approach (Marshall et al. 2016b). The implementation of context-awareness requires: (1) a mapping between the contextual features monitored by the system, the devices sensing them and the types of possible events generated by the sensors and (2) if–then–else rules that check the current sensed context and fire the corresponding system behaviour. In the Atlantic Wall exhibition, the NFC tags in the smart replicas and the NFC readers in the interactive stations are the devices that recognise users’ actions. If–then–else rules monitor the events that may determine the play and stop of the content according to what is sensed by the devices and update a representation of the context accordingly.

Adaptivity is the more sophisticated form of personalisation; it requires the dynamic modelling of the ongoing interaction and the dynamic changing of the system behaviour to adapt to the changing situation. Examples of adaptivity include the use of the history of the actions and contents consumed by a visitor to derive presumed interest to select additional content or to dynamically generate summaries of the visit (Not et al. 2017). Adaptivity requires more complex behaviour rules. In the Atlantic Wall exhibition, adaptivity rules use the visiting history (content start and stop, sequence and length of interactions) to generate a personalised postcard and an online personal page with a visit summary and multiple activities based on this model (Petrelli et al. 2016b). In a different scenario that was implemented with similar hardware (interactive stations with an NFC reader and three buttons) adaptivity rules were used to implement a personalised quiz on natural history topics: thematic cards placed on the interactive station represent multiple-answer questions to which users can reply via buttons. Users responding incorrectly to many questions are modelled by the system as less proficient and are presented with optional review information to reinforce learning.

With meSch we aimed at delivering a flexible system able to handle all possible combinations of these three personalisation forms leaving the author in control of which combination is more suitable when. To achieve this vision, we first had to devise a formalism that was both powerful and flexible to model combinations of personalisation forms (described below in 4.3) and to design a user interface easy enough to be used by CHPs with confidence and minimal training (discussed in 5).

4.3 A formalism to represent content, context and interaction

The Experience Schema¹¹ formalism has four components: (1) the narrative: a set of curated digital content items, annotated with semantic dimensions that express the context in which they can be used, i.e. whether these items refer to a specific point of interest (e.g. an object, a place), belong to specific thematic narrative threads, elaboration details, optional parts, or are suitable for certain categories of audience or language; (2) the appliance: a declarative specification of the capabilities of the technology embedded in the museum premises such as detecting presence, proximity, or object handling; (3) the device: the hardware, possibly a bespoke assembly of electronics, to implement the appliance and deploy the experience (e.g. ultrasound to detect proximity); (4) the interaction script: the rules to control the delivery of the content in context such as when a presentation should start or stop or how to automatically adjust the experience to visitors’ behaviour (e.g., to project the most appropriate content about an exhibit only when visitors are close by).

This deconstruction of the interactive experience into components is motivated by several needs. (1) As emerged from the co-design work described in Sect. 3, CHPs with different expertise want to focus on certain parts of the experience and ignore others (e.g. curators and communicators on the content, designers and makers on the preparation of the interaction and the form factor). A more abstract interaction (the appliance) could be easier for the CPHs to grasp without the need to understand the many technical components needed (the hardware devices). (2) We also wanted to be independent from the hardware as to be “future proof”, so that if a better sensor comes along we can swap the old device for the new and leave the rest untouched. (3) The separation between the content and the logic for its delivery in context enables us to control different combinations of personalisation types, as we can clearly specify the contextual dimensions used for adaptivity or customisation (different semantic dimensions), the events that control context-awareness (declared in the appliance), and the logics for personalisation instantiation at runtime (in the interaction script). (4) A modular representation of the experience also favours the reuse and recombination of the single ingredients to create novel experiences, facilitating, for example, the combination of the same content with multiple types of interactions and vice versa.

We will now explain in more detail the meaning of the different parts of the Experience Schema.

4.3.1 Narrative: the content and its context

The narrative in the experience schema captures both the content and some aspects of the context in which the content will be delivered. The context is modelled via features that affect which piece of content is chosen at presentation time (such as the language) and a number of valid configurations of the context (e.g. the language could be English or Dutch). At editing time, the CHPs decide which content will be delivered in which context and prepare the narrative accordingly. The three elements that compose a narrative are more formally represented with a triple n = (D, V, C), where:

• D is a set of semantic dimensions D = {d₁,…,d_n} that represent multiple contextual conditions (or features) that may influence how the content is selected (e.g. the type of audience, the preferred language of the user). The set of dimensions depends on the communicative goals of the CHPs, for example to offer different content to different visitors groups, to serve an international audience, or to map specific content to artefacts or locations in the museum.

• The set V = {V₁,…,V_n} collects the values for each semantic dimension, where V_i= {v₁,..,v_k} are possible symbolic values for the dimension d_i. As an example, V_i= {english, german, dutch} is the set of values for the semantic dimension d_i= language.

• C is a set of content elements in the form c = (item, context, rels). The item lists the properties of the content used for its identification and rendering, specifically item = (media_type, length, title, description, file_url). Then context specifies the configuration in which the item is to be delivered; it is a set of tuples in the form context = {(d_i,w_i) | d_i∈ D, w_i∈ P(V_i)} where d_i, is a semantic dimension, w_i is one or more of its possible values and P is the symbol for the power set. In other words, context lists the values for the semantic dimensions for which c is applicable; for example the context = {(language, {english}), (audience, {adult, child})} prescribes that a certain media item is appropriate when the preselected language is English and the visitor is classified as adult or child. The last part of the representation for c is optional and encodes the discourse relations with other content elements as rels = {(r_i, c) | r_i∈ R, c ∈ C} where R is a subset of the rhetorical relations defined in (Mann and Thompson 1987). Rhetorical relations are used to express, for example, that two content items should be presented in a certain “sequence” or that an item provides an “elaboration” of the previous content and are useful when composing presentations with sophisticated forms of adaptivity [see (Not et al. 2017) for an example of experience where rhetorical relations were used].

Figure 3 shows the formal representation of the narrative of the Atlantic Wall experience. The semantic dimensions D include the interactive stations (the “points-of-interest”), the perspectives of the German soldier, the Dutch civilian and the Civil servant (the “themes”), and the supported languages. The sample content items listed show a different content is planned for the same interactive station and the same language but a different theme.

Fig. 3

The formal representation of the narrative for the Atlantic Wall experience. This narrative structures content into multiple layers of information. Three semantic dimensions are used: for each point of interest corresponding to a multimedia interactive case (10 values), alternative content is prepared for each of theme (3 values) and for each output language (2 values), for a total of 60 content items

4.3.2 Appliance: the interaction abilities of the system

The appliance spells out how the system recognises interaction events, identifies the context configuration and actuates the output modalities. Our concept of appliance builds upon the notion of context awareness in ubiquitous computing by Greenberg et al. (2011) that defines proximity of entities (people, devices and things) in five dimensions: distance, orientation, movement, identity (i.e. the recognition of entities), and location. The data captured by sensors associated to these dimensions are aggregated into abstract events used to design and implement proxemic interactions through the programming of alternative scripts (e.g. to implement a media player surface that reacts to different movements and actions of users). In our case, context-awareness as defined by Greenberg et al. is expanded beyond proximity to include also events triggered by tangible means (e.g. the pressing of a button). The abstraction of the events from the sensor data coming from multiple devices is what makes the appliance: we declaratively specify the set of contextual dimensions as made available by the devices. Therefore each appliance is a tuple a = (events, output, actions, params) whose components have the following meaning:

• events is a set of elements in the form e = (t, d) where t∈ E specifies a type of event that the system is able to recognize and d = {d₁,.., d_k} ∈ P(D) is a subset of semantic dimensions whose value is instantiated by the event e. For example, e = (object_present, {point_of_interest, language, theme}) specifies that the system is able to recognize an action of placing an object at a certain point of interest and the language and theme preferences associated to that object; when this type of event occurs, the actual values of the semantic dimensions d_i become available. Other types of events in the catalogue E that we used are: proximity, the event that detects the presence (of a person or a device) within a specified distance from a centre; identification, the recognition of a certain object (for example via visual markers); choice, the display of a preference (for example via button pressing).

• output specifies the outputs supported by the system; for example an appliance with output = {video, sound, light, vibration} can play video material or sound as well as generate luminous and vibrotactile feedback. This information is useful to perform consistency checks that ensure the prepared content matches the presentation abilities of the appliance.

• actions is a set of elements in the form act = (t,d) where t ∈ Act specifies a type of action that the system is able to perform and d = {d₁,.., d_k} ∈ P(D) is a subset of semantic dimensions whose value is passed as a parameter for the correct execution of the action. For example, act = (play_content, {point_of_interest, language, theme}) specifies that the system is able to play a content element c when the context matches the actual values of the parameters. Other types of actions we used in the Atlantic Wall are: stop_content, to stop a presentation when the context changes; print_postcard, for the dynamic creation of the souvenir.

• params indicates a set of values that can be tuned to adjust the interaction experience. For example, the proximity event has distance range as parameter that controls when the user reaches a significant position.

Figure 4 shows the formal representation of the appliance of the Atlantic Wall experience, that is to say it is the abstract representation of an interactive case.

Fig. 4

The formal representation of the appliance for the Atlantic Wall experience. This appliance senses when users put down and take away objects at certain locations: these actions generate information about the point of interest associated to the location and the theme and language preference of the user. The system is capable of playing and stopping content and printing a summary of the visit

4.3.3 Device: the actual hardware

The device describes the combination of hardware elements that implements the appliance abilities and is represented by a couple dev = (components, mappings) where:

• components is a list of hardware devices with sensing or actuating elements that are able to support the recognition of the events and the execution of the actions described in the appliance. For example, components = (NFC_reader1, NFC_reader2, NFC_tag1, NFC_tag2, NFC_tag3, display1, projector1) indicates that a certain experience is deployed by means of two NFC readers and three NFC tags to recognise events, plus two different output devices to deliver content.

• mappings is a list of pairs in the form m = (comp, context) where comp is an element in components and context = {(d_i,w_i) | d_i∈ D, w_i∈ P(V_i)} describes a context configuration that is associated to the component. For example m₁=(NFC_reader6, {(point_of_interest, {C6_Harbour})}) associates an NFC reader with the sixth point of interest in the Atlantic Wall exhibition, whereas m₂=(NFC_tag1, {((language, {dutch}), (theme, {soldiers})}) associates the first NFC tag to the value pair dutch and soldiers for the semantic dimensions language and theme respectively. In other words, this declarative specification expresses the fact that the NFC_reader6 is installed at the interactive station of the Harbour and visitors who choose the beer-mug that conceals NFC_tag1 express a preference for the Dutch language and the German perspective.

Figure 5 shows the formal representation of the device of the Atlantic Wall experience. By keeping the device specification separate from that of the appliance, we reach a greater flexibility and reusability, as different devices may be experimented with to support the same appliance and the same type of experience.

Fig. 5

The formal representation of the device for the Atlantic Wall experience. Six different NFC tags are used to identify the objects that may be carried by users and each interactive station is equipped with an NFC reader. The mapping prescribes how the NFC tags are associated to the possible thematic and language choice made by the users and which NFC reader identifies which point of interest

4.3.4 Interaction script: the rules of the system’s behaviour

The interaction script is the part of the experience schema that encodes the rules that monitor the interaction input collected by the sensors; it also decides which actions the system should perform. The interaction script has two components:

• a library that implements the events and the actions declared in the appliance by means of the lower level capabilities of the hardware, e.g. an object_present event is recognised when an NFC tag is placed over one NFC reader and the values of the output parameters of object_present are computed by using the mappings declared in the device for the NFC tag and the NFC reader.

• a set of rules that implement the logic of the experience, e.g. “when an object_present event occurs and provides values for dimensions d₁, d₂, d₃, then execute play_content (v₁, v₂, v₃)”.

Figure 6 shows the formal representation of the interaction script of the Atlantic Wall experience.

Fig. 6

The formal representation of the interaction script for the Atlantic Wall experience. The rules prescribe that when visitors place an object over an interactive station, the content item associated to the current point of interest, theme and language is played. When the object is removed, the playing of the content stops. When the object is placed over the check-out station, the printing of a visit summary is activated

4.4 Assessing the flexibility and expressiveness of the formalism

The formalism for the Experience Schema is open and flexible: it does not prescribe the type of personalisation, visitors type, heritage or devices to include. It is the author who decides which semantic dimensions capture the case at hand, who defines the personalisation expressivity and makes an experience schema specific for their heritage, hardware configuration, or visitor profile (e.g. whether to consider just static features like the child/adult distinction or multiple levels of user interest). By isolating the narrative, the schema enables the curators to focus on the quality of the content and its annotation, deferring the editing of the rules or the hardware settings to another stage possibly managed by other people. Similarly, the declarations in the appliance specify the type of interactions and behaviour independently from the actual hardware details and the low level sensor logs, thus enabling experimentation with different technological solutions. For example, in the Atlantic Wall exhibition, the setting of the language and the theme could be done using two devices: the language could be set by visitors’ proximity while the smart replica then represents only the theme. At the entrance the visitor would receive a brooch embedding a Bluetooth beacon associated with their preferred language; in addition to the NFC-reader, the interactive case should be equipped with a Bluetooth detector to set the current language by proximity. The part of the experience schema that needs change is the description of the device and the library of code that implements the low level management of sensors.

More in general, the modularity of the experience schema formalism facilitates the reuse and repurpose of (or parts of) already deployed experiences. Multiple experience schemas can be composed by merging their elements to create more complex interactive experiences, for example to deploy interactive cases, audio narrations activated by proximity and post-visit souvenirs in one digitally augmented visit.

As part of the meSch project the research team deployed six installations used by over 20,000 visitors in three museums for a period of up to 6 months. A further fifteen installations have been created with partners outside the project or for the purpose of exploring the potential of personalisation in other domains such as retail and tourism (Cavada et al. 2018). The variety of applications allowed us to specify several forms of personalisation (Not and Petrelli 2018) and confirms the great potential of the experience schema formalism as a method to support the representation of a wide range of tangible, adaptive and interactive experiences for the Cultural Heritage domain. This extensive experimentation within and outside the project team allowed the identification of templates of narrative structures and templates of interaction rules that are often used in combination, or reused across different hardware setups. The types of narrative structures most used by meSch prototypes and case studies are:

• Multilayer content discovery This is the narrative strategy in the Atlantic Wall exhibition and was used in several prototypes, see Table 1. It has a set of content items each annotated with values of multiple semantic dimensions (e.g., point of interest, language, theme, type of audience), as shown in Fig. 3. The narrative is in the baseline form n = ({d₁,…,d_n},{V₁,…,V_n},C) where each content element c ∈ C, c = (item, context, {}), has a context that associates a value for each semantic dimension d_i and an empty set of rhetorical relations. The set C of content elements may be complete, i.e. with one content item for each single combination of possible values of the semantic dimensions, or incomplete, i.e. when some combinations of features are not used, for example if one or more themes are not available for all of the Points of Interest.

  Table 1 The modularity of the schema formalism facilitates the recombination of schema elements to create new experiences

• In-depth content discovery The multilayer content structure above is enriched with additional content items that provide further information to deepen the discourse in certain contextual conditions. More formally, this can be expressed in the narrative with some content elements c₁∈ C, c₁= (item, context, {(elaboration, c₂)}), where c₂ contains information that elaborates on the contents in c₁.

• Sequence of steps Used to deliver a quest-like experience, this narrative is composed by a set of content items C = {c₁, c₂,…, c_k} annotated to specify a logical order of presentation through the “sequence” rhetorical relation and providing a clue for the next point of interest to find in the exhibition space. For example, the representations c₁= (<presentation for object1>, {(point_of_interest, {POI1})}, {(sequence, c₂)}) and c₂= (<image of object2>, {(point_of_interest, {POI1})}, {}) describe that the content element c₁ contains information suitable for presentation at point of interest POI1 and can be followed by content element c₂ that provides a clue for finding the second object in the visiting path.

• Q&A This narrative strategy creates presentations in the form of dialogues based on pairs of questions and answers. Q&A is a popular way to deliver content in science museums, for school and family audience. For each point of interest, one content item maps the question and another maps the answer, e.g. c₁=(<“What is Heracles hunting for?”>, {(point_of_interest, {POI1}), (type_of_content, question)}, {}) and c₂=(<“First, he had to kill the Nemean lion…..”>, {(point_of_interest, {POI1}), (type_of_content, answer)}, {}). Whether the two items are presented together or in automatic sequence or after a user action (e.g., the visitor pressing a button) depends on the specific device and interaction chosen.

• Adaptive storytelling This is an articulated narrative template, composed by a set of content items associated to story pieces. Semantic annotations describe the first layer of applicability conditions for the story pieces (e.g. the output language or the audience type) while rhetorical relations define the main temporal succession of content items and elaboration annotations indicate optional details. This narrative structure creates articulated stories with possible in-depth paths and alternative branches, the delivery of which depends on the visitors’ profile or their behaviour during the visit. For example, the following representations are three content elements of a narrative used to compose personalized souvenir postcards for an exhibition on a WWI fortress (Not et al. 2017):

  c₁ = (“In the Artillery section of Museo della Guerra you have heard the stories of various people who have worked or lived at Forte Pozzacchio, the last Austro-Hungarian fortress.”, {(language, {EN}), (content_type, {Introduction})}, {(sequence, c₂), (sequence, c₄), (sequence, c₅)})

  c₂ = (“Close to a cannon, Albino, Mario and Amalia have told you how the fort and the war changed the life of civilians and the economy of the valley.”, {(language, {EN}), (content_type, {Theme_description}), (theme, {Civilians})}, {elaboration, c₃})

  c₃ = (“For example you have heard the story of Mario Rippa: ≪I left the first day of August 1914. We were on the mountains cutting the hay, …≫” {(language, {EN}), (content_type, {Example}), (theme, {Civilians}), (point_of_interest, {Mario_Rippa})}, {})

  c₁ represents the introductory sentence preceding the description of what visitors have experienced at each interactive station. c₂ is relevant only for visitors who have used the interactive station recounting stories of civilians during WWI. The third content element is optional and describes one of the stories played onsite that may be included in longer visit summaries.

The variety of content structures above was used in conjunction with a range of different types of interaction strategies, for example:

• Free exploration and selection implements a visit where visitors are free to choose what’s next (as opposed to follow a trail), as was in the Atlantic Wall exhibition (Fig. 6). Tested on both indoors and outdoors, visitors were able to express preferences on the type of content (customisation) and controlled the presentations (start/stop) while the system reacted to user actions or movements (context-awareness). This simple interaction was implemented in several installations (Marshall et al. 2016a; Not et al. 2017).¹² Script variations included the context-awareness with respect to different proximity distances to play sounds or a story depending on the distance from a point of interest (Ciolfi et al. 2013; Marshall et al. 2016b).¹³ A straightforward extension is to include adaptivity rules to condition the play of a content item to what the visitor has already heard, for example to avoid repetitions, to include comparisons or to adjust the amount of details according to interest (see in Fig. 7 a sample extension of the Atlantic Wall interaction script).

  Fig. 7

  

  The pseudo-code of a script implementing the “Free exploration & selection” interaction strategy including a rule for adaptivity. The script prescribes that when visitors place an object over an interactive station, the content item associated to the current point of interest, theme and language is played. A history of the interaction is maintained so that when a visitor comes to the same point of interest a second time, a content item providing further details is presented

• Guided visit helps visitors to orient themselves throughout an exhibition. It has behaviour rules that pace the visitor’s experience as the visit progresses: visitors are given a suggestion for the next object to see and, when at a point of interest, receive the correct content (context-awareness) while more information is presented according to interest (customisation on visitors’ choice or automatic adaptivity), as in the interaction script in Fig. 8. This interaction was implemented in an Augmented Reality trail prototype and tested with the visitors of an archaeology museum (Van der Vaart and Damala 2015).¹⁴

  Fig. 8

  

  The pseudo-code of a script implementing the “Guided visit” interaction strategy. When an event is detected and a certain point of interest has been identified, a feedback content on the reached location is played. More information is shown up on user request. The system then shows the next point of interest in the trail

• Story generation Interaction scripts of this type aim at generating an articulated story which is personalized according to the visit history and user preferences (adaptivity) and is delivered in its entirety to the visitor at a certain point of the visit, or after the visit. The rules in the script traverse the content network evaluating the conditions at optional nodes or alternative branches of the story against the contextual model built up to that point. This interaction strategy was experimented in conjunction with the storytelling narrative strategy described above (Adaptive Storytelling) on the generation of personalized souvenir postcards based on the choices made by the visitors at different points in the exhibition and the many stories they listen to. Figure 9 shows sample rules that assemble the text in an adaptive process. First the content network is traversed to assemble an introductory text for the postcard. Then, for each interactive station used by the visitor, a sentence is included summarising the associated theme, in the same order of station usage (adaptivity with respect to interaction history). For stations where the visitor heard more than three stories, one of the stories is included in the postcard (adaptivity with respect to interest). By changing the rules in the script, experiments were made with different text genres and degrees of verbosity, with the possibility of producing personalized online summary pages as described in detail in Not et al. (2017).

  Fig. 9

  

  The pseudo-code of a script in the category “Story generation”. Different content nodes are selected and properly ordered according to the interaction history and interest of the user

The list of strategies above results from the classification of the many developed installations that implement a variety of narratives and interaction modes. It shows that very different experiences can be modelled with the same formalism demonstrating the flexibility of the personalisation approach and representation formalism we defined. Table 1 shows a selection of prototypes that share some elements in the schema: by changing the narrative strategy, the interaction strategy, the device, or simply the form factor, very different digitally augmented experiences were implemented.

5 The meSch authoring environment

The previous sections describe in some detail our understanding of the potential users of the meSch authoring environment (Sect. 3) and the formalism that supports an open design and development process (Sect. 4). To summarise:

• The team involved in the design and installation of exhibitions is not homogeneous; members have different backgrounds and belong to different institutions; each member has a specific expertise, role and set of skills; even the less technical members of the team are willing to use a digital tool if this is designed to match their expertise and role and requires limited learning. The expertise split is between content (curators and communicators) and interaction (designers and makers). We therefore defined a formalism that distinguishes the creation of the content (the narrative and the related semantic dimensions) from the interactive installation (the appliance, the devices and the interaction script).

• The tangible and embodied experiences should invite the visitors to be active (manipulating, moving), take decisions and do things. The content should be thought provoking; the narratives structure should be multi-layered and flexible as the stories can be related to objects on display but also to abstract themes or targeted to certain visitor types, e.g. families, schools. The technology has to be thought of at the same time as the experience is being designed, as it affects the visit. This calls for a very flexible tool where every installation can be designed independently to respond to specific needs. The formalism allowed us to implement an open system where it is the user who decides which semantic dimensions are meaningful for the case in hand. This gives compete freedom to the team to design the most suitable experience for their specific heritage and visitor group(s).

• The creation process alternates online and offline, individual and teamwork activities. The authoring environment should accommodate as much as possible the way the specific team works, different expertise and roles; it should leave the curatorial team in full control of the content. It should allow them to reuse and repurpose the elements from past exhibitions such as content, hardware and interactions. It should allow testing and fast prototyping well before installation and continuous updates after it. The elements in the formalism enable the reuse of each component independently as well as fast prototyping.

In designing the meSch authoring environment, we took into account the user needs for a simple interaction and our desire for a powerful new formalism that gives the author complete freedom in defining personalised interactions. As the design was holistic, meaning that we considered the CHPs and the system at the same time, it is impossible to completely separate the user interface from the underlying formalism. In this section, we first use the graphical user interface to discuss how we integrated elements of the experience schema formalism with the process of designing interactive installations (as discussed in 3.3). We then report on the user evaluations the research team carried out, both a formative usability test with partner CHPs and a summative evaluation open to the cultural heritage community at large.

5.1 A user interface for reuse, experience editing and deployment

5.1.1 Finding inspiration

We use the metaphor of a ‘magazine’ of ‘recipes’ to show a catalogue of installations curators can browse through to find implementations that use a specific hardware, are by a similar heritage or were used for a similar audience (Fig. 10). The magazine collects the recipes that have been shared. The preview of a recipe lists the ingredients (the required technology), information on what it does and the target audience (e.g. a quest for children vs. a free visit for all). Simple recipes with tutorials are intended to invite experimentation and build confidence. The magazine holds the recipes shared by the community and therefore could have hundreds of different recipes designed for different types of heritage, different visitors groups, or different experiences. A user browsing through the recipes (the CHP editing the recipe) is then supported by traditional search filters and by recommendation algorithms (Fig. 10, left) that reorder the recipes in the magazine, for example by selecting a set of diversified recipes for a richer inspiration or by placing recipes from organizations with similar profiles on the top. The user profile is filled in at registration time and is used at different recommendation points, as discussed below, to assist the author in choosing a recipe and composing its content. The profile includes descriptions of the institution and the type of managed exhibitions such as:

Fig. 10

The catalogue of installations (magazine of recipes) (left) and the information page of a simple recipe for practice (right)

• Indoor versus outdoor—the outdoor having issues of power supply, lack of Wi-Fi connection, wet weather, vandalism, etc. and therefore being more restrictive on suitable hardware or devices.

• Size¹⁵—small, medium or large: size matters as small heritage and museums may be seen in full in one visit while medium and large would need multiple visits to be covered in full.

• Permanent collection versus temporary exhibition to distinguish presenting a few star objects from the collection or to organise a full exhibition on a theme proposed by a curator.

• Type of heritage institution impacts on the way visitors behave and what they expect, e.g. children and their families make science museums noisy, living history museums offer opportunities for enactments and live storytelling, visitors of art galleries usually expect peace and quiet—heritage type can be used to filter in/out media types, e.g. to avoid sound in art galleries but use it in living history museums. Moreover, our co-design work showed that science museums have a greater degree of in-house technical expertise in comparison to historical museums; this can, in turn, be used by the meSch system to recommend to the CHPs more or less technically complex solutions. In essence, organizations that operate in the same cultural domain (e.g. archaeology, art, ethnography), that manage similar types of museum spaces (e.g. history museum, botanical garden, historical cemetery), with specific types of physical constraints (e.g. no Wi-Fi, no electricity), and with similar classes of visitors (e.g. school groups, families, elderly people) are likely to have similar needs, thus the recipe recommender should promote recipes that have been deployed by organizations with a similar profile.

The recipe recommender exploits the user profile as follows. By definition, recommendation technology uses information about (1) users’ personal characteristics, needs and preferences, (2) semantic features of products and/or evaluation ratings, and (3) possibly the behaviour of the community of users at large, to suggest products that best suit user’s needs and preferences in a given situation and context (Resnick et al. 1994; Resnick and Varian 1997; Burke 2007). The algorithm applied to the magazine is an adaptation of the approach described in (Ricci et al. 2006), a hybrid recommendation system that integrates collaborative-based and knowledge-based models, by exploiting both the interactions of other users (collaborative aspect) and the rich descriptions of organization profiles and recipes (knowledge-based aspect). In our implementation, a recipe r_i in the catalogue is ranked high in the recommendation list if another organization with interests, needs and choices very similar to the current user has already selected a recipe very similar to r_i. The double similarity computed both on organization profiles and on catalogue elements provides chances to be recommended also to newly created recipes which have not been adopted by any organization yet. This provides opportunities for innovative interactive experiences published within the authoring environment to be suggested to potentially interested heritage institutions. More in detail, a case =(clf, cnq, recipe) is associated to each author organization and contains the following three main components:

• Collaborative features (clf) store information about the author profile features described above and the list of already authored recipes with their descriptive metadata (type of narrative and interaction strategy, environment of deploy, type of devices). Collaborative features are used to measure author similarity.

• Constraint queries (cnq) collect information about the possible search constraints the author has specified for searching the magazine (e.g. type of experience; type of hardware; preferred physical setting).

• Current experience instance (recipe) includes a description of the in-progress authored recipe, i.e. content and the devices that the author has already selected for it (this element may be still empty).

The baseline recommendation algorithm contains the steps in Fig. 11. The algorithm steps execute: (1) the computation of the most similar authors and (2) the identification of what recipes they have chosen for their museum institution. In the loop in (3) each recipe identified in second step is used in step (4) as seed to look for and rank other similar recipes in the catalogue that satisfy the query constraints expressed by the current author. In step (5), the merging of the ranked recipe sets computed in (3) returns a list of all the recipes that satisfy the author search query, boosting at the first positions recipes potentially more interesting for them. In the original recommendation algorithm described by Ricci et al. (2006) a modified Euclidean distance metric was used to compute similarity. However, this metric does not take into account the frequency of appearance of features in the recipe catalogue or in the authors’ profiles, therefore rare features would not get boosted in the computation of similarity. In our case, the possibility of boosting terms by frequency is particularly important for unusual heritage institutions such as a historical cemetery or a battlefield to ensure that the limited number of recipes is not a hindering. For this reason, for similarity computation we adopted the vector space model that uses term frequency and inverse document frequency (TF-IDF) (Manning et al. 2008) and we implemented it via the opensource Apache Lucene library¹⁶ as integrated in the Elasticsearch engine.¹⁷

Fig. 11

Recommendation algorithm based on double similarity

A different recommendation strategy was implemented for unregistered users that do not have a stored profile so their collaborative features (clf) are empty and cannot be used for recommendation. We expect such users to be CHPs who are curious and intrigued but not yet interested and committed to using the authoring environment. Therefore, in this case, the appropriate algorithm to inspire new users is one that favours diversity (Venturini and Ricci 2006); in this way the curious user would be shown the potential of meSch via a variety of different implemented installations. The set of recipes to suggest is built iteratively by randomly selecting a seed item from the catalogue (step 1 in the algorithm in Fig. 12), and by choosing at each step an additional item that minimizes the similarity with those already included in the set (loop 4 in Fig. 12): the top selection of recipes then shows variety with respect to the domain of heritage, type of environment, type of interaction and hardware devices to help the author understand the potentials of different narrative, interactive and technical solutions.

Fig. 12

Recommendation algorithm for inspiration

To evaluate the quality of the recipe recommendation for inspiration we run a repeated experiment and we measured: (1) diversity, i.e. how well recipes of different types get a chance to be shown; (2) item space coverage, i.e. how well all the recipes get a chance to be displayed in top positions; (3) user space coverage, i.e. how well different types of institutions are served by the recommendations. For the experiment, a controlled catalogue was created with 6 institutions from different heritage domains and 23 recipes of varied interaction experiences in their collections, with 12 of those recipes marked as “public” (shared) and therefore recommendable to other users. The list of recommendations displays more variety in the top positions than a normal search. A comparison of the results of 60 repeated calls to the recommender as a new, non-profiled user showed the following phenomena:

• Item and user space coverage All the shared recipes in the magazine had a chance to appear in the first position of the result list meaning that all organizations with public recipes in the meSch magazine are periodically featured in the top position; this happens with a frequency which is proportional to the number of recipes marked as ‘public’ in their individual repository.

• Diversity In 80% of the 60 trials the first 4 results are from different heritage domains out of 5 in the trial database. In 81% of the runs, the top 5 positions are for different types of experiences out of 6 in the trial database. When considering the combination of domain of heritage and type of experiences the number of diverse samples before repetition is high: 5 (51%) or 6 (36%) out of 10 possible different combinations of domain and experience type.

Despite the limited sample set used for the evaluation test, we found that the recommendation by diversity represents a valid means to offer at a glance an impression of the variety of visitors’ experiences that can be created using the meSch platform and therefore it satisfies the goal to provide inspiration to new users.

5.1.2 Reuse and repurpose

Our co-design research clearly showed the importance for CHPs of reusing and repurposing both hardware and devices from previous installations as well as content. In the meSch authoring environment, the possibility of reusing and repurposing an existing recipe is core. One can reuse own recipes, stored in “My cookbook”, or copy an inspiring one from the magazine. Overall a recipe provides an abstract visualisation of the Experience Schema: by allowing users to reuse an existing and working recipe, we empower them to jump ahead of their current understanding of how the system works and support a ‘learning by doing’ approach, as discussed in the following Sect. 5.2.

When reusing an existing recipe, the first step is to adjust or change the semantic dimensions of the experience schema at the basis of the recipe. For example, the recipe for the Atlantic Wall (Fig. 13) lists: target audience, language, theme, and points of interest that capture the interaction described in Sect. 4.1. What the author will change from the original recipe depends on the intended reuse. For example, the author reusing the recipe may decide to add another language or another theme or to split the audience in adult and children. These changes only require adding new semantic dimensions or new values for existing ones that are used as tags for the content. A museum that intends to replicate the same experience but in a different setting can keep the same classification but change the values, for example the languages can be localised for other countries. Of course, such a museum will change the number of points of interest or the themes to match the exhibition under development.

Fig. 13

The “Recipe” tab shows the semantic dimensions and their values (left); “Edit” opens up the editing panel (on the right)

To exploit the complete flexibility of the underling formalism, it is possible to add new semantic dimensions (via ‘add classification’, Fig. 13 right) and their specific values (for example add ‘genre’ as a new semantic dimension and ‘emotional content’ and ‘historical facts’ as values; or add ‘detail level’ as a semantic dimension with values ‘main description’, ‘details’ and ‘example’). These semantic dimensions can be used to represent any personalisation feature the CHPs want to use (e.g. the adaptation of the presentation with respect to the output language, type of audience, current position of the visitor, preferred theme, interest level etc.). This enables us to move away from a system that decides a priori which personalisation features are allowed [as it has been so far (Not and Petrelli 2014)]; instead it enables CHPs to model any situation and to experiment with a very wide range of personalisation settings (Not and Petrelli 2018), some of which are reported in Table 1.

5.1.3 Structuring content

In the previous step of the recipe creation process, the semantic dimensions and their values used by the rules to control the personalisation have been defined: it is time now to structure and annotate the content. It was an explicit design choice taken early in the project not to create a media editor, but to import content files. This choice was motivated by the will of the CHPs to control the content, both in what is said and how it is said. Our initial study of CHPs practice showed museums generally commission to external media companies the production of high quality content: the meSch authoring environment was then designed to upload any file format and to apply the personalisation formalism on the file itself as to enable museums to maintain their current working practice. However, a fine granularity of personalisation (e.g. at the sentence level) is still available to authors and we experimented with text generation for a personalised postcard souvenir (Not et al. 2017).

The content files can be uploaded from the local drive, searched on the museum database or downloaded from public repositories such as Europeana. Searching own databases and Europeana could benefit from contextual search techniques: CHPs talked about time pressure when preparing an exhibition and this might prevent them from spending much time searching for new material and, more often than not, they tend to use sources they already know. Therefore, a search component with recommendation features able to take into account the current editing and provide recommendations that complement the already selected content could help the CHPs in finding new content and getting to know better their online resources. In line with these thoughts, meSch trialled algorithms that use information about the institution and the task in hand (i.e. the subject-matter domain, frequently used search strings, content already selected) to suggest a filtered list of results potentially more relevant for the current authoring task (Hashemi et al. 2016).

When the content has been uploaded into the authoring environment, semantic tags are used to model the condition under which this content will be delivered: the semantic tags (listed on the right-hand side of the interface) are the specific values of the semantic dimensions defined in the previous step. Figure 14 shows in the “Content” tab (left) the list of files marked with different semantic tags, and (right) the editing of the metadata when the file is uploaded (by clicking on the pen icon).

Fig. 14

The “Content” tab shows the list of tagged content files and the semantic dimensions (left) and the panel for loading and describing a content file (right)

The editing of the content and its conditions based on an abstract representation of files and tags was the outcome of an investigation whether the authoring of a recipe should be grounded on a representation of the exhibition space or not. A map with drag-n-drop functionalities to mark hotspots and add related content may seem an obvious choice and, as discussed in Sect. 2, it is now common for the commercial creation of museum guides. Despite its appeal, however, a map-based interface would not fit many settings of tangible interaction that are not defined by the space such as single interactive installations (Wolf et al. 2015; Not et al. in press), single object augmentation (Damala et al. 2016), or the printing of personalised tangible souvenirs (Petrelli et al. 2016a; Not et al. 2017). Furthermore, designing with a physical map of the exhibition precludes the use of the authoring environment for online experiences where the same content can be organised in different ways (further online exploration after a visit based on the topics of interest (Petrelli et al. 2016b) or online summaries of the visit); it also makes it difficult to adjust an exhibition to a different physical environment (e.g., travelling exhibitions) thus limiting reuse and repurpose. For this reason, an approach that grounds the authoring process around the narratives (content) and the actions for their release (context features) was chosen.

5.1.4 Defining the tangible components

The “Appliance” tab supports the specification of the tangible and embodied interaction and how the physical components map the semantic tags to implement the personalised visit, exposing the authors to what in the experience schema is formally represented in the appliance and device parts. This section of the authoring environment (Fig. 15) is to be used by the exhibition designers who define, within the system, the technical specification of the physical interaction. In symmetry with the content file, in the “Appliance” tab each device in the IoT platform has a set of tags that specify what that device represents when it comes to recognise a given condition. Figure 15 shows the case in which BeerMug, the smart replica of a German beer mug, is associated to the a visitor preference for the theme “Soldiers” and the output language “Dutch” (Fig. 15 left), showing the formal representation in Fig. 5 in a user friendly way. When the BeerMug is edited (by clicking the pen icon) the panel (Fig. 15 right) shows its details: a specific NFC tag whose MAC address (“60AF81CB8”) is automatically included in the hardware configuration section that implements the device within the experience schema. Similarly, the interactive case number 8 in the exhibition maps the Point of Interest “International Zone” (POI8), a bespoke assemblage of NFC reader, microprocessor and audio–video output. When the BeerMug is placed on the “International Zone” their tags are combined and the content file that has the matching set of tags is played.

Fig. 15

The “Appliance” tab shows the devices used and their tagging that specify the context they represent (left) and the editing panel for adding details about the specific hardware (right)

By splitting the description of the hardware into its logical representation (the appliance, Fig. 15, left) and its physical components (the device, Fig. 15, right) we enable easy replacement of the hardware itself. Adding new hardware to the current IoT configuration can be done in few easy steps via “Load Available Devices” (Fig. 15, left) that displays the devices currently active in its surroundings; a click imports them into the editing area ready to be tagged and edited. “Load Available Devices” calls a specialised onsite platform, meSchUp, that orchestrates the IoT level providing access to the smart things and their capabilities independently from specific differences such as different communication protocols or different brands (Kubitza and Schmidt 2017). In other words, it provides “a unified view on I/O capabilities of heterogeneous smart devices” (Kubitza and Schmidt 2017) and offers to the meSch authoring environment a set of capabilities rather than the explicit device specification. This means that the editor can use an appliance that detects proximity while it is transparent which sensor is used to detect it, e.g. infrared versus ultrasound versus video thermography. While meSchUp can be used via its own user interface (Kubitza and Schmidt 2017), the meSch authoring environment communicates with it via the Cloud and delegates to it the management, control and coordination of the different devices in that specific IoT configuration, Fig. 16. When the Appliance for a recipe is set up, the meSchUp communicates to the authoring environment the smart objects configuration, including their capabilities; when the recipe is deployed (as explained below) the experience schema (including the smart objects configuration) is sent from the authoring environment to the meSchUp ready to be tested via the smart objects.

Fig. 16

The overall architecture of the meSch platform: the authoring environment (with functionalities for managing the magazine and recipe editing) communicates with the IoT level, managed by the meSchUp onsite platform, via the Cloud. The IoT elements are then embedded into the tangible elements within the exhibition design

Even though we have focussed on objects and places augmented via IoT sensors and actuators, the authoring environment also supports the configuration of smartphones and tablets as devices the visitor interacts with. For example, to implement a location-aware mobile guide that delivers content via smartphone when the visitor reaches a hotspot (implemented via a beacon—Bluetooth Low Energy emitter), the author uploads an Android device in the appliance tab and as many beacon devices as the number of hotspots. A bespoke app, a reduced version of the meSchUp server for smartphone, executes the behaviour rules from the interaction script of the recipe and plays the items from the narrative content network. This mechanism integrates smartphones and tablets within the meSch ecosystem as preconfigured clusters of sensors (Bluetooth receiver, NFC reader,…) and actuators (to play audio, video) and to program them within a unified authoring environment. A prototype based on this setup was used for an indoor guided visit for two churches in the Netherlands.¹⁸

5.1.5 Controlling the interactive behaviour

The “Behaviour” tab (Fig. 17) contains the actual instructions for execution, the interaction script. As for the IoT configuration, editing the behaviour requires technical skills in both understanding what is already there and being able to modify it if the code needs changing. As discussed in Sect. 3.4, the authoring environment exposes the interaction rules directly in their Javascript format, for their editing by interaction designers with programming experience (Fig. 17). Our study shows that, although they have a good understanding of the logics of the interaction, curators and communicators prefer not to be involved in the actual preparation of the script (that might also involve repeated testing and debugging), as this requires the learning of skills quite distant from their own heritage background and daily activity. For this reason no visual support was designed for the representation of visitor models, interaction history and adaptivity rules, that thus remain internal to the script itself and therefore completely controlled by the technical members of the team. This is the place where components for advanced user modelling could also be integrated as external services, for example to compute group visiting style by aggregating individual visitor logs (Kuflik and Dim 2013), to determine visit fatigue (Bitgood 2010), or to dynamically compute which is the next visit step to recommend (see for example (Alexandridis et al. 2019) and (Sansonetti et al. 2019) for a review of different approaches). Personalisation rules could also embed advanced forms of cognition-based user modelling, as proposed by Raptis et al. (2019).

Fig. 17

The panel for the actual editing of script rules

When the editing of all the parts of the recipe is complete, the package of the content items (the narrative) and the rules that govern their context-dependent play (the interaction script) are automatically transferred to the meSchUp onsite server that manages the execution (Fig. 16). The “Deploy” tab is where the result of the completeness test is shown to ensure the checking on the overall content-interaction-devices mapping is correct before deployment. This is important as identifying failures in the IoT is still very difficult due to the many layers of communication protocols involved (Rowland 2015a, b).

The deployment is instantaneous so that changes and additions in the contents, annotations or parameters can immediately be tested onsite (Kubitza and Schmidt 2017). This is a step change in the way in which interactive installations are designed, as it allows avoiding simulation and enables fast prototyping therefore inviting CHPs to experiment more broadly with personalisation, as discussed in Sect. 5.2.2 below.

The recipe is now complete and the final installation can be taken onto the exhibition floor for the visitors to interact with. This is when the authors can decide to share their new recipe in the magazine for other CHPs to see and reuse. A step-by-step process, that starts from the “Recipe” tab (Fig. 13), asks to specify the metadata associated to the recipe so as to enable the recommendation mechanisms, i.e. to specify the profile features of the organization that created and tested it and the environment features of the place where it was deployed.

5.2 Formative and summative evaluations

This section discusses the evaluation of the authoring environment. Two different evaluations were carried out: a usability test with CHPs from the project team used the first version of the prototype to find out usability issues that needed changing (formative evaluation); the final version of the user interface (the one discussed in 5.1) was evaluated with a larger set of CHPs from a wide range of heritage institutions to assess the use of the authoring environment in the overall context of setting up a new tangible interactive installation (summative evaluation).

5.2.1 Usability evaluation

A controlled task-based evaluation was performed on the first working prototype of the authoring environment. The evaluation was run as a usability test. The task carried out by participants fit the scenario of use we had identified in our study with the curators: starting from an existing recipe with two interactive cases as implemented in the Atlantic Wall exhibition, participants had to add a new interactive case (attaching two new devices—the NFC reader and the projector—to create the new appliance) and upload the required content. The uploaded content and the appliance had to be properly tagged to enable the interaction. The evaluation was carried out individually and CHPs that are members of the meSch team took part; a think aloud protocol was used (Preece et al. 1994) and a researcher sat next to the participant taking notes. During the evaluation the three CHPs that took part in the evaluation were invited to comment on the user interface and on what the meaning of the different elements was, the actions they wanted to perform, and what they expected the system to do next. The think aloud protocol affects the performance time, and therefore, efficiency measures such as time on task were not taken into account. Instead, the think aloud technique allowed us to get a deeper understanding on what did or did not work for the participants. At the end of the task, the observer questioned the participants on specific behaviours observed during the evaluation. Participants were also invited to provide comments and suggestions.

The issues that emerged from the evaluation were classified according to Nielsen’s ten usability heuristics (Nielsen 1994). Of the 81 comments collected from the CHPs, the most relevant related to:

• the lack of visibility of the system status (21% of the problems, e.g. lack of feedback after pressing “Save” in the “Content” editing panel);

• inconsistencies across different interface tabs and the importance of complying with standards (27%, e.g. different ways to add a new item in the “Content” tab and to add a new device in the “Appliance” tab);

• the persistence of relevant information through recognition rather than recall (18%, e.g. relevant menu information that disappears with scrolling).

The issues were then addressed in the revision cycle that brought to the second prototype described in previous Sect. 5.1.

Besides usability, the study allowed us to assess the attitude of curators to modifying the narrative and the customisation features that control the personalisation. The upload of new content, the adding of new values for semantic dimensions and the change of the tags did not present any difficulties, thus demonstrating that curators easily grasped the tagging system that is the foundation of the content annotation.

As part of the evaluation we also questioned their confidence in changing the script that controls the interaction and specifies context-awareness and adaptivity. Consistently with the distinction of roles and what we found in the specific study with CHPs external to the project (see Sect. 3.4), the three curators that took part in the evaluation were very concerned with the responsibility of making changes to the Behaviour part of the recipe. Even the modification of a simple parameter like the proximity range to tune the distance for the recognition of visitors’ presence, was considered out of the scope of curators’ work.

5.2.2 Observing CHPs at work on the authoring environment

The summative evaluation intended to assess how the authoring environment would be used by CHPs in their everyday job. The set up was then as naturalistic as possible meaning that we simulated the process of designing a new interactive installation. Editing by means of the authoring environment was then part of a more extended and open task. To this purpose, four 2-day events were organised between July 2016 and January 2017 in The Netherlands, the UK and Ireland. The first was a pilot and meSch was one of the many technologies available, the other three were focussed on the meSch platform.¹⁹ The pilot was used to refine the program for the subsequent major events and gather a first impression of what would happen when CHPs unfamiliar with the meSch platform would have to use it to create interactive installations. Overall, over 40 professionals used the meSch authoring environment to create bespoke tangible and embodied interactions. These workshops provided us with a better understanding of the potential that the meSch authoring environment holds for the cultural heritage sector, how easy it is to use, and how it fits into the creative process of designing interactive exhibits.

In all the events creativity was the driver and the purpose was to make an interactive installation by the end of the event. There were small variations in the setup of the four studies although all had a similar organisation of activities over the 2 days and the ambition of providing examples of use of the meSch authoring environment as part of a realistic situation.

Participants were recruited following an open call in relevant mailing lists such as Museum Computer Group in the UK or the Irish Museum Association. When expressing their interest in taking part, potential participants were requested to state their experience and background. The organising team then selected those that best represented the different CHP profiles discussed in 3.1 as to bring in a variety of skills. Selected participants were sorted in groups that resembled exhibition design teams with expertise on content creation (curators and communicators) and exhibition setup (designers and makers). Each group was assigned a facilitator and technical support. Members of the meSch team were available for discussion and any implementation needs. Participants worked in small groups of 2–5, they had available the editing environment as well as the meSch kit, a set of bespoke devices that implement the Internet of Things: the kit was connected (via the cloud) to the editing environment and the magazine (Fig. 16). To create the installations participants had to handle both the editing environment and the kit that included: NFC readers and NFC tags; an Android phone set up for AR and a set of visual markers; Bluetooth beacons, buttons and lights; a node to connect to an output screen; a multimedia node to control pico-projectors and loudspeakers; a small thermal printer and a processing unit running the meSchUp server (Fig. 18, top left).

Fig. 18

The work of participants at the event, the kit (top left) and the editor (top right) they used and two of the installations implemented (bottom line)

The activities of the 2 days were roughly split in concept creation (day 1) and implementation (day 2). As most participants did not know each other, the first was an ice-breaking activity. They were then introduced to the meSch vision of tangible interactions and were shown examples from meSch case studies in order to inspire their creative work. Then each group was invited to choose an exhibit (from a selected collection) around which to create a new interactive experience. Creativity was the driver: each object came with a comment attached that was expected to trigger curiosity, e.g. a mystery object, maybe a fake archaeological finding, some curious facts. Any discussion or decision on what technology to use was expected for day 2 when the concept was established. The rest of day 1 was spent finding inspiration from the object selected and brainstorm around possible engaging interactions. When the concept was well formed, both the technical implementation and the installation setting were discussed with the meSch support team that advised on the best way to implement the concept (day 2). This implied to specify the narrative structure and the interaction strategies. Albeit the concepts were envisaged independently from any specific hardware and software solution, in most of the cases the imagined interactions could be implemented by adapting existing experience schemas already supported by the authoring environment, e.g. by changing the number or the type of semantic dimensions in the narrative; or by keeping the interaction script but changing the type of hardware (i.e. a different appliance).

None of the participants had seen or used the meSch authoring environment before. As part of the introduction to the workshop (day 1), they were shown a demo of the editing and the deployment, but then they focussed on the creative activities. In anticipation of the events, the meSch team had prepared material for participants to familiarise with the authoring environment. The familiarisation tasks were of two types, video-based tutorials (used in The Netherlands and Ireland) or paper-based simple tasks (used in the UK), and were available to all participants. In the 2 days, there were times for the group to work on their specific project as well as to discuss and present to each other. While logs were recorded across the three meSch events,²⁰ different qualitative data sets were collected by the three organising teams, e.g. video recordings, observations and questionnaires in UK; observations and group-debrief in The Netherlands; observations and shared discussion in Ireland.

The 13 groups prototyped personalised interactive tangible installations at various levels of completeness and sophistication, they designed and deployed their ideas and some became part of public exhibitions. Of the 13 new recipes edited by the participants, all contained elements of context-awareness with respect to user actions, 4 supported visitor’s customisation according to the preferred theme, 2 included a customisation choice of the output language, and 2 envisaged automatic adaptivity (one made use of the interaction history to generate a personalised visit receipt and one exploited the user profile to select the type of content played).

All groups were able to use the authoring environment to assemble and annotate their multimedia content without any or with minimal support, demonstrating accessibility for CHPs to personalisation of visitors’ experiences. An analysis of the logs showed that all groups reused an existing recipe; they all removed or changed the values of the semantic dimensions and extended the existing “Point of Interest” and “tag-#” while none changed the dimensions themselves. The logs show that all the groups edited the content by uploading files and tagging them and, when inspected, the vast majority of the recipes were judged well-formed; this shows confidence and understanding by the participants in creating a narrative structure suitable for their intended interaction. Similarly, appliances and devices were also changed and tagged, apart from one group that needed a new device purposefully built for their installation. Interestingly all groups, to various degrees, looked at the interaction script, and 2 changed it as a different behaviour was needed to implement the concept. Apart from three groups that requested bespoke devices to be assembled on the day, all other groups deployed their recipes multiple times and independently showing they were able, over 2 days, to create tangible personalised interactive installations.

The logs and the observations show that different groups displayed different attitudes when using the authoring environment: some made a change at a time and tested it in full (thus followed precisely the sequence of steps suggested in the familiarisation task), while others found their preferred way such as uploading all the content files and then tagging all files in a single go before deployment. It is also clear that mistakes were made, e.g. tagging the wrong files, but the groups were able to troubleshoot, quickly recovering and fixing the issues. On average, the groups spent 3 h working on their recipe over 2 days; again, some preferred to do all the design first and implement everything in one go while others swapped frequently between the editing environment and the making stage. The logs clearly show that all the groups kept exploring the functionalities of the authoring environment over the 2 days, although how much and what depended from their individual expertise and interest. In the logs, we also looked for evidence of exploration and inspiration, e.g. searching the magazine, looking at the tutorials or opening multiple recipes. Although we identified different explorative behaviours, most groups showed an interest in the searching, browsing and inspecting recipes beyond what they needed for the task in hands. They looked both at the collection of recipes available to them as a group as well as to all the recipes in the magazine. An interesting behaviour was displayed by one group that repeatedly looked at the original recipe they cloned while editing their own, possibly indicating that they were repeatedly checking their actions were correct and they were learning-by-doing.

While for nearly all groups it was a matter of adapting existing recipes to their ideas, in three cases their concept required the use of sensors that were not provided by the hardware kit available, such as a magnetic switch or a short-range proximity sensor. The technical support was able to compose those new sensing devices that were then included in the authoring environment and used by the groups. This is a further demonstration that the proposed framework is flexible to the point of adding new devices on the fly and then personalise the content on those.

The feedback collected at the three events was very positive with the most frequent comments referring to the novelty of the tangible aspect integrated with digital media; the ease of use and its immediacy (“great to be in control of an easy UI and see ideas instantly in action”); the pleasant interface layout; how easy and fast it was to arrive to a working prototype from a concept. The flexibility of the platform that allowed customisation, change and the update of existing installations was widely noted. The possibility to create new recipes in an independent way by using only their own knowledge and expertise was also highly valued. Finally, the learning-while-using approach was also pointed out as very positive; this extended to the possibility of splitting the effort and collaborating around the same task.

Participants with a technical knowledge prised the seamless integration of the IoT element with the editing of digital content; the novelty of tagging “as a mode of coding”; the modular approach and the easy way to add new elements, both content and hardware. Among the limitations pointed out by the technical participants were the constraints imposed by the simple behaviours available. They also expressed the wish to create new scripts, but, to do this, substantial documentations must be created.

Participants with a curatorial experience were enthusiastic about the novel interactions that meSch enables and they imagined how such a platform could be used in their museums to “make you see things in a different way”. Questions on how to add new sensors or actuators or to trigger multiple media simultaneously were the most frequent requests showing they were already envisaging how meSch could be used in their own institution. Indeed they put forward ideas of using it with the audience to create new exhibitions or with children to let them explore hands-on new technological frontiers (namely IoT and Cloud Computing). These were for us most unexpected but point to new possible uses of the meSch platform. The criticism put forward by the curators and communicators was the lack of documentation and limited number of illustrated examples as only a few recipes among the dozens offered had a video tutorial. Practical tips on how to set it up such as a reliable WiFi or a list of possible suppliers were also voiced as useful and missing.

From the observations, we noted how the changes of simple elements in the interaction needed a technical person able to amend the interaction script. For example, a switch on the interface could control if multimedia content is played to the end or if it is stopped when the NFC tag is removed, but in the current implementation this simple change had to be coded in the script. A clear limitation observed was the bottleneck created when groups wanted a new component built and integrated, e.g. a short-range proximity sensor, or the display of a hologram. While we had a highly technical person at each event, this was often not enough if two groups asked for bespoke solutions or the request came too late in the process for the expert to implement it.

6 Conclusion

In this paper, we have described the 4-year work that was carried out in the meSch project to design, develop and evaluate an organic platform that supports cultural heritage professionals in creating and deploying personalised, tangible an embodied experiences for visitors for the most diverse heritage settings. We took the decision of starting afresh and did not make any assumption of what personalisation in cultural heritage means, neither on how we could support its creation. With a radical approach, CHPs were involved in a co-design process that empowered them to influence the editing environment under development. We started our research by questioning heritage professionals to give a meaning to the personalisation of visitors’ experiences: we were then able to clearly identify the different types of personalisation we wanted to model, namely customisation, context-awareness and adaptivity. Studying the process of exhibition design, we gathered a deep understanding of the current professional practices and discovered that very different professionals are involved at different points in time, with completely different skill sets and roles. This means that those responsible for the content creation were not involved in the technical development, although consultation among the team was frequent. We concluded that an environment for editing personalised and interactive installations should strive to support the teamwork in the most seamless way. However, this ambition was challenged by the great disparity of attitude to technical aspects: on the one hand we wanted to offer an easy-to-use editing environment so that curators and communicators could create the content and decide under which conditions it would be delivered; on the other hand we wanted to be able to offer a powerful tool for creating personalisation. This double-aim guided us in the process of defining a flexible formalism while at the same time designing a user interface and the interaction that could fit with the abilities, expectations and current practice of curators and communicators that are not interested in learning how to master the technology.

In its final implementation, the meSch authoring environment supports CHPs in the whole cycle of creating an interactive and personalised exhibition. It provides examples of installations offering both inspiration and a means for learning what is possible and how to do it. To simplify the editing while maintaining a powerful representation, we deconstructed the complex process into a limited number of elements and phases. Focussed studies on specific critical activities such as the willingness of curators and communicators to edit the interaction script (that controls the behaviour of the final installation), enabled us to take informed decisions on what tasks should be proposed to the user and how those should be carried out. We believe that the co-design approach enabled us to design and implement a far superior editing environment than the one we would have been able to create had we followed a more traditional approach. This was evident in the evaluation: a large group of CHPs showed confidence in using the authoring environment to deploy the narratives on the devices that composed the interactive installation and independently troubleshoot it if this was needed. The independence and autonomy of the CHPs was one of the early motivation of the meSch project (Petrelli et al. 2013); the behaviour of our participants, the way they were looking at other recipes to better understand the system and experimenting with their own ideas makes us believe this aim was achieved.

From a technical point of view, the representation formalism we developed proved to be open and flexible. We deployed a number of very different installations within the cultural heritage sector and we have now started to apply personalisation in areas such as tourism and retail (Cavada et al. 2018). The simplicity of the final solution should not be underestimated as it is the outcome of an integrative and lengthy study on the many aspects involved in personalising tangible and embodied experiences for heritage.

Our final reflection is on the co-design process we followed. Starting this journey with the CHPs allowed us to step back from our own experience in personalisation and question the decisions and results the community have made in the past 25 years. Looking to this challenge through the eyes of the professionals we were able to clearly define the key elements that could change the perception of the technology needed to bring personalisation to the exhibition floor. On the premises of shared decision making implicit in co-design, we had to accept that the most advanced personalisation techniques (and possibly the most exciting for us as researchers) were not sought by the professionals who instead were interested in simple but effective ways of designing visiting experiences inspired by their own creativity and ambitions. Our goal as researchers then shifted toward designing and developing a platform that could take personalisation for cultural heritage to the people who have the interest and abilities to use it.

The user-centred approach intended to foster as much as possible the actual adoption of the authoring environment within the community it was designed for. This is a strength, but at the same time it also represents one of the limitations. Although the underlying formalism for experience representation is general and can be applied to other interaction domains, the editing functionalities implemented in the graphical user interface are obviously tailored to the needs of users in the cultural heritage domain (and close domains such as cultural tourism). Furthermore, we specifically focussed on the requirements imposed by tangible and embodied interaction, that is to say experiences based on the synergy between physical and digital elements. Other digital-only experiences might call for alternative authoring solutions.

Another limitation derives from the very same openness and flexibility of the authoring process and the underlying formalism that was pursued to enable the creation of very different types of interaction experiences. An authoring environment that is very flexible provides fewer constraints on how the different ingredients are chosen and structured, thus allowing also the creation of clumsy aggregations of contents and devices that only an experienced author is able to further edit. The extensive evaluation studies carried out in the project provide evidence that this risk is limited, as CHPs prefer to reuse and customize an existing recipe to their own scenario and goals instead of working in a completely unconstrained way. This suggests a possibility to create “closed” versions of the authoring environment, specialized to work with a bespoke hardware kit (with a fixed appliance configuration) and with a dedicated magazine of reusable recipes optimized for that kit to provide more guidance to authors with reduced creative needs.

Interesting directions for future work were suggested by the same CHPs who tested the system. Particularly valued was the possibility to use the authoring environment to engage the audience in the participatory creation of new exhibitions, in line with the principles of the participatory museum (Simon 2010), or to organize maker sessions for young visitors. Another strand of research is to investigate how the authoring environment can be adopted by companies working in the creative industry sector to practically demonstrate to museums the potential of IoT technology and do fast prototyping, before large-scale deployments are made onsite.