Keywords

1 Hybrid Environments for Multimodal Experiences

“People don’t actually design experiences, but environments in which experiences happen.” [1]. Digital design has been playing a huge role in the field of creation, communication and curation of cultural artefacts, experiences, and systems. The relationship between design and cultural and historical institutions, with authors, artists, and people – visit-actors or [2] or visit-author [3] (shifting the concept from consumer, according to marketing perspective, into visitors with a proactive role, according to a more cultural-centric point of view – innovating the field of preservation, valorisation, and exhibit as well. Besides, the two-year pandemic emergency with the lockdowns and other restrictions has been a (forced) opportunity for GLAMs – Galleries, Libraries, Archives and Museums – as well as for the educational world to rethink the in-presence, or site-specific visits, and to explore new modalities to keep in touch and engage the public in the digital-mediated realm. In fact, the dichotomy presence/distance that possibly opens the notion and challenges concept of space itself – physical, relational, blended, or cognitive – as well as tangible/immaterial [4] are the spectrums in which the cultural experience is trying to find new possibilities [5]. Cultural venues, closed due to the pandemic, are opening thanks to storytelling, curatorship, or participatory and choral construction [6], which digital, social, and other onlife [7] interactions make fertile ground for new ways of enjoying them. Collections, itineraries, and exhibitions are rearticulated thanks to the aggregation of hashtags with which people search, label and share a narrative mashup [8] that creates original stories and bottom-up curations which seem to fullfill the original promise of the web 2.0’s folksonomies [9] and grassroot disintermediation [10]. A good example of these new practices is the path taken by the Uffizi Museum Centre in Florence, already before 2020. The strategic vision of the director and the social media manager and the synergies created within the curatorial culture itself have led the museum to explore and innovate its online/onsite communication, creating a space for dialogue and participation especially for the digital native generations. Communication moved along two main assets. The first exploited the channels and online environments used above all by Gen Z (1997–2010) and Gen Alpha (born after 2010), i.e. Instagram and TikTok, leveraging the great evocative power and iconicity of the museum’s master pieces elaborating transversal narratives that break the historical linearity to recompose it in aggregations of meaning. The two social media based on images and audiovisual reels facilitated the enhancement of a visually engaging artistic heritage and an active and interactive participation played on the thread of humour to make culture accessible and engaging. The second asset was the co-involvement of one of the world’s best known influencers – Chiara Ferragni – who visited the museum during the lock-down, portraying herself in front of the most famous paintings and generating, both through the museum’s official channels and within her own social base of followers a viral effect in terms of communication and visibility on segments of the public that currently “escape” traditional/institutional cultural communication. The effect was much less virtual, i.e. a +27% increase in access by young and very young people in presence-attendance during the first weekends of reopening.

The online fruition, which is deprivatory – compared to a site visit – of the possibility of moving around in the physical exhibition and of the spontaneous interplay between people and the space, nevertheless it enriches the interaction mediated by the two-dimensional interface, to which the act of planning and curatorship must be reduced. On the contrary, limitation becomes an opportunity to restore complexity to the experience. In fact, interaction is enriched by different channels of fruition that, although deprived of direct sensory experience, offer a generous [11] multimodal synesthetic and cognitive pathway.

2 Fragments, Clusters and Narrative Plots

Under the push and the necessity of collapsing or blending the actual cultural locations and their remote access through digital devices and tools, the natural evolution of the repository space offered by the net – mainly based on the hypertextual structure of navigation [12] and according to an archive paradigma – is shaped by a different perspective that ephaties the narrative side of the overall architectural structure. A path according to which people are free, but at the same time “supported” and accompanied in “making sense” [13] of the knowledge chuncks disconnected from the visit and proxemic flow or experienced outside their natural context. The fourth revolution, in fact, has made possible to deconstruct not only the physical unity of the museum location, but also and above all, the conceptual space of the collection and the constrains of belonging. The art works and historical artefacts become primordial cells, atoms of culture and memory that aggregate, disassociate and recompose themselves in conceptual clusters and meaningful connections other than the static and given exhibition space. The concepts become nomadic and transversal, they intertwine in a continuous movement that stratifies their readings, relations and contaminations. It is the utopian space in which the same painting, the same fragment, the same document can simultaneously belong to a physical place and to multiple virtual places that recompose in a dialogic manner the relationship between the self and the similar, the self and the other. It is the dominance of meta-data, ontology and semantics that, in the face of the single object, compose infinite reading planes: the chimera of multidimensional cataloguing that generates without ever being able to exhaust, almost like an Escherian architectures, umpteen reading keys, like Warburg’s visual atlases or the multi-faceted universe of Ranganathan’s classifications [14].

3 Method: The Research Approach

If the relationship between GLAMs and the multimedia and internet world has been established since the beginning of the digital era, the shift created by the mobile devices and the actual phygitalisation and the diffused ubiquity of technologies suggests a further conceptualisation of the field. The proposed subdivision between /onsite/ and /online/ [15] seems no more suitable to describe the complexity of the ongoing transformations. As suggested by Turkle [16] back in the 1990s and labeled as onlife by Floridi [7] in 2015 the continuum between the online/0n-life/off-life – according to the new definition given by philosopher Caffo [17] – questions the paradigms of cultural experience we explored so far. According to this perspective, the research explores a first mapping activity to understand how the cultural institutions are facing the transition thank to the analysis of case studies. In particular it develops as an explorative model, a radar graph to qualitatively categorise the projects listed and selected, according to three criteria/axis, i.e. Space (a); Storytelling (b); Technologies (c). The work in progress activity has, as inbetween goal, the creation of a taxonomy of phygital storytelling in the field of cultural heritage to infer guidelines for future projects, mainly for small heritages, not always able to invest in developing new strategies of communication and engagement.

a) The concept of space is considered here in its multiple possible declinations that include both physical aspects - in this sense, experienced, in situ, situated and embodied places - and virtual aspects, understood as two- and three-dimensional, immersive, simulative and virtual (AR, VR, XR) digital places where technology is the experiential medium, hybrids, i.e. where systems (cognitive and instrumental) contaminate each other to the point of blurring and collapsing. b) Storytelling is understood both in its narrative dimension, in the strict sense, i.e. the development of a plot that follows sequential, metaphorical or archetypal paradigms – also in the recent interpretation proposed by Baricco [18] in the short essay La via della narrazione – but above all in its dialogical dimension. c) Technologies, which are often at the centre of many projects, are studied as an enabling and augmentative part – not in the sense of augmented reality – but as an enhancement of the possibilities of access to information and knowledge, enrichment and extension of the experience. In the light of these interpretative keys, three projects were chosen from among the possible ones mapped, representing as many paradigmatic interpretations of the intertwining of the parameters.

While traditional interaction models allow for a staging in fixed, pre-established spaces, strictly separated from the place where the user is and from which he or she orchestrates the interaction, the experience remains simbolic and is shaped by metaphors dictating laws and principles that the interaction must follow. It is the direct manipulation of the non-existent in which, in order to interrogate the physical object, we interact with a virtual and conceptual space loaded with information. It is in this recomposed discontinuity that the loss of context and structure is compensated for by the story-telling dimension of digital display and interaction. The nomadic and isolated fragments weave new and multiple relationships within a curatorial narrative fabric, translating physical scenarios into cognitive territories and migrating statutes into relational meanings.

3.1 The Cloud Pavillion, Narrative Dialog Between Space and Avatars

The panorama of experience and experimentation in this field goes back as far as the 1990s with experimental art projects in which the exploratory, meta-design and reflective component coexisted with installations and exhibitions itself, the object and subject of research aimed to translate into new languages and fruition the spatial and digital realm. The traditional descriptive dimension that accompanies the visitor on a didactic, when not didactic, path of knowledge becomes a dialogue between object/exhibit, subject/experience and space/context. The scenery of immersive experiments in the physical and virtual museum scenario is broadened and structured in the work of Studio Azzurro [19] or in the info-installations of iO16 in which the physical, proxemics and proprioceptive interaction of our relationship with the environment leaves the monitor’s mise-en-scène and pervades the space of reality, superimposing itself like a layer of knowledge.

A paradoxical example of this duality is well represented by the German pavilion at the 17th Venice Architecture Biennale 2021, the so-called Cloud Pavilion. The structure appears empty and bare to the visitor, who wanders among the rooms feeling that sense of estrangement so familiar in that onlife that characterized the lockdown. A being present in another space, here deprived of any other sensory dimension other than pure walking, the founding act of the exhibition experience, this time without a path, a goal or a specific direction other than a vain exploration. On the contrary, an offlife [20] which, denying itself to dialogue, dramatically reminds us of the simulacral value of the period we lived through. The interaction, however, takes place. In a a-site-specific dimension: another space, accessible via a personal device and a QR code or directly from the app. Here people find themselves in the three-dimensional simulation of the exhibition space projected in the near future 2038 in which many of the problems of contemporaneity seem to be solved in one of the possible [21] or desirable [22] futures, built from, or as an alternative to, the present. Two artificial intelligences accompany, chat and guide people through new scenarios – edenic as well as dystopian (!) – both in the form of abstract avatars and genderless voices. The narration becomes dialogue, in a continuous reference in which the real and the virtual are reflected, integrated and where they merge and confuse. The remediation of the space [23], then, is the crucial turning point of this new hybrid realm.

3.2 TeamLab Planet, Osmosis, Convergeses, Performances

The role of space thus becomes the nerve centre for these intertextual writings and for the collapse of individual, relational and collective layers. Questioning the ontological qualities of space and our being intrinsically and inextricably situated in it is outside the scope of this research, yet its connotation becomes a prerequisite in design terms. As in the deferred and dislocated and osmotic performances of Florian Feigl and Christopher Hewitt already realised in 2014 [24] – for instance Dissolved. The Uncanny Valley telematic performance held at the same time in Berlin and London – or the multi-sensorial and synthetic experiences of the TeamLab collective’s installations – one among all the Tokyio’s Planets (https://planets.teamlab.art/tokyo/) installation that inaugurates the strand in 2018 – real and digital collapse according to two processes that are at once divergent and convergent. The use of mobile devices that fragments the fixity of the monitor-computer-keyboard posture [25] and brings people back into space, so that technology comes out of the “black box” and ditributes itself thanks to Ubiquitus Compiuting and IoT (Internet of Things) on the “surface of the world”, making data and information always accessible and often contextual. Environments become sensitive and sensory [26], hybrid and phygital and as such embrace new paradigms of experience, interfacing and interaction. The hybridocene [27], a possible derivation or development of a technology-mediated and technology-enabled reality [28] explores narrative ways of reconnecting past and future in the situated and embodied present. Curiously, in this blended space, interaction returns to be strongly sensorial, tactile and multi-aesthetic. Virtuality, or rather syntheticity, in its simulative [29] and mimetic meaning, reopens our being located at the centre of the perception and knowledge process. in the case of Planets, for example, even the more virtual and abstract dimension, created by the dialogue between the people who impute and perturb the environment via their smartphones and the space dynamically ‘dressed’ by the artificial intelligence algorithm, generate an environment made of light and hyperreal space. The individual and intimate dialogue between the agent-person and the platform-algorithm generate a participatory narrative that never repeats itself and simulates at the same time the ever-equal and ever-different infinite cycle of natural the seasons. Althought yet mediated by a device and projected in a blended space, the people interaction shifts from an input-output exchange, to a choreography played in the space [30]. The space itself is the interface, and immaterial surface that enable the contact between the agent and the scope, here abstract, immaterial and emotional. The exhibition doesen’t show anymore, but rather creates a feel to be explored as any engaging story, able to create new and immaginific worlds. The typical linear flow of the narration [31] – of course, whit ups and downs and the gaussian shape – leave room to a network of possible touch-points distributed in the hybrid ecosystem of communication, more similar to a score to be played, interpreted according to one’s own personality and sensibility. The phygital space – physical, hybrid and digital – is then the stage where people acting and interacting improvise their particular time-site-based co-authorial curation. In the end, it is the performative act that creates the story similar, but unique of every exploration.

3.3 Arch of Light, Associative Plot, Metaverses and NFTs

If this is, more or less, the actual scenario and part of the debite regarding the relationship between cultural heritage – material or intangible – the new wave, or the possible further revolution of the Web3 is adding new variables to be discussed. Beyond the many potentials and variables, certainly at least two phenomena seem both promising and ambiguous with respect to the development of curatorial practices and studies in the field of cultural dissemination (Fig. 1).

Fig. 1.
figure 1

Le Gallerie degli Uffizi: official social media account on TikTok (a) and Instagram (c) and bottom up interactions (b) generated by visitors.

Full size image

.

On the one hand, meta-verses represent new ways and new worlds to show, visit or experience. The three-dimensional configuration, haptic-prossemic simulation, multi-modal languages, simulative and immersive capabilities can become a field of experimentation to the extent that they can break free from a low-resolution/hyper-realistic mimetic model. Augmented, immersive and virtual exploration can in fact be the site of a narrative that transcends and coexists with the physical limits of reality. The very patterns of behaviour and interaction are, in some ways, connected to our natural proxemics on the one hand, but also contiguous to the imagery of gaming and entertainment, i.e. akin and familiar to the new generations accustomed to the convergence of playfulness [32] and learning, play and culture. How and what stories the GLAMs will be able to tell is still to be explored and – as always with the arrival or reappearance of new technological waves – although we are beginning to see some experiences more fascinated by the technical potential than by the dialogue with people, which are in any case at the heart of any experience. However, it can be hoped that – mindful of the lesson of MUDs (the multiplayer real-time virtual worlds) of the pre-Internet era and SecondLife of the early 2000s – the world of Cultural Heritage will find a original and appropriate way to exploit the narrative potential of three-dimensional virtual spaces (Fig. 2).

Fig. 2.
figure 2

The Cloud Pavillion / 2038 The New Serenity, Germany Pavillion at the 17th Venice Architecture Biennale 2021 / digital app.

Full size image

But the phenomenon – or the buzz word of the moment – that is changing both the world of traditional digitised art-historical assets and those already natively virtual are Non-Fungible Tokens. Within the narration of cultural heritage, NFTs add a novel variable that further hybridises the physical space and object dimension. One of the earliest examples of the phygital experience and its potential is the first monument – the Arco della Pace in Milan – which earlier this year (more precisely between 31 December 2021 and 1 January 2022) became the first one to enter the metaverse as cryptoart as an NFT [33]. Realised by the Instanbul-based collective Ouchhh Studio and curated by Reasoned Art (an gallery focused on digital art), the monument/installation – the paradigms of the 20th century now seem inadequate to describe the mixing and overlapping of genres coexisting in the hybridocene – came to life in a multiformity of modalities and channels. Video-projection, immersive experience, live-happening or data-sculpture created by an Artificial Intelligence algorithm, the event was live-streamed on Youtube, instagram and Facebook. AI DATAPORTAL_ARCH OF LIGHT – this is the title of the work – “extrapolated elements from more than 20,000 works of art created over a wide period of time and belonging to different artistic currents that developed from Byzantine art to contemporary art. In addition, Ouchhh made use of the data from the map of the Italian firmament collected by NASA and the digitised Italian literary heritage from more than 700 years of history. Masterpieces from the history of Italian art by masters such as Raphael, Leonardo, Caravaggio, Titian and many others are brought to life in the form of a new work of art that combines art and science. Using digital data as colours and the algorithm as a paintbrush, Ouchhh Studio aims to connect the physical and virtual worlds by redefining the future of art.” [34]. If this work represents an orginal experiment in phygital heritage storytelling, NFT represents, from a certain point of view, an artistic product in itself and a new form of heritage. Whether originally “analogue” or natively “virtual” – whether this classificatio still makes sense – the transformation into NFT enshrines not only the ownership/uniqueness of the object, but above all its digital existence, to the extent that new museum structures are springing up to display them. It is the case and the case of the Seattle NFT Museum, founded in 2022, the first of its kind “designed to bring together artists, creators, collectors, and the broader blockchain community. […] to display their NFTs for visitors in a highly contextual, physical setting.” [35] The interesting – and perhaps somewhat dystopian – aspect looking at the images of the exhibitions on the website is that the NFTs are displayed framed and hung along the walls of the galleries in which the space is organised, as if they were paintings rather than digital experiences.

It is precisely on this last aspect that the future challenges posed by the phygital will be played out. On the one hand, the reflexive ability to conceptually rethink the world of dissemination and participation in the world of cultural heritage, but above all, the search for new languages and original, autonomous solutions for cultural storytelling that fully exploit the potential offered by technologies.

4 Discussion and Conclusions

By comparing the three case studies as paradigms of as many possible case studies and approaches, we can outline some ways of declining the three parameters adopted in the research. In the first case analysed, the narrative is played out in a dialogical manner between the real and the virtual, be it the absent space, or the digital environment in which one interacts by transposition. it is in the latter that people and the avatar interact in a “dramatic” manner according to the idea of laurel [29], that is, a dialogue that finds in the relationship with an artificial intelligence an original plot from time to time and a narrative that is dynamically generated. In the second case, however, the narration takes the form of a performative act, almost a happening, in a continuous loop of stimuli/inputs mediated and in betweenby sensorial feelings and digital devices, as well by the interaction with an actual and – at the same time – projective space. Finally, the third examples plays a role in creating and mixing the new statute of the relationship between the cultural heritage and its unique identity questioning the concept of cultural object in itself. The stratification of different fragments creates original meanings offering possibilities of narrative paths according to the key chosen to decodify the layering both of the physical space as well as the one generated and superimposed by the artificial intelligence.

A phygital approach, then, seems to be not only a promising, but an unavoidable one. On the one hand, phygital represents an opportunity to integrate and augment the planes that, through technology, intertwine, i.e. the real understood as physical and bodily situated experience, and the digital, i.e. the mediated or virtualised, two-dimensional, dialogic or immersive experiential and cognitive dimension. On the other hand, potential developments will be mainly related to generational change, a change that in scope goes beyond the physiological turnover, and configures new ways. On the other hand, potential developments will be mainly related to generational change, a change that in scope goes beyond the physiological turnover, and configures new modalities. In fact, the real change is in the design perspective, if it is explored by designers who are not only digital natives - we could say third generation by now - but above all parameterised on new modes of interaction that find their scope in the digital, but in the fragment, in the playful approach, in the streaming mobile.