Keywords

1 Introduction

Immersion and reflection are often considered as opposing [7] or alternating states [9] of experience, and while these aspects of perception in lived experiences have been investigated long before the emergence of digital media, the development of augmented- and virtual reality technologies has revitalized investigations into the interrelated concept of presence [30]. In a technology-mediated environment, there is often no escaping the existence of devices, equipment and interfaces, the technological materiality that drives and delivers digital content. Attention to these aspects of the experience can remind participants that they are engaging with a generated or semi-synthetic environment and hamper the sense of immersion. Likewise, in the evaluation of experiences that elicit immersion and flow, these states can potentially break when a participant is interrupted to answer questions or reflect. Study of these experiences relies heavily on post-experience questionnaires and qualitative debriefings. However, research in cognitive science has shown that post-experience reporting can be biased by a number of factors, leading to a condition known as the experience-memory gap [26]. In this paper we propose a new method of Experience Probes (EP) that integrates design, experience and evaluation, engaging participants in the virtual world through reflective, investigative activity. EP is demonstrated with a pilot study in an immersive soundscape installation to demonstrate the potential for examining participants’ perceptions from within an immersive experience through the analysis of participant-authored mapping artifacts.

One key observation of this work is that visitors to the soundscape installation, when engaged in listening and sound mapping exercises (reflecting), overwhelmingly describe ambience and experience of the virtual place rather than that of the installation, often expressing a feeling of ‘being there’ (immersion). Another important finding from these preliminary results is that the sense of presence is proportionally represented in participant reporting even when specific attention is given to the technological materiality of the experience. Results are examined with reference to the reality-virtuality continuum [28] and portray participants’ momentary experiences, untainted by the experience-memory gap. This work contributes to the investigation of technology-mediated environments with a method to bind the experience of immersion in a technology-mediated environment with the activities of evaluation.

2 Related Work

Definitions of immersion range from “the experience of being transported to an elaborately simulated place” [27] to “unreflective absorption in an activity” [23]. These two definitions mirror the shift in focus in two decades of HCI research, from usability to experience [3, 18]. Presence, initially examined as the perception of being ‘in’ a place, achievable by effectively masking the medium of experience [30] has similarly evolved in HCI literature to address perceptions of agency within that place, including social, cultural and material dimensions [33]. Contemporary study of technology-mediated environments questions what exactly constitutes immersion and experience and how to measure them [1, 2, 12, 16, 20, 29, 34]. Kristina Höök addresses the challenges in [16] decrying the mismatch between traditional HCI methods that are largely concerned with usability and the concept and aims of evaluation in interactive art. Her recognition that traditional HCI methods need expanding to encompass the lived experiences of technology are reinforced in [22, 24], specifically concerning interactive art installations and mixed reality environments in [35] experience design in [36] and prototyping interactions in [5].

Immersion in any form, be it narrative, theatrical or technology-driven is related to the ‘willing suspension of disbelief’, the idea that an observer consciously disregards the fact that a situation is beyond the ordinary reality in order to follow or participate, accepting the logic of the experience regardless that it may deviate from that of the real world [25]. This is essential to understanding technology-mediated experiences as virtual worlds, games and interactive installation involve hybrid situations that fluctuate along the reality-virtuality continuum [28]. The real and the virtual in this model are located at the extremes and the space between regarded as mixed reality. Augmented reality, the overlay of virtual elements on the real world is positioned towards the real environment. Augmented virtuality, the implementation of objects and elements from the real world within the virtual, is located towards virtuality.

Considering immersion as a complete captivation [27], reflective thought and contemplation of the immersive experience might seem to inhabit the opposite end of the continuum [15]. This is congruent with Schon’s perspectives of reflection-in-action, the conscious renegotiation of knowledge and practice within an activity, and reflection-on-action, the post-experience review of knowledge gained through an activity [31]. However there remains considerable debate over design and evaluation of immersive experiences. Numerous studies [2, 6, 11, 17, 18] examine participation and collaboration in interactive art installations attempting to quantify emotional reactions and social engagements with techniques such as Positive and Negative Affect Schedule (PANAS), Collaborative Analysis Framework and the Repertory Grid. Other methods employ video-cued recall [2] to supplement questionnaires and multimodal investigations such as Gaver’s design probes, cultural commentators [14] and polyphonic assessment [13] draw on various lines of qualitative analysis taking into account the subjective interpretations of participants. The Sensual Evaluation Instrument (SEI) [21] introduces nonverbal assessments of emotions and the affective properties of computational artifacts, another unique approach to addressing the methodological deficit. However the majority of HCI research into experience evaluation relies on post-experience reporting which fails to access the thoughts, feelings and emotions of participants from within the lived experience.

The importance of extracting participants’ perspectives momentary impressions is related to the potential, and sometimes marked dissonance between the emotions, sensations and the perceptions of the moment and those recalled afterward [19]. This dissonance is referred to as the memory-experience gap [26] and it has driven the development of methods to address the immediate experience in cognitive science research [10] and behavioral medicine [32]. The discrepancy between momentary and retrospective data in reporting is attributed to a variety of experiential, environmental, memory and cognitive factors [26]. An excellent introduction to the topic in the framework of HCI and user experience evaluation can be found in [4].

3 The Technology-Mediated Environment: Soundscape

This pilot study of Experience Probes examines the immersive soundscape installation Listening to the Walkable City that was constructed in an underground car park as part of the COOP2016 conference in Trento, Italy (Fig. 1). Its material components were a circular array of six audio monitors on stands, a north arrow marked on the floor in the center and a bamboo street-sweeping broom. Several chairs and tables were located to the sides supporting computer, audio equipment, and a coffee machine. The car park was accessible by stairs leading down from a small courtyard at the conference venue. The real environments portrayed in the soundscape were pedestrian stairs in Hong Kong that ascend relatively steep inclines in the city and are capped at each end by auto traffic. These are public spaces interspersed with small parks, public toilets and lined with trees, local small businesses and outdoor restaurants. The social and cultural activity of the stairs are the focus of the research project “Hong Kong Stair Archive: Documenting the Walkable City” [8] that grounded initial investigations into the stairs soundscapes.

Fig. 1.
figure 1

Installation environment in Trento, Italy and real environment, Hong Kong

Full size image

The soundscape was composed of street recordings made on the stairs following several listening surveys of the area. Recordings were made mornings, afternoons and evenings on three separate stairs, and the final soundscape installation condensed a nearly 300 m transect of the city into an exhibition circle approximately eight meters in diameter. The soundscape installation loosely reflected the real world situation in Hong Kong, with an outdoor restaurant located at the bottom of the stairs on a small street with delivery trucks and pedestrian traffic. The stairs led ‘up’ through the installation area, with sounds of local businesses on each side, and footsteps tripping up and down slightly off-center. At the southern, top of the staircase could be heard the sounds of a heavily trafficked road, with further in the distance a jackhammer and school playground. The street sweeper could be heard in two locations on the stairs.

Sound recordings on site were made with a handheld audio recorder with background sound s of traffic, construction, birdsong, rain and the outdoor restaurants captured by a wide audio image. Isolated sounds were recorded using a very narrow audio image and included footsteps, the rattling of delivery carts and the street sweeper. Audio clips were filtered and assigned unique positions and animated trajectories within the soundscape installation to mimic the natural dynamics of the stairs’ sonic environment. Additional sounds were contributed by the installation space, such as the operation of the coffee machine, voices of participants and researcher, the occasional opening of the car-park garage doors, and the action of a broom sweeping the concrete floor.

4 Experience Probe

A total of 34 participants undertook the experience probe. Demographic data was not collected, however participants were roughly balanced male and female, between 30 and 60 years of age, mostly self-selected from the conference attendees. The conference was not sound specific, though many of the participants had prior experience with qualitative data collection methods. Once in the installation space, participants were allowed a few minutes to explore before being offered a clipboard and pen and invited to take part in the evaluation. The probe activity was orientated on a printed form with basic instructions and a map circle with a ‘North’ arrow. This arrow was replicated on the floor of the installation and orientates between the virtual soundscape, the car park installation space and the evaluation map (Fig. 2).

Fig. 2.
figure 2

EP mapping form and north marker on floor

Full size image

Participants were asked to “Describe (in any language) some of the sounds that you hear. Number them 1,2,3… and mark their position on the map.” Additionally they were asked to “Describe the ambience that this installation presents.” and after, to “Describe your experience, impressions, or comment on this installation.” Resulting evaluations were written in English, Italian, German, and one in Danish. Participation was largely individual with little discussion or collaboration during the mapping exercise. Several participants sat in one of the chairs to write in more detail, though most remained standing. Instructions were left deliberately simple, and participants took various approaches to completing the exercise.

The probe produced 34 participant-authored maps that were translated into English by native speakers. Transcripts were thematically analyzed using AtlasTi, first grouping descriptions of sound sources together, such as “broom” and “sweeping”, “auto”, “car” and “traffic” and then examining for inscriptions of senses of self and place. Self-perception was identified for example in “I feel”, “feeling” and “I am”, references to memories such as “in some ways it remembers a bit ​Beijing or Hong Kong” and creative or imaginative declarations, as in “These sounds are really inspiring for writing poetry!” and “…crimescene”. Sense of place was noted in comments such as “It seems to be in a shop overlooking the street in a pedestrian district…” and “People passing by or exiting the workshop; happy because their work time has finished.” Frequencies were determined not by word occurrences, but by the number of participants reporting in each category, as many participants repeated words and phrases. Participants’ maps were further sorted by the method used to map, for example numbering of sounds or speakers, showing motion, and text entered direct on the map (Fig. 3).

Fig. 3.
figure 3

Experience Probe sample results

Full size image

5 Results and Discussion

The descriptions provided by participants reflected a true-to-life perception of the actual place in Hong Kong, highlighting the capacity of soundscape to accurately transmit ambience. The most frequently recognized sound within the installation was the street sweeper, reported by 31 of 34 participants (91%). This was followed by sounds of traffic (82%), water and rain (70%), nature sounds (67%), voices (61%), footsteps (52%) and noise (52%). Birdsong, construction and car horns were among the other frequently reported sounds. Participants’ ability to identify the elements of the soundscape, however, was to be expected, as the installation was a straightforward arrangement of familiar urban sounds in an enclosed listening environment.

Our main interest is in participants’ reporting of these sounds in relation to their awareness of self and presence in the environment. Self-perception (41%) was identified in statements such as “If I close my eyes I have the impression of find myself elsewhere” and “Impression that around you is an oriental market in action”. Sense of place, or first-hand experience of the ambience of the soundscape (38%) was marked in texts such as “It seems to be in a shop overlooking the street in a pedestrian district […]” and more imaginatively, “A gutter, a water loss slips unseen in a silent narrow street, on the margin of the big and noise metropolitan arteries […]” Nearly a third (29%) of participants expressed both self and place, clear signs of presence in the virtual that signal immersion. Several participants addressed the overlap of real and virtual presence directly, for example “During the walk eyes and site are not important: I was looking at the ceiling the man next to me kept eyes closed. It is strange the difference between actual location (car parking) and the world presented by sounds.” and “I felt I was walking a bit in circles; maybe moving back-and-forth; am I looking for an address?; The sounds move from one side to another; but I felt more like it was me that was moving.”

Examining how the participants enacted their mapping, we observe another important overlap between reality and virtuality on the continuum. Of the 34 participants, one group of 18 (52%) numbered their sounds and marked them on the map, while a second group of 10 participants numbered the six audio monitors on the map and described the sounds they perceived were emanating from each source. Many of this second group of participants stood in front of each audio monitor and listened intently, clearly orientating on the material element of the installation, towards the reality end of the continuum. Yet reporting of self-perception and sense of place in the virtual remained consistent across both groups, shown in Table 1.

Table 1. Number and % of participants, map style, reporting self and place
Full size table

The percentage of participants simultaneously reporting reflection and immersion remained constant, regardless if individuals were consciously attending to the installation environment or were focused on the virtual world of the soundscape. This supports the idea that a hybrid state of real-virtual, reflective-immersive experience is being initiated through the probe. That 38% of participants described movement within the soundscape, and 26% depicted motion on the map further suggests that participants were cognizant of their own position, as listeners, in relation to both the real and the virtual environments.

This preliminary investigation of the Experience Probes (EP) technique demonstrates the potential to engage participants in reflective practice and gather data from directly within the immersive experience. The probe’s participant-authored maps provide sufficient detail to establish that nearly one third of participants experienced self reflection and sense of place, with some participants reporting from a state that clearly fluctuates along the reality-virtuality continuum. The proposition that one can reflect while immersed is borne out even among participants acting with acute awareness to the technological mediation of the installation. Though the use of pen and paper mapping worked well for the small-scale investigation of soundscape installation, it may not be suitable for examining all immersive experiences, particularly those with active visuals or tangible interactions. Yet the EP method is applicable to other domains with some adjustment to the form of data collection. Where a handheld device is appropriate for engaging participants, EP can be enacted through a digital interface to facilitate data collection, storage and filtered retrieval for analysis and visualization. If collaboration or cooperation is fundamental to a particular experience, EP can be implemented as a game or playful engagement. In any form, central to implementing a successful probe is to ensure that the reflective practice that produces data is mediating and not disturbing the immersive experience.

6 Conclusion

With the increasing technological mediation of our everyday lives, we inhabit what has become a Mixed Reality Environment. HCI as a field has pressing needs to develop new practices to design for and evaluate experiences in this environment. This research presents Experience Probes, an approach to momentary assessment of participant experiences in transition zones between reality and virtuality. The case study presents a soundscape installation and describes the facilitation of a hybrid state among participants in which the experience of immersion is investigated through conscious real-time reflection on a mapping exercise. Expressions produced in this state have been examined for insights into participants’ perceptions and we describe how participants express a sense of self-perception as well as a sense of place, intimating their presence and immersion in the virtual space. We surmise from this that with this method the reflective activity of reporting may not necessarily detract from immersion. Experience Probes thus represent an expansion of traditional HCI methodologies to include assessment of participants’ momentary perceptions, bypassing the memory-experience gap to evaluate technology-mediated immersive experiences.