Haunting Space, Social Interaction in a Large-Scale Media Environment

Abstract

The Immersive Lab is a platform for the development and experience of large-scale audio-visual and interactive media arts. In this article we investigate questions of audience engagement, artistic strategies, and interaction principles, as well as the effects of embodied and social interactions that become evident in this media environment. Using the catalogue of artistic works developed for this platform within the past five years as our material, we carry out qualitative inquiries through interviews and categorisations. The emerging insights generate a clear perspective on the convergence as well as discrepancies between the artist’s intentions and the behaviours of visitors in the media space and allow us to, if not definitively state, then at least speculate about universal aspects that each encounter in the media arts context entails.

1 Introduction

“Our body is not in space like things; it inhabits or haunts space. It applies itself to space like a hand to an instrument ... For us the body is much more than an instrument or a means; it is our expression in the world, the visible form of our intentions. Even our most secret affective movements ... help to shape our perception of things.” [31, p. 5, our emphasis]

When working in interactive media, with technological installations, we rarely get to engage directly in social situations of shared exploration and learning. Conventional contexts, means, and media are more oriented towards producing finished works that produce predictable single user experiences where the mode of experience falls within a standard range of attitudes, such as cinema, TV-series, and video-games. Shared presence within interactive media spaces and installations enables the audience to enter into a direct engagement as a group. Through the experience of exploratory processes, the social dynamics of shared exploration come to the foreground.

In this article we discuss the ‘Immersive Lab’ (IL) platform, the concepts and activities developed therein, as well as a series of artistic works that were developed for this platform within the last five years. Focussing on the question of understanding social interactions patterns in reaction to abstract, algorithmic, or narrative media-art works, we investigate through qualitative methods some of the categorisations and salient dimensions that constitute social interaction in media environments. This covers two principal perspectives: the concepts and processes used by the artists for creating work, and the effect of the works on the audience’s interaction- and engagement-behaviours. Through qualitative analysis processes based on interviews with artists, experts, and the observation of audience behaviour in exhibition situations, a multi-perspective field of interpretations emerges that can serve as a starting point for developing categories that structure elements of social interaction and engagement.

The IL platform we are developing since 2010 is a vehicle for the exploration of spatial media-arts work, combining the modalities of musical and visual surround presentation with a full-scale interaction surface. Central in the installation is the perceptual fusion of the three sensory modalities of vision, audition, and touch, thus providing a seamless interactive experience. It is sufficient to say about the design and constructing the installation itself that the dimensions and arrangement of the elements was guided by the intention to provide a human-sized space which fosters multi-sensory, embodied engagement (see Fig. 1).

Fig. 1.

Interaction in the ‘Immersive Lab’. July 2015: Haunted ‘Mirror’.

Furthermore, a core idea of this project is to provide a platform for a wide variety of artists to experiment in and develop artistic works specific to this multi-modal configuration. From an artistic point of view, the most important challenge concerns the development of an interaction model that takes advantage of the particular setting of the installation. This is a classical interaction design tasks, but applied to a situation that presents a particular set of demands and eschews some of the classical themes of interaction design. The works need to address group and embodied interaction and shared experience in a model that is based on dynamic interactions and algorithm-driven content generation. To date, through residencies and workshops, approximately thirty artists of various disciplines, backgrounds, and expertise levels have developed more than two dozen works. This growing catalogue of works forms part of the materials for the analysis carried out in this article.

2 Background

The platform of the IL can be situated in the field of interactive media art and is oriented towards the general public. The installation system intended for a variety of teaching scenarios that include teaching in the domain of creative coding, interactive media, as well as computer music and algorithmic composition. It has been presented in exhibitions and showings to the general pubic as well as specialised audiences with specific interests such as scholarly or scientific investigations.

Similar media infrastructures that address immersive surround content in various configurations–some larger, some smaller than the IL–exist in several places worldwide. Prime examples are the StarCAVE systems at UC San Diego [8], the Allosphere at UC Santa Barbara [2], the RML Cinechamber project¹, and the Graz immersive media lab running the extended view framework.² The specific characteristics of the IL that distinguish it from other platforms are its intimate, human-scale size fit for small groups of visitors, and the emphasis on tangible interaction with the entire screen surface. The IL is a space for artistic experimentation, community building, and a research platform enabling investigations into creative processes, multi-modal perception, and multi-user interaction.

Immersion in its original sense means being submerged or enveloped, usually in water. In media arts and theory this term has been extended to mean envelopment by mediated contents, be they visual, sonic, or sometimes tactile. We may consider frescoes set in architectural spaces [1] and panoramic paintings [16, p. 62] to be older forms of mediated immersion. The concept of ‘virtuality’ [32] is a central topic in the discourse about immersion and can be summarised as the idea that mediated contents generate an artificial envelopment. Cinema has been for a long time the principal vector for immersive experiences for a large public [36] and today pushes further into that domain by the application of 3D and stereoscopic techniques [46]. Video games in general and the recent resurgence of virtual reality headsets have become another important way of experiencing virtuality in an actively engaged manner [21, p. 81].

A further dimension that plays a role is embodied presence [43]. It informs the perception of the digital image or abstract objects where they are integrated into the body’s process of perceiving itself [17] or its environment [15]. The importance of embodiment can better be appreciated when considering the enactive position, as does O’Regan [34]. He postulates that cognition arises through the body’s fundamental intertwining with the environment [20].³ The intertwining of body and environment is present in the biological domain as a regulating principle called autopoiesis [29]. The body’s ability to perceive itself and to adapt its relation to the environment informs all sub-personal processes of perception and human experience [14].⁴

These capabilities play an important role in the media context where abstract simulated content needs to be engaged with in an active, intentional manner [42].⁵ Here, intellectual reasoning fails to account for an important part of the experience. It is thanks to the embodied, enactive [40] connection to the environment that experience arises; based on the physical presence and sub-personal perceptions, the affective [37] and embodied capabilities [43] inform experience and provide models for understanding abstract or metaphorical media content. By relating with the installation through a direct bodily behaviour the non-semantic aspects of a work can have their effect. Think for example of encountering body-sized figures or hands: this prompts physical reactions of touching and mimicking in an involuntary engagement.⁶

This is why an important aspect of any type of cross-media work and cultural interaction scenarios is the development of metaphorical relationships [25] and blended spaces of signification [13], which surpass the concrete ‘mediality’ of any given situation. Thus, the visitor’s innate and acquired skills of recognising relationships is applied to the simulated, mediated representations appearing in the media space. This process leverages the complex educational, cultural, and social assets and occurs “by building on visitors’ pre-existing knowledge of the everyday, non-digital world ... employ themes of reality such as visitors’ understanding of naïve physics, their own bodies, the surrounding environment, and other people” [22].

By looking at the challenges and demands of designing interactions and simultaneously creating aesthetic experiences that get shared by a group of visitors in a common real space, principles of design and psychological foundations of sociality come into play. Considering the combined affordance of the IL media space together with the often idiosyncratic artistic ideas that the visitors are confronted with, the perspectives of activity theory [23, 26] are more appropriate, rather than for example structured task analyses [3]. The agency of the visitor in combination with a need or motivation creates the intentional relation to the object in the world. This activity responds to affordances in the (natural) environment [15], which are defined as the potential for action or perception that is on offer; in the discourse about design this concept gets differentiated between perceived and objective affordances [33], between cognitive, physical, sensory, functional affordances [18], and affordances of control [35]. The visitor’s learning of the interaction modality happens through a reinforcing action-perception coupling (or a circular causality [30, p. 15][11]), where they clearly perceive their own agency [14, p. 237], and see or hear a clear dependency between an action and the response by the (artistic) work, i.e., the media environment.

Interaction, within the context of media arts as well as broader technical design “means the degree of access to model parameters at runtime” [45] and thus the mode of exerting influence on a system’s behaviour. A mark of an interactive engagement is that a degree of autonomy is present in the system, which mimics agency, in order to appear opposite the person in an inter-subjective [9] and thus social [7] relationship. If therefore a “social action is action in which the other is addressed in the visitor’s acts” [6], then on the level of the human-machine interaction, the ‘other’ is first perceived within the technical system. Simultaneously this perception includes other persons present, be it the (implicitly present) author or another person entering into the interactive situation. This side-by-side presence is based on different types of social interaction skills that “include verbal and non-verbal communication, the ability to exchange physical objects, and the ability to work with others to collaborate on a task” [22]. A specific case of this relationship arises when several visitors participate in the exchange with the system. In addition to the topics of translation and behaviour between man and machine we enter into a triangular social situation, where the attention of the visitors gets divided between the engagement with the system and the other people. Through the joint attention arising in the social situation [10], the social bond is strengthened and a mode of negotiation through social dynamics emerges. This is not comparable to an on-screen individual situation or the sharing in the virtual sphere of the social web, but takes place in an actual physical situation that occurs in the installation setting and engenders a shared experience between visitors. The occurrence of these situations is specific for a determined context, in what installation artist Snibbe calls “social immersive media: immersive media that favors interaction in a shared social space using a person’s entire body as the ‘input device,’ unencumbered by electronics or props” [39].

The media situation presented in the IL falls within this category. Much of the exploratory engagement by the visitors is based on these behaviour and learning patterns. Given that the principal mode of engagement is by touching the screens, the visitors, even first-time ones, can refer to earlier experiences with touch surfaces, be it mobile phones, tablets, or touch-screen kiosk applications [5].⁷ These fresh cultural norms, that have become pervasive since the rise of the touch-screen mobile device, serve as experiential points of reference for exploring the interaction in the IL.

Developing work, in particular of artistic nature, without concrete task and result imperatives, means taking into account the biological, psychological, and cultural elements contributed by each visitor, as well as the shared gestural iconicity and familiarity with a certain type of abstraction in the presented works.

3 The ‘Immersive Lab’

The ‘Immersive Lab’ (IL) installation is the fruit of several years of development, investigation, and artistic creation. Originating from research into surround sound and algorithmic composition [4], it has evolved to become the multi-sensory and fully interactive space we present here. The platform serves for experimentation in the artistic domain, as well as a device for generating experiences to be investigated from a point of view of composition, systems theory, and above all interaction and social behaviour within media environments.

Currently, the IL installation is a modular platform consisting of four free-standing frames carrying curved rear-projected screens. The frames also carry the multi-speaker audio system stacked in two circles of eight around the perimeter of the screens (see Fig. 2),⁸ as well as the infrared illumination that is necessary to transform the screens into touch-interfaces. The interaction is implemented by tracking the visitor’s touch from behind the screens by means of a camera-based system (OpenCV⁹ in OpenFrameworks¹⁰). The tracking system surveys the entire screen surface of the installation, which spans more than ten meters in width and 1.5 m in height.

Fig. 2.

A rendering of the ‘Immersive Lab’ platform in its current configuration.

The circular setup used in the past four years creates an enclosed space that generates an immersive field of image and audio. Stepping into the space immediately exposes the visitors to projected light and sound from all sides and envelops them in an image that exceeds the natural field of vision. Apart from a spatial envelopment by image and sound, additional levels of immersion are generated for the visitors: they enter into a dedicated physical space, the direct tactile interaction with the panoramic surface enhances their personal engagement, and finally group behaviours and social interactions emerge within the shared space.

The main activity carried out in this environment has been the development of artistic works by a growing number of artists. Their task is to develop pieces that combine panoramic image and surround sound with a focus on interaction. The curatorial guidelines established early on have led to a catalogue of artistic works that all present live-generated, multi-visitor, non-linear, audio-visual interactive pieces. These decisions put a focus on generative graphics, not photo-realistic artistic work or the use of little photographic or pre-produced video materials. The sound domain consists also mostly of non-linear modules rather than static musical sequences.

Each artistic idea uses specific materials and broadens the spectrum of ideas, generating a wider scope of experiences and evoking different metaphors. In the best case a piece exposes the inherent materiality of the installation (for example by showing the machine-perspective and using the sounds of the screen material, as is the case in ‘Mirror’, see Fig. 1); the pieces always constitute metaphorically blended [12], algorithmically driven models, exhibiting behaviours in media that give rise to the perception of machine agency and autonomy [38].

The perceived spatiality in the IL is not ‘virtual’ in the sense of a virtual reality. Given by the physical space that is created through the disposition of elements, a real spatial experience is generated, which provides depth, perspective, and boundaries that do not need to be simulated. In the few cases where a ‘world’ is created in the abstract realm (Hyperborea, Star Camber), the screen surfaces obtain the character of windows that can be approached and touched but also observed from a distance. Even then, the perception of immersion does not primarily suggest physical presence as avatar in the simulated world. Its function is more similar to a flight simulator where you look out through the window at the world, rather than inhabit a first-person perspective like in a game.

For the visitor to the installation, the first mode of engagement is assessing what and how content is being presented. Depending on the artist’s concept, there might be an algorithmic piece running that modulates its behaviour and content without visitor intervention (Star Chamber; Trails), and only changes its overall state upon direct contact by a visitor. Other pieces depend on a specific type of interaction to get started (Clocks and Clouds), posing the question of audience engagement and visitor guidance. The engagement with each piece is based amongst others on the ability to ignore that the contents are computer-generated and projected as mere images onto the screens, in a willing suspension of disbelief [41]. This ability builds on the acquisition of cultural techniques [28, 44].¹¹ The emerging curiosity to engage with the presented idea and metaphor arises from the recognition of the materials and contents arrayed by the artist. The more concrete and realistic these elements are, the quicker recognition occurs; the less mysterious or abstract the piece appears, the faster patterns of engagement are tried out. Only when abstraction reaches a level of simple, primitive shapes does engagement return to a direct, playful interaction.

However, unlike Snibbe’s work in museum environments [39], where content is used to convey meaning, a message, and tell a story, in our artistic context, fewer constraints operate, thus giving the artists who develop work a greater degree of freedom. Those conceptual constraints that are imposed originate from the aforementioned curatorial intent, which is aimed at keeping the experiences coherent and with comparable degrees of interactivity, so that the visitors may transfer the accumulated experience from one piece to the next.

The same kind of learning that we observe in the visitors also occurs for the artists who engage with the challenges of making work for the IL. In this case, the knowledge-gain may be located in the shift in understanding, which displaces them from their original naïve idea to the actual implementation, based on an iterative loop of implementation, experience, and adaptation to the observed behaviours of the audience. To facilitate this process, we provide a software simulation environment, where sketches can be visualised and pre-recorded interaction patterns can be applied to the model for testing purposes.

Fig. 3.

Exploration and co-learning within the installation. September 2014, ‘Clocks and Clouds’.

A short discussion of the piece Clocks and Clouds’ (see Fig. 3) shall serve as an example for a detailed insight into artistic ideas, the development processes, experiences, and problems, in particular with regard to social interactions. The basic idea for this piece is that of an algorithmic clockwork driven by visitor interaction. Through the layering of numerous identical elements, each with separate parameters, a dense texture emerges, that is a direct reflection of audience activity. In reaction to the visitor’s touch gestures, the circular space gets filled with acoustic and visual pulse-trains that are running at differing intervals and exhibiting different life spans. The combination of the interactions by several visitors generates a complex web of overlaid visual and sonic voices’, which merge in perception into clusters or masses. The guiding idea in creating this piece is a curiosity about perception of temporal, spatial, and sonic density and the way spatial and temporal textures depend on establishing in the visitors an understanding of the generative principles and the importance of group interaction. The stark and abstract quality of the piece poses a challenge to perception. It serves as an investigation into how our perception is capable of separating visual and acoustic streams and how sensory overloading has the effect of forming clusters, fused objects, or gestalts [24] that consciously emerge through the abstract phenomena perceived. The central topic of interest is social interaction between several visitors, in particular in relation to the reduced sonic and graphical elements, and the richness of the spatial envelopment arising from several people interacting at the same time. The development process of the piece was typical of working in this environment. Through a series of iterations, the original sketch slowly gained in complexity, up to the point where adding more elements, intricate state- or behaviour-mechanisms became counter-productive. Evaluating the effectiveness of engagement and interaction and judging when to stop adding complications was only possible in situations where a number of visitors entered into unguided play with the piece.

4 Analysing Models, Interactions, and Behaviours

In the catalogue of artistic work that have been developed for the IL thus far, a number of different interaction models are explored. Although the technical framework and physical infrastructure remains the same, a variety of metaphors and principles appear. The works developed thus far serve as base material for our inquiry into social interaction, mainly in the form of video documentation of public installation settings, through interviews with the contributing artist, a well as an interview with an expert from the field of perception.

Approaching the question of social interaction in interactive media installations is possible principally through qualitative, experience-based methods. Setting up empirical experiments for a data-driven quantitative approach is rendered difficult on the one hand by the nature of the presented artistic content, and on the other hand by the psychological dimensions of social interaction. These may manifest themselves in group dynamics that only become visible and measurable in verbal exchanges, gaze- and gesture-synchronisations but require knowledge of the behaviours for interpretation. It seems hardly possible to formalise and break this down into atomic tasks suitable for empirical experiments in the given situation. Inevitably, we use our judgement for establishing categories and attributing interpretations to systematisations. The use of common and clearly identified terms when establishing categories and the collection of key aspects through the interviews grounds this investigation in the experience of multiple people, be they the artists, audience members, experts, or the researchers themselves.

4.1 Rating Works

In order to subject the various works that have been realised for the IL to a comparative and systematic analysis, we introduce a set of categorical dimensions. The full set comprises: media content, narrative structure, system behaviour, visitor experience, interaction principles, and social setting. For the purpose of this publication, we restrict our analysis and discussion to the latter three categories. Each of these categories contains several attributes with which the works can be examined and rated. In a preliminary process by the authors ratings were attributed by judging the functional, experiential, and social aspects of each work. From the collected ratings a selection and reduction process was carried out.

Table 1. Analysis of interaction principles across the catalogue of works. Piece titles abbreviated: Dolphy-Coltrane DC; Hyperborea HY, Trails TA; Clocks and Clouds CC; Mirror MI; Star Chamber SC; FORMBIT FB; Trees TE; Sozio-Natürlich SN; Mushroom Holzburger Paradise MP; The Unattainable/The Intimate UI; Nothing to Hide NH; Monument for San Francisco MS.

Table 1 collects the insights from the primary categorisation process by identifying representative works of each category that sit at the lower and upper rating boundaries. This is complemented with a statement about the average and its significance to interpretation. This table provides an overview about the diversity or similarity of the various artistic approaches and their relation to visitor experience. In addition, it provides the means to identify how the IL as an installation setting that is common to all works expands, constrains, or pre-configures the range of artistic, experiential, and social possibilities. These considerations are addressed in the discussion section in juxtaposition with terms and categories obtained from further analysis processes. The discussion is directly based on the analysis organised and subdivided along the same categorical lines and attributes. The following sections explore the catalogue of pieces as categorised in Table 1.

Visitor Experience. (1) Affective – Intellectual During the visitor’s initial encounter with a work the intellectual curiosity as well as a piece’s artistic strategy are determining for the engagement. In all cases the principles of interaction and multimodal correlation can only be discovered through a process of exploration and discovery. (2) Individual – Shared Works that play with the concept of intimacy focus on individual experience whereas works that create atmospheric situations or a visceral immediacy emphasise shared experiences. Most pieces alternate between these extremes by providing direct feedback to interaction as well as integrating the system’s response into a global and shared setting. (3) Disengaged – Engaged Almost all works establish an interaction that depends on and rewards a high level of engagement. The focus on engagement is particularly strong in works that show little autonomy or those that appropriate game principles. In some cases, uninterrupted engagement is detrimental to developing diversified and extended responses to interaction.

Interaction Principles. (4) Predictable – Unpredictable Predictability caters to expectations and the rewards provided by a piece, in particular where an instrumental tool-action provides the basis for interaction. The relationship between the mechanism of the work and the diversity of visitor behaviours can be identified in the predictable forms of interaction. (5) Persistent – Changing Static interaction modes provide a rewarding comprehension for visitors, emphasise the dynamics of social interrelations among visitors, but also weaken the interactivity’s role as driver of a piece’s development. Changing interactivity fosters development and maintains an audience’s attention and curiosity. (6) Conventional – Idiosyncratic Most pieces don’t adhere to conventional HCI principles and those that do, do so deliberately as part of an appropriation through artistic strategy. Nevertheless, among the artists a set of conventions has emerged that is shared by several pieces, the most prominent being the use of trigger zones. (7) Explorative – Affording Most works don’t reveal specific interaction zones or principles as an affordance. Rather, these principles need to be discovered by exploring or observing of other visitors; this generates a focus on shared forms of engagement. Where graphical clues are given, it is in order to shift the focus away from exploration of interaction towards the exploration of content.

Social Setting. (8) Scales to Multiple Visitors Most works favour interaction by multiple visitors, but often the number of interacting visitors is limited, either because the piece cannot accept more input or because the legibility of the work’s responsive behaviour would suffer from concurrent interactions. (9) Requires Multiple Visitors Works that provide instrument-like interaction or social game situations benefit from multiple concurrent interactions. Those works that require the presence of multiple interacting visitors do so as part of their content progression principle. (10) Requires Coordination Most pieces don’t require a strict coordination between visitors but benefit from a synchronised alternation between interaction and observation modes. Where coordination is emphasised, it is in order to establish a collaborative interaction setting, which in some cases represents an artistic appropriation of multi-user game conventions. (11) Social Dynamics Drives Complexity Works that operate with the metaphor of a musical instrument establish their complexity directly via the social dynamics of the musical performance situation. The role of social dynamics is less important for pieces with autonomous and inherently complex behaviour. (12) Thematises Social Issues The specific properties of the installation setting make the social aspects of group interaction implicitly relevant to all works. However, its explicit thematisation is rare and gets chosen by artists with a background in participative performance or social media.

4.2 Interviews

Artist Interviews. In order to better understand the needs, processes, and specifics of artistic development in this project, a number of artist interviews were carried out.¹² They cover general questions about concepts and metaphors of the works, the mode of engagement, the significance of interaction, unexpected experiences, lessons learned about artistic processes, and opinions about particularities of the IL.

‘Hyperborea’ is described by its author as “an imaginarium”, stating that “the intentions of the users are converted to a medium value”, that the“infrastructure becomes transparent, as you enter the space” and that visitor can show “an observing or active engagement.”

Regarding ‘Dolphy-Coltrane’, the artist states that “the concept [is to] look for a relation between two languages that are closely related, the graphical and the musical”; it is a process of “translation, transcription from sounds to colours” and that the effect is to “touch the colours and make them sound directly, instead of playing them with an musical instrument.”

Commenting on their piece ‘Trails’ the artists state that “the interaction model [is] that a touch has attraction forces, and works with few touch actions,” however “with a larger number people a competitive situation arises.” The size of the installation influences perception in that “the 360-degree projection increases the dynamics through the enveloping characteristic and exerts an almost hypnotic effect.” A specific observation is that “frictions emerge between the different forms of interactions within the simulation model and between the actively engaged and the passively observing visitor” and that “through the model itself the interaction situation is being made evident.”

The author of ‘Star Chamber’ notes that “the space is a unique space, you cannot really understand until you’re in it ... its a very intimate space, [which] changes the complexion of the piece.” The basic model of interaction is influenced by the fact that “the algorithm is very unstable, dynamic, the context of the data influences the changes”, which implies interaction as well as autonomous behaviour. The visitors can “experience a sense of play ... an emotional experience ..., [the] sense of being dislocated and overwhelmed, [and] become part of the space and enter another world.”

Reflecting on his piece ‘FORMBIT’, the artist states that “it’s a visually dominant audio-reactive [piece].” He remarks that the perception has a specific role in the installation: “peripheral vision is completely immersed, so I can sit in it and ... the audio is encapsulating me,” and that by “touching in the periphery, it is being filled with the feedback image and we can continue to have that interaction ... it’s an experience I couldn’t have anticipated.” A central idea emerges: “in such an environment I think it’s open ground for how people play with it.” Finally, during the development process a reduction occurs and he “realised: keep it simple, stupid, because you really want the [interacting] person not to be frustrated.” So from an artistic point of view “it’s a good challenge trying to figure out how simple is enough that it’s engaging and where to stop [when it is] too much.”

These artist’s statements make evident several central elements: the unique spatial experience; the acts of translation that are necessary between the media; the main mode of interaction is that of play [19], that reduction and simplicity is essential in the layered media and interaction setting; and that collaboration as well as competition is a recurring social pattern.

Expert Interview. A expert was invited to experience works in the IL and provide insights through an interview. His expertise is the psychology of perception as well as cognitive effects of multi-sensory interaction. The selection of works shown was made in order to provide the experience of the basic types of contents and interactions: ‘Mirror’ provides body-sized, media-inherent feedback to touch gestures, ‘FORMBIT’ provides a strongly synesthetic audio-visual experience with abstract lines and synthetic sound, and ‘Mushroom Holzburger Paradise’ provides the experience of a rich auditory immersive environment with an almost naturalistic visual situation. The inquiry was directed at discerning levels of engagement, modes of interaction, multi-modal fusion in perception, the change of experience over time, the expectations brought to the pieces, and the balance between affective impact and intellectual curiosity.

About ‘Mirror’ he states that “you don’t feel alone, you always try to connect with the figures in the projections” and that there is a “social level of engagement, social gesture, bodily communication and interaction.” The effect according to him is to “interact with a piece of art, with a riddle, try to explore this space, interact with the space and its inhabitants.”

In his reaction to ‘FORMBIT’ he notices that it presents “a flat but interesting learning curve” and that the “experience doesn’t really end, there is something to explore afterwards” in memory and the resonance of the perceptual field. This is due to the fact that “the perception of movement is strong; a felt kinaesthetic movement” is produced that acts on a bodily level in the same way that“you can feel the sound energy on the touch panels.” Overall he states that “[the piece] focuses more on the intellectual curiosity and playfulness” in the way the interaction is set up.

For ‘Mushroom Holzburger Paradise’ he notes the process and unfolding over time. “You start out with black and have to start to touch ... you have to engage through sounds, have to unlock the visual sphere, try to find correlation between the touch and the sound.” This leads him to state that within this process “you realise ... that you won’t get surprised, [it’s] more an exploration of something established,” and that it is “stimulating to have this visual feedback to touch, the fluid simulation. [You] want to trigger it again, [this is the] playful element.”

The new terms that appear in these interviews are the concepts of social bond (not feeling alone), bodily communication, the riddle and surprise, the notion of perceptually unlocking correlations, the different learning curves, the kinaesthetic, bodily sensations produced by the works, and finally both curiosity and playfulness. He also remarks that in each piece the experience changes over time while the learning process and the playful engagement evolves. His main point concerning expectations is that some of the intentions of the artists remain hidden and that in some cases the pieces do not evolve enough, thereby leaving his curiosity unsatisfied.

4.3 Visitor Observation

A separate mode of investigation is the observation of the behaviour of visitors towards a given work and their interaction amongst each other. This process can be carried out either live or in video-captures. To provide repeatable observations, we carried out the observations on the documentation videos captured during the project exhibitions; they are available on the project’s website (see below). In each of these videos, the principal modes of engagement and exploration become discernible. The main criteria for selecting categories are their reoccurrence in several works and the involvement of at least two people. The main attitudes, behaviours, and effects that we observed and collected were: contemplation/observation, exploration, showing-doing, explanation, and competition/collaboration.

The following video excerpts shall stand for many other situations where the same behaviours and effects can be identified:

In the video for ‘Dolphy-Coltrane’,¹³ between 00:27 and 00:46, two visitors, who happen to be professional musicians, are seen exploring collaborative playing and shaping the musical structure in the piece. Their communication and synchronisation occurs through sound, which is a habitual mode of paying attention between musicians. In the same video, between 00:54–01:10 the different modes of observing, trying out (exploring), joint attention and communicating (showing-doing) between members of a general, non-expert audience are clearly visible.

Similar behavioural dynamics are visible in ‘Mirror’.¹⁴ Here, a group of expert musicians discover (exploration) the piece for the first time and show each-other elements that they find interesting (showing-doing). A more conventional situation of passive observation and sharing understanding (explanation) can be seen in the video documenting the piece ‘Star Chamber’,¹⁵ between 00:30 and 00:54. Here the interaction modality is structured by a model that only allows state-changes and the slowly evolving generative processes puts the visitors into a passive, contemplative state.

In the video of ‘Monument for San Francisco’,¹⁶ with a technology-savvy audience, between 03:35 and 03:50, visitors can be seen explaining (showing-doing) to each-other what they see in the presented imagery, thus engaging in a communication situation (explanation) that goes beyond purely exploratory interaction. Similarly, in ‘Nothing to Hide’, in the same video, between 05:55 and 07:20, the group interaction and joint interaction is brought to a maximum, in particular with the piece’s concept of exposing social media contents in an interaction model resembling ‘whack-a-mole’.

The mode of competition but also collaboration is most visible in ‘Hyperborea’,¹⁷ between 05:08 and 05:40, where each segment of the screen-space presents a separate but linked interaction element that is used to navigate the piece (Table 2).

Table 2. Collected keywords from the three qualitative investigation strands.

5 Discussion

After these detailed analyses the next step is to bring together, compare, and synthesise the insights. The collected keywords as well as the analysis of interaction principles generate a map of interaction mechanisms and effects, which needs to be deciphered. The installation platform and the inherently artistic situations assembled in the work catalogue form a well defined framework, which informs the observed situations. Basing our investigation on these materials enables a comparative approach across a number of cases, yet at the same time prevents the generalisation into other interactive media settings. Nevertheless, the combination of elements assembled in the IL platform represents a valid experimental setup with which to carry out the intended analysis about social interaction.

The four investigation strategies carried out on the corpus of work each represent a specific perspective. At the intersection of these four fields lies the question about social engagement in interactive media installations. Several dimensions of sociality can be observed in this intersecting field: exchanges among artists, influences between artist and the (imagined and projected) visitor, perceptual links between the engaging visitor and the abstract entities presented in a piece, and finally the different interactions taking place between several visitors. In each of these cases the relational nature of the situation is central, but their rapport is not always a direct and immediate one. On the contrary, through the different temporalities that are present in the artistic development process, in the maturation of the ideas and experiences, as well as the time of a visit to the IL, the different subjects relate to each other in circular, adaptive, and influencing loops that mark the experience. Only between members of the audience does a direct social interaction emerge, in all other cases the technology as well as the metaphors and models used to construct the works mediate the experience and the inter-relationships.

An important question arising from this state of affairs is to what degree the social interactions perceptually dominate the other elements. Does the spatial, multi-sensory, yet evidently artificial media environment not impose a stronger impact than the (group) experience that is possible within its confines? Can we deduce that an inter-subjective link provides an indispensable influence on the visitor’s behaviour or does it influence it in an oblique manner? Does the action-perception loop enable a perception of lateral social interactions between visitors, or does the perception only occur when it is metaphorically established with the media work within the circular causality of interaction?

Even though it is hardly possible to provide a definitive answer to these questions within the given framework, the evidence that was collected in the analysis shows that on the different levels of interaction, within the different relations, the social or at least the inter-subjective rapport forms a core part of the experience. Whether this relationship supersedes the impact of the media environment depends on a variety of factors: the number of visitors present at the same time, the interaction principle and the artistic language of a given work, the familiarity of a person with interactive installations, and a general affinity to abstract, metaphorical, game-like scenarios.

Judging from the collected interviews and the observations of audience behaviour (see also Fig. 4) we are confident that in many cases the inter-subjective exchange forms the basis for interaction. Framed by the initial curatorial brief, in none of the works does a purely linear, utilitarian tasks fulfilment principle prevail. The abstract playful engagement [27]¹⁸ establishes both on real and metaphorical levels the subjective engagement of the visitor with a situation where an agent provides the subjective vis-à-vis. This subject can be perceived as the (inherently present) author or the agency-endowed, simulated entities within the interactive system.

Lateral social interaction or at least influences between persons in large-scale installations are always present; we haven’t been able to observe a single situation where a visitor did not engage in a dialogue with other people present. The setting is, contrary to personal devices and small screens, not amenable to solitary exploration, and if a single person is interacting there is always another person partaking who becomes the recipient of commentary and self-reports of experience.

The complexity of the emerging phenomenon in the resulting interactions depends on and reflects the complexity of social situations and group behaviour patterns. This is the case both during artistic development processes and in exhibitions with visitor interactions. The only solitary moment occurs during artistic development processes, where activities of sketching and constructing ideas takes place. But even in that phase the evaluation of the models and ideas requires the presence and interaction of another person. This particularly concerns an understanding of how synchronisation, coordination, and competition function within the interactive situations. These effects can only be seen and understood in the full installation scenario with an adequate number of visitors. Furthermore, the scale and scope of immersion is difficult to anticipate. The understanding about mechanisms and effect of interaction models on this dimension of experience can only be achieved through exposure to the actual setting. All this to emphasise the absolute necessity to experiment in the actual setting and adapt the original concept and idea to experienced and observed behaviours and interaction patterns.

Fig. 4.

Large Group interaction inside the ‘Immersive Lab’. November 2015 at the Gray Area Foundation for the Arts, San Francisco, CA.

6 In Closing

The ‘Immersive Lab’ platform provides a framework for artistic work as well as investigations into foundational principles of interaction and audience engagement. By laying out the framework, it’s conceptual background, and the specificities of creating artist work for this interactive media environment, we have set up an experimental device with which to investigate core question that are relevant to designers, HCI-researchers, as well as artists interested in mastering this particular mix of disciplines.

The main insight from this inquiry may be that there is always mutual showing and learning occurring in the social setting of a large scale installation, and that this complex, charged media environment enables the observation of behaviours, the extension of experience, and the encounter with archetypical categories of interaction, that are provoked by the specific, particular, idiosyncratic demands and reactions inherent to each artist’s vision.

Future work with the IL platform will extend the installation framework into other configurations, to enter into research and dissemination collaborations with scientific partners who investigate perceptual phenomena of proprioception, action-perception coupling, and musical performance principles. In addition, the pedagogical impetus will be broadened to include a variety of students, ranging from high-school to post-graduate arts and design students. And finally, the ‘Immersive Lab’ will be shown in a number of exhibitions where the general public will be able to experience the effects of human-scale, multi-modal interaction with spatial audio-visual media.