Keywords

1 Introduction

The increasing pervasiveness of embedded and mobile digital components in urban environments has led to spaces and objects with more fluid than fixed behavior. These objects and spaces are capable of sensing, computing, and acting in real-time; they can change their behavior in relation to the state of their own system, their histories of past actions and interactions, the behavior of humans and machines within their reach, or environmental parameters. These environments are no longer static backdrops to human activity but have the potential to become interactive elements displaying a dynamic behavior and engaging with their human occupants [1, 2].

This potential meets a concrete demand in real world situations: Workplaces, health care environments, public spaces, or settings for social entertainment are increasingly required to flexibly adapt to continuously changing modalities of usage. However, physical settings often lag behind in their capability to respond dynamically [3].

Technologies capable to activate environments have reached a considerable level of maturity and are being deployed widely in everyday environments. Different domain names (ambient intelligence, responsive environments, pervasive computing, etc.) express different biases but what they largely have in common is the objective of developing embedded digital systems capable of supporting people in their activities by engaging in ways that are more akin to human interaction. These systems are composed of a wide range of sensing, processing and actuation technologies and aim at displaying context aware and adaptive behavior [4–7].

One of the biggest challenges of these systems on a phenomenological level consists in mediating successfully between the uniqueness of real world situations and the modalities of actuation these systems are capable of. Systems tend to perform well when goals, sensor input and input-output mapping can be clearly defined. However, when humans are involved in the interaction, these basic elements are typically uncertain. They rather emerge from the interaction itself in a situated way. The activity itself, the sequences of interaction, impacts and alters the condition of both subject and object as they engage in collaboration in an ongoing dialogue [8].

The very term “interaction” tends to be employed in a broad way and used to describe anything from a simple click of a button triggering a predetermined effect, to the sophisticated dynamics underlying human dialogue. Dubberly et al. [9] illustrate different models of systems that help in grasping the nature of interactivity. On one hand, in its most basic form, we have a human-computer interaction based on a single feedback-loop. A person having a goal, acting upon the environment, the person perceiving the effect of that action and comparing it to the initial goal to inform further action. More than ‘interactive’, the authors refer to this as ‘reactive’ - “in ‘reaction’ the transfer function (which couples input to output) is fixed; in ‘interaction’ the transfer function is dynamic, i.e., in ‘interaction’ the precise way that ‘input affects output’ can itself change.” On the other hand, Dubberly et al. identify learning systems in the form of second-order systems in which one feed-back loop interacts with another feedback loop, with the possibility of further levels of nesting feedback-loops within them. Here, “the second system sets the goal of the first, based on external action. We may call this learning - modification of goals based on the effect of actions.” In the presence of multiple first-order systems within one or more second-order systems, the second-order system can exert a choice as to which first-order system to engage with depending on its current goal and past experience with the response mechanisms of the first-order systems to work towards them.

In this more comprehensive model of interaction goals are defined dynamically as the system parts interact, past experiences feed into choices made on the system’s behavior and processes of learning are set in place. People interacting with augmented spaces based on such a model engage in what we an refer to as conversational dynamics in which meaning is co-created and space is appropriated every time anew for specific situated activity. When developed as truly interactive systems, their behavior can be described as improvisational, as is ultimately that of their human counterparts.

To talk about an ‘interface’ was an effective conceptual model to contemplate on human-product interactions before the dawn of computerization, when configurations were fixed (buttons, levers, sliders, telephone dials, etc.). The plasticity of the nature of human-computer interfaces created new challenges and opportunities. The notion of “conversation” was used as a new framework, suggesting a model where meaning is negotiated and co-created between human and computer through interactions. In this sense interface is no longer a simple notion by which human and computer represent themselves to one another. Rather, it forms a shared context for action in which both are agents in an Aristotelian sense. Laurel [10] describes how the ‘interface’ has developed into a theatrical ‘stage’ for the performance of intentional activity in which both human and computer have a role. The limitation of theatre as a model being that in scripted theatre, unlike real life, the process of choice and decision making takes place during rehearsal and practice and not during the actual staging of the performance. In the sense that drama formulates the enactment and not the action, it is un-like real life. However, in improvisational techniques used during rehearsal, as in real life, anything can happen, actions are situated in context and always in flux, situations are essentially unique, the focus is on dynamic choice in a dynamic environment. Improvisation is a model that more closely mirrors the process of human-computer interaction in real life.

Today, the stage that Laurel describes has left the confinement of the theatre and has become the city. The technologies she discusses are embedded and mobile, connected, and ubiquitous in urban environments. This paper articulates a mindset for the analysis and the design of interactive urban environments based on the process of improvisation. Following an overview of related work and the analysis of the nature of the improvisation process, a systems view of improvisation is adopted. The fundamental elements of the Viewpoints improvisation technique for theatre actors are then mapped to design stages based on a second-order systems perspective to develop a framework for the design of truly interactive environments.

2 Related Work: Improvisation in HCI

Improvisation is part of the performing arts and there has been considerable exploration within that domain over the past two decades that explore different models of relationship between performer and technology systems. The following studies focus on HCI and improvisation with a particular bias towards research involving Viewpoints improvisation theory.

In the Viewpoints AI (VAI) project [11] a person can engage in movement improvisation with a virtual actor in the context of a projection screen. The virtual actor is projected from one side of the screen, the real person’s shadow is cast onto the same screen from the other side so that both co-exist in the real/virtual space of the projection screen. As the interactor moves, the system monitors the motion using a Microsoft Kinect device and analyses the interactor movements through procedural representations using the Viewpoints theory as a reference model. It responds to the interactor movements by choosing from a number of strategies or response modes including past experience with the same as well as other interactors. The system is claimed to work with almost no predefined content.

Corness and Schiphorst in [12] explore how the interaction between theatre performers and an autonomous generative music system can use synthetic sounds of breath as a cue for an upcoming action to incorporate the performer’s sense of intuition. Breath is used as expression of intention for action/behavior. Before performing an improvised musical phrase the system emits a breath sound conditioned by the structure of the improvised musical phrase about to come. The improvisation as a whole is conditioned by performers and by environment and is modeled on some aspects of Viewpoints theory in that it monitors two aspects: performer motion and architectural shapes (lines and shapes) detected in the environment. The results obtained from interviews with performers indicate that when the system improvised and ‘announced’ the improvisation by its synthetic breath, the improvisation was perceived as more understandable, facilitating to connect with it.

In a study by Lu [13], the objective is to develop a physical vocabulary for human-robot interaction by using trained actors as models for appropriate timing and gestures. The approach proposed is to study human actors and their timing of gestures in modes of interaction and then program robots’ behavior based on time measurements taken from actors behavior.

The mechanical ottoman experiment described in [14] explores behavior types of autonomous objects and people’s reaction to them. Specifically the paper describes an experimental setup in which a hand or remote controlled ottoman is moved in ways as to prompt a person sitting in an armchair to put the feet on it, to take the feet off, etc. While a hand and remote controlled behavior in the experiment setting, the study points towards an autonomous object that would improvise its behavior in response to context and behavior of a person. In the initial experiment where the ottoman was controlled by a human, the object presented indeed an improvised behavior (even if enacted by a human through the object), the later study transferred the motion control entirely to the object by automating the ottoman, however scripting the behavior rather than enabling it to generate responsive patterns of behavior.

These related works illustrate a range of research carried out over the past years at the intersection between HCI and improvisational performance. Predominantly it is HCI and autonomous systems research that is brought into the domain of the performing arts [11–13], exploring systems that become actors on a stage with human performers. This paper proposes an inverse operation: to bring the structures from improvisational theatre and the embodied knowledge from trained improvisational actors into the context of everyday urban environments to enhance interactive processes between people and their built environment. Viewpoints improvisation theory serves not only to categorize some behaviors but is taken as a foundational conceptual and structural framework.

3 A Systems View of Improvisation

Improvisation is a process that is often misunderstood. A common interpretation is that when something is improvised, it is to make up for a lack of something. To extemporize, a common synonym is indicated by Webster’s dictionary as doing something in a makeshift manner, which in itself is referred to as a usually crude and temporary expedient [15]. To improvise, that is, is to get by in some way until the plan that was lost can be recovered.

Instead, in the arts and social sciences, the practice of improvisation has been studied extensively for some time now. Jazz is one of the domains intensively vested in this phenomenon. The development of Free Jazz in the 1950’s and 1960’s led to extensive experiments of composing in real-time [16]. Its Latin root ‘proviso’ indicates a condition attached to an agreement, a stipulation done beforehand. Together with the prefix ‘im’, improvisation indicates that, which has not been agreed upon before, which lacks a prior stipulation of contract or provision. Improvisation deals with the unforeseen, that which has not been provided or planned for, that which presents itself as unexpected.

Contrary to a common perception, the improvised act does not come out of nowhere. We underestimate the investment in attention, study and practice that is the foundation for every improvised performance. Together with an astute awareness of self and of others, an improvisation is based to a critical extent on the artist’s past practice and experience. As artists improvise, they elaborate on existing material in relation to the unforeseen ideas that emerge out of the context and the unique conditions of the performance. In this way, variations are created and new features are added every time anew [17, 18]. While not following a previously formulated plan as such, improvisation does acquire in this way some form of consistency in that it connects with what has come before in an ongoing process of repetition and variation [19].

The phenomenology of the moment for improvisational performers is as much material for their art as is their past training and practice of structures and procedures. Improvisational performers not only pick up on gestures, sequences played and acted by their fellow performers, but they also develop a capability to recognize form when it is in the making. Improvisational actors do not only read the initial movement of another actor’s hand gesture as such, but they attribute meaning to the completed form of which they see the seed. Whether that hand gesture would have been executed according to this attribution of meaning or not is irrelevant. The attributed meaning and the action based on it become the novel elements an actor contributes to the collective process. Misunderstandings and errors are constructive elements in improvisation. They are the noise that leads to the emergence of new structures.

In systems theory, emergence describes the appearance of something new. Something, of which arrival could not be anticipated, could not be expected and foreseen, but something that was born out of the interaction between previously present elements. This new arrival emerges from non-simple interactions between many different parts that interact both serially and in parallel, forming a complex system. A system that is self-organizing and which complicates boundaries between interiority and exteriority. A system, that is neither fixed nor static but that evolves and adapts [20, 21].

Adapting a systems view of improvisation makes the tension between the notions of stability and variation a productive one. The dynamics of change and variation become an integral part of the repetitions of interaction among human actors or musicians and their environment. It is through repeated interactions between them and their environment that new forms emerge.

In several instances of the literature on improvisation in different fields, parallels are drawn between the way actors or musicians improvise and the phenomenology of the spoken language, of discourse and conversations. “It’s like language: you’re talk-in, you’re speaking, you’re responding to yourself. When I play, it’s like having a conversation with myself” (Paul Berliner quoting drummer Max Roach) [17].

Also in the context of systems theory, the interactions between the constituent parts of a system and between systems are likened to the process of a conversation. Gordon Pask discusses how “structures may be designed (as well as intuited) to foster a productive and pleasurable dialogue” [22]. In this text he invests the architect with the role of designing systems instead of buildings that follow rigid typologies. Since the human occupants of manmade structures and environments change, evolve, and adapt, Pask sees an imperative for these manmade structures to reach similar capabilities to main relevant and effective. By formulating a ‘cybernetic design paradigm’, he indicates the stages of design in the context of ‘adaptive environments’ which can be synthesized into these three:

  1. 1.

    Goal and purpose: These will be underspecified since they are not fully known in evolving systems but a loose specification will provide constraints.

  2. 2.

    Invariants: Selection of system elements that present some stability (such as physical materials, etc.), and which become relevant basic properties in the man-environment dialogue.

  3. 3.

    Principles for system learning and evolution: Specification of what the environment will learn about and how it will adapt following evolutionary principles.

It falls to the designer of such a system to specify what the environment will learn about and how it will learn as well as how it will be able to evolve in terms of evolutionary principles. The designer of adaptive environments as discussed in Pack’s writing will thus not be designing the environment as such but rather the terms upon which such an environment organizes itself over time and in an ongoing interaction with its human occupants and other factors. The designer in this view loses his position as a controller, and instead instills his creations with the structural and procedural capabilities to evolve.

Conversational processes between man and his environment, as discussed by Pask, and the process of improvisation, as described above, share fundamental characteristics: they both build on elements that are present prior to the action. These elements can be structural elements or procedural sequences. Actions from past experiences are repeated and iterated under an astute awareness for the special circumstances (internal and external) of a unique situation leading to variations in the repetition. New features emerge and are added to those that came before and the systems involved (both human and machine) change, adapt and evolve – they learn.

While the development of machines capable of this kind of interaction is challenging, improvisational performers have developed their art to a high degree. Also, techniques and training methods have been formalized that provide a detailed and structured understanding of the learning process to develop the capabilities to improvise. It is on these premises that I propose to look at improvisation as a mindset to inform the design of responsive and truly interactive environments in the context of cities.

4 Viewpoints for Ambient Interaction

Among the wide range of improvisation techniques in several of the arts the technique of Viewpoints improvisation seems particularly interesting to the scope of this discussion. It was first developed by Mary Overlie in the 1970’s and later formalized by Anne Bogart and Tina Landau. “Viewpoints is a philosophy translated into a technique for (1) training performers; (2) building ensemble; and (3) creating movement for the stage” [23].

Viewpoints offers an alternative to conventional approaches to acting, directing, playwriting and design. It represents a defined procedure and attitude that is non-hierarchical, practical and collaborative in nature [23]. It overcomes the often found dynamic in acting where directors want actors to do certain things (“I want you to come in and walk across the stage like this…”). Instead, Viewpoints focuses on engaging actors as co-creators in the collaborative process of collectively making choices on stage. The choices are made in response to what the play wants in Bogart’s terms and what we can paraphrase in our context as what the situation wants. A process in which acting is experienced as a collective discovery of acts that dynamically and recursively respond to the questions that arise (or emerge) during rehearsal. View-points is based on the “trust in letting something occur onstage, rather than making it occur. The source for action and invention comes to us from others and from the physical world around us” [23].

In Viewpoints, individual and collective activity emerges in real-time, based on actors’ heightened awareness and immediate response to any of nine Viewpoints that are temporal and spatial in nature: Tempo, Duration, Kinesthetic Response, Repetition, Spatial Relationship, Topography, Shape, Gesture, Architecture.

4.1 Temporal Viewpoints

Tempo is concerned with how fast or slow an action is. It does not matter what the action is, it explores extremes of tempo - very fast and very slow - together with the medium. The attention is guided to meaning created by the tempo of an action (slow to touch or fast to grab, etc.). It brings awareness to the inner and outer tempo of an actor, to remain calm inside while acting fast and vice versa, to engage in fast collective action while keeping a slow pace at an individual level.

Duration works on an awareness of how long an action lasts, developing a sense for how long is long enough to make something happen or how long is too long so that something starts to die. Again, it explores the extremes as areas from which we tend to shy away - something that lasts too long or too short - while intuitively retreating towards a medium comfort zone.

Kinesthetic Response brings attention to other bodies in space, to their movements, exploring ones own behavior as being impacted by these external mobilities. The focus is now on when you move instead of how fast or slow and for how long.

Repetition is experimentation with letting when, how and for how long actors move be determined by repetition. Repeating someone else, someone close, someone far away, repeating off two people simultaneously. Actors experiment with repetition over time, recycling movements carried out by others in the near or distant past, reproducing forms and figures and repurposing them for the dialogue in that moment.

4.2 Spatial Viewpoints

Spatial Relationship works on the distance between actors, between bodies in motion. Taking distance from others, getting closer, close, too close. Experiencing dynamic distance and its effects.

Topography brings the actors attention to the shape within which they move. Several exercises are based on movements along an imaginary grid. Then this grid is changed, broken, distorted. Actors are asked to imagine and become aware of boundaries within which they move. They move on three dimensional grids, paint shapes on the floor with their movement, work with the shape and size of the playing space and patterns within that space.

Shape is created by the group of actors following an input from the trainer or by actors becoming aware of lines and curves that emerge. Actors train their awareness on legibility of those shapes both from within the ensemble and from outside. Shapes in space and in motion of this kind are held constant over time and through continuous slow or fast or even increasing tempo of movement. They might dissolve, actors experiment with changing location within the shape or trigger novel shapes through their action.

Gesture is explored by investigating behavioral and expressive gestures. The former are declinations from everyday life (pointing, waving, saluting, scratching, etc.) while the latter relate to the interior, expressing feeling, desire, an idea or a value. Bogart and Landau relate to them as either prosaic or poetic.

Architecture refers to the work with the spatial qualities that are already there. It puts attention on how the awareness of the physical environment affects actor movement and behavior. Actors enter quite literally into a dialogue with a room, a space to let movement evolve out of the surroundings. Architecture is broken down into 5 domains referring to solid mass (walls, floors, doors, furniture, windows, etc.), Texture (in regards to the material composition of the solid masses), Light (the source of it as well as the shadows it casts, etc.), Color (of solid mass objects but also light), Sound (that created directly from the architecture, e.g. sound from feet walking on different surfaces, creaking of a door, etc.). Actors also engage in working with objects within that space, to create with and through these objects.

4.3 Viewpoints for Interactive Environments

These nine Viewpoints formalized by Bogart [23] seem an ideal framework to both conceptualize and develop the character and the behavior of responsive elements that form interactive environments. In reference to Gordon Pask’s cybernetic design paradigm described above, we can now map the following elements from the Viewpoints method to Pask’s design stages:

(1) Goal and purpose: These will be underspecified since they are not fully known in evolving systems but a loose specification will provide constraints.

In Viewpoints improvisation training, the goal and purpose of the performed interaction is provided by the input from the trainer. The trainer guides the actors’ attention to particular Viewpoints and provides stimulating input. In Viewpoints improvisation (without trainer), instead, goal and purpose are dynamically generated by the improvised actor-actor, actor-space, actor-ensemble interactions.

In the design of interactive environments it is difficult to see goal and purpose to emerge entirely from system interactions. Depending on the context of the system, the designer will specify whether the system is intended to provide lighting or sound in an environment, whether it configures spatial arrangements, whether it provides information for collaborative processes, etc.

However, in line with Bogart and Landau’s call to move beyond the traditional director request for “I want you to walk across the stage in this way…”, the focus can be on what the system does and not on what the system does it for.

(2) Invariants: Selection of system elements that present some stability (such as physical materials, etc.), and which become relevant basic properties in the man-environment dialogue.

In Viewpoints improvisation, Architecture is the Viewpoint that refers to the space and all its physical characteristics that Viewpoint actors find already there and with its formal articulation into Solid Mass, Texture, Light, Color, Sound, as outlined above, this Viewpoint addresses the core of this design stage. In addition, the Viewpoints can be viewed as basic and invariable building blocks themselves. The Viewpoints actors engage with the world, with each other and with themselves through the nine Viewpoints as filters, and so would this be the case for interactive environments. In the constant tension between stability and variation, movement and behavior elaborated in response to shifting attention to Viewpoints and the situation at large will become temporary building blocks for more complex behavioral structures. While not invariant on a global variant, these will present some stability on a local level both spatially as well as temporally.

(3) Principles for system learning and evolution: Specification of what the environment will learn about and how it will adapt following evolutionary principles.

How systems will learn and evolve is the point that addresses the issue of the earlier distinction between truly interactive and merely reactive systems. In Viewpoints improvisation the ensemble performance evolves through the actors’ heightened awareness of their presence in relation to each other, to their movements and actions, as well as to their environment. They work with this attention through the framework of the nine Viewpoints. In the design of interactive environments, the framework of the nine Viewpoints becomes a filter to extract information from the environment and the dynamics within it in order to extract meaningful elements that trigger change in the systems setup and behavior.

Mapping the capabilities of the Viewpoints improvisation technique to the design stages for the development of interactive environments provides a path to disclosing in practice the potential that the mindset of improvisation can bring to the design of urban interactive environments.

5 Discussion

As outlined so far, improvisation and in particular the Viewpoints improvisation technique offer an interesting approach for the development of interactive urban environments. Viewpoints trained actors engage with other ensemble members and their environment in ad-hoc interactions, allowing behaviors to emerge from these very interactions in ways that can be described a second-order system. In the way that for both, improvisation and second-order systems, goal and purpose evolves and can emerge from the very interactions between the elements of the system, it can be a challenge to evaluate the effectiveness of such a system based on underspecified goals. When can we assess such a system to work or not to work? The more narrowly defined the goal is, the harder it is to valorize the value of improvisation and the wider the goal is defined, the harder it is to assess the validity of the system’s behavior.

Equally, the more critical the tasks are that such a truly interactive environment performs, the bigger the impact will be felt of the irreversibility of a system’s actions. The way an interactive system performs in an art performance, in a gallery or in a museum will have limited functional impact on people’s activities, whereas a system that supports critical tasks such as in emergency situations or even workplace situations might encounter resistance towards unpredictable behavior - much of these contexts are based on rigid planning and adherence to planning. Structure that emerges from improvisation will benefit some contexts and operations while presenting a liability for others.

Tempo is one of the Viewpoints actors use to explore ranges of speed of their actions and interactions. For musical performers it is clear how the speed of improvised performances impacts the capability to improvise. It is hard to improvise at very high tempo of performance, leading to an inclination to employ predictable patterns and techniques that were tested before. “At extremely fast tempos there is no choice but to use preplanned, repetitive material to keep the performance going. This suggests that there are upper limits to improvisation” [18].

6 Conclusion

This paper proposes a framework for the analysis and the design of interactive environments based on a systems view of improvisation and in particular the View-points improvisation technique for theatre actors.

A study of the phenomenology of improvisation reveals that it shares fundamental characteristics with conversational dynamics often referenced in second-order systems theory. In this paper we adopt a systems view of improvisation which allows us to identify key elements, structures, and procedures in improvisation that become graspable and that are meaningful for the work on human-machine interaction.

While the development of machines capable of this kind of interaction is challenging, improvisational performers have developed their art to a high degree and can be used as a rich resource to inform this process.

Specifically, the Viewpoints improvisation for actors is identified as particularly relevant for the purpose at hand due to its well articulated technique based on nine spatial and temporal Viewpoints: Tempo, Duration, Kinesthetic Response, Repetition, Spatial Relationship, Topography, Shape, Gesture, Architecture. By mapping the foundational elements of Viewpoints improvisation to the design stages for interactive environments based on Gordon Pask’s cybernetic design paradigm, this paper proposes a framework to valorize the practice and the understanding of improvisation for the development of urban interactive environments.