Beginnings and endings—dance phrase edges in an interactive dance study

Abstract

This article describes a study exploring the expressive and creative potential of dance phrase onsets and endings in interactive dance, using an artistic research approach. After describing the context and background of interactive dance and relevant perceptual issues, the technical setup is presented, both in terms of the performance area, sensors and other hardware, and software. The main part of the article is concerned with the specific movement-sound mappings and their relation to the dancer’s movement choices in four different sections in the performance that the project resulted in, entitled Beginnings and Endings - Study I. Subsequently, the process and performance are evaluated, specifically focusing on the performer’s perspective. The most interesting points from the evaluation are then discussed with reference to the presented mappings, related movement choices, and relevant research literature. Findings include that the dancer experienced an increased awareness of beginnings and endings in different sections of the performance, that the affordances of different mappings were explored in different ways and to different degrees, and that some of this could be ascribed to a dynamic between liberation and restriction.

1 Introduction

Expressive and embodied interaction has been a growing field within ubiquitous computing and HCI in the last few decades, especially as a part of the so-called third wave of HCI [1,2,3]. As technology has become more integrated into our daily lives, there has been a renewed interest in studying the role of the body in human-computer interaction which has led to new avenues of research and design, such as health and well-being, gaming, sports, and artistic practices [4,5,6,7]. As a part of the latter category, interactive dance provides a unique opportunity to explore real-time interactions between dancers and technology. This journal article describes an artistic research project that has aimed to explore the expressive and artistic potential of movement onsets and endings as part of the development of an interactive dance performance entitled Beginnings and Endings - Study I. Following a line of interactive dance practices using audio feedback to allow the dancer to play and thereby in turn be affected by the music [8], this work has explored methods and technologies for tracking a dancer’s phrase onsets and endings, and then how this could be mapped to musical features in esthetically interesting and relevant ways. In addition, the work has grown out from a set of artistic ideas relating to beginnings and (potential) endings on a more global scale, which will also be presented.

I will begin the article by presenting relevant contexts and background for the work, focusing on the field of interactive dance. Because onsets and endings are related to phenomena like pauses, segmentation and salience, I will also relate my work to these. Since this is a work of artistic research, I will also be presenting the artistic ideas and processes behind the work. Subsequently, I will go more into the details of the project, describing the technical setup, the design of the mapping between movement and sound, and how this has affected the choice of dance movements. In an evaluation of the work, I have emphasized the performer’s perspective through an interview of the dancer, which is subsequently analyzed and discussed in relation to the mappings and movement choices identified from the documentation of the performance. The article is rounded off with some thoughts about future research this work might entail and some conclusive remarks.

This article is an extension of an earlier conference paper from AudioMostly2022 [9] and partly draws upon a conference paper from the Sound and Music Computing Conference 2022 about the same topic [10]. Compared to these publications, this article goes a lot more into detail about most aspects of the project, including background and context, artistic intentions and inspirations, process and development, technical setup and solutions as well as discussion and evaluation of the results.

2 Background

2.1 Interactive dance

The term Interactive dance refers in some contexts to dance practices that engage with its audiences so that they actively participate or in other ways contribute to the performance itself.¹ In other contexts, like this article, however, the term instead refers to the inclusion of technologies to promote interaction: Interactive dance, then, refers to a performance in which dancers’ movements, actions and/or bodily processes/states are tracked by some form of sensor technology, and technological tools are then applied to transform this into some form of output (video projections, sound, text, graphics, robotic movement, etc.) that shapes the performance environment in real-time, in turn affecting the dancers’ actions [8, 12,13,14].

Variations V from 1965, by John Cage and several others is generally considered as one of the earliest examples of an interactive dance piece [15, 16]. In the development of interactive dance following Variations V, both sound, visual, and other forms of output like the ones mentioned above have been connected to dancers’ bodies in multiple ways. Although the use of interactive visuals, especially in the form of computer graphics has perhaps received the most attention, the focus in this context will be interactive dance with sound and music output, as defined by Siegel [8]. The feedback loop that is created when the sonic output affects the movements of dancers, sometimes also referred to as bidirectionality [14, 17] is at the core of what is considered interactivity [18], and has several important ramifications for the dance and the dancer. I will briefly discuss four of these.

Firstly, if the dancer is affecting how the music sounds, then it would make a lot less sense to have a fixed choreography, since one could then simply rely on pre-determined or pre-recorded music. As Birringer notes: “The space is not ‘set’ for a fixed choreography, but programmed for potential interactions and movements in which partners behave within a network of relays and responses, and in which technologies and media generate realities and perceptions” [18]. Summing up what he regards as the eleven characteristic esthetic concerns of the art form of interactive dance, Toenjes also lists improvisatory elements [13]. Such elements might, as Toenjes adds, lead to the dancers being engaged in a more immediate and complex way than in conventional dance, also giving “an ‘edge’ to each performance.”

Secondly, when the dancer controls the musical output of the system, listening becomes highly important, and might even become the main source for the dancer’s decision-making [19]. Listening in an interactive context deals with a lot more than synchronization of the inner clock. Since the dancers are the ones who, at least to some degree or in some respect, are making the music, they also have to acquire a sensibility to the wide gamut of musical traits. This was demonstrated by Brown and Paine who found that tango dancers playing music with their interactive tango system gained skills in both musical listening and musical response [20]. Listening is also more broadly related to a sense of awareness, particularly toward the kinesthetic and proprioceptive senses that interactive sonification of movements might invoke. Wilson and Bromwich see this as something where listening and body awareness comes together in what they call “audiokinetic” awareness [21]. Furthermore, as François and colleagues could report in their study, both knowledge and experience with the system, the sounds’ ability to invoke mental imagery, as well as mental focus and attention were factors related to their participants’ experienced kinesthetic awareness, and the authors highlight the importance of designing the interaction with this particularly in mind [22].

Thirdly, when dancers become responsible both for the composition of dance movements and the musical output, this can be regarded as an empowerment of the dancers, and is something that has been articulated in the literature by several authors. Robert Wechsler comments how one in an interactive dance setting can both hear and see the dancers’ phrasing, and that the dancers therefore are empowered as ever before [23]. Akerly reported how the dancers engaged in her interactive dance work Vertigo expressed a sensation of empowerment in their ability to make choices and directly live in the results of their choices [24]. This form of empowerment is nevertheless something that can be a matter of design, and therefore might vary between interactive dance environments. For example, Hahn and Bahn designed the interactive dance performance Pipika with the intent of empowerment of the main character [25].

Finally, depending on how the interactive system is designed and which sensors are used, certain properties of the dance movements can be emphasized, that were less perceivable or salient in conventional dance contexts. For example, with the use of biosensors, invisible or partly visible bodily features like heart rate, breath, or muscular activation can be tracked and sonified [19, 26,27,28,29]. Another example is the tracking of micro-movements, i.e., more or less involuntary movements when one tries to be still, which has been explored both in research and in artistic settings including interactive sonification [19, 22, 30, 31]. The focus of the current project, movement onsets and endings, might also fall into this category. I will therefore go into more detail about this in the following section.

2.2 Pauses, onsets, and endings in interactive dance

Since pauses or moments of stillness in a sequence of dance movements inevitably will insert onsets and endings into the sequence, this is a related topic in the literature about interactive dance and interactive movement sonification. For example, Masu and colleagues instructed their dancers to improvise with alternating movements with moments of stillness [32], but left out any reflection on how these are performed or experienced. The interactive sound installation still, moving focused on stillness and movement, but used an adaptive process that worked like a “kinesthetic zoom” with the system’s sensitivity increasing when the level of activity decreased [29]. Thus, the transition between stillness and movement was seen as a continuum, effectively diminishing the relevance of movement onsets and endings. Erdem and Henriksen similarly explored the contrasts between stillness and movement in their piece, Vrengt, but payed little attention to transitions between the two [19].

In Brown and Paine’s interactive system for tango dancers, movement onset features were given consideration by measuring certain characteristic “whipping kicks” and mapping the acceleration of these to note velocity for the triggered melodic fragments [20, 33]. Interestingly, they devised what they call a unified three dimensional Tango Movement-Music Perceptual Continuum, where the second dimension referred to the degree to which dancers performed their steps smoothly (legato) or sharply (staccato). Moreover, they related this continuum to the contrast between smooth or rough timbres, and to the presence of third and fourth derivative components of the position vector, where a low intensity of these indicated smoothness and a high intensity indicated sharpness or roughness. Although not directly referring to onset and/or ending qualities, one can also note how this relates to studies of fluidity as a movement quality in dance movement sonification, usually in opposition to jerkiness [34,35,36].

All in all, however, there seems to be a lack of research literature discussing these phenomena in greater depth, and this project is a response to this.

2.3 Segmentation and salience

There are also several research areas that seem to have indirect relevance to the topic in this article by focusing on perceptual phenomena related to onsets of temporal auditory and audio-visual events.

One general phenomena that seems particularly relevant here, is salience. Salience as a general phenomenon refers to the properties of an object or event that makes it stand out relative to its immediate context and therefore tend to involuntary attract our attention [37,38,39]. The concept is related to other phenomena such as scene analysis, segmentation, chunking, and Gestalt principles, and the key factor here is how perceptual contrasts and marked edges, which might be temporal and/or spatial, separate an entity from whatever surrounds it [40].

If we consider the auditory domain, we can observe how qualitative temporal discontinuities in the audio signal, and particularly between sound and silence, are important. If such discontinuities are sufficiently abrupt they can be a source for perceptual segmentation, or alternatively, chunking [41]. On the other hand, more gradual changes would tend not to cause segmentation/chunking [42]. However, segmentation/chunking can happen also without abrupt transitions in the stream of sound, e.g., with the recognition of familiar musical units or schemata, such as a motif or a metric pattern [41]. Segmentation is also something that is believed to be continuously ongoing and that takes place on several hierarchical levels, from detecting coarse, high-level boundaries, to fine, lower-level ones [43].

When perceiving dance and music together, one naturally needs to consider how the auditory and visual modalities work in tandem. Whereas the visual modality tends to dominate over the auditory modality in bimodal (audio-visual) spatial perception, the auditory modality tends to dominate over the visual modality in bimodal temporal perception [44]. Also, in experiments, researchers have found that sound can (1) increase the salience of visual events, that is, if a sound is added to a visual event, it is more likely to attract attention [45], and (2) audition can affect how visual stimuli are perceived, especially when sharp transient sounds coincide with the visual event [46]. This latter phenomenon is also referred to as auditory capture [47]. These findings, along with considerations related to segmentation, salience and accents, might have an impact on the design of sonification of movements in an interactive dance setting, to which we will now turn.

2.4 Sensor technologies in interactive dance

A number of different sensor technologies have over the years been applied to track the movement and bodily states of dancers in interactive dance, and a comprehensive review of these would greatly exceed the limits of this article. I will therefore just give a brief outline of some very general approaches and technologies. In her book Composing Interactions (2022), Marije Baalman has written extensively about different types of sensors and their practical application in interactive art, and the following of these are the most relevant for interactive dance [48].

Using Baalman’s categorization, some of the most frequently applied sensor technologies for interactive dance include:

• Force, bend and stretch sensors

• Ultrasonic and infrared sensors for distance, including marker-based motion capture

• Capacitive sensing of touch and distance

• Motion and orientation, including accelerometers, gyros and inertial measurement units (IMUs)

• Biosignals, including EMG and heart rate sensors

• Video-based systems, analyzing video images with computers

Thus, there exists a broad range of sensors that can be applied in interactive dance. At the core of the matter for most of them, except several of the biosignal sensors, is these technologies’ abilities to represent in real-time (and without too much latency) experienced features of bodily movements or states. There are a number of other aspects that could affect the choice of the sensor in each case, for example, (1) movement feature of interest, (2) the number of dancers one would like to track, (3) non-obtrusiveness, and (4) practical matters like price, availability and ease of use, robustness, and more. I will return to several of these matters in Section 4 below.

3 An artistic research project

Even though this project to considerable degree has been informed by academic research within several fields, this project was first and foremost conceived as an artistic research project. It was conducted in the belief that its creative and artistic processes will produce new insights, understandings, and products, in line with Borgdorff’s notion of artistic research [49]. Moreover, my role in the project fits well with Grey’s notion of the “practitioner-researcher”: “The role is multifaceted — sometimes generator of the research material — art/design works, and participant in the creative process; sometimes self-observer through reflection on action and inaction, and through discussion with others; sometimes an observer of others for placing the research in context, and gaining other perspectives; sometimes co-researcher, facilitator and research manager, especially of a collaborative project” [50]. The latter point has been important since the work described in this paper was done in collaboration with Seh Yun Kim, who contributed as a dancer and choreographer. Kim was recruited through a request to the Master Dance, Choreography program at the ZHdK, where she was a student at the time. She had an earlier career as a dancer at the highest artistic level, including being the principal/soloist dancer with the Dutch National Ballet, the Spanish National Ballet, and more. She participated in the project as a part of an internship class at the program, which allowed her to devote time and effort to it during her studies. Kim has been a co-creator in the sense of giving me responses and opinions of how the interaction has worked for her, as well as coming up with ideas for sounds and ways of interacting with them, during the development and rehearsal phases, which I will describe in more detail below.

3.1 Artistic ideas and inspirations

The development of the work was done during a three month residency at the Computer Music and Sound Technology (ICST) at the Zürich Hochschule für der Künste (ZHdK) in the fall of 2021. The project was initiated with the intention of gaining knowledge and practical/artistic experience through experimentation and testing, but with a goal of finishing a performance study that could be presented for an audience. The final presentation of the work was held at the Konzertzaal 1 at ZHdK, November 28, 2021. The performance was recorded on video and audio. A video of the full performance can be seen here: https://youtu.be/fkOZaT2pS-k.

The artistic ideas of concepts that were to shape this performance, as well as the inspiration from earlier pieces in the interactive dance have both been important factors in this project, and I therefore will address these in turn.

3.1.1 Seminal artistic ideas

While the focus on onsets and endings have given an overall focus to many of the elements of the process and results, related narrative and thematic threads have also oriented important aspects of the work. One central idea has been not only to look into onsets and endings on the meso-level of movement and sound phrases and events, but to increase the scope to the entire history of humanity and consider conceptions of its beginnings and (potential) endings, thereby also intending another interpretation of the title of the pice. Partly inspired by Yuval Noah Harari’s book Sapiens - A Brief History of Mankind, discussions about whether we are in the middle of a sixth mass extinction (see, e.g., [51]), and current research in biology, the topic chosen was the changing relationship between humans and other species during our (pre-)history, and the way in which the spread and increase in numbers of homo sapiens on the planet gradually have made a huge number of species extinct, something that might end up in threatening our very own existence [52,53,54]. The original idea was then to present different imaginary stages of humanity’s development and its different approaches to interacting with its environment, starting with a more perceptive and humble approach, and gradually becoming more assertive, invasive, and threatening. Along with this development, I intended to apply movement-to-sound mappings that were pretty clear and intuitive in the first interactive section, and then progressively present more and more complex mappings, thus metaphorically projecting humanity’s increasingly complex interactions with itself and its environment through history.

Initially, I imagined a nine-part interactive piece with preceded fixed media prelude, postlude and interludes, where each section represented one of these stages, starting with the most distant past and ending in an imagined future. The original plan was then to present a preliminary version with sketches for some of the sections during the fall of 2021, and then to finish the piece later. Four of the sections were eventually considered interesting enough to present to an audience, and these had the working titles Discovering, Imitating, Wheels, and Metals. Whereas a short version of the originally imagined Fires section was included as an ending of the performance, as seen in Fig. 1, another section called Gathering was considered unfinished and therefore abandoned during the last weeks of the rehearsal period.

Fig. 1

Sections in Beginnings and Endings - Study I. The blue rounded square encircles the sections described in this article

A way of putting the issue of the extinction of species to the fore without presenting a too obvious narrative was to use recordings of read or sung species names in Latin in the interactive instruments. With the aid of resources online like The IUCN Red List (https://www.iucnredlist.org/) and The World Wildlife Funds species directory (https://www.worldwildlife.org/species/directory), I selected 21 extinct species names in Latin, where humans were among the most likely causes of extinction due to activities like hunting, agriculture, and keeping live stock, and where the species names also had a connotative and/or musical potential, for example Eclectus infectus, Mekosuchus inexpectatus, and Nestor productus. I will elaborate further on how this was used to generate the sound material in the piece in Section 5.1.

3.1.2 Inspirations from the interactive dance repertoire

There are a number of performances I have studied on video that have been inspirational for the focus on dance phrase onsets and endings. These both deal with the active use of contrast between stillness and movement and articulation of different onset and ending qualities in the dance phrase. Three very good examples are Seine hohle Form by Palindrome Dance Company [55, 56], Two Pandoras by K.Dance and Stocos [57, 58], and Blood Music by Roosna and Flak [48]. In these pieces, one can observe how in some sections dance movements are interspersed with stillness to create temporal incisions and accents, which appears to have the effect of highlighting the interactive nature of the expression for an observer.

3.2 Artistic development

The process from initial ideas to final performance had several affinities with an iterative design process [59]. It took place as an iterative cycle between technical/musical development that was done alone, and practice and evaluation sessions that was done together with Kim. While the technical and musical development was mostly done in a music studio setting, most practice sessions were done in a dance studio with appropriate flooring, but with a simplified technical setup, to speed up the setup process and allow for more practice time with the limited availability of those studios. Typically, a practice session would begin with a brief presentation and explanation of the latest development, which was then followed by open exploration by the dancer. Kim would try out how the different interactive environments responded to different movements, and then try out choreographic ideas that could work with these. Subsequently, we would then typically discuss aspects of the interaction like:

• What was the general experience of the interaction?

• How were the sounds related to experienced feelings?

• How did the emotions affect the movements?

• Was anything missing, felt awkward, or difficult with the interaction?

• How would the interaction work in the context of the whole piece?

Based on these discussions, I would take hand written notes, which I would then bring with me into the development period up to the next session. These hand written notes were sometimes interspersed with sketches and drawings. Along with this, I kept a digital log in this period, mostly to reflect on different challenges related to the day-to-day work and to keep track of the more technical aspects of the development process. I would either work on purely technical issues, work with sound production, editing, processing, organization, and spatialization, work on new movement-sound mappings, or the improvement of those already tested on the basis of our practice sessions.

After the performance, I did an interview with Kim about the her experiences to bring forth insights and reflections related to the performer’s perspective that could be applied in the evaluation and further development of the project. The content of the interview will be discussed in more detail in Section 6 below.

4 Technical solutions

The technical setup used for making the study included movement sensors, equipment for receiving wireless sensor signals, computer hardware, and software, including existing frameworks as well as software developed in the process, and on-site facilities at the ZHdK in the form of performance and studio spaces equipped with multi-channel loudspeaker setups. I will go through these components in turn.

4.1 Performance area and loudspeaker setup

When planning the final performance of the study, it was considered important to make it into an immersive experience, using the 3D sound facilities at the ZHdK along with relevant spatialization techniques. The Konzertsaal 1 (Fig. 2) was considered a highly fitting venue, having both a relatively large floor space with its 174 square meters, and an existing loudspeaker setup of 23.4 channels. The layout can be seen in Fig. 1, with 8 speakers on floor stands, 10 mounted on the walls, and five mounted in the ceiling. In addition, the four sub-woofers were placed in the corners of the room.

Fig. 2

Konzertsaal 1, ZHdK. Loudspeaker layout, performance, and control area

To give the audience the most immersive experience along with a high spatial resolution, we decided to locate the audience close to the so-called sweet spot in the center of the room and to locate all chairs with the back toward the center of the room and faces pointing out from the center. We could then define the performance area as the whole area surrounding the audience, with an exception of a small control area in the back of the room (see 2). The circular shape of the performance area could also work as a metaphor for the circular image of the earth as a “playground” for humanity that fits well with the topic of the piece. Finally, we decided to also include two perpendicular passages through the chair of the audience area, thus making it possible for the dancer to cross through it.

4.2 Choice and placement of sensors

The choice of sensors and their placement on a dancer’s body will naturally have an effect on the interaction possibilities and types of movement that an interactive dance system affords (see, e.g., [17, 60]). In Section 2.4 above, I reviewed a wide range of sensor technologies that have been applied in interactive dance and discussed some of the issues that might be relevant when choosing a sensor for a particular project. For this project, a combination of artistic, technical, and practical matters played a part in the choice of sensors.

As for the artistic matters, it was crucial to have sensors that represented the dynamic qualities of movement well, since these are the most important when focusing on the onsets and endings of movements. Therefore, having relatively low latency and high sampling frequency was something that I would prioritize.

As it turned out, other practical matters played a significant role in the choice of sensors for the project. Having only three months for developing artistic concepts, technical solutions, recording and treating sounds, programming sensor communication and sound processing, and dealing with all other practical issues, made time a critical factor. Thus, ease of use and my own technical familiarity with the sensor technology became highly important. Moreover, the constraints of having to move between different performance spaces in the rehearsal and performance phases made it important to have a light-weight setup that was easy and fast to set up and pack down.

Ultimately, NGIMU inertial sensors from X-IO (https://x-io.co.uk/) and Myo sensor armbands from Thalmic Labs were chosen for the project. I had worked with both sensors in other projects before, so putting them to use in this project wouldn’t require learning to use any new hardware or software. As for the NGIMUs, they are battery powered, have a size of 56$\times$39$\times$18 mm, a weight of 49 g, and are relatively unobtrusive for a dancer to wear and move around with. Moreover, they are fitted inside a hard plastic casing that makes them withstand shocks, contact pressure, and moisture from sweat fairly well. When once configured with a wi-fi router, they will automatically connect when you turn them on, which makes them very easy and fast to use and set up. In terms of the data they provide, they have an accelerometer, gyro, magnetometer, barometer, and thermometer, as well as an onboard AHRS sensor fusion algorithm that provides orientation data (quaternions and Euler angles), linear and earth acceleration, and a rotation matrix, which are all transmitted over UDP using the Open Sound Control (OSC) protocol at data rates up to 400Hz. Although there are no official measurements of latency, I have informally tested it and found it to approach 8 ms in optimal conditions. Through testing we also found the range to cover the performance space without any issues.

The Myo armbands were primarily chosen for their electromyographic (EMG) sensing possibilities, which, as mentioned above, was interesting when it came to especially the onsets of movements. The sampling rate for the EMG part of the armband was 200 Hz [61], and the measurements seemed very responsive in allowing for the detection of muscular activation produced by even very small movements of the hands and fingers, as well as muscle tension without movement. The armbands are connected to a computer via a small Bluetooth dongle. We also found the Myo sensors to have sufficient range for our performance space.

Fig. 3

Placement of sensors on the dancer’s body

Fig. 4

Hardware setup

Even though the original plan was to locate sensors on the arms, torso, and legs, the sensors on the legs were unfortunately abandoned due to time constraints, with the final placement as seen in Fig. 3. This naturally led to a focus on the upper body throughout the process, something which I will get back to in the evaluation and discussion sections below. Nevertheless, the placement of the NGIMUs on the dancer’s wrists was seen as good for the expressive use of arms and hands, both regarding the choreographic and the musical aspects. The third NGIMU sensor was placed on the central upper chest. This placement was seen as interesting since it would allow for getting both torso direction and angle in a very straightforward way. The Myo armbands were placed on the upper part of the forearms, as devised by the manufacturer.

4.3 Hardware

The sensors were integrated in a hardware setup as seen in Fig. 4, and connected to a laptop computer via Bluetooth and Wi-Fi as discussed in the previous paragraphs. Due to the relatively heavy computationally demand on the system, including real-time audio signal processing and spatialization, the software was distributed over two computers communicating using OSC over an ethernet hub. A MIDI-controller, Faderfox UC-44, was connected to Computer 2 to enable hands-on activation of different sections of the work along with control the volume of different components in each section. In the final performance, Computer 2 was connected to the Dante audio interface at the venue to route the sound output to the 23.4 audio channels.

4.4 Software

The software used in this work falls into two main components: (1) Sensor data management and processing, and (2) Sonification and spatialization, with these two components running on computers (1) and (2), respectively. A simplified illustration of the structure and signal flow of the software is given in Fig. 5.

Fig. 5

Structure and signal flow of the software. Sensor signals are depicted in blue, and audio signals in red. The number of arrows representing audio signals does not represent the number of audio channels

4.4.1 Sensor data management and processing

Sensor data management and processing was programmed in Max (https://cycling74.com). The main patch included a simple GUI to set up and connect the sensors. With a battery capacity of about 2–3 h for the NGIMU and somewhat more for the Myos, it was necessary to have monitoring of battery levels in the GUI. Finally, a button was added to be able to set the point of 0 degrees azimuth for the NGIMU chest sensor with the directions of the room.

Several subpatches were embedded in the main patch to receive, monitor, process and transmit sensor data to Computer 2. While data from the NGIMUs could be received directly in Max with the udpreceive object, the Myo data was only available as OSC data after an initial procedure using the proprietary Myo connect software and the Myo package for Max (https://github.com/JulesFrancoise/myo-for-max). Simple treatments of the data from the five sensors, like derivation, summing, filtering, conditional operations, and normalization were done with objects from the MuBu (https://forum.ircam.fr/projects/detail/mubu/) and RIoT (https://ismm.ircam.fr/riot/) packages from IRCAM. More details about the sensor processing will be presented in Section 5 below.

4.4.2 Sonification and spatialization

The sonification part of the software along with some additional sensor data processing was developed in the music computing system Csound (https://csound.com) and ran on Computer 2. The Csound code was wrapped in two identical VST plugins generated with Cabbage, each with 16 channels of audio output (https://cabbageaudio.com/). These two plugins were running on separate tracks inside the digital audio workstation software Reaper (https://www.reaper.fm/). Each instance of the plugin was then linked to either the left or the right sensors receiving sensor data over the OSC protocol from Computer 1.

The last stage of the sonification was the spatialization, which was done using the ICST Ambisonics 4th order encoder and decoder plugins (https://ambisonics.ch/page/icst-ambisonics-plugins). The 16 audio channels of each of the two plugins were routed to individual objects in the ICST encoder, which was set to encode with distance simulation. The decoder then received OSC messages from the sonification instruments controlling the individual objects’ azimuth, elevation, and distance. The output from the decoder was then routed to the 23.4 output channels over the Dante protocol.

5 Development of movement-sound mappings

Using the presented technical setup, the performance consisted of one fixed media section acting as an overture to the piece, and four interactive sections. For these sections, the movement-sound mappings are highly important for the artistic result. I will therefore describe the development and main features of these for each of the sections, supported by accompanying demonstration videos. Prior to this, I will briefly describe the conception and production of the sound material used in the piece, since most of the mappings were directly dependent on the nature of the recorded material. For each section, I will also make some observations about the movement choices and elements.

The reading of this section should be accompanied by watching demonstration videos of the mappings in each section (links will be presented in each section), and the video documenting the whole performance (available at https://youtu.be/fkOZaT2pS-k).

5.1 Sound material

An open call for participation to generate vocal material resulted in the recruitment of six vocalists, five female, and one male. All were recorded in the sound studios at the ZHdK, and subsequently awarded with a gift card of 10 CHF for their participation. They were instructed to articulate the species names or syllables in those names with different vocalization styles and qualities, including singing, whispering and speaking.

Additionally, metal sounds were considered fitting to evoke associations to humanity’s extraction and transformation of the earth’s resources into technologies of different kinds. A number of different metal objects from the ZHdK metal workshop were recorded.

The sound material from all sessions were thereafter processed with noise removal software, aligned in gain, and then edited into audio files containing shorter segments with durations mostly between 0.1 and 5 s. This material was then categorized and sorted according to sound source, text content (when applicable), voice quality (when applicable), and intensity.

5.2 Section 1: Discovering

The mappings in this section attempted to focus on small movements and movement onsets that could otherwise seem to have little significance. This gave a high degree of control and consistency in the sonic output, and a clear causal link between the performer and the sonic output, something which according to Toenjes can lead to a greater enjoyment and understanding of interactive dance works if occurring near the beginning of the work [13]. The mappings were further constructed to let high-salience onsets be followed by smooth continuations of indefinite length and with sonically interesting modulations and to end in a somewhat less salient way. After a brief description of the most important technical details and signal flow of the movement-sound mappings, I will give a brief account of the movement choices. It is recommended to accompany the reading of this section with the demonstration video found on https://youtu.be/BjW4027YKko?feature=shared and 0:00-4:15 in the performance video (link above).

5.2.1 Signal flow and technical details

Figure 6 shows the signal processing steps to achieve a mapping with very abrupt onsets and smooth continuation.

Fig. 6

Signal flow delineating the mapping between movement and sound in Section 1, Discovering

The raw EMG data used in this section (1) was filtered with a Bayes filter from the MuBu package in Max (2). The filtered values were then averaged (3).² The second derivative of the average (4) was compared to a threshold set empirically, which created a trigger if the value was above it (5). This trigger, in turn, activated an attack envelope (6a). This envelope was cross-faded (7) with the smoothed average of the EMG sum (6b) thus giving it a gradual onset. Using this cross-faded value to scale the volume of a phase vocoder-based sound file playback engine³ (8) with an arbitrary controllable time pointer, it was possible to prolong the sound files indefinitely as long as the muscle activity was held above a certain threshold. When the averaged value fell under a threshold, a relatively brief (50 ms) fade-out envelope would be imposed on the sound resulting in a relatively instant, albeit not marked, ending of the sound.

Taken together this can technically be interpreted as a convergent or many-to-one mapping, as described by Hunt and colleagues [62], although including the additional mappings described below would position it as a many-to-many mapping. Admittedly, mapping threshold crossing of sensor values to initiate musical events like the playback of samples is definitely not something new (see, e.g., [63, 64]), but the highlighting through choice of sound samples and a fast attack envelope, in this case served the purpose very well.

The mapping was further developed to allow Kim to “infinitely” prolong the sustained vowel part of the sound, and then adding sonic interest by mapping some types of modulation and effects to the sensor data (not depicted in Fig. 6):

• Muscle activation moves time pointer through the sound file

• Muscle activation creates a gentle modulation of pitch

• Any large movements of the arm gives “chorus” effect and adds reverberation⁴

• When the continuous sounds are prolonged over many seconds, a duplicate of the sound file transposed an octave down is gradually faded in

5.2.2 Movement choices

First of all, the way that the triggering thresholds were set for the EMG sensors, made it necessary that Kim had to start from a fully relaxed state to reliably trigger the gentle and short phrases planned for the opening, something which caused some frustration during the practice sessions. After trial and error, she soon learned that the best way to do this was leaning the torso forward and then with protruded shoulders letting the arms hang straight down while fully relaxed. Thus, both the placement of the sensors and the design of the mapping clearly affected the way she moved.

Moreover, Kim also learned how letting her arms down while relaxing her forearm was a reliable way to end a phrase, and in the performance, this can clearly be observed at many moments (see 2:42, 2:54, 3:08, 3:15, 3:51, and 3:56). In fact, this contributes to giving a lot more audio-visual salience to the endings of the phrases than was originally intended in the mapping. In other phrases, however, onsets and endings would be performed in less salient ways (2:57, 3:43-44, 3:52-53).

5.3 Section 2: Imitating

In the second section, the mapping aimed to enable close to imperceptible onsets and endings, partly as a sort of antithesis to the first mapping. Again, it is recommended to confer with the video demonstrating the mappings (https://youtu.be/Y0FabW2zQNk) and the video from the performance (4:15-9:40, link in Section 5).

A central strategy for the first aim was to introduce a “carpet” of sound that was always present independent of any movement, and then construct the mapping so very gentle arm movements would bring this carpet gradually to the foreground. In the same manner, as in the first section, recordings of sustained single syllables were used, but this time with very gradual onsets. The interactive instrument played groups of three different layers together, giving a hint of a choral quality, which also added richness to the sound. A playback engine with time pointer control made it possible to prolong the sound indefinitely. Here, sounds with temporal variations in the form of sibilants following vowels, and sounds with gliding pitch were used, so that a moving time pointer through the files would create interesting modulations of the sounds.

5.3.1 Signal flow and technical details

Fig. 7

Signal flow delineating the mapping between movement and sound in Section 2, Imitating

The signal processing of the mapping for the gradual onset and ending part of this sonification is seen in Fig. 7. A general movement intensity or activity value was calculated using the absolute value (3) of the delta (2) of the quaternions (1) from the two NGIMU sensors on the dancer’s wrists. These values were summed (4) before being treated with a simple noise gate (5), implemented as a linear transfer function set to 0 up to the noise threshold. This activity value was sensitive even for very low-velocity movements. By adding this to a small constant (7) which was used to scale the volume of the sound file (8), the sound would always be playing. This, combined with the sensitivity to slow movements, made it possible to achieve very soft, and gradual onsets and endings of the sound, thus exploring the minimal range of salience. To achieve timbral variation and interest throughout the interaction, the stack of vocal sounds were rotated every time the arms of the dancer were still. Moreover, the dancer could directly control the time pointer into the sound files by changing the azimuth angle of each of her arms.

To obtain high-salience accents, a computational model of salience called the rarity index was implemented in Csound, based on Niewiadomski and colleagues [65]. The algorithm was slightly modified to only give high output values only when there was a large positive change in input magnitude, i.e., from zero or a lower value to a higher value, and not the other way around. This rarity index value was then mapped to a pitch transposition and spectral arpeggiation effect, including the volume of the effect. This mapping worked as a highly expressive means for the dancer when going from relative stillness to the upper-intensity range of her movements. When the dancer produced a sudden burst of activity with her arm, the sounding result was almost like a short phrase played on a flute with a marked timbral and dynamic contrast compared to the relatively quiet vocal layers.

5.3.2 Movement choices

Looking at the performance video, one can observe how Kim’s movement choices were definitely affected by the mappings. First of all, to highlight the very gradual nature of the onsets, she decided to start this section lying on the floor, curled up like a ball, and with only one arm extended (4:33). She would then move her extended arm back and forth very slowly to play the sound a few times before “uncurling” and engaging her head and other around 5:02. Several other times later in this section (6:62-7:02, 7:45-7:50, and 9:06-9:29) she engages in the same very slow movement with her arms, thereby bringing forth the imperceptible onsets and endings of the vocal layers.

In between these very slow movements and brief moments of stillness, Kim explores a range of different movement dynamics and velocities, which does not necessarily deal directly with onsets and endings, but which nevertheless contributes greatly to the overall artistic impression. For example, we can also clearly observe several moments where she activates the salient accents (5:52, 7:22-7:27, and 8:56-8:59), often through raising and/or lowering her arms, and sometimes also combining this with pirouettes. Lastly, we can also observe how she, especially in the first half of this section, keeps her arms still, to change sounds in the vocal stack, as explained above.

Apart from the movement choices that more or less directly can be related to the movement-sound mappings, it is also clear how Kim creatively transcends the affordances of the interactive setup and adds choreographic elements more freely, especially in the latter half of this section. For example, she often engages in complex sequences of steps and gestures, combining pirouettes with arms raised or to the sides, bending knees slightly or stretching the whole body toward the ceiling. Also, the latter half of the section is highly dynamic in the sense that it varies a lot between a very quick execution of the movements and almost being still.

In sum, this sonification explored the high and low ends of movement salience, combining a “carpet” of vocal sounds for the very slow and gradual movements, and more dramatic spectral effects for the highly salient movements. Other movement components can be ascribed to either aspects of the topic of the piece or Kim’s creative and expressive improvisation of elements less related to the mappings.

5.4 Section 3: Wheels

This section was intended to be dominated by rotational movement on many levels: both for Kim’s body and for her movement in the performance area surrounding the audience. Together, this was intended to metaphorically project how the invention of the wheel allowed humans to increase their mobility and to travel to new places in the world. Also to have a much less direct movement-sound relationship than the preceding sections. The primary sensor input was from the NGIMU sensor on the torso, whereas the NGIMUs on the arms took on more secondary roles. As in the preceding sections, vocal sounds were used, but this time without the sustained articulation with the species names being presented in a recognizable form. Instead, single syllables of some of the species names were used, performed with three different vocal qualities: (1) sung, (2) spoken, and (3) whispered.

Again, reading should be accompanied by watching the demonstration video (https://youtu.be/vDu9aU7bCuY), and 9:40-13:30 of the performance.

Fig. 8

Signal flow delineating the mapping between movement and sound in Section 3, Wheels

5.4.1 Signal flow and technical details

The signal processing of the onset part of the movement-sound mapping in this section is shown in Fig. 8. (1) The y-axis from the gyroscope on the NGIMU provided a value corresponding to the rotation of the dancer’s torso. In (3), this value was multiplied with a constant with a very low value representing a momentum weighting factor, m, before (2) being fed back one step in the signal chain. Subsequently, this value was multiplied by a second constant representing a friction weighting factor, f in (5), before once more being fed back one step in the signal processing chain in (6). With the constants, m and f being properly tuned empirically, the steps (2)–(6) then effectively worked as a simple simulation of wheel rotation, with the angular velocity of the rotation of the dancer giving a spin to the wheel, and then a subsequent gradual deceleration due to friction. If, as shown in step (7), this value was over a minimal threshold, T, the value controlled the frequency of a metronome, (8). This metronome was in turn used to trigger sound files. This allowed the dancer more freedom to move independently of the sound, since after a forceful and swift rotation of the torso, the sound would continue for quite a while before coming to a halt. A rotation in the opposite direction of the current rotation would effectively act as a brake. Thereby, if at any point, the dancer wanted to kill the sound, she had to make a rotation the opposite way and with the same force. To give more interaction possibilities two additional mappings were added: (1) bending of the torso would affect sound duration, and (2) arm height would affect the vocal quality (sung, spoken, whispered) of the sample.

5.4.2 Movement choices

In the performance video it can be observed how Kim actively takes advantage of the possibilities inherent in the mappings in this section. For example, she sometimes is able to start the triggering of vocal sounds through first rotating in one direction and then the opposite, just enough to stop the sound (9:46-9:52, 11:07-11.12). Other times, however, the rotation is not quite matched and the sound keeps on playing after she has stopped (10:10-10:20, 11:27-11:32). She also has a part (10:33-10:52) where she bends forward, thereby shortening the vocal syllables while rotating first very slowly, and then faster. The mapping feature changing the quality of the voice when raising the arms is also something that Kim applies very effectively in several phrases: At both 11:40-11:48 and 12:03-12:13 she raises her arms at the very end of a phrase, and in that way giving the phrases an emphatic ending.

Compared to the previous section, there are relatively few choreographic elements that are not related to the mappings. The most striking of these is perhaps when Kim is running a full revolution around the audience (11:48-11:58). Different combinations of steps in between all the pirouettes and two moments where Kim is close to perfectly still for a few seconds (11:12-11:18, 12:11-12:13) can also be mentioned here. While both these moments of stillness contribute to emphasizing the ending of the phrase and the onset of another one, the latter is especially emphatic since it has both stillness and two accentuated vocal sounds.

5.5 Section 4: Metals

The goal of the final sonification was to achieve a rich, complex, and varied sonification of dance phrases with two types of high-salience endings. In this section, the endings of the movement phrases were the focus of the interaction, and two contrasting types of endings were included in the design. The first type of ending was achieved with a sudden muting of the sound, whereas the second was achieved using an accent sound of the impulse-resonance type according to Smalley’s spectromorphological typology [66].

A demonstration of the sonification can be found at https://youtu.be/3yd6XDe7zVU.

5.5.1 Technical details and signal flow

A simplified signal flowchart for the mapping is shown in Fig. 9. The calculation is based on delta quaternions from the NGIMU on the torso (1), as shown in points 1–6 of Fig. 7. This value is then sent to a switch (2), which will compare it to a threshold value and decide whether to mute or make an accent at the ending of the sound: it will proceed with the left part of the flowchart if lower than the threshold, or the right if higher than the threshold.

Fig. 9

Signal flow delineating the mapping between movement and sound in Section 4, Metals

The first three steps (3)–(5), are similar for both paths: The linear acceleration (3) values from the NGIMU sensors on the wrists are sent to the intensity object in the Max software.⁵ If this value is above a threshold, it will produce a trigger according to the chosen path. For the muting path, the peak intensity (6) will be scaled (7), and used to set the duration of the muting (8a) before being applied as an envelope of the sound file (9). For the accents path, it will simply trigger a sound file with an accent sound. Thus, the dancer could end her phrases with a forceful movement to produce these two kinds of endings, with the muted ending when she had been relatively still for the last few moments, and an accent ending when she had been moving more.⁶ The trigger of the accent would also initiate a muting of the metal sounds for a few seconds.

As for the general design of this sonification, it is a form of granular concatenative synthesis based on the author’s earlier work [68, 69]. The sonification uses a great number (200+) of short sound files, in this case, recordings of different metal objects, organized to make up a timbral continuum. These sound files are triggered by a metronome whose frequency is controlled by the delta quaternion-based intensity parameter (implemented as in Fig. 7, pt.(1)–(6)). Additionally, the horizontal orientation of the NGIMU sensor on the torso was mapped to the selection of the sound file.

5.5.2 Movement choices

There are several choreographic elements that can be related to the mappings in this section. Especially, in the first half minute (13:42-14:15), the temporally distinct arm gestures coupled with the relatively direct movement sound of the concatenative synthesis gives the impression of playing or controlling the metal sounds. Later in this section Kim also triggers the accents by making forceful gestures following semicircular trajectories with one or two arms, keeping the elbows straight, frequently also combined with turning (e.g., 15:06-07, 15:13-14, and 15:39-40).

Choreographic elements and the sounds they produced were not always related in a way that was intended in the mapping design, and this was to some degree the case for the mapping of the endings. Firstly, even if the accents were designed to work as phrase endings, they would not necessarily always take this function, but instead often also function as within-phrase accents. For example, in the phrase starting with Kim having both arms raised, then performs several of these cutting gestures while turning (incl. one pirouette) before she ends bent forward with the arms crossed (15:35-46), there is a total of three accent sounds, with only one of them taking place at the end of the phrase. Secondly, despite how the muting ending mapping was demonstrated and to some degree practiced in rehearsals, I have had trouble finding examples where Kim actively seeks to perform this variant.

There also seem to be several movement choices that have been only partly related or not related to the mappings. One example is her use of gestures where one or two arms are thrust linearly away from the body with the hands in a stretched out and flat shape. This gesture can resemble a form of cutting, and especially when Kim makes two consequent gestures of this kind where her arms end up crossing, this association is particularly strong (e.g., 15:20-25, 15:35-38).

As was the case in the Wheels section, Kim also here frequently applies pirouettes together with different combinations of steps in many parts of this section, especially after the first one and a half minutes or so. Other seemingly mapping-independent movements stand out due to their strong associative potential, namely the brief part where she first holds one hand in front of her eyes, and then her mouth, while the other hand, at least for the first part, is stretched out in the direction where she is walking, while at the same time doing a kind of wavering movement with the hand.

Taking these observations together, the movement choices in this section are perhaps less bound to the design of the mapping compared to the other sections, with some deviations from the intentions of the design.

6 Evaluation and dancer feedback

Depending on the research questions and approaches, interactive dance projects can be evaluated from different angles and with different methods. In the same way as, e.g., [19, 70, 71], I have chosen to focus on the dancer’s experiences and feedback during and after the project in the evaluation of this project.

6.1 Interview and thematic analysis

To collect Kim’s experience of her participation in the project, I performed an interview of her on Zoom, 3 weeks after the performance. The interview was then transcribed, and imported into NVivo (https://lumivero.com/products/nvivo/), a computer-assisted qualitative data analysis software (CAQDAS). I then performed first a preliminary coding of the interview and then re-coded the whole interview with the aim of refining code categories and achieving consistency. After further analytical steps based on principles from thematic analysis [72], several themes emerged:

• 1. Using stillness

• 2. Onsets and endings

• 3. Understanding how things work

• 4. Thinking as a musician

• 5. Being led by the sound/interaction

These themes are often partly interrelated, but I would still like to go through them one by one in the following and to exemplify each of them with statements from the interview, as well as relating some of them to my own observations and notes from the process.

6.1.1 Theme 1: using stillness

A recurring theme in the interview was the use of stillness in Kim’s performance and how it related to any musical stillness. The work in this project thereby went against the conventions she had grown accustomed to in her earlier career, and this seemed to invoke some discomfort in her, especially since it was also often directly related to the music:

Looking back, I realize that when I am still the sound is many times still, so I think that kind of affected me in a way that I felt like I had to do something because that I know that I am the only person who can stop this stillness or silence, and as a performer, I’m like (gasp) - the audience is not getting anything.

Kim expressed, however, that during the work with this project, she gradually got accustomed to using stillness as a part of her artistic expression:

[...] if I’m still, [the audience] is not really getting anything else, at least that’s what I thought. [...] And later on, when I was on purpose making the stillness, that was then a different thing. Then I felt that I was giving people a little break, or time to breathe.

It was interesting to note also how Kim experienced how her stillness took another and interesting role when it wasn’t corresponding with musical stillness, as was the case in the Wheels section:

I think the third part was interesting because it was not really matching the movement. And as well, coming back to the stillness, my stillness was being seen more because of contrast of the music [...] So, I start the music, and all of the sudden I stop, but the music is continuing, and that was an interesting thing to experience.

Moreover, she mentioned how this situation of having a mapping where she initiates a movement, and then that it takes some time before it disappeared, enabled her to do “much more diverse things.”

6.1.2 Theme 2: onsets and endings

The second theme obviously relates to the previous point and directly to the focus of the project. In the interview, it seems that Kim has become more aware of the substructures in dance through working with the project, i.e., the onsets and endings of single movements, phrases, and sounds:

Whenever I did some small movement, it was a new beginning for me, because the music is appearing with me or I am making it, so it was actually a slightly weird sensation that I have. [...] each movement felt like I was starting again, starting again. I think there were a lot of beginnings and as well as endings. But I guess that we did it in a way so that [...] each movement, each sound — there were a lot of beginning and ends, microscope-wise.

Interacting with sound in the way that Kim did in this project also engaged her a lot, something she several times expressed in the practice session. In the interview, one specific positive experience was related to the ending accents of the Metals section:

I felt really cool actually in the last beat, [...] in the very last section of Metals, it was a very cool feeling to do it, it was like hitting a tennis ball and really hitting it on the right place many times - it feels good.

Getting to know the sensors and the mappings in the different sections also made Kim reflect on what happens just before a movement is initiated:

It’s not actually the sound that gives the beginning, it’s the logic of making the sound that starts the beginning of the movement. It definitely shapes it - and the ending as well.

Thus, “the logic of making the sound,” i.e., Kim’s cognitive understanding of the interaction, is thereby something that has a lot to do for how she would execute a movement to produce a certain sound. This brings us to the next theme.

6.1.3 Theme 3: understanding how things work

This theme partly dealt with insights of how Kim’s understanding affected her action, and partly reflections on how all this knowledge affected her cognitive load. As for the first of these, she expresses quite poignantly how her understanding of how the interaction works directly affects her choice of movement:

I guess the sound comes as a result, but the understanding of how to make the sound and how to stop the sound, it actually makes what kind of movement is gonna happen.

As for the second point, it was clear that Kim experienced that all this knowledge put a strain on her cognitive processing. Here she is explicitly referring to the Imitating section:

It was difficult, because I was very busy [...] because I was busy about the concept of making the animal-like mimicked movements, I was busy with the sounds that were coming from each speaker and the background, and the sound that I was making, how I was stopping, and how it was also getting into a different level. [...] So it was a very busy moment there for me.

6.1.4 Theme 4: thinking as a musician

The fourth theme also dealt with mental processes, but this was more directly related to the sound and the musical processes in the piece. It was evident from the interview that Kim became aware of her own shaping of the dynamics of the musical phrases she generated with her movements, and how she would more or less consciously start to think like a musician:

Well, I think, you know the most way of common music progression, you know coming up, and then slowing down, I think subconsciously I was trying to mimic it. Nobody told me to, but I was always starting slow, and then going up, and then slowing down a little, to go back to the place and finish it at the highest point, or sometimes going down and then closing it.

Although Kim here assumes this musical thinking could be subconscious, she also reports that parts of this musical thinking were conscious, since she wanted to repeat a certain approach she found valuable:

And there were some moments that I really liked, so I really tried to repeat that in each time.

6.1.5 Theme 5: being led by the sound/interaction

This theme dealt with Kim’s descriptions of her experience that her choreographic or improvisatory choices were, at least partly, based on the interactive instruments and the sound that came from them. This theme, then, is therefore related to the third theme, but focuses more on the effects knowledge and understanding have on her movements. In the Wheels sections, for example, she felt that her movements were greatly affected by the movement-sound mappings:

I would never have rotated if I didn’t know that rotation would make sound, you know.

Also, the placement of the sensors on her torso and arms, clearly affected the degree to which she was activated her legs:

[...] a lot of thing are on the upper body, there was not so much movement in the legs. [...] in ballet we use a lot of legs, and really those kind of movements didn’t happen. Because it was kind of like, either make sound that is not intended, or disturb other elements that is happening within the rules.

In other words, she experienced that using her legs extensively would interfere with the controlled and intentional upper body movements, steered by her knowledge of the interaction, which were also heavily linked to the sonifications.

7 Discussion and future research

As stated by Hank Borgdorff, “artistic research — embedded in artistic and academic contexts — is the articulation of the unreflective, non-conceptual content enclosed in esthetic experiences, enacted in creative practices and embodied in artistic products” [49].

Although a lot of the value of this project has lied in gaining experiences, skills, movements, and sensibilities that are non-conceptual, embodied, and pre- or unreflective, like Borgdorff notes here, I would still attempt to tease some insights and reflections that hopefully could be of interest to communities that are engaged in interactive dance, and hopefully also beyond.

One very basic and obvious observation that goes throughout this project is how there are two opposite perspectives to the topic. On one hand, there is one perspective where one can see “dance phrase edges,” like the title of this article refers to, as one phenomenon with some shared properties. On the other hand, onsets and endings are in many cases both technically and qualitatively quite different from each other.

Let’s start with the first. We observed in this project how the synchronization of movements and sounds with a relatively precise temporal placement could increase the salience of both onsets and endings as temporal events. Both when a marked sound accompanied a small movement, as the onsets in Discovering, when a marked movement accompanied a relatively gentle fadeout, as in the endings in Discovering, and the marked movement triggered a marked accent, as in the endings in Metals, we experience these as salient multi-modal events, that we can relate to theoretical concepts such as auditory capture, as discussed in Section 2.3. And since salience is about making something stand out and attract attention, designing mappings that emphasize the dance phrase edges, might also increase the awareness of them, and even assist movement learning and memorization [73].⁷ That the phrase edges became more salient can also be inferred from Kim’s interview, where she expressed how there were a lot of onsets and endings — “microscope-wise,” and in that way attests to an increased awareness of these events. Of course, Kim’s perception would probably in this case involved more than vision and sound, such as kinaesthetic awareness and proprioception [22, 29, 74]. One could to some degree observe the negation of that phenomenon in the Imitation section, where the transitions between stillness and movement, silence and sound, were made blurry, partly by redefining silence as a very low sound, and partly by setting up a contrast to salient events, the accents, in the middle.

As for the second perspective, our project demonstrated how onsets and endings also can have clearly distinct qualities. A phrase, be it a music or a dance phrase, is something much more active and activating than stillness and silence, and starting it is something qualitatively different than ending it. Godard’s notion of pré-movements, denoting the preparatory postural adjustments taking place before a movement [75] highlights exactly this. In our case this had an additional dimension, however, since Kim’s knowledge and understanding of the interactive system (cf. theme 3), at least from an experiential and embodied point of view, was partly guiding her movement preparation and adjustment of posture (leaning forward, protruding shoulders). As Kim stated it, “the understanding of how to make the sound and how to stop the sound, it actually makes what kind of movement is gonna happen.” An important part of Godard’s notion of pré-movement is the largely unconscious initiation and preparation of movement, but this, as we see here, might have additional conscious aspects in interactive dance.

Compared to starting an movement, ending it, at least if we assume some brief moment of stillness before the next starts, would have instead to operate with opposite principles, engaging inhibitory muscles aiming to put energy to rest and attain a stable posture. In that respect, Kim’s strategy for ending the phrases in the Discovering section by letting her arms down worked very well, since that movement was naturally confined by the maximum extension of the arms, and relaxation would easily follow. Our strategy for triggering both the ending muting and accents in Metals, however, was perhaps less effective. Since both mappings were based on making a brief energetic movement burst, a “flick,” at the end of the movement, this would potentially make halting the movement in itself more demanding. We might speculate that this partly can explain the observations that many of the intended ending accents were instead performed inside phrases in this section. Arguably, one could also see this in the light of Kim’s experience of empowerment, so that the very positive feeling of making a movement accent that triggered a powerful sound in this section, might simply have made her want to do it more often. It might even have been the reason why she didn’t integrate the muting ending among the choreographic elements in the performance.

We could also draw the experiences from the Wheels section into this discussion. Here the interaction was clearly much more demanding in the sense that Kim had to think about the both the energy and the direction in which way she initiated the simulated wheel to be able to stop it — otherwise the wheel would keep its virtual “spinning” for a little while before it would stops. Moreover, since the speed of the wheel was effectively audible only through the frequency of the triggering of the sound files, very low frequencies would also imply long pauses between each triggered sound, and this clearly made it difficult to estimate the velocity required to stop the virtual wheel. In the HCI world, this might be considered as poor design, but as Kim mentioned in the interview (theme 1), she found the lack of correspondence between movement and sound to emphasize her stillness through contrast, and that this mapping actually enabled her to do “more diverse things.” Thus, instead of feeling the difficulties as restricting for her movement, there was instead a degree of liberation.

Kim’s interaction with the mappings mentioned can be seen in the light of the concept of affordance, a concept that is often brought into the discussion in HCI (see, e.g., [76, 77]). Based on Norman’s [78] idea of perceived affordances, Magnusson sees affordances as potential applications derived from the agent’s embodied relationship with the object in the enactive sense [77]. This seems like an highly fitting description of the way in which Kim in this project has built an embodied knowledge and understanding of the possibilities of the interactive system throughout the process, and how this knowledge has led her to prefer certain movements before others. Still, we also had a few examples where she chose to ignore some affordances (the muted endings), tweaked some (use Metals accents within instead of at the end of phrases), and discovered others (how letting her arms down could make a salient ending in Discovering). Thus, Kim’s actions in the face of the affordances in different cases attest to how the individual in the face of different affordances may act adaptively and creatively.

Another thing that might play a part here is constraints. That interactive systems impose restrictions upon dancers might be seen in a negative light as something impoverishing for choreography, and some of Kim’s statements in the interview also indicated this (Theme 5). However, such constraints might also be seen as a creative possibility. Jung [79], who has applied fairly intrusive technology (Gametrak controllers) connecting dancers to one or several cables to control music and visuals in what she calls a poetic of restriction, instead sees restrictions as a way of escaping established habits and well-rehearsed moves, and thereby something that can introduce new kinds of movement into the dance [79]. Thus, even if there seems to be an inherent conflict between the purely choreographic concerns and a dancerly response to the workings of interactive dance systems, the latter also implies certain creative possibilities at least from a certain perspective. It might be interesting to address the dynamics between liberation and restriction in future studies.

7.1 Future research

The explorations of onsets and endings in this interactive dance study invite further studies in a number of areas. While all the mappings explored in the current study were focusing on one type of temporal dynamic for an isolated body part, it might be interesting to explore more complex temporal dynamics, either consisting of combinations of different temporal dynamics in one single body part, one single body part but with several sensors, or in several body parts at the same time. Moreover, while it could be observed in this study how pre-movement became a necessary condition for a certain movement onset, it would also be interesting to investigate how certain postures might afford certain types of onsets and how certain movements might afford certain types of endings (cf. the discussion on endings in Section 7).

While this work took place in an artistic research context, there are also aspects that could be studied in a more dedicated research setting to gain insights that are more generalizable. For example, it could be interesting to let groups of users test out the different interaction mappings that were developed, either with a focus on the users’ kinaesthetic awareness and proprioception, their enjoyment or with other research agendas.

Although the focus on the performer and mappings are both important, another important perspective missing from this article is the audience’s. Like studies by Alaoui [80], Salter and colleagues [81], and Golz and Shaw [82], show, including surveys, interviews or other forms of feedback from an audience can bring up many interesting issues and questions. In retrospect, it would have been interesting and relevant to have the audience’s viewpoints on many of these as well as other issues related to the onsets and endings of dance phrases in interactive dance. As far as I am aware of, there are no such studies available at the time being, and this will therefore be something I will hopefully pursue in future studies.

8 Conclusion

In the design of mappings and in noting the dancer’s response to these, this work has brought forth insights related to phrase onsets and endings in interactive dance, both related to stillness, pré-movement as well as the onsets and endings themselves. Questions regarding affordances, restrictions versus liberation and the conflicts between weighing the dancerly or musical sides of the interaction have all been highlighted here.

The movement-sound mappings presented in this paper are sonifications based on the temporal dynamics of the movements, with special emphasis on the onsets and endings. While they have had many aspects that were designed with an esthetic motivation, this might accommodate the user in receiving the information, simply by making the sonification more engaging to use and less prone to annoyance. The precise mechanisms involved will still have to be verified by an empirical evaluation in the future. Moreover, while several of the mappings presented seemed to have clear high-salience as well as low-salience events, further details of the landscape in between remain to be explored. In the further development of the project, one can hopefully enrich and inform the technologically and esthetically challenging art form of interactive dance, demanding time-consuming close attention to details in a long chain from movement to artistic experience.