Keywords

1 Introduction

Neon Meditations is an ongoing collaboration with visual artist Per Hess. It is an improvised piece, a recurring performance, and a crossing point between music and visual art. Its background is a curiosity about colour, sound, and their possible relationship. We read the colours of neon light with colour sensors and make music with a modular synthesizer, which both performers control simultaneously in a way that easily causes confusion about action and sound.

The sound design can be described in simple terms as excitation and resonance, or source and filter: the excitatory signals from the analogue modular synthesizer are distributed to exciters attached to vibrating objects which colour and distort the sound. Neon Meditations may be an inconvenient example to illuminate Godøy's concept of sound–motion objects and the motor theory perspective, since these ideas were developed primarily with acoustic music in mind. Nevertheless, it may be revealing to reconsider sound–motion objects from the perspective of live-electronic music.

First, I will describe Neon Meditations from a technical and aesthetic point of view, with a focus on sound, and finally, I will discuss some topics from Godøy's research on sound-related motion as it applies to live-electronic music in general, and illustrate it with the example of Neon Meditations.

2 Colour and Sound

Per Hess has been particularly occupied with colour in his work as a visual artist. When we first met at one of his exhibitions in 2017, we began discussing a possible collaboration where I would contribute music or sound. As we soon realised, it is by no means obvious how to combine static visual art with music as a form unfolding in time, at least if one is to avoid making one part illustrative and subservient to the other. In Scriabin's landmark work Prometheus, a colour organ dynamically lights the space in different colours following the music. The simultaneous use of coloured light and music as mood-inducing devices has become ubiquitous in concerts and movies, to the point of being barely noticed. Our approach is seemingly less spectacular by the restriction to constant colours, which are unaffected by the music.

Hess has produced a series of neon tubes segmented into fields of different colours. Neon by itself glows with an orange-red, and various fluorescent pigments inside the glass tubes bring out a range of different colours. In fact, what many artists casually refer to as ‘‘neon” might also include argon, which produces a bluish colour. Eventually, we decided to use sensors to read colours and control an analogue modular synthesizer with these signals. By doing so, we have turned the neon tubes into a visually appealing part of a musical instrument, although they still can be exhibited on their own.

A long period of experimentation and practice preceded our premier performance. Technical problems had to be solved, such as constructing colour sensors and designing a patch on the modular synthesizer, but we also faced aesthetical choices, such as how to map colour to sound. Actually, “mapping” is a misleading word, since we ended up with a rather complex relation instead of a simple one-to-one correspondence between colour and sound.

The notion of synaesthesia tends to come up in discussions about colour and sound. In the strong form, hearing a sound may induce the visual impression of a particular colour (so-called photisms), or vice versa, in completely idiosyncratic ways. However, there is evidence of more widespread forms of synaesthesia (Marks 1975). Temperature is associated with colours when we describe red and yellow as “warm” or blue and green as “cold.” Bright colours are regularly associated with a high pitch and dark colours with a low pitch. Vowels are sometimes associated with colours; in particular, the second formant frequency, and the spread between the first and second formants, appear to be related to colour. According to Marks, synaesthesia is a cross-modal manifestation of meaning in a purely sensory form and is not fundamentally different from non-synaesthetic cross-modal meaning or even abstract verbal meaning.

It may be revealing to consider the various visuo-auditory correspondences from their temporal aspect. Colour doesn't ‘‘happen” in time; we typically experience it statically. Hence, for its auditory correlate, we should expect something that can be extended over time. Gestures are completely different; with the closure of their beginning, trajectory, and end, they can be related to musical Gestalts such as tones or short phrases, or musical objects in Pierre Schaeffer's sense (Godøy 2018).

It is commonplace in contemporary art to explain what the work is about. If a trite verbal description exhausts its meaning, one may wonder, what is the point of creating the work? Usually, there is a remainder that resists explanation. Artistic research, in the sense of solving problems posed by the realisation of the artwork, is an intrinsic part of artistic creation. There is also a narrower sense of the term, related to how artistic projects have been adopted in an academic setting with expectations to produce knowledge. However, such knowledge may be specific to the project, highly subjective, or difficult to generalise and share. Peter Osborne has rightly identified some other perils of the academic form of artistic research. Rather than producing critically significant works, it may lead to a new kind of academic art (Osborne 2021). Posing and solving research questions does not necessarily contribute to attaining artistic goals. In Neon Meditations, it is fair to say that the work concerns translating colour into sound. But by no means is it a pedagogical illustration of particular synaesthetic colour-to-sound correspondences, a topic better left to researchers in cognitive psychology. Nor have we been particularly occupied with the relation between gestures and sound. Nonetheless, it turns out that Godøy's research on sound-related motion can shed some additional light on our performance—and perhaps also the other way around.

3 The Patch

To begin with, we had to construct colour sensors, which we did using the simplest means. We use two hand-held colour sensors (Fig. 1), which can point independently in different directions. This design decision already eliminates any straightforward one-to-one correspondence between colour and sound. Instead, there would be a mapping from colour pairs to sound, except that both performers influence the sound.

Fig. 1.
figure 1

Neon lights and sensors. Video still, reproduced by permission of Guro Berger.

Full size image

Each colour sensor receives a constant voltage which passes through a photoresistor. In front of the photoresistor, there are colour filters made of plastic film, which ensure that the sensors actually register hue and not only brightness. The varying control voltage (CV) is returned to the modular synthesizer in a rather complex patch. First, there is some processing of the CV signals from the colour sensors. Both sensor signals are sent to a comparator which outputs a gate signal when one voltage is higher than the other. The sensor signals also directly control filter cutoff, oscillator frequencies, and amplitude. In parallel, the other performer controls the modular synthesizer through knobs, faders, and an expression pedal. The audio path includes two oscillators, VCAs, filters, and a few other modules with complex cross-modulation. The output is sent to an external mixer via an amplifier, to exciters attached to vibrating objects.

Over the years we have performed the piece, the rough structure of the patch has remained the same, but some modules have been exchanged, causing necessary adjustments. There is no score in a traditional sense, but there is documentation of the patch that I try to follow, which gives the work a certain cohesion and makes it recognisable through its successive iterations.

Two things are crucial to notice here. First, two performers control the same instrument, even with the same parameters. It is a multi-agent system, with predecessors such as the network ensemble The League of Automatic Composers, who connected micro-computers in a network around a kitchen table in the 1970s, even before the internet and the MIDI protocol became available (Rohrhuber 2007). Second, left to its own devices, without our active control, the patch would only produce a monotonous drone. With our aid, it produces a somewhat variable drone.

4 Sound Design

Arguably, sound design may cover the entire process from composition to interpretation. Sometimes it makes sense to distinguish sound design from composition proper. This may be the case if the composition is conceived as an abstract structure, represented by a score or other symbolic information, which is given concrete flesh in an interpretation or realisation. Indeed, the term musique concrète itself reflects this focus on the actual sounding music (Schaeffer 1966; Godøy 2021a).

Some of my electroacoustic pieces have been created first as a timbrally crude sketch that I have submitted to acoustic processing before mixing a final version (Holopainen 2021a). Acoustic processing here refers to using real acoustic spaces and vibrating objects to colour the sound. I would play sound files through loudspeakers or exciters attached to the resonating bodies of instruments such as guitars, drums, or other objects with a prominent acoustic character and then record it again. This technique is also used in Neon Meditations. Since it is a live performance, the exciters and vibrating bodies are included onstage (Fig. 2).

Fig. 2.
figure 2

Exciters and resonators. Video still, reproduced by permission of Guro Berger.

Full size image

Schematically, the sound design of Neon Meditations can be considered as separated into excitations and resonances. Excitations can be understood as related to what we do and mental images of actions, whereas resonances can be related to the effects of what we do and images of materials (Godøy 2001). Some acoustic instruments have feedback from resonances to excitation, which complicates their modelling as separate stages, but in our case, the separation is justified.

In the patch for Neon Meditations, excitations come from spectrally rich sawtooth waveforms and white noise, fed into analogue filters with variable cutoff frequencies. The filters provide a first stage of resonances, but this signal acts like an excitation in the following stage of acoustic processing. The output from the modular synthesizer is fed to the exciters, usually attached to a frame drum, a tambourine, and cardboard boxes (other resonators may be used if available). These resonating bodies strongly colour the sound, and at loud levels, they also distort it by adding rattling or buzzing sounds of their own. By adjusting the position of the exciters on the object, various vibratory modes can be emphasised. The amount of extraneous rattling depends not only on the volume and frequency; it can also be controlled to some degree by how tightly the exciter is fastened to the vibrating object. The use of physical vibrating systems could be taken much further. For example, Daniel Wilson has built feedback systems around exciters and resonators with contact microphones in what he calls a post-electronic modus operandi (Wilson 2012).

The technical and aesthetic ideal of hi-fi treats the loudspeaker as a transparent window into an imaginary world. We are not supposed to notice the presence of the loudspeaker but focus on the music. Michel Chion recalls how cinema sound was once characterised by a cavernous resonance and a wavering sound caused by the uneven speed of the film projector (Chion 1994, p. 99). Modern movie theatres have solved these problems; a powerful deep bass can be produced with little distortion. At home, such deep bass tones would make the furniture or dishes shake. Chion also makes a useful distinction between fidelity and definition (ibid, p. 98). Fidelity is more of a selling argument than a verifiable notion; it would require hard to arrange comparisons between the original and the reproduction. Definition is a more technical and precise term. High-definition sound covers a broad frequency range, particularly high frequencies that can transmit a sense of acuity and presence, as well as a large dynamic range. In sound distribution by exciters on sounding bodies, the distinction seems apt: clearly, it distorts the sound too much to be considered hi-fi, but on the other hand, the added rattling noise that extends the high-frequency range with a shimmering provides high definition. The exciters and their associated resonators are also point sources, well localised in the spatial field, in contrast to the diffuseness of stereo panning achievable with an ordinary pair of loudspeakers.

In Neon Meditations, the exciters and vibrating objects replace transparent loudspeakers. Their presence onstage is noticeable, both visually and soundwise. Acousmatic listening, another of Schaeffer's famous notions (Schaeffer 1966, ch. 1), refers to any listening situation where we cannot see the source, such as sound played over loudspeakers. Supposedly, it helps the listener focus on the sound as such and be less preoccupied with the causality of the sound source. With the stage presence of rattling tambourines and buzzing shoe boxes, it is far from certain that acousmatic listening is an adequate term, since these objects partly become sound sources of their own while also transmitting sound.

Neon Meditations may be described tentatively as a drone piece, although perhaps not typical of that genre. Our performances typically last about 20 min and consist mostly of a sustained, wavering sound. In Schaeffer's typological classification, it would be called a thread (“Trame” in French). The thread type can be encountered not only in natural environments such as waterfalls, but is also common in orchestral music (Chion 1983, p. 134), typically as a background texture. In drone pieces, what otherwise serve as background elements are brought into the foreground.

Drone pieces, in general, are characterised by a total lack of development or any sense of large-scale form. There may be a high density of micro-events or textural and timbral articulations, but very little happens at the phrase level. Many contemporary music genres, including drone pieces, pose certain challenges to the listener's perception and memory (Wanke and Santarcangelo 2021). Retention and protention are intentionally put to the test in these pieces, which have also been described as engaging in a form of”memory sabotage.”

Schaeffer's most basic classification of sound objects divides them into three categories: Sustained, iterated, and impulses. Schaeffer also has a category of objets convenables, or suitable sound objects, which are of medium length and easy to memorise. Given the drone character of Neon Meditations, we tend to stay away from the objet convenable category. For Schaeffer, there was a normative aspect to this category, these sound objects were deemed suitable for music.

5 Interaction

If we ever veer away from the sustained sound type in Neon Meditations, it is by brief passages of iterated sound objects that may become sufficiently separated to be perceived as impulses; it is done simply by turning the frequency knob of an oscillator down to the sub-audio range. As Godøy points out, there are phase transitions between different kinds of bodily motion (Godøy 2021b), directly corresponding to the sustained, iterative, and impulsive types of sound objects. With an electronic instrument, these transitions are producible in one smooth movement, simply by turning a knob, although the resulting sequence of sustained pitched tones, iterations, and separate impulsive sounds remain perceptually distinct categories.

Physiological constraints make it impossible to increase the rate of a bow tremolo to audio frequencies. Electronic instruments do not share our physiological limitations, but should they simulate them?

Here it is important not to confound what is and what ought to be. Granted, we have distinct perceptual and motoric regions of sustained, iterative, and impulsive types, and electronic instruments afford a seamless transition across the range. From these two facts, someone might suggest that electronic instruments ought to be designed such that these distinct perceptual types can only be produced by dedicated types of gestures, as they are in acoustic instruments. Someone else might instead celebrate the fact that electronic instruments afford a new, non-natural connection between gesture and sound. I believe both views have their merits.

However, there is another aspect of live electronic interaction that needs to be considered. Although much of the writing on the topic specifically addresses interactive computer music, much of it also applies to analogue or hybrid electronic instruments such as modular synthesizers. In any case, interactivity consists of relegating certain tasks to the machine and letting the performer play a role that can be described as that of a supervisor, pilot, or collaborator. In my experience, the interfaces that allow for the most expressive performances are those that permit a detailed control of all aspects of sound and relegate as little as possible to automation. In Sergi Jordà's words:”A good instrument should also be able to produce ‘terribly bad’ music, either at the player's will or at the player's misuse” (Jordà 2007, p. 104). Such instruments require more practice to master and allow for bad performances, but that is precisely the point.

Virtuoso instruments don't correct the performer's mistakes. In Neon Meditations, there is another reason for not granting the machine too much autonomy. Instruments or systems for generative music can be designed to create a stream of varied output with little to no input, a goal that has been pursued in various media involving feedback, so-called interfaces for self-organising music (Kollias 2018), and in more algorithmic approaches using monolithic systems that merge sound synthesis and slower processes (Holopainen 2021b). In modular synthesizers, self-generative patches can produce endless musical variation with no input. However, in Neon Meditations, where the point is to translate colour readings into sound, such an additional layer would unnecessarily obscure an already complicated gesture-to-sound relation. Furthermore, the multi-agent nature of our system creates a complexity of interaction comparable to what can be achieved by sophisticated interactive digital or analogue computer systems.

6 The Motor Theory Perspective

Godøy (2018) lists four types of music-related motion that may be expected in performances of instrumental music:

  • Excitatory motion: transfer of energy from musician to instrument

  • Modulatory motion: dynamically changing pitch, timbre, loudness

  • Ancillary motion: avoiding strain, etc.

  • Communicative motion: between performers or toward an audience

Examples of excitatory motion include blowing air into a wind instrument, plucking or bowing a string, or tapping a drum membrane. In live electronic music, where the sound production is already taken care of, it is still possible to simulate excitatory motion, as is commonly done on keyboard instruments where depressing a key produces a sound. In Neon Meditations, on the other hand, we do not even try to simulate such correspondences; it is all about modulatory motion. Both performers modulate timbre, pitch, and loudness. Ancillary actions do not produce sound, at least not on purpose, but are more or less necessary to accommodate the playing. Our performance requires rather static postures and mostly looking down on the instrument. Therefore our communicative motion is minimal while we are engaged in the performance.

In an acousmatic listening situation, when we listen to a recording of musicians, we see none of their motions. Nevertheless, the first two categories of music-related motion are more directly involved in producing and shaping the sound we hear than the last two. We might infer the excitatory and modulatory actions from listening only; at least, we might imagine probable sound-producing actions (see Godøy 2001). However, we are unlikely to guess all sorts of ancillary or communicative motions the musicians were making before the microphone in the studio.

7 Sound-Motion Objects in Live-Electronics

Godøy describes sound–motion objects as multimodal, including sound and corresponding body motion; they typically occur on medium time scales of 0.3–5 s; and they may involve complex motor schemata such as complicated, rapid passages which have to be practised before being performed automatically without conscious control (Godøy 2021b). It is no coincidence that their time range corresponds to Schaeffer's objet convenable.

The theory of sound-motion objects suggests that we tend to imagine a plausible physical motion, often a body motion, corresponding to sounds we hear. This is practically unquestionable in the case of singing and acoustic instrumental music but becomes more conjectural in electroacoustic music with less immediate connections between sound production and perception.

In live electronic music, the gesture-to-sound relation may become confused, depending on the mapping from controller to sound production. Some actions correlate with sounds, but there may be ancillary motion with no causal relation to the sound. For an audience without expert knowledge about the controllers and mappings used in the performance, it may be impossible to distinguish ancillary motions from those that modulate or trigger sounds. The antennae of a theremin controller make no distinction between motions with modulatory, communicative, or ancillary intention. If you move at all sufficiently close to the antennae, they register it. Furthermore, some sounds in live electronic music may not correspond directly to performance actions, such as automated sequences or pre-recorded parts that only need to be started and perhaps stopped. In Neon Meditations, two performers at once influence the sound. It is quite unlike playing the piano with four hands, where each pianist knows which part they are playing; in our performance, we may both control filter cutoff or oscillator frequencies, and the audience is likely to have trouble deducing which actions are responsible for the timbral changes that result. Indeed, we also found this confusing at first and had to spend time practising before our first performance.

According to the motor theory perspective, sounds correspond to imagined actions (Godøy 2001). This may hold even in electroacoustic music since we can imagine whatever we want. But it also happens that sounds in live electronic music contradict what is seen. Usually, after a performance, we engage the audience in a dialogue and answer their questions. We have had audience members compare our sound to a car or motorbike. The colour sensors are sometimes mistaken for microphones, which they admittedly resemble. Some audience members, therefore, speculate that they pick up sound directly from the neon tubes, which they do not. The point is not to call out the audience for not getting what we are doing. On the contrary, the active search for causal links may contribute to making the performance an engaging experience.

8 Assessment of the Sound-Motion Object

The notion of sound-motion objects is rooted in the praxis of acoustic instrumental music, which explains some of its biases; it has a certain focus and possibly a few blind spots or limits to its applicability. I will briefly summarise those that come to my mind.

  1. 1.

    Acoustic instruments have been with us for a considerable time. We interact with them as we interact with the rest of our physical surroundings; excitations and resonances inform us about forces that set objects into vibration and about the material properties of these objects. Grounding the theory in ecological perception gives it a general, broad validity.

  2. 2.

    Sound-motion objects, like Schaeffer's objet convenable, emphasise the medium duration range of about 0.3–5 s. This makes drone pieces inconvenient examples to illustrate the idea. The focus on gestures and sound–motion objects downplays processes over longer time spans. On the other hand, sound–motion objects are situated at a level above the intermodal concept of texture. Spatial textures of various coarseness should be easy to imagine as timbral qualities of varying roughness. This touch-to-sound correspondence does not seem to necessarily involve motion.

  3. 3.

    Sound–motion objects take acoustic instrumental music as their model; the concept is therefore not a priori equally relevant for electronic music. The freedom to introduce arbitrary mappings from controllers to synthesis parameters may destroy the unity of perception and performance, which can be taken for granted in acoustic music.

  4. 4.

    Even the concept of coarticulation, which is best known from phonetics but is also a reality in vocal and instrumental music, must be reconsidered in live electronic music. Coarticulation involves the fusion of otherwise distinct motions, and prepared actions, such as performers placing their fingers in the correct position on an instrument before playing (Godøy 2021b). This has certain consequences for sound production in vocal and instrumental music. In live electronic music, the role of coarticulation in shaping the performance may be much less important, or at least very different, depending on the specifics of the mappings and interfaces used.

  5. 5.

    The theory of sound-motion objects does have interesting things to say about virtuosity, idiomatic writing and playing (Godøy 2018), but it seems almost overqualified, yet not quite to the point when it comes to motorically less challenging improvised live-electronic performances. Live coding is perhaps the most striking example, where mental effort largely supplants bodily effort; the typing motions, although also involving motor skills, have a most indirect relation to the sound.

  6. 6.

    A single performer is implicitly assumed responsible for the sound production, not two or several performers as in multi-agent systems, nor a hybrid combination of performer and algorithms or other kinds of automation (As Godøy reminded us during the 2022 seminar, the mechanical organ originally needed two performers, one of whom was treading the bellows. The bagpipe also apparently frees the performer from excitatory motion while playing so that only modulatory motion is required. To these examples, one might add wind chimes, a mechanical instrument that requires no human performer.)

  7. 7.

    The focus is on low-level perception and the physicality or physiology of sound production. This choice of focus is understandable as a complement to, or reaction against, a previously prevailing overly abstract and”disembodied” flavour of music theory. As always, focusing on something is fine as long as it does not replace an old myopia with a new one or pushes other questions worth asking and methods worth pursuing into the background, such as sociological, historical, and aesthetic perspectives on music.

In summary, the concept of sound–motion objects most aptly deals with acoustic instrumental and vocal music. The motor theory perspective offers the plausible view that we experience motion and sound as interconnected, almost synaesthetic aspects of a coherent phenomenon. This is most obvious in acoustic music and may still, to some degree, be true when listening to electronic music, where distinct sound types may be associated with suitable imagined sound-producing actions. In live-electronic music with its arbitrary mappings from gestural controllers to sound, on the other hand, it is a matter of artistic choice whether the motion–sound correspondence should be upset and quite illogical, or follow our expectations by simulating the functioning of acoustic instruments. Maybe live-electronic music deserves its own addendum to the theory of sound-motion objects, wherein we distinguish between the motion we would typically imagine as we hear the music, and the actual, arbitrary mappings from gesture to sound. What complicates it is that these two levels are superimposed and may provide mutually conflicting cues.

As for Neon Meditations, the project has turned out to be surprisingly long-lived. We are less preoccupied with developing the performance than maintaining it and adapting it to new circumstances. For each new performance, we solve the practical matters of sound design in slightly different ways.