Over recent years, the use of data sonification has become increasingly widespread. There is a new, or perhaps renewed, consciousness of the particularities of aural perception, and we are learning to consider clicks, beeps, varying pitches or chords as carriers of significant information. Although much of this evolution is taking place in the technical realm, as a way of enhancing a user’s perception of important data, notably when their other senses are occupied, interest in data sonification is also increasingly apparent in the realms of art and music. Artists are using sonification to introduce “real-world” or “real-time” elements into their work, and composers are abandoning human decision-making and fixed scores to leave space for variation derived from incoming data.

Locus Sonus is a research group attached to the art colleges of Aix-en-Provence and Bourges (France). Our aim is to further knowledge in the domain of audio art. We are particularly concerned with the relationship of sound to space and the evolution of this paradigm through contemporary technology and networks. Locus Sonus combines experimental and exploratory artistic practice, technological development, critique and theory to produce publicly presented artworks and other forms of dissemination.

In March 2010, Locus Sonus organized, in collaboration with MMSH, CRISAP and IMERAFootnote 1 its 6th international symposium, entitled Sonification—What Where How Why. We are happy and honored to be invited to edit this special issue of AI & Society, which can be considered as a sequel to the symposium of the same name.

The field covered in this review is simultaneously extensive and limited:

Contributions explore a broad range of artistic practices, scientific research and theory related to artistic practice, the aim being, not so much to adopt a curatorial or critical stance, but rather to provide a survey of the field. We have chosen not to apply a strict definition to the word sonification, and therefore, contributions include references to audification, algorithmic composition and beyond.

The field is limited in the sense that the choice was made to restrict the subject to sonification and environments. This definition is open to interpretation, and we find references to environments as diverse as the articulated hardware, software space of a computer booting (Valentina Vuksik), the human brain listening to music (Stephen Barrass) or the electrical probing of the Elephant fish (Jean Cristofol); at the same time, it eliminates the more practical, if potentially artistic, aspects of sonification that include electronic instrument design or human computer interaction.

This issue of AI & Society has been organized into two main parts: The first section is dedicated to artists’ pages providing short descriptions of art works and artists’ positions, intended to give an overview of different practices. This is followed by a series of original articles, some of which are written by contributing artists, reflecting their personal engagements, and others by theorists from different domains.

If, at first glance, contributions might not seem have much in common—indeed, there is no desire here to propose a new artistic “genre”—the reader will find a certain commonality in the problems dealt with, albeit from radically different angles. The result is a fascinating crisscrossing of paths between art, philosophy, music theory and science where ideas converge and fuse (or at times clash). The articles contain an element of debate, undoubtedly fueled by the fact that most contributions were written in continuum from the Locus Sonus symposium. Original documentation of these round table debates can be found on the Locus Sonus website and additional video interviews with participants on the author’s website.

1 What

In sonification for art, the choice of the data to be used is fundamental, often serving as the conceptual mainstay of a piece. A distinction appears here between the use of recorded, often recuperated, scientific or technical data sets and the use of real-time data.

There is a long tradition of seeking artistic inspiration in the infinitely vast—for example, we can find references relating the motion of the planets to music dating back to antiquity where composers and philosophers sought to explain the organization of our world and our universe through the fundamental rules of harmony (Jerome Joy, Florian Grond and Thomas Hermann). We find this reflected in several contemporary works presented here. Scot Gresham-Lancaster in Remap (McCall.DEM) uses the relief of a real-world terrain to generate his sounds; Jens Brand reads the circumference of the earth like a vinyl using data sets generated by the satellites orbiting above us. Marty Quinn has worked with NASA using data from solar storms as a source, and Richard Kroland-Martinet, Solvi Ystad and Mitsuko Aramaki sonify cosmic particles—invisible but constantly present in our environment. Inversely, Victoria Vesna, in Blue Morph, uses nano-data, from the metamorphosis of a butterfly, to awaken us to the infinitely small. Peter Gena describes his work sonifying the human genome (here too we find a case of collective consciousness, as numerous artists have realized DNA sonifications, a phenomenon for which Florian Grond and Thomas Hermann offer us a possible explanation).

Real-time data sonification and sonification of recorded data sets share what is possibly the only common denominator in sonification works, the delegating of at least part of the responsibility of the art work to the chosen data. The case of real-time sonification, however, implies a conscious decision to insert the artwork into a present situation. This will be dealt with in more detail in the following Sect. 2.

It is worth noting that Lorella Abenavoli’s installation, The Pulse of the Earth, offers a hybridization of these two uses of data, real-time and the recorded. The installation reads continually updating vibrations of our planet but compresses them in time in such a way as to make them audible to the human ear, enabling her to invite us to “drop in and listen to the earth”. In keeping with this, during the symposium, astrophysicist Roger Malina demonstrated the necessarily relative nature of the concept of real time by playing a sonification of the first 100 million years of the universe. It all depends on scale and resolution.

2 Where

We can extend the question of real time to that of time and space, as developed theoretically by Jean Cristofol and Stuart Jones, both of whom confirm that sonification readily lends itself to being a vehicle for these difficult to pin down agents. This is reflected in works that use real-time, in situ data, to mediate hidden aspects of the environment in which the piece is playing. In Stuart Jones’ installation Bop!, data capturing current conditions: light, heat, human frequentation… drive the composition. In her performance piece Tripping Through Runtime, Valentina Vuksik explores the way in which software programs inhabit the architecture of computer hardware, guided by human interaction. My current work, Road Music, uses data from a car ride to simultaneously generate music for that ride. In John Eacott’s Flood Tide, data from tidal movements of the River Thames modify the score being interpreted by an orchestra on the riverbank. In these cases, it is the relationship between the unfurling of the piece and the proximity of the data source that generates the art.

Data can also be transported from phenomena that are distant in time and/or place, thus extending our spatial/temporal perception. While projected moving image is strongly localized and appears to us as a window through which we observe, sound is enveloping, we can enter a sound environment, or sound can enter ours from elsewhere, creating interpenetrating spaces. Andrea Polli puts us in contact with extreme climatic conditions through her project Sonic Antartica. In another project heat and the heartbeat of the city, she employs reductive techniques to process meteorological data collected over a relatively long period that have the potential to awaken the inhabitants of Manhattan to a new understanding of their local environment.

3 How

The reader will find a diversity of descriptions concerning methods employed in data sonification. Some authors offer details of techniques they have employed, and others have preferred to focus on the articulation between sonification and artistic process. Approaches vary from that of audifying data (Lorella Abenavoli—The Pulse of the Earth, Victoria Vesna—Blue Morph), through the adaptation of data sets as musical patterns (Peter Gena—EGERYA), to the use of data as the starting point for a musical work (Stuart Jones—Chesterfield Starfield). Recounting the wonderful project Listening to the mind Listening Stephen Barrass describes a chain of interpretation: neurological data, recorded from the brain of a person listening to a piece of music, was proposed as raw material to a number of composers who then produced works following various compositional strategies, which were finally performed live before an audience. In a subsequent piece Baroque EEG for Cello, Heart and Mind by Stephen Barass, Geoff Gardner played a Baroque Basso Continuo on Cello, accompanied by the sonification of his pulse and brain activity.

Peter Gena suggests that the origins of sonification can be found in algorithmic composition and the pioneering days of computer music. Further, he reminds us of the fact that it was only recently that musical composition, with the advent of recording and synthesis, became a question of manipulating sound. Up until then, it was a question of manipulating notes or in other words—data. This is an interesting point in the debate, on whether sonification belongs to the realm of music, art or science (or all three). And it is worth noting that Richard Kroland-Martinet, Solvi Ystad and Mitsuko Aramaki, in their rigorously scientific approach, employ what might be considered reverse engineering techniques—the analysis of expression in musical interpretation—in their studies leading to the creation of models for “applied” sonification. Scot Gresham Lancaster and Stephen Barrass tackle the sense of the term sonification in depth querying Thomas Hermanns’ recent definition, which tends to exclude an artistic usage. I should hasten to add that in Aesthetic Strategies in Sonification presented here by Florian Grond and Thomas Hermann, a rather more “artist friendly” notion of sharing a common problem is put forward. Beyond this, the paper offers a careful, historical discussion of the nature of audio representation.

Jerome Joy’s interrogation of NMSAT (Networked Music and Sound Art Timeline) offers a multitude of cross-references, allowing us to trace the evolution of ideas and techniques leading to today’s practices of sonification.

4 Why

Why use data to produce sound? What does it signify? Artistic motivation is varied. In much of the work presented here, the use of data is considered as a counterpoint to personal choice and, as such, refers to artistic positions which reflect on determinism and freewill as deployed by landmark composers such as John Cage and Iannis Xenakis. Contributing composers—Peter Gena, Scot Gresham Lancaster and Stuart Jones, all of whom describe having had dealings with John Cage during their formative years, are able to consider without undue difficulty that sonification is a natural step in the evolution of music, though as the reader will discover the paths they follow are quite different. While Andrea Polli’s use of sonification does not exclude such ideas, her equation of audio environment to social and political engagement perhaps owes more to Murray Schafer for the invention of soundscape and to Joseph Beuys for his art of the social. Thus, her articulation of elements of interdisciplinary research is used to promote environmental and social awareness. Jens Brand treats the question of sonification with humor but still uses the fact that the data being interpreted is on the scale of the planet to get his message (and brand) across, while Marty Quinn believes in a future where a universal language of sonification will interpret data for the blind community. More concretely still, Richard Kroland-Martinet, Solvi Ystad and Mitsuko Aramaki at LMA aim to identify audio semiotics, through fundamental research combining acoustics and brain imaging techniques for use as design factors in industrial sonification.

From a more philosophical point of view, Jean Cristofol suggests, through a series of quite beautiful metaphors, a field of artistic endeavor related to information flux, which resonates in today’s connected society. Stuart Jones, primarily concerned with real-time data, debates whether sonification can articulate ontological and phenomenological approaches, building on Henri Bergsons’ notion of the intuition of passing time.

This brings us to the question of audience perception: do they need to know where the data are coming from? If a general consensus is possible, we would say that most artists engage in some sort of strategy to allow a comprehension, at least partial, of the provenance of the data being sonified. However, John Eacott has suggested that possibly 1 day the general public will be so used to music including elements of sonification that they will spontaneously seek the origins of the data being used upon arrival at a concert. The question of audience perception is debated in depth in the interviews that constitute John Eacott’s paper.

Possibly as a sort of epilogue, we have included an interview with Dr Bruno Debien, which we hope, offers insight into a world of sonification where the questions: What? Where? How? Why? are clearly answered.