Prelude

I began my professional life as a painter, but when computers became available from 1984 or so, I began to use them in various ways as tools and media. I made my first computer-controlled interactive piece in 1994 (Fig. 8.1), and was immediately struck by the powerful and unexpected responses shown by some participants.

Fig. 8.1
figure 1

Under the shirt of a happy man (Interactive installation 1994–96)

Full size image

This stimulated my interest in perception, and in the psychological aspects of the complex experience of interactivity, but also introduced me to the broader framework of the digital revolution, media criticism, sociology, philosophy, and so on. In 2001, several participants in my large-scale interactive installation Infonoise [1] (Fig. 8.2) showed puzzling signs of distorted consciousness, and so I began to explore the transdisciplinary field of consciousness studies [2]. It was against this background that I began work on another large-scale interactive piece, Fugue, and once again found myself asking questions about the strange effects of digital interactive technology.

Fig. 8.2
figure 2

Infonoise (Interactive installation 1998–2001). Left: 3D simulation. Centre: building the installation. Right: inside the installation

Full size image

Fugue

An Arts Council England Individual Grant in 2004 brought modest funding for Algorithmica, originally titled City Portrait. Firmly grounded in research, it aimed to critically address the form of the mass-media industry. A spontaneous, non-tactile interaction was to be based on the biological principles of interaction among cells, and a game-based software architecture would operate within the set of a 3D London Tube map. Dr Peter Bentley, a UCL-based expert on computational models of the human immune system, joined the project, along with Anthony Ruto, an expert on 3D modelling also from UCL, and my long-term collaborator Rainer Linz, the Australian new music composer. We agreed to develop the piece in phases, and to present and exhibit each step as a different stand-alone integral work within the overall conceptual framework. Largely because of Peter’s influence, the piece soon came to focus on the complex cellular interactions within the immune system, with the core engine being a specially written version of his artificial immune system software.

Algorithmica then evolved into Fugue [3], a project with two potentially conflicting goals: creating an artwork, and developing an audiovisualisation of the immune system for scientists. In my view, audiovisualisation offers significant advantages for understanding complex systems, because, as Dombois [4] notes, it offers a much wider bandwidth than vision alone, and engages both serial and parallel modes of perception. I was also keen to explore the potential of interactive technologies for enabling users to engage with the production of phenomena, rather than merely observing them passively.

We decided that our focus in Fugue would be on understanding and applying the principles of biological processes rather than creating photo-realistic ‘beautiful imagery’, or merely re-representing scientific findings as visualisations or sonifications. My friend, the immunologist Dr Nada Pejnovic, a research fellow at St. Mary’s Hospital, gave me a detailed introduction to the field, and I was able to discuss with her my artistic interpretations of scientific subjects, my sketches inspired by medical sources, and the concept for the eventual piece. Real-time generated images originating in computational processes would set the framework for the visuals, with a major practical requirement being to reduce the typically heavy computation to a minimum. To meet this condition, and to achieve an abstract, symbolic representation of the actors in the immune system drama, I looked back to the cell-like egg-shaped and spherical structures that had appeared in my paintings from the 1990s (Fig. 8.3) and I suggested making some clay models, as I had done for my early paintings. Anthony scanned the clay models in 3D as the starting point for the final look of the inhabitants of our virtual immune system. To emphasise the focus on processes, and the distinction from the aesthetics of the gaming industry and commercial computer graphics, I suggested a black-and-white approach. Both scientists found this idea problematic, because, in their own words, they ‘could think of the immune system only in red’. As a compromise, our first prototype was indeed monochromatic, but in red.

Fig. 8.3
figure 3

12th view (Oil on canvas, silk screen print computer generated algorithm)

Full size image

The combination of Anthony’s expertise in creating 3D wire-frame models of the human body, his taste for abstract visual art, and my experience as a trained painter of using techniques of perspective, colour and sfumato to suggest depth, and to upgrade the crudeness of the computer generated image with the attributes of traditional visual aesthetics, gave rise to an enjoyable creative process. We replaced the red with a greyscale approach, which was now accepted as being congruent with the overall conceptual framework (Fig. 8.4). To complete the basic system, Rainer designed the sound software around a series of customised audio players that he called Fugue Players, which responded in real time to changes within the artificial immune system.

Fig. 8.4
figure 4

Fugue (Images generated in real time by the computational model of the immune system)

Full size image

The first outcome of the team’s collaboration was a paper [5] presented at a scientific conference. However, our concept received little attention, as most of the scientists categorised it as ‘non-scientific’, although a small number praised its fresh approach.

Variations on the Theme of Interactivity

After this encounter with scientists, we concentrated on developing the artwork. At the time, I wrote:

The title ‘Fugue’ is a metaphor for the trans-disciplinary nature of the work, and for the method applied: interweaving the different perspectives of artists and scientists. The emergent, evolving nature of the artificial immune system algorithm, the use of repetition in the form of a succession of variations of immune system ‘events’, and the complex structural and functional interrelationships between the individual elements and processes are strongly related to the musical form of counterpoint, which formed one of the inspirations for the artistic concept for Fugue. The Artificial Immune System software creates the dynamics of the virtual immune system drama, and also constructs and implements the architecture of the Fugue by providing the functional structure for the communication channels between the visuals and the sound. [5]

We were engaged in intensive online work shaping the architecture and aesthetics of the installation. Online communication imposes numerous limitations, from the inevitable time lag to the lack of direct face-to-face discussion. After a period of excitement and enthusiasm, we entered a phase when tensions ran high. This was caused by conflicts between the demand for scientific accuracy, and the artistic interpretation of scientific data, and it escalated to a level that threatened to end the collaboration, and the project. It took a lot of effort from all of us to reach a consensus, and we decided from then on to limit our comments to our own area of expertise.

Rainer and I had conceptualised the interactive component of Fugue as a dialogue between the rhythms of the piece and the biological rhythms of the participants – between the computational model and the living body. The engineer Zoran Milkovic designed the sculptural form for the set – a screen system shaped like a truncated hexagonal pyramid, and large enough to contain several participants, with the visuals being back-projected onto three adjacent walls to achieve an immersive effect. Zoran also produced a 3D computer model that served both as a virtual maquette for designing the interaction software, and as a guide for the final production. This brought a much needed new impetus. Richard Newcombe, a University of Essex computer scientist now at the University of Washington, designed the hardware and software to support the interactive component of Fugue. The participants were lit from above by an infrared light source, and the resulting images were streamed from an infrared video camera. Richard then modified his particle filtering and computer vision software, originally designed for scientific research, to track the positions and movements of the participants, and to send the data in real time to the immune system algorithm, which would then respond in a complex time-delayed way through the visuals and the sound. After a few more weeks of intense work on the project, developing a common language with our new collaborating scientist in the process, we had our piece fully developed and tested.

The usual venues for art|science projects are of two kinds: a closed circle of specialised events with well informed audiences, which never engage with the wider art scene; or science education events, which usually target a very young audience and address scientific issues in a predominantly pedagogical manner. The first large-scale interactive Fugue installation (Fig. 8.5) in Belgrade in August 2006 was rather different, as it took place in the ULUS fine art gallery. Exhibiting the piece in a fine arts context produced both expected and unexpected results. The expected responses came mainly from artists and scientists, each group initially seeing the piece as the province of the other – a perennial problem with art|science. What was quite unexpected was the degree of suspicion and fear showed by some members of the public, who hesitated at the threshold of the enclosure and refused to enter it, demanding to know what personal data were being collected from them, on whose behalf, and for what purpose. The causes of the intensity of their reaction, and their suspicion of the medical establishment, has become a topic for investigation in future developments of Fugue.

Fig. 8.5
figure 5

Fugue. A participant is about to enter the Belgrade installation

Full size image

Variations on the Theme of Immersion

Fugue was also exhibited in the large-scale group show entitled “Infectious: Stay Away” at Trinity College Dublin’s Science Gallery. Oddly enough, it was the first time that Fugue had been shown in the context of the public engagement with science. The curatorial concept for this intriguing exhibition was marked by spectacular design and an imaginative theatrical script. An excellent balance of varied and clearly distinct takes on the theme of infection included straight scientific demonstrations along with artworks of a more reflective nature, and provided a perfect framework for both the artistic and scientific aspects of Fugue to unfold and communicate.

The Science Gallery exhibitions always attract the general public in large numbers (nearly 9,000 came in the first week or so of Infectious), and the 3 month duration of the show demanded a robust, durable and safe design. We therefore decided to show a new version of Fugue, a screen-based presentation of the free-running real-time generated sounds and visuals in a non-interactive and non-immersive environment. For this show, we wanted to guide the audience towards the meditative properties of the piece, and to focus on the complexity of the interplay between the artistic expressions, and the free-running real-time generated dialogue between the artificial immune system components. The intimate set consisted of a large plasma screen in a small cubicle, juxtaposed with the overall spectacular setting of the gallery and the other pieces. We invited visitors to “watch and listen, and become attuned to the processes and rhythms that are mirroring what might be happening right now, inside your own body”; this invitation had a certain immediacy, as the outbreak of the swine flu epidemic coincided with the opening of the exhibition. The response from both the visitors and the curatorial team was very positive, with every indication that we had achieved our aims.

However, what was most remarkable was that the senior scientists curating the exhibition took Fugue into the heart of the scientific establishment by interviewing me, and publishing the outcome in the prestigious journal Nature Immunology [6]. I believe this showed that they had accepted that the legitimacy and value of the art|science concept for science goes beyond public engagement with science, and that it is capable of making a distinctive contribution to both of its core disciplines in its own right. A similar softening of disciplinary boundaries can be seen in subsequent invitations to present Fugue as an artistic contribution to “Risk inSight”, an exhibition dealing with the scientific and social aspects of risk [7], and also at “State of Mind”, an exhibition on aspects of consciousness [8] presented by the Sackler Centre for Consciousness Science at the University of Sussex.

Quodlibet

Although the content and form of Fugue has attracted most attention, it is perhaps the nature of the technologies employed that carries the deepest message. It is becoming clear that the digital revolution has changed the nature of our perceptual processes, and this in turn has changed our conscious experience of the physical world, inducing changes in cognition on a scale that is still unknown. Some of the most radical insights into the essence of the problems arising from a digital culture come from the media critic Paul Virilio. He identifies the economic and political origins and aspects of the digital revolution, and its socio-political effects, particularly globalisation and global militarisation, mediated perception and new forms of alienation. He paints a dark and accurate picture of the current world, with an even darker vision of the future: “One day the day will come when the day won’t come” [9]. His disturbing, dramatic warnings about the potential remodelling of humans by means of technology carry a strong message and call for a revolt against the tyranny of real time interactivity and media, questioning the ethics of both the arts and the sciences.

But of course, issues similar to these have been explored across all art disciplines. An increasing number of artists working with technology are investigating and experimenting with the phenomena arising directly from the interplay between our senses and technology – for example, Stelarc through his concepts of obsolete bodies and exoskeletons [10], Char Davies with her pioneering bio-feedback VR [11], Rainer Linz analysing the physiological aspects of electronic music [12], and Margaret Dolinsky using digital art to study cognitive recognition and perceptual shifts [13].

We might think that science should provide some answers to these questions, but with very few exceptions, mainstream science has ignored the possible cognitive consequences of these dramatic environmental changes. For example, the majority of scientists investigating perception, particularly within the discipline of neuroscience, still refer to the ‘real world’ as a constant, an axiom, unchanged through time. But there is a further problem within neuroscience itself: the dominant doctrine, recognising slow evolutionary processes as the only way for qualitative changes to happen in a species, has combined with the established view that the brain is a closed and static system with fixed locations for particular functions, leaving no space to consider changes occurring within periods of 10 years – or even less. It is assumed that we come into this world with a genetically predetermined design for the brain, with exact locations for different functions, and that this structure remains constant throughout our lives.

However, over many years, a small number of scientists have been breaking out of this rigid context to show that the brain is not a closed, unchangeable system. We are now seeing growing scientific evidence that the brain is in fact almost nakedly open to external influences, and is capable of rapid and radical change by remodelling itself through learning and interaction with the environment. The field of neuroscience is now yielding evidence that may revolutionise not only the science of cognition, but also the wider view of the relationship between humans and the environment, and even the role and nature of culture. The brain can no longer be regarded as a fixed, closed, passive receiver of information from the senses – a mere processor for the information that controls our body through a kind of one-way communication. Rather, it is intrinsically plastic, in a process of constant change and growth through its interaction with the environment, and through a variety of learning processes.

It is certainly too extensive and complicated a matter to be reviewed adequately in this brief paper, but we can pick out some pioneers. The late Dr. Paul Bach-Y-Rita [14] was one of the first neuroscientists to work on what is now called neuroplasticity. His approach was not just theoretical, but practical: he worked with technical experts to construct electronic devices that would enable the brain of a patient with severe sensory problems to recover the lost functions. His method was to provide the patient’s brain with the missing information through a different sensory channel. In 1969 he provided blind people with ‘visual’ information by transferring a camera image to the patients’ skin using an array of vibrating pins, and his success led to the radical concept of ‘seeing with the brain’.

We also now have a wealth of scientific evidence that shows that the way in which we use and exercise our brains really does matter. Another neuroplasticity pioneer, the neuroscientist Michael Merzenich [15], argues that learning and practising certain skills can rapidly change hundred of millions of connections in our brain, improving and speeding up a wide variety of cognitive abilities. His experiments over the years have delivered strong arguments against the idea of fixed functions in fixed locations in the brain. He has been particularly active in discovering how new learning can stimulate the brain to counteract age-related deterioration, or the effects of serious brain injury, or language impairment in children. Most importantly, he has found that the most powerful way of delivering the learning tasks is through the use of digital technology: the speed and flexibility of his interactive computer-based training scheme enable the delivery of more effective rewards, in turn speeding up the rate of learning.

In a recent book on neuroplasticity, “The Brain that Changes Itself”, Norman Doidge [16] seems to offer a roadmap for future connections between disciplines grounded in neuroplasticity. In the chapter ‘The Culturally Modified Brain’ he writes:

As we use an electronic medium, our nervous system extends outwards, and the medium extends inwards. […] Because our nervous system is plastic, it can take the advantage of this compatibility and merge with the electronic media, making a single, larger system. Indeed, it is the nature of such systems to merge whether they are biological or man-made. [16], p. 311

In the case of an electronic device playing the role of a substitute for a lost capacity, or as an assistant in its regeneration, our body’s response can take a dramatic form, because the way in which electronic and digital devices transmit information is in essence quite similar to the basic function of our nervous system – the almost instantaneous transmission of electrical impulses. Due to its capacity for plastic changes, our nervous system easily rewires itself and makes use of this alternative nervous system. In a passage that could have come from the filmmaker and media critic Peter Watkins, Doidge notes that it is actually the form of electronic media, and not so much the content, that affects our cognitive processes:

It is the form of the television medium—cuts, edits, zooms, pans and sudden noises—that alters the brain, by activating what Pavlov called the ‘orienting response’, which occurs whenever we sense a sudden change in the world around us, especially a sudden movement. […] The response is physiological […]. [16], p. 309

Elsewhere, Merzenich emphasises the unprecedented opportunities that now exist for digital technologies to affect our brains:

The internet is just one of those things that contemporary humans can spend millions of ‘practice’ events at, that the average human a thousand years ago had absolutely no exposure to. Our brains are massively remodelled by this exposure—but so, too, by reading, by television, by modern electronics, by contemporary music, by contemporary ‘tools’, etc. [15]

Merzenich’s remarks date from 2005. Since then we have seen the introduction of the iPhone (2007) and other smartphones, and the development of the iPad (2010) and other tablets. In conjunction with developments in social networking, such as the opening of Facebook to anyone over 13 with an email address (2006 – it was previously limited to students), and Twitter (2006), these changes have created an explosion in the use of ‘always-on’ mobile devices, especially among young people, and so Merzenich’s comments are now more relevant than ever.

Because we now have this solid evidence that interaction with electronic media can not only affect our perception and cognition, but can also produce rapid and irreversible changes in our brains, the potentially damaging nature of this aspect of modernity confronts and challenges humanity with a set of serious problems. However, as far as I am aware, the true nature and extent of the influence of the modern urban environment, whether private, public, or workspace, has not yet been the subject of an in-depth scientific analysis. We can, of course, exercise choice even in the face of this onslaught – and neuroplasticity also tells us that the extent to which we can shape our own lives through the ways we choose to use our brains is far larger than we once thought it was.

Coda

It is clear that as artists we must continue to engage with this theme, although we know that this is a difficult task in this age of immaturity, kindergarten states and the hyperreality of the profit-driven post-digital revolution. For artists, however, there may be problems closer to home. This new understanding of cognitive processes is warning us that experiencing art, and especially electronic art, or technology-enabled art, is far from being an innocently entertaining or aesthetically pleasing experience extending for a limited period of time. The disturbing evidence of neuroplasticity raises the possibility that experiencing particular forms of art may itself affect and mark our cognition – perhaps with irreversible and unknown changes. But of course we cannot abandon technology: we must instead seek a deeper understanding of its effects on humanity by looking at all its aspects, positive, negative, and unknown. And here, the dramatic shift in neuroscience brings with it a fascinating opportunity to explore and analyse the effects of electronic media through scientifically informed art, which could give rise to an entirely new art form: neuroplastic art [17].

The concept of neuroplastic art opens a future for scientifically articulate artists and artistically articulate scientists to work closely together, with a full awareness of both the potential and the danger that emerges from the parallels between the nature of our nervous system and the characteristics of digital technology and electronic media. It may be possible to structure artworks according to new scientific evidence, and to fuse scientific knowledge with imagination to exploit the nature of electronic media to create platforms for experiences that have never existed before. Bringing together the scientists’ knowledge about the brain, and our knowledge of the properties of electronic media, we can envisage art works that will become in a way tuneable complex instruments, serving both art and science. Only then will imagination and creativity transcend today’s mere fascination with state-of-the-art technology, and use both technology and brain science as a means to express ideas. And perhaps this will even uncover new and benign ways of linking our brains with, and through, technology.

However, there is a hidden danger lurking within the concept of neuroplastic art. Does its involvement with neuroscience conceal a tacit acceptance that neuroscience alone can bring the final answers to all perception-related questions, the enigma of digitally enabled artefacts included? This is not necessarily the case, and it is reassuring that a new and strong critique that challenges many current dogmas of neuroscience has now appeared from the field of neurophenomenology, a discipline embracing recent research in neuroscience but also firmly grounded in the philosophy of Maurice Merleau-Ponty. A key figure in neurophenomenology is Alva Noë, who is part philosopher, part cognitive scientist, and part neuroscientist. Together with Evan Thompson and others, he offers a hypothesis about perception in action which builds on Merleau-Ponty’s idea of perception as a process of interaction between the embodied and situated human and the world. He also rejects the idea that artists should just be objects of scientific investigation, as they are in neuroaesthetics, and believes that they should actively contribute to the fields of perception and consciousness studies.

Noë’s iconoclastic views can be sampled in his recent book ‘Out of our Heads’ [18]. A key passage reads:

Our culture is obsessed with the brain—how it perceives; how it remembers; how it determines our intelligence, our morality, our likes and our dislikes. It’s widely believed that consciousness itself, that Holy Grail of science and philosophy, will soon be given a neural explanation. And yet, after decades of research, only one proposition about how the brain makes us conscious—how it gives rise to sensation, feeling, and subjectivity—has emerged unchallenged: we don’t have a clue.

Among other things, he strongly criticises the limitations of current brain scanning technologies, emphasising the fact that they are incapable of accessing the processes occurring during an individual’s freely moving spontaneous interaction with the environment. This is significant, because such a technology is precisely what is needed for enabling a deeper understanding of the nature of interactive environments and artistic installations – especially if they involve neuroplasticity.

In our future development of Fugue, we intend to continue our earlier explorations of intuitive interactivity, focusing on the situation of a participant engaged in a spontaneous communication with another entity, in the form of a non-verbal dialogue with the largely virtual body of the digital interactive installation. The technological and scientific aspects of the work will concentrate on turning Fugue into a tuneable interactive instrument, like those mentioned above, which will allow the visual and aural elements to be calibrated in terms of the participant’s response. If successful, this will give us a way of controlling the audio-visual and kinaesthetic experiences and provide us with a specific tool for analysing the phenomena in question from both the artistic and scientific points of view. Our main problem, identified by Noë, is that the technology that we need for monitoring the relevant brain responses of a freely moving participant is not yet available. (However, wearable technologies for the real-time remote monitoring of major physical and physiological variables have been developed in a number of contexts, including their use in interactive media installations [19].)

But help may be at hand, because we are not the only group with these requirements. The current European project CEEDS ‘The Collective Experience of Empathic Data Systems’ [20] is committed to developing ‘unobtrusive multi-modal wearable technologies to measure people’s reactions…including users’ heart rate, skin conductance, eye gaze, observable behaviours, speech characteristics, and brain activity.’ We have opened a dialogue with one of the project partners (the University of Sussex) and we hope to have access to this equipment within a reasonable time-frame. This will not solve all of our problems – it is unlikely to extend to direct readings of brain activity at the levels sought by Noë, at least initially – but it should be capable of testing and validating our approach, and preparing the way for our ultimate vision.

In summary, what have we learned from Fugue so far? Firstly, in relation to the tensions between art and science, many scientists are increasingly ready to accept the validity of artistic contributions, and no longer see them as being limited to a role in the public understanding of science. Such scientists make excellent collaborators for artists, as long as each group respects the discipline-specific expertise of the other. Secondly, we must accept that the way a piece is perceived and experienced will differ for different audiences, and we must actively seek and respond to those perceptions and experiences – for example by accepting invitations to exhibit in unusual contexts – rather than trying to impose our own reading. And thirdly, some scientific ideas, such as neuroplasticity, can have a power and a relevance for technologically-enabled art that goes well beyond serving as the inspiration for individual artworks, and we must engage with this science as well as with the technology.

In conclusion, as a last comment on our responsibility and opportunity as artists to engage with these issues, we should perhaps recall the prophetic words of Heidegger in one of the most frequently analysed philosophical texts on technology [21]:

…essential reflection upon technology and decisive confrontation with it must happen in a realm that is, on the one hand, akin to the essence of technology and, on the other, fundamentally different from it. Such a realm is art.