FuXi: A Fish-Driven Instrument for Real-Time Music Performance

  • João CordeiroEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9027)


In this paper we present a system for real-time computer music performance (live electronics) and live visuals based on the behavior of a fish in an aquarium. The system is comprised of (1) an aquarium with a fish; (2) a computer vision module; (3) a visual display of the fish overlaid by graphical elements controlled by the user, (4) a sound synthesis module and (5) a standard MIDI controller. The musical expression and graphic generation is a combination of the fish movements and decisions made by the performer in real-time. By making use of a live animal, the system provides indeterminacy and natural gestures to the sound being generated. The match between sound and image shows some semantic redundancy, aiming at a more narrative compositional approach where the fish is the main character. The system is targeted to soundscape composition and electroacoustic music featuring a high degree of improvisation.


Fish-driven instrument Electroacoustic music NIME HCI Improvisation Indeterminism 



We would like to acknowledge the valuable artistic inputs coming from Prof. Álvaro Barbosa, Dr. José Alberto Gomes and Ana Cristina Dias; the expertise and encouragement from Professor David Flores Gonalves in the field of biology and fish behavior; and the financial support from GAES, which made possible the presentation of this work abroad.


  1. 1.
    Taruskin, R.: Music in the Late Twentieth Century: The Oxford History of Western Music. Oxford University Press, New York (2009)Google Scholar
  2. 2.
    Chadabe, J.: Electric Sound: The Past and Promise of Electronic Music. Pearson Education, Upper Saddle River (1996)Google Scholar
  3. 3.
    Cage, J.: Silence, 1st edn. Wesleyan University Press, Middletown (1961)Google Scholar
  4. 4.
    de Leeuw, T.: Music in Twenty Century: A Study of Its Elements and Structure. Amsterdam University Press, Amsterdam (2005)CrossRefGoogle Scholar
  5. 5.
    Collopy, F.: Color, form, and motion: Dimensions of a musical art of light. Leonardo 33(5), 355–360 (2000)CrossRefGoogle Scholar
  6. 6.
    DeWitt, T.: Visual music: searching for an aesthetic. Leonardo 20(2), 15–122 (1987)CrossRefGoogle Scholar
  7. 7.
    Paradiso, J., O’modhrain, S.: Current trends in electronic music interfaces. J. New Music Res. 32(4), 345–349 (2003). (Guest Editors’ Introduction)CrossRefGoogle Scholar
  8. 8.
    Peacock, K.: Instruments to perform color-music: two centuries of technological experimentation. Leonardo 21(4), 397–406 (1988)CrossRefGoogle Scholar
  9. 9.
    Franco, E., Griffith, N., Fernström, M.: Issues for Designing a flexible expressive audiovisual system for real-time performance & composition. In: Proceedings of the 2004 Conference on New Interfaces for Musical Expression (2004)Google Scholar
  10. 10.
    Levin, G., Lieberman, Z.: Sounds from shapes: audiovisual performance with hand silhouette contours in the manual input sessions. In: Proceedings of the 2005 Conference on New Interfaces for Musical Expression, pp. 115–120 (2005)Google Scholar
  11. 11.
    Levin, G.: Painterly interfaces for audiovisual performance. Ph.D. thesis, Massachusetts Institute of Technology (2000)Google Scholar
  12. 12.
    Lew, M.: Live Cinema: designing an instrument for cinema editing as a live performance. In: Proceedings of the 2004 Conference on New Interfaces for Musical Expression, pp. 144–149 (2004)Google Scholar
  13. 13.
    Schott, G.: Magia universalis natura et artis. Number, vol. 1 in Magia universalis natura et artis (1657)Google Scholar
  14. 14.
    Caccuri, V.: Canções Submersas (2008)Google Scholar
  15. 15.
    Quiet Enseble: Quintetto (2009)Google Scholar
  16. 16.
    Walker, B.N., Godfrey, M.T., Orlosky, J.E., Bruce, C., Sanford, J.: Aquarium sonification: Soundscapes for accessible dynamic informal learning environments. In: Proceedings of the 12th International Conference on Auditory Display, pp. 238–241. Citeseer (2006)Google Scholar
  17. 17.
    Walker, B.N., Kim, J., Pendse, A., Others: Musical soundscapes for an accessible aquarium: Bringing dynamic exhibits to the visually impaired. In: Proceedings of the International Computer Music Conference (ICMC 2007), Copenhagen, Denmark (2007)Google Scholar
  18. 18.
    Pendse, A., Pate, M., Walker, B.N.: The accessible aquarium: identifying and evaluating salient creature features for sonification. In: Proceedings of the 10th International ACM SIGACCESS Conference on Computers and Accessibility, Assets 2008, pp. 297–298. ACM, New York (2008)Google Scholar
  19. 19.
    Bruce, C., Walker, B.: Designing effective sound-based aquarium exhibit interpretation for visitors with vision impairments. In: Proceedings of the 12th International ACM SIGACCESS Conference on Computers and Accessibility, pp. 251–252. ACM (2010)Google Scholar
  20. 20.
    Jeon, M., Winton, R., Yim, J., Bruce, C., Walker, B.: Aquarium fugue: interactive sonification for children and visually impaired audience in informal learning environments. In: Proceedings of the 18th International Conference on Auditory Display, Atlanta (2012)Google Scholar
  21. 21.
    Jeon, M., Winton, R.J., Henry, A.G., Oh, S., Bruce, C.M., Walker, B.N.: Designing interactive sonification for live aquarium exhibits. In: Stephanidis, C. (ed.) HCII 2013, Part I. CCIS, vol. 373, pp. 332–336. Springer, Heidelberg (2013) CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  1. 1.Faculty of Creative IndustriesUniversity of Saint JosephMacauChina
  2. 2.CITAR-Research Center in Science and Technology of the ArtsPortuguese Catholic University of PortugalOportoPortugal

Personalised recommendations