Skip to main content


A multimodal tangible interface for the sonification of phenological data at multiple time-scales


The study of periodic biological processes, such as when plants flower and birds arrive in the spring is known as Phenology. In recent years this field has gained interest from the scientific community because of the applicability of this data to the study of climate change and other ecological processes. In this paper we propose the use of tangible interfaces for interactive sonification with a specific example of a multimodal tangible interface consisting of a physical paper map and tracking of fiducial markers combined with a novel drawing interface. The designed interface enables one or more users to specify point queries with the map interface and to specify time queries with the drawing interface. This allows the user to explore both time and space while receiving immediate sonic feedback of their actions. This system can be used to study and explore the effects of climate change, both as tool to be used by scientists, and as a way to educate and involve members of the general public in a dynamic way in this research.

This is a preview of subscription content, access via your institution.


  1. 1.

    Bell B (1970) The oldest records of the Nile floods. Geogr J 136:569–573

    Article  Google Scholar 

  2. 2.

    Bimber O, Raskar R (2005) Spatial augmented reality: merging real and virtual worlds. A K Peters, Natick

    Google Scholar 

  3. 3.

    Bregman AS (1990) Auditory scene analysis: the perceptual organization of sound. MIT Press, Cambridge

    Google Scholar 

  4. 4.

    Costanza E, Robinson J (2003) A region adjacency tree approach to the detection and design of fiducials. In: Video, vision and graphics, pp 63–69

    Google Scholar 

  5. 5.

    Doi H, Katano I (2008) Phenological timings of leaf budburst with climate change in Japan. Agric For Meteorol 148(3):512–516

    Article  Google Scholar 

  6. 6.

    Fiala M (2010) Designing highly reliable fiducial markers. IEEE Trans Pattern Anal Mach Intell 32(7):1317–1324

    MathSciNet  Article  Google Scholar 

  7. 7.

    Fitch T, Kramer G (1992) Sonifying the body electric: superiority of an auditory over a visual display in a complex, multi-variate system. In: Proceedings of the first international conference on auditory display, pp 307–326

    Google Scholar 

  8. 8.

    Flowers JH, Whitwer LE, Grafel DC, Kotan CA (2001) Sonification of daily weather records. In: Issues of perception, attention and memory in design choices, Espoo, Finland, pp 222–226

    Google Scholar 

  9. 9.

    Funakoshi S, Kanda F (2000) Phenological study of common purple lilac flowering as a study program for environmental education. J Environ Educ 3:77–81

    Google Scholar 

  10. 10.

    Harding C, Kakadiaris IA, Casey JF, Loftin RB (2002) A multi-sensory system for the investigation of geoscientific data. Comput Graph 26(2):259–269

    MATH  Article  Google Scholar 

  11. 11.

    Harding C, Souleyrette RR (2010) Investigating the use of 3d graphics, haptics (touch), and sound for highway location planning. Comput-Aided Civ Infrastruct Eng 25(1):20–38

    Article  Google Scholar 

  12. 12.

    Ishii H, Ullmer B (1997) Tangible bits: towards seamless interfaces between people, bits and atoms. In: Proceedings of the CHI 1997

    Google Scholar 

  13. 13.

    J P, F I, Comasx P (2002) Changed plant and animal life cycles from 1952 to 2000 in the Mediterranean region. Glob Change Biol 8(6):531–544

    Article  Google Scholar 

  14. 14.

    Jordà S, Geiger G, Alonso M, Kaltenbrunner M (2007) The reactable: Exploring the synergy between live music performance and tabletop tangible interfaces. In: Proceedings of the international conference tangible and embedded interaction [TEI], pp 139–146

    Chapter  Google Scholar 

  15. 15.

    Landy L (2007) Understanding the art of sound organization

    Google Scholar 

  16. 16.

    Lieth H (1974) Phenology and seasonality modeling. Springer, New York

    Google Scholar 

  17. 17.

    Menzel A, Sparks TH, Estrella N, Roy DB (2006) Altered geographic and temporal variability in phenology in response to climate change. Glob Ecol Biogeogr 15(5):498–504

    Google Scholar 

  18. 18.

    Ness SR, Reimer P, Krell N, Odowichuck G, Schloss WA, Tzanetakis G (2010) Sonophenology: a tangible interface for sonification of geo-spatial phenological data at multiple time-scales. In: International conference on auditory display, pp 55–62

    Google Scholar 

  19. 19.

    Oviatt S (2002) Multimodal interfaces. Erlbaum, Lawrence

    Google Scholar 

  20. 20.

    Post E, Stenseth NC (1999) Climatic variability, plant phenology, and northern ungulates. Ecology 80(4):1322–1339

    Article  Google Scholar 

  21. 21.

    Quinn M (2001) Research set to music: the climate symphony and other sonifications of ice core, radar, dna, seismic and solar wind data. In: Laboratory of acoustics and audio signal processing and the telecommunications software and multimedia laboratory, Helsinki University of Technology, pp 56–61

    Google Scholar 

  22. 22.

    Walker BN (2007) Consistency of magnitude estimations with conceptual data dimensions used for sonification. Appl Cogn Psychol 21:579–599

    Article  Google Scholar 

  23. 23.

    Walker BN, Kramer G, Lane DM (2000) Psychophysical scaling of sonification mappings. In: International community for auditory display, international community for auditory display, Atlanta, GA, USA. Proceedings/2000/WalkerKramer2000.pdf

    Google Scholar 

  24. 24.

    Walther G, Post E, Convey P, Menzel A, Parmesan C, Beebee T, Fromentin J, Hoegh-Guldberg O, Bairlein F Ecological responses to recent climate change. Nature 416 (2002)

  25. 25.

    Zhao H (2006) Interactive sonification of abstract data: framework, design space, evaluation, and user tool. PhD thesis, College Park, MD, USA, AAI3236730

  26. 26.

    Zhao H, Plaisant C, Shneiderman B Data sonification for users with visual impairement: a case study with georeferenced data. ACM Trans on Comput-Hum Interact 15(1) (2008)

Download references

Author information



Corresponding author

Correspondence to Steven Ness.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Ness, S., Reimer, P., Love, J. et al. Sonophenology. J Multimodal User Interfaces 5, 123–129 (2012).

Download citation


  • Sonification
  • Tangible interfaces
  • Geo-spatial data
  • Phenology