Interactive hierarchy-based auditory displays for accessing and manipulating relational diagrams

  • Oussama MetatlaEmail author
  • Nick Bryan-Kinns
  • Tony Stockman
Original Paper


An approach to designing hierarchy-based auditory displays that supports non-visual interaction with relational diagrams is presented. The approach is motivated by an analysis of the functional and structural properties of relational diagrams in terms of their role as external representations. This analysis informs the design of a multiple perspective hierarchy-based model that captures modality independent features of a diagram when translating it into an audio accessible form. The paper outlines design lessons learnt from two user studies that were conducted to evaluate the proposed approach.


Auditory displays Diagrams Hierarchies Representational models Multiple perspectives Accessibility Interaction strategies 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Avanzini F, Crosato P (2006) Haptic-auditory rendering and perception of contact stiffness. In: Haptic and audio interaction design vol 4129, pp 24–35 CrossRefGoogle Scholar
  2. 2.
    Bauer MI, Johnson-Laird PN (1993) How diagrams can improve reasoning. Psychol Sci 4(6):372–378 CrossRefGoogle Scholar
  3. 3.
    Bennett D (2002) Effects of navigation and position on task when presenting diagrams to blind people using sound. In: Diagrammatic representation and inference, pp 307–319 Google Scholar
  4. 4.
    Best V, Van Schaik A, Carlile S (2003) Two-point discrimination in auditory displays. In: Brazil E, Shinn-Cunningham B (eds) Proceedings of the 9th international conference on auditory display (ICAD2003). Boston University Publications Production Department, Boston, pp 17–20 Google Scholar
  5. 5.
    Blattner MM, Sumikawa DA, Greenberg RM (1989) Earcons and icons: their structure and common design principles. Hum-Comput Interact 4(1):11–44 CrossRefGoogle Scholar
  6. 6.
    Bregman AS (1994) Auditory scene analysis: the perceptual organization of sound. MIT Press, Cambridge Google Scholar
  7. 7.
    Brown A, Pettifer S, Stevens R (2004) Evaluation of a non-visual molecule browser. In: Proceedings of the 6th international ACM SIGACCESS conference on computers and accessibility (Assets ’04). Atlanta, GA, USA, pp 40–47 Google Scholar
  8. 8.
    Gaver WW (1992) Using and creating auditory icons. SFI studies in the sciences of complexity. Addison/Wesley/Longman, Reading Google Scholar
  9. 9.
    Gurr CA, Lee J, Stenning K (1998) Theories of diagrammatic reasoning: distinguishing component problems. Minds and Machines 8(4):533–557 zbMATHCrossRefGoogle Scholar
  10. 10.
    Harary F, Norman RZ, Cartwright D (1965) Structural models: an introduction to the theory of directed graphs. Wiley, New York zbMATHGoogle Scholar
  11. 11.
    Horstmann M, Lorenz M, Watkowski A, Ioannidis G, Herzog O, King A, Evans DG, Hagen C, Schlieder C, Burn AM, King N, Petrie H, Dijkstra S, Crombie D (2004) Automated interpretation and accessible presentation of technical diagrams for blind people. In: New review of hypermedia and multimedia, vol 10. Taylor & Francis, London, pp 141–163 Google Scholar
  12. 12.
    Kramer G (1994) Auditory display: sonification, audification and auditory Interfaces. Addison-Wesley, Reading Google Scholar
  13. 13.
    Larkin JH, Simon HA (1987) Why a diagram is (sometimes) worth ten thousand words. Cognitive Science 11(1):65–100 CrossRefGoogle Scholar
  14. 14.
    Leplatre G, Brewster S (2000) Designing non-speech sounds to support navigation in mobile phone menus. In: Cook PR (ed) Proceedings of the 6th international conference on auditory display (ICAD2000), Atlanta, GA, USA Google Scholar
  15. 15.
    Mansur DL, Blattner MM, Joy KI (1985) Sound graphs: a numerical data analysis method for the blind. J Med Syst 9(3):163–174 CrossRefGoogle Scholar
  16. 16.
    McGookin D, Brewster SA (2006) Soundbar: exploiting multiple views in multimodal graph browsing. In: Proceedings of the 4th Nordic conference on human-computer interaction (NordiCHI ’06). New York, NY, USA, pp 145–154 CrossRefGoogle Scholar
  17. 17.
    Metatla O, Bryan-Kinns N, Stockman T (2007) Using hierarchies to support non-visual access to relational diagrams. In: Proceedings of the 21st British CHI group annual conference on HCI (BCS-HCI ’07). British Computer Society, Swinton, pp 215–225 Google Scholar
  18. 18.
    Metatla O, Bryan-Kinns N, Stockman T (2008) Constructing relational diagrams in audio: the multiple perspective hierarchical approach. In: Proceedings of the 10th international ACM SIGACCESS conference on computers and accessibility (Assets ’08). Halifax, Nova Scotia, Canada, pp 97–104 CrossRefGoogle Scholar
  19. 19.
    Mynatt ED, Edwards KW (1992) The Mercator environment: a nonvisual interface to x windows and UNIX workstations. Tech rep GIT-GVU-92-05 Google Scholar
  20. 20.
    Mynatt ED, Weber G (1994) Nonvisual presentation of graphical user interfaces: contrasting two approaches. In: Proceedings of the SIGCHI conference on human factors in computing systems (CHI ’94). Boston, Massachusetts, USA, pp 166–172 Google Scholar
  21. 21.
    Norman DA (1981) Categorization of action slips. Psychol Rev 88(1):1–15 MathSciNetCrossRefGoogle Scholar
  22. 22.
    Norman DA (1993) Things that make us smart: defending human attributes in the age of the machine. Addison-Wesley, Reading Google Scholar
  23. 23.
    Palmer SE (1977) Hierarchical structure in perceptual representation. Cogn Psychol 9(4):441–474 CrossRefGoogle Scholar
  24. 24.
    Petrie H, Harrison C, Dev S (2005) Describing images on the web: a survey of current practice and prospects for the future. In: Stephanidis C (ed) Proceedings of 3rd international conference on universal access in human-computer interaction Google Scholar
  25. 25.
    Scaife M, Rogers Y (1996) External cognition: How do graphical representations work. Int J Hum-Comput Stud 45(2):185–213 CrossRefGoogle Scholar
  26. 26.
    Shimojima A (1996) Operational constraints in diagrammatic reasoning. In: Logical reasoning with diagrams. Oxford University Press, New York, pp 27–48 Google Scholar
  27. 27.
    Shneiderman B (1996) The eyes have it: A task by data type taxonomy for information visualizations. In: Proceedings of the 1996 IEEE symposium on visual languages (VL ’96). IEEE Computer Society, Washington, p 336 CrossRefGoogle Scholar
  28. 28.
    Simon HA (1980) Cognitive science: The newest science of the artificial. Cogn Sci 4(1):33–46 CrossRefGoogle Scholar
  29. 29.
    Stevens RD, Edwards ADN, Harling PA (1997) Access to mathematics for visually disabled students through multimodal interaction. Hum-Comput Interact 12(1):47–92 CrossRefGoogle Scholar
  30. 30.
    Stevens SS (1946) On the theory of scales of measurement. Science 103(2684):677–680 CrossRefGoogle Scholar
  31. 31.
    Stockman T, Metatla O (2008) The influence of screen-readers on web cognition. In: Proceedings of Accessible design in the digital world conference (ADDW 2008), York, UK Google Scholar
  32. 32.
    Wall SA, Brewster SA (2006) Tac-tiles: multimodal pie charts for visually impaired users. In: Proceedings of the 4th Nordic conference on human-computer interaction (NordiCHI ’06), New York, NY, USA, pp 9–18 CrossRefGoogle Scholar
  33. 33.
    Weber G (1993) Adapting direct manipulation for blind users. In: Conference companion on human factors in computing systems (CHI ’93: INTERACT ’93 and CHI ’93), Amsterdam, The Netherlands, pp 21–22 CrossRefGoogle Scholar
  34. 34.
    Yu W, Kangas K, Brewster SA (2003) Web-based haptic applications for blind people to create virtual graphs. In: Proceedings of 11th symposium on haptic interfaces for virtual environment and teleoperator systems (HAPTICS 2003), pp 318–325 CrossRefGoogle Scholar
  35. 35.
    Zhang J (1996) A representational analysis of relational information displays. Int J Hum-Comput Stud 4(16):59–74 zbMATHCrossRefGoogle Scholar
  36. 36.
    Zhang J, Norman DA (1994) Representations in distributed cognitive tasks. Cogn Sci 18(1):87–122 CrossRefGoogle Scholar
  37. 37.
    Zhao H, Plaisant C, Schneiderman B, Duraiswami R (2004) Sonification of geo-referenced data for auditory information seeking: Design principle and pilot study. In: Barrass S, Vickers P (eds) Proceedings of the 10th international conference on auditory display (ICAD2004), Sydney, Australia Google Scholar

Copyright information

© OpenInterface Association 2011

Authors and Affiliations

  • Oussama Metatla
    • 1
    Email author
  • Nick Bryan-Kinns
    • 1
  • Tony Stockman
    • 1
  1. 1.School of Electronic Engineering and Computer ScienceQueen Mary University of LondonLondonUK

Personalised recommendations