Universal Access in the Information Society

, Volume 12, Issue 4, pp 339–368 | Cite as

Equivalent representations of multimodal user interfaces

Runtime Reification of Abstract User Interface Descriptions
Long paper
First Online:

Abstract

While providing non-visual access to graphical user interfaces has been a topic of research for over 20 years, blind users still face many obstacles when using computer systems. Furthermore, daily life has become more and more infused with devices that feature some kind of visual interface. Existing solutions for providing multimodal user interfaces that ensure accessibility are largely based on either graphical toolkit hooks, queries to the application and environment, scripting, model-driven user interface development or runtime adaptation. Parallel User Interface Rendering (PUIR) is a novel approach based on past and current research into accessibility, promoting the use of abstract user interface descriptions. Based on a single consistent conceptual model, PUIR provides a mechanism to render a user interface simultaneously in multiple modalities. Each representation of the UI provides equivalent semantics to ensure that collaboration between users of different modalities is facilitated. The possible application of this novel technique goes well beyond the realm of accessibility, as multimodal interfaces become more popular and even needed. The design presented here has been implemented as a prototype for testing and further research in this fascinating area of HCI.

Keywords

Accessibility UIDL Universal access Multimodal interfaces HCI 
This is a preview of subscription content, log in to check access.

Notes

Acknowledgments

The research presented in this paper is part of the author’s doctoral work at the Katholieke Universiteit Leuven, Belgium, under supervision by Jan Engelen (ESAT-SCD-Research Group on Document Architectures).

References

  1. 1.
    Ali, M.F.: A transformation-based approach to building multi-platform user interfaces using a task model and the user interface markup language. Ph.D. thesis, Virginia Polytechnic Institute and State University (2004)Google Scholar
  2. 2.
    Barnicle, K.: Usability testing with screen reading technology in a Windows environment. In: Proceedings of the 2000 Conference on Universal Usability, CUU ’00, pp. 102–109. ACM (2000)Google Scholar
  3. 3.
    Bergman, E., Johnson, E.: Toward accessible human-computer interaction. In: J. Nielsen (ed.) Advances in Human-Computer Interaction, vol. 5, pp. 87–113. Ablex Publishing Corp., New York (1995)Google Scholar
  4. 4.
    Bishop, J., Horspool, N.: Developing principles of GUI programming using views. In: Proceedings of the 35th SIGCSE technical symposium on Computer science education, SIGCSE ’04, pp. 373–377. ACM (2004)Google Scholar
  5. 5.
    Blattner, M., Glinert, E., Jorge, J., Ormsby, G.: Metawidgets: towards a theory of multimodal interface design. In: Computer Software and Applications Conference, 1992. COMPSAC ’92. Proceedings., Sixteenth Annual International, pp. 115–120. IEEE Computer Society Press (1992)Google Scholar
  6. 6.
    Bouillon, L., Vanderdonckt, J., Chow, K.C.: Flexible re-engineering of web sites. In: Proceedings of the 9th International Conference on Intelligent user interfaces, IUI ’04, pp. 132–139. ACM (2004)Google Scholar
  7. 7.
    Bouraoui, A., Soufi, M.: Improving computer access for blind users. In: K. Elleithy (ed.) Advances and Innovations in Systems, Computing Sciences and Software Engineering, pp. 29–34. Springer, The Netherland (2007)CrossRefGoogle Scholar
  8. 8.
    Braille Authority of North America: Braille Formats: Principles of Print to Braille Transcription 1997. American Printing House for the Blind (1998)Google Scholar
  9. 9.
    Brunet, P., Feigenbaum, B.A., Harris, K., Laws, C., Schwerdtfeger, R., Weiss, L.: Accessibility requirements for systems design to accommodate users with vision impairments. IBM Syst. J. 44(3), 445–466 (2005)CrossRefGoogle Scholar
  10. 10.
    Buxton, W., Gaver, W., Bly, S.: Auditory interfaces: The use of non-speech audio at the interface (1994). Draft manuscriptGoogle Scholar
  11. 11.
    Calvary, G., Coutaz, J., Thevenin, D.: A unifying reference framework for the development of plastic user interfaces. In: Proceedings of the 8th IFIP International Conference on Engineering for Human-Computer Interaction, EHCI ’01, pp. 173–192. Springer, Berlin (2001)Google Scholar
  12. 12.
    Calvary, G., Coutaz, J., Thevenin, D., Limbourg, Q., Bouillon, L., Vanderdonckt, J.: A unifying reference framework for multi-target user interfaces. Interact. Comput. 15(3), 289–308 (2003)CrossRefGoogle Scholar
  13. 13.
    Calvary, G., Coutaz, J., Thevenin, D., Limbourg, Q., Souchon, N., Bouillon, L., Florins, M., Vanderdonckt, J.: Plasticity of user interfaces: a revised reference framework. In: Proceedings of the First International Workshop on Task Models and Diagrams for User Interface Design, pp. 127–134. INFOREC Publishing House Bucharest (2002)Google Scholar
  14. 14.
    Chapman, M.: Create user interfaces with glade. Linux J. 87, 90–92,94 (2001)Google Scholar
  15. 15.
    Congress of the United States of America: 42 U.S.C.—The Public Health and Welfare, Section 1382(a)2). GPO (1997)Google Scholar
  16. 16.
    Coutaz, J., Nigay, L., Salber, D.: Multimodality from the user and system perspectives. In: Proceedings of the ERCIM’95 Workshop on Multimedia Multimodal User Interfaces (1995)Google Scholar
  17. 17.
    Draheim, D., Lutteroth, C., Weber, G.: Graphical user interfaces as documents. In: Proceedings of the 7th ACM SIGCHI New Zealand chapter’s International Conference on Computer-human interaction: design centered HCI, CHINZ ’06, pp. 67–74. ACM (2006)Google Scholar
  18. 18.
    Edwards, A.D.N.: The difference between a blind computer user and a sighted one is that the blind one cannot see (1994). Interactionally Rich Systems Network, Working Paper No. ISS/WP2Google Scholar
  19. 19.
    Edwards, A.D.N., Mitsopoulos, E.: A principled methodology for the specification and design of nonvisual widgets. ACM Trans. Appl. Percept. 2(4), 442–449 (2005)CrossRefGoogle Scholar
  20. 20.
    Edwards, W.K., Mynatt, E.D., Stockton, K.: Providing access to graphical user interfaces—not graphical screens. In: Proceedings of the First Annual ACM Conference on Assistive Technologies, Assets ’94, pp. 47–54. ACM (1994)Google Scholar
  21. 21.
    Gajos, K., Weld, D.S.: SUPPLE: automatically generating user interfaces. In: IUI ’04: Proceedings of the 9th International Conference on Intelligent User Interfaces, pp. 93–100. ACM Press (2004)Google Scholar
  22. 22.
    Gaver, W.W.: The sonicfinder: an interface that uses auditory icons. Hum.-Comput. Interact. 4(1), 67–94 (1989)MathSciNetCrossRefGoogle Scholar
  23. 23.
    Guerrero-Garcia, J., González-Calleros, J.M., Vanderdonckt, J., Muñoz-Arteaga, J.: A theoretical survey of user interface description languages: preliminary results. In: Latin American Web Congress, 2009. LA-WEB ’09, pp. 36–43 (2009)Google Scholar
  24. 24.
    Gunzenhäuser, R., Weber, G.: Graphical user interfaces for blind people. In: K. Brunnstein, E. Raubold (eds.) 13th World Computer Congress 94, vol. 2, pp. 450–457. Elsevier, Amsterdam (1994)Google Scholar
  25. 25.
    Haneman, B., Mulcahy, M.: The GNOME accessibility architecture in detail (2002). Presented at the CSUN Conference on Technology and DisabilitiesGoogle Scholar
  26. 26.
    Harness, S., Pugh, K., Sherkat, N., Whitrow, R.: Fast icon and character recognition for universal access to WIMP interfaces for the blind and partially sighted. In: Ballabio, E., Placencia-Porrero , I., Bellcasa, R.P.D.L. (eds.) Rehabilitation Technology: Strategies for the European Union (Proceedings of the First Tide Congress), pp. 19–23. IOS Press, Brussels (1993)Google Scholar
  27. 27.
    Hollins, M.: Understanding Blindness: An Integrative Approach. Lawrence Erlbaum Associates, Boca Raton (1989)Google Scholar
  28. 28.
    Institute of Electrical and Electronics Engineers: 610.12-1990, IEEE Standard Glossary of Software Engineering Terminology. IEEE, Los Alamos, CA (1990)Google Scholar
  29. 29.
    International Organization for Standardization: ISO/IEC 9126, Information Technology, Software Product Evaluation, Quality Characteristics and Guidelines for their Use. ISO, Geneva (1991)Google Scholar
  30. 30.
    International Organization for Standardization: ISO/IEC 9241-11, Ergonomic Requirements for Office Work with Visual Display Terminals (VDTs), Part 11: Guidance on Usability. ISO, Geneva (1998)Google Scholar
  31. 31.
    Jacob, R.J.K.: User interfaces. In: Ralston, A., Reilly, E.D., Hemmendinger, D. (eds.) Encyclopedia of Computer Science, 4th Edition. Grove Dictionaries, Inc., Williston (2000)Google Scholar
  32. 32.
    Kawai, S., Aida, H., Saito, T.: Designing interface toolkit with dynamic selectable modality. In: Proceedings of the second annual ACM conference on Assistive technologies, Assets ’96, pp. 72–79. ACM (1996)Google Scholar
  33. 33.
    Kay, A.C.: User interface: A personal view. In: Laurel, B. (ed.) The Art of Human-Computer Interface Design, pp. 191–207. Addison-Wesley Publishing Co, Boston (1990)Google Scholar
  34. 34.
    Kochanek, D.: Designing an offscreen model for a gui. In: Zagler, W., Busby, G., Wagner, R. (eds.) Computers for Handicapped Persons, Lecture Notes in Computer Science, vol. 860, pp. 89–95. Springer, Berlin/Heidelberg (1994)Google Scholar
  35. 35.
    Kost, S.: Dynamically generated multi-modal application interfaces. Ph.D. thesis, Technische Universität Dresden, Dresden, Germany (2006)Google Scholar
  36. 36.
    Kraus, M., Völkel, T., Weber, G.: An off-screen model for tactile graphical user interfaces. In: K. Miesenberger, J. Klaus, W. Zagler, A. Karshmer (eds.) Computers Helping People with Special Needs, Lecture Notes in Computer Science, vol. 5105, pp. 865–872. Springer, Berlin/Heidelberg (2008)Google Scholar
  37. 37.
    Kurniawan, S.H., Sutcliffe, A.G.: Mental models of blind users in the Windows environment. In: K. Miesenberger, J. Klaus, W. Zagler (eds.) Computers Helping People with Special Needs, Lecture Notes in Computer Science, vol. 2398, pp. 373–386. Springer, Berlin/Heidelberg (2002)Google Scholar
  38. 38.
    Kurniawan, S.H., Sutcliffe, A.G., Blenkhorn, P.L.: How blind users’ mental models affect their perceived usability of an unfamiliar screen reader. In: M. Rauterberg, M. Menozzi, J. Wesson (eds.) Human-Computer Interaction INTERACT ’03, pp. 631–638. IOS Press, Brussels (2003)Google Scholar
  39. 39.
    Laberge-Nadeau, C.: Wireless telephones and the risk of road crashes. Accid. Anal. Prev. 35(5), 649–660 (2003)CrossRefGoogle Scholar
  40. 40.
    Lauridsen, O.: Abstract specification of user interfaces. In: Conference companion on Human factors in computing systems, CHI ’95, pp. 147–148. ACM (1995)Google Scholar
  41. 41.
    Lévesque, V.: Blindness, technology and haptics. Technical Report TR-CIM-05.08, McGill University, Centre for Intelligent Machines, Haptics Laboratory (2008)Google Scholar
  42. 42.
    Limbourg, Q., Vanderdonckt, J.: UsiXML: A user interface description language supporting multiple levels of independence. In: Matera M., Comai S. (eds.) Engineering Advanced Web Applications, pp. 325–338. Rinton Press, Paramus (2004)Google Scholar
  43. 43.
    Limbourg, Q., Vanderdonckt, J., Michotte, B., Bouillon, L., Florins, M., Trevisan, D.: UsiXML: A user interface description language for context-sensitive user interfaces. In: K. Luyten, M. Abrams, J. Vanderdonckt, Q. Limbourg (eds.) Proceedings of the ACM AVI’2004 Workshop ”Developing User Interfaces with XML: Advances on User Interface Description Languages”, pp. 55–62 (2004)Google Scholar
  44. 44.
    Limbourg, Q., Vanderdonckt, J., Michotte, B., Bouillon, L., López-Jaquero, V.: UsiXML: A language supporting multi-path development of user interfaces. In: R. Bastide, P. Palanque, J. Roth (eds.) Engineering Human Computer Interaction and Interactive Systems, Lecture Notes in Computer Science, vol. 3425, pp. 134–135. Springer, Berlin/Heidelberg (2005)Google Scholar
  45. 45.
    de Melo, G., Honold, F., Weber, M., Poguntke, M., Berton, A.: Towards a flexible ui model for automotive human-machine interaction. In: Proceedings of the 1st International Conference on Automotive User Interfaces and Interactive Vehicular Applications, AutomotiveUI ’09, pp. 47–50. ACM (2009)Google Scholar
  46. 46.
    Morley, S.: Window Concepts: An Introductory Guide for Visually Disabled Users. Royal National Institute for the Blind, London (1995)Google Scholar
  47. 47.
    Mynatt, E.D.: Transforming graphical interfaces into auditory interfaces for blind users. Hum.-Comput. Interact. 12(1), 7–45 (1997)CrossRefGoogle Scholar
  48. 48.
    Mynatt, E.D., Edwards, W.K.: Mapping guis to auditory interfaces. In: Proceedings of the 5th Annual ACM Symposium on User Interface Software and Technology, UIST ’92, pp. 61–70. ACM (1992)Google Scholar
  49. 49.
    Mynatt, E.D., Weber, G.: Nonvisual presentation of graphical user interfaces: contrasting two approaches. In: Proceedings of the SIGCHI Conference on Human factors in Computing Systems: Celebrating Interdependence, CHI ’94, pp. 166–172. ACM (1994)Google Scholar
  50. 50.
    Newell, A.F.: CHI for everyone. Interfaces 35, 4–5 (1997)Google Scholar
  51. 51.
    Nigay, L., Coutaz, J.: A design space for multimodal systems: concurrent processing and data fusion. In: Proceedings of the INTERACT ’93 and CHI ’93 Conference on Human Factors in Computing Systems, CHI ’93, pp. 172–178. ACM (1993)Google Scholar
  52. 52.
    Parnas, D.L.: On the criteria to be used in decomposing systems into modules. Commun. ACM 15(12), 1053–1058 (1972)CrossRefGoogle Scholar
  53. 53.
    Pontelli, E., Gillan, D., Xiong, W., Saad, E., Gupta, G., Karshmer, A.I.: Navigation of HTML tables, frames, and XML fragments. In: Proceedings of the Fifth International ACM Conference on Assistive Technologies, Assets ’02, pp. 25–32. ACM (2002)Google Scholar
  54. 54.
    Rose, D., Stegmaier, S., Reina, G., Weiskopf, D., Ertl, T.: Non-invasive adaptation of black-box user interfaces. In: Proceedings of the Fourth Australasian User Interface Conference on User Interfaces 2003—vol. 18, AUIC ’03, pp. 19–24. Australian Computer Society, Inc. (2003)Google Scholar
  55. 55.
    Sacks, O.: The Mind’s Eye: What The Blind See. The New Yorker, New York, pp. 48–59 (2003)Google Scholar
  56. 56.
    Sadato, N., Pascual-Leone, A., Grafman, J., Deiber, M.P., Ibañez, V., Hallett, M.: Neural networks for braille reading by the blind. Brain 121, 1213–1229 (1998)CrossRefGoogle Scholar
  57. 57.
    Sadato, N., Pascual-Leone, A., Grafman, J., Ibañez, V., Deiber, M.P., Dold, G., Hallett, M.: Activation of the primary visual cortex by braille reading in blind subjects. Nature 380(6574), 526–528 (1996)CrossRefGoogle Scholar
  58. 58.
    Savidis, A., Stephanidis, C.: Building non-visual interaction through the development of the rooms metaphor. In: Conference Companion on Human Factors in Computing Systems, CHI ’95, pp. 244–245. ACM (1995)Google Scholar
  59. 59.
    Savidis, A., Stephanidis, C.: Developing dual user interfaces for integrating blind and sighted users: the HOMER UIMS. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’95, pp. 106–113. ACM Press/Addison-Wesley Publishing Co. (1995)Google Scholar
  60. 60.
    Savidis, A., Stephanidis, C.: The HOMER UIMS for dual user interface development: fusing visual and non-visual interactions. Interact. Comput. 11(2), 173–209 (1998)CrossRefGoogle Scholar
  61. 61.
    Seybold, J.: Xerox’s ”star”. The Seybold Report 10(16) (1981)Google Scholar
  62. 62.
    Smith, D.C., Harslem, E.F., Irby, C.H., Kimball, E.B., Verplank, W.L.: Designing the Star User Interface. BYTE, pp. 242–282 (1982)Google Scholar
  63. 63.
    Souchon, N., Venderdonckt, J.: A review of XML-compliant user interface description languages. In: Jorge, J.A., Jardim Nunes, N., Falcāo e Cunha, J. (eds.) Interactive Systems. Design, Specification, and Verification, Lecture Notes in Computer Science, vol. 2844, pp. 391–401. Springer Berlin / Heidelberg (2003)Google Scholar
  64. 64.
    Stephanidis, C., Savidis, A.: Universal access in the information society: Methods, tools, and interaction technologies. Univ. Access Inf. Soc. 1(1), 40–55 (2001)Google Scholar
  65. 65.
    Sun Microsystems: GNOME 2.0 desktop: Developing with the accessibility framework. Technical report, Sun Microsystems (2003)Google Scholar
  66. 66.
    Theofanos, M.F., Redish, J.G.: Bridging the gap: between accessibility and usability. Interactions 10(6), 36–51 (2003)CrossRefGoogle Scholar
  67. 67.
    Trewin, S., Zimmermann, G., Vanderheiden, G.: Abstract user interface representations: how well do they support universal access? In: Proceedings of the 2003 conference on Universal usability, CUU ’03, pp. 77–84. ACM (2003)Google Scholar
  68. 68.
    Trewin, S., Zimmermann, G., Vanderheiden, G.: Abstract representations as a basis for usable user interfaces. Interact. Comput. 16(3), 477–506 (2004)CrossRefGoogle Scholar
  69. 69.
    Vanderdonckt, J., Limbourg, Q., Michotte, B., Bouillon, L., Trevisan, D., Florins, M.: UsiXML: a user interface description language for specifying multimodal user interfaces. In: WMI ’04: Proceedings of the W3C Workshop on Multimodal Interaction (2004)Google Scholar
  70. 70.
    Weber, G.: Programming for usability in nonvisual user interfaces. In: Proceedings of the third international ACM conference on Assistive technologies, Assets ’98, pp. 46–48. ACM (1998)Google Scholar
  71. 71.
    Weber, G., Mager, R.: Non-visual user interfaces for X Windows. In: Proceedings of the 5th international conference on Computers helping people with special needs. Part II, pp. 459–468. R. Oldenbourg Verlag GmbH (1996)Google Scholar
  72. 72.
    Weber, G., Petrie, H., Kochanek, D., Morley, S.: Training blind people in the use of graphical user interfaces. In: W. Zagler, G. Busby, R. Wagner (eds.) Computers for Handicapped Persons, Lecture Notes in Computer Science, vol. 860, pp. 25–31. Springer, Berlin/Heidelberg (1994)Google Scholar

Copyright information

© Springer-Verlag 2012

Authors and Affiliations

  1. 1.Department of Electrical EngineeringKatholieke Universiteit LeuvenHeverleeBelgium

Personalised recommendations