Universal Access in the Information Society

, Volume 9, Issue 4, pp 337–356 | Cite as

A unified methodology for the evaluation of accessibility and usability of mobile applications

  • Marco BilliEmail author
  • Laura Burzagli
  • Tiziana Catarci
  • Giuseppe Santucci
  • Enrico Bertini
  • Francesco Gabbanini
  • Enrico Palchetti
Long Paper


This article reports a unified methodology developed to evaluate the accessibility and usability of mobile computing applications, which is intended to guarantee universal access as far as possible. As a basis for the methodology, this paper presents an analysis of the accessibility guidelines, conducted to take into account the specificity of mobile systems, as well as a set of usability heuristics, specifically devised for mobile computing. Finally, it presents the results of the application of the proposed methodology to applications that have been semi-automatically developed by the MAIS Designer, a new design tool that provides applications suited to different mobile devices.


Accessibility Usability Mobile computing 



Research partially supported by the MIUR/FIRB Project “MAIS” ( and the EC NoE G038-507618, “DELOS” ( We would like to thank Alan Dix for many fruitful discussions and comments on earlier drafts of this paper and the anonymous Reviewers for their useful suggestions.


  1. 1.
    Bertini, E., Calì, A., Santucci, G.: Automatic interface generation through interaction, users, and devices modeling. In: Proceedings of 6th International Conference on Computer-Aided Design of User Interfaces CADUI2006, Bucharest, Romania, June 5–8, 2006Google Scholar
  2. 2.
    Stephanidis, C., Savidis, A.: Universal Access in the Information Society: Methods, Tools and Interaction Technologies. Univers. Access Inf. Soci. J. 40–55, Springer-Verlag (2001)Google Scholar
  3. 3.
    Emiliani, P.L., Stephanidis, C.: Universal access to ambient intelligence environments: Opportunities and challenges for people with disabilities. IBM Sys. J. 44(3) (2005)Google Scholar
  4. 4.
    Gray, W.D., Salzman, M.C.: Damaged merchandise? A review of experiments that compare usability evaluation methods. Hum. Comput. Interact. 13(3), 203–261 (1998)CrossRefGoogle Scholar
  5. 5.
    Lang, T.: Comparing website accessibility evaluation methods and learnings from usability evaluation methods, December 2003
  6. 6.
    ISO/IEC Guide 71Guidelines for standards developers to address the needs of older persons and persons with disabilities. International Organization for Standardization (ISO) (2001)Google Scholar
  7. 7.
    ISO 9241-171Ergonomics of human-system interaction—Part 171: Guidance on software accessibility (2006)Google Scholar
  8. 8.
    ISO 9241-20 Ergonomics of human-system interaction—Part 20: Accessibility guidelines for information/communication technology (ICT) equipment and services. (2006)Google Scholar
  9. 9.
    Wegge, K.P., Zimmermann, D.: Accessibility, usability, safety, ergonomics: concepts, models, and differences. Universal Access in HCI, Part I, HCII 2007, LNCS 4554, pp. 294–301, Springer-Verlag, (2007)Google Scholar
  10. 10.
    Henry, S.L.: Introduction to web Accessibility. (2005)
  11. 11.
    Pernici, B. (ed.): Mobile Information Systems: Infrastructure and Design for Adaptivity and Flexibility, p. 237. Springer, Berlin (2006)Google Scholar
  12. 12.
    Petrie, H., Kheir, O.: The relationship between accessibility and usability of websites. CHI ‘07: In: Proceedings of the SIGCHI conference on Human factors in computing systems, pp. 397–406, ACM Press, USA (2007)Google Scholar
  13. 13.
    Thatcher, J., Waddell, C.D., Henry, S.L., Swierenga, S., Urban, M.D., Burks, M., Regan, B., Bohman, P.: Constructing Accessible Web Sites. Glasshaus, San Francisco (2003)Google Scholar
  14. 14.
    Shneiderman, B.: Universal usability. Commun. ACM 43(5), 85–91 (2000)Google Scholar
  15. 15.
    Shneiderman, B.: Promoting universal usability with multi-layer interface design. In: Proceedings of the 2003 Conference on Universal Usability (CUU 2003)Google Scholar
  16. 16.
    Antona, M., Mourouzis, A., Stephanidis, C.: Towards a Walk- through Method for Universal Access Evaluation. Universal Access in HCI, Part I, HCII 2007, LNCS 4554, pp. 325–334, Springer-Verlag (2007)Google Scholar
  17. 17.
    Theofanos, M.F., Redish, J.: Bridging the gap: between accessibility and usability. ACM Interact. 10(6), 36–51 (2003). ACM PressCrossRefGoogle Scholar
  18. 18.
    Chisholm, W., Vanderheiden, G., Jacobs, I.: Web Content Accessibility Guidelines 1.0 W3C Recommendation, ( May 5, 1999
  19. 19.
    Henry, S., Grossnicklem, M.: Just Ask: Accessibility in the User-Centered Design Process. Georgia Tech Research Corporation, Atlanta, Georgia, USA, 2004. On-line book:
  20. 20.
    Brajnik, G.: A comparative test of web accessibility evaluation methods. In: Proceedings of ASSETS’08, ACM Press, Halifax, 13–15 Oct 2008Google Scholar
  21. 21.
    Souza, R.: Design Accessible Sites Now, Forrester Report, December 2001,1338,11431,00.html
  22. 22.
    Zimmermann, G., Vanderheiden, G.: Accessible design and testing in the application development process: considerations for an integrated approach. Univer. Access Infor. Soci. Int. J. 7, 117–128 (2007). Springer-VerlagCrossRefGoogle Scholar
  23. 23.
    Manko, J., Dey, A.K., Hsieh, G., Kientz, J., Lederer, S., Ames, M.: Heuristic evaluation of ambient displays. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: CHI’03, ACM Press, pp. 169–176 (2003)Google Scholar
  24. 24.
    Baker, K., Greenberg, S., Gutwin, C.: Heuristic Evaluation of Groupware Based on the Mechanics of Collaboration. In: Proceedings of the 8th IFIP Working Conference on Engineering for Human-Computer Interaction (EHCI’01). (May 11–13, Toronto, Canada), 2001Google Scholar
  25. 25.
    Abowd, G.D., Mynatt, E.D.: Charting past, present, and future research in ubiquitous computing. ACM Trans. Comput. Hum. Interact. (2000)Google Scholar
  26. 26.
    Dix, A., Finlay, J., Abowd, G., Beale, R.: Human-Computer Interaction, 3rd edn. Prentice Hall, Upper Saddle River (2004)Google Scholar
  27. 27.
    Abowd, G.D., Mynatt, E.D., Rodden, T.: The human experience. IEEE Perva. Comput. pp. 48–57, (2002)Google Scholar
  28. 28.
    Gabrielli, S., Mirabella, V., Kimani, S., Catarci, T.: Supporting Cognitive Walkthrough with Video Data: A Mobile Learning Evaluation Study. MobileHCI 05 Conf., Salzburg (Austria), September 2005Google Scholar
  29. 29.
    Duh, H.B.-L., Tan, G.C.B. et Chen, V.H.-H.: Usability evaluation for mobile device: a comparison of laboratory and field tests. In: Proceedings of the 8th conference on human-computer interaction with mobile devices and services (Helsinki, Finland), pp. 181–186, ACM Press (2006)Google Scholar
  30. 30.
    Kaikkonen, A., Keklinen, A., Cankar, M., Kallio, T., Kankainen, A.: Usability testing of mobile applications: a comparison between laboratory and field testing. J. Usabil. Stud. 1(1), 4–17 (2005)Google Scholar
  31. 31.
    Kjeldskov, J., Stage, J.: New techniques for usability evaluation of mobile systems. IJHCS 60(5–6), 599–620 (2004)Google Scholar
  32. 32.
    Roto, V., Oulasvirta, A., Haikarainen, T., Lehmuskallio, H. et Nyyssnen, T.: Examining mobile phone use in the wild with quasi-experimentation. Helsinky Institute for Information Technology (HIIT), August 2004, Technical Report 2004-1Google Scholar
  33. 33.
    Waterson, S., Landay, J.A. et Matthews, T.: In the lab and out in the wild: remote web usability testing for mobile devices. CHI’02 conference on Human Factors in Computing Systems extended abstracts (Minneapolis, Minnesota, USA), pp. 796–797, ACM Press (2002)Google Scholar
  34. 34.
    Po, S.: Mobile usability testing and evaluation. Master’s Thesis, University of Melbourne, Australia (2003)Google Scholar
  35. 35.
    Vetere, F., Howard, S., Pedell, S., Balbo, S.: Walking through mobile use: Novel heuristics and their application. In: Proceedings of Conference of the Computer-Human Interaction Special Interest Group of the Human Factors and Ergonomics Society of Australia OZCHI2003 (2003)Google Scholar
  36. 36.
    Bertini, E., Gabrielli, S., Kimani, S., Catarci, T., Santucci, G.: Appropriating and Assessing Heuristics for Mobile Computing. In: Proceedings of the International Conference in Advanced Visual Interfaces (AVI) (2006)Google Scholar
  37. 37.
    Bertini, E., Catarci, T., Dix, A., Gabrielli, S., Kimani, S., Santucci, G.: Appropriating heuristic evaluation for mobile computing. Int. J. Mobile Hum. Comput. Interact. 1(1), 20–41 (2009)Google Scholar
  38. 38.
    Thorp, J., Henry, S.L.: Web Content Accessibility and Mobile Web: Making a Web Site Accessible Both for People with Disabilities and for Mobile Devices.
  39. 39.
    Rabin, J., McCathieNevile, C.: Mobile Web Best Practices 1.0. W3C Recommendation 29 July 2008
  40. 40.
    Smillie, D.: Web accessibility: is it just a merry-go-round?. In: Proceedings of the 2006 International Cross-Disciplinary Work- Shop on Web Accessibility (W4A). pp. 90–91. ACM Press, Edinburgh, 2006Google Scholar
  41. 41.
    Harper, S., Yesilada, Y.: Building the mobile web: rediscovering acessibility? Univers. Access Infor. Soci. Int. J. 6, 219–220 (2007). Springer-VerlagCrossRefGoogle Scholar
  42. 42.
    Trewin, S. Physical usability and the mobile web. In: Proceedings of the 2006 international cross-disciplinary workshop on Web accessibility (W4A). pp. 109–112. ACM Press, Edinburgh, 2006Google Scholar
  43. 43.
    Owen, S., Rabin, J.: “W3C mobileOK Basic Tests 1.0—Can didate Recommendation” November, 2007
  44. 44.
    Garofalakis, J., Stefanis, V.: MokE: a tool for Mobile-ok evaluation of Web Content. In: Proceedings of the 2008 international cross-disciplinary conference on Web accessibility (W4A), Beijing, China, pp. 57–64. ACM press, 2008Google Scholar
  45. 45.
    Vigo, M., Aizpurua, A., Arrue, M., Abascal, J.: Evaluating Web Accessibility for Specific Mobile Devices.In: Proceedings of the 2008 international cross-disciplinary conference on Web accessibility (W4A), Beijing, China, pp. 65–72. ACM press, 2008Google Scholar
  46. 46.
  47. 47.
  48. 48.
    Chandler, E., Dixon, E., Moniz Pereira, L., Espadinha, C.: A Comparison Study between a Heuristic Evaluation Technique and End User Trial for Mobile Phone Accessibility—Published by COST (2006)Google Scholar
  49. 49.
    Cockton, G., Lavery, D., Woolrych, A.: Inspection-based evaluations. In Jacko, J.J., Sears, A. (eds.). The Human-Computer Interaction Handbook (2003)Google Scholar
  50. 50.
    Nielsen, J.: Usability Engineering. Morgan Kaufmann Publishers, San Francisco (1994)Google Scholar
  51. 51.
    Caldwell, B., Chisholm, W., Vanderheiden, G., White, J.: Web Content Accessibility Guidelines 2.0 W3C Working Draft, ( November 23, 2005
  52. 52.
    Jacobs, I., Gunderson, J., Hansen, E.: User Agent Accessibility Guidelines 1.0. W3C Recomendation, ( 17 December 2002
  53. 53.
    Guidelines containing the technical requirements, the definitions of the different accessibility levels and the technical methodologies for the testing of Web site accessibility. Law 4 of 9/1/2004, Article 11, Paragraph 1, Letters a) and b) December 2004 (Version 3)
  54. 54.
    Chuter, A.: Relationship Between Mobile Web Best Practices 1.0 and Web Content Accessibility Guidelines, (
  55. 55.
    Po, S., Howard, S., Vetere, F., Skov, M.B. In: Brewster, S., Dunlop, M. (eds.). Heuristic Evaluation and Mobile Usability: Bridging the Realism Gap. In: Proceedings of Mobile HCI 2004, pp. 49–60, 2004Google Scholar
  56. 56.
    Kjeldskov, J., Graham, C., Pedell, S., Vetere, F., Howard, S., Balbo, S., Davies, J.: Evaluating the usability of a mobile guide: The influence of location, participants and resources. Behav. Inf. Technol. 24(1), 51–65 (2005)CrossRefGoogle Scholar
  57. 57.
    Bertini, E., Santucci, G.: Modeling user-system data exchange to design adaptive interfaces. In: Proceedings del International Workshop on Plastic Services for Mobile Devices 18 PSMD05, Rome, Italy, September 12, 2005, during the10th International IFIP Conference on Human-Computer Interaction Interact 2005Google Scholar
  58. 58.
    MAIS Deliverable R.7.3.4: Specifiche del Prototipo per la Generazione di Interfacce Utente, 2004Google Scholar
  59. 59.
    MAIS Deliverable P.7.3.8: Prototipo per la generazione di interfacce utente, 2005Google Scholar
  60. 60.
    MAIS Deliverable R.7.3.8: Specifications of the Second Level Prototype of the MaisDesigner, 2005Google Scholar
  61. 61.
    Kjeldskov, J., Graham, C.: A Review of Mobile HCI Research Methods. In L. Chittaro (ed.). Proceedings of Mobile HCI 2003, pp. 317–335, Springer-Verlag, 2003Google Scholar
  62. 62.
    Battestini, Flanagan, J.A.: “Modelling and Simulating Context Data in a Mobile Environment”, Workshop on Context Awareness for Proactive Systems (CAPS), 2005, pp. 127–136, Helsinki, FinlandGoogle Scholar
  63. 63.
    De Waard, D., Brookhuis, K.A., Toffetti, K.A. (eds.): Developments in Human Factors in Transportation, Design, and Evaluation. Field Versus Laboratory Usability Testing: A First Comparison, pp. 205–212. Shaker Publishing, Maastricht (2006)Google Scholar
  64. 64.
    Nielsen, C.: Testing in the Field. In: Proceedings of APCHI 98, pp. 285–290, (1998)Google Scholar
  65. 65.
    Kjeldskov, J., Skov, M.B., Als, B.S., Høegh, R.B.: “Is it Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware Mobile Systems in the Field”. In: Proceedings of 6th International Conference on Human Computer Interaction with Mobile Devices and Services, September 2004, University of Strathclyde, Glasgow, ScotlandGoogle Scholar

Copyright information

© Springer-Verlag 2010

Authors and Affiliations

  • Marco Billi
    • 1
    Email author
  • Laura Burzagli
    • 1
  • Tiziana Catarci
    • 2
  • Giuseppe Santucci
    • 2
  • Enrico Bertini
    • 2
  • Francesco Gabbanini
    • 1
  • Enrico Palchetti
    • 1
  1. 1.Consiglio Nazionale delle Ricerche, Istituto di Fisica Applicata “Nello Carrara”Sesto Fiorentino (FI)Italy
  2. 2.Universià di Roma “La Sapienza”, Dipartimento di Informatica e SistemisticaRomeItaly

Personalised recommendations