Quantitative assessment of mobile web guidelines conformance

  • Markel VigoEmail author
  • Amaia Aizpurua
  • Myriam Arrue
  • Julio Abascal
Long Paper


Conformance metrics for the mobile web can play a crucial role as far as engineering mobile websites are concerned, especially if they are automatically obtained. In this way, developers can have an idea in numeric terms of how suitable their developments are for mobile devices. However, there are a plethora of devices with their own particular features (screen size, formats support, etc.) that restrict a unified automatic assessment process. This paper proposes a tool-supported method for device-tailored assessment in terms of conformance with Mobile Web Best Practices 1.0, including the definition of five quantitative metrics for automatically measuring mobile web conformance: Navigability, Page layout, Page definition, User input and Overall score. The behaviour of these metrics was analysed for different devices and different web paradigms, both mobile web pages and their equivalent desktop pages. As expected, the results show that mobile web pages on more capable devices score higher. In addition, 20 users took part in an experiment aimed at discovering how conformance-based scores relate to usability dimensions. The results demonstrate that automatic scoring approaches strongly correlate with usability scores obtained by direct observation, such as task completion time and user satisfaction. This correlation is even stronger for the device-tailored assessment than the one that assumes a general profile for all devices. For instance, results show a strong negative correlation between Overall score and task completion time: ρ (9) = −0.81, (p < 0.05) for the generalist approach and ρ (9) = −0.88 for the device-tailored one, entailing that mobile web guidelines and the metrics based on their conformance capture usability aspects. This result challenges the widely accepted belief that conformance to guidelines does not imply more usable web pages, at least for web accessibility conformance.


Mobile web Usability Metrics Device-tailored evaluations 


  1. 1.
    Abou-Zahra, S., Squillace, M.: Evaluation and report language (EARL) 1.0 schema. W3C Evaluation and Repair Tools Working Group. (2009). Accessed Sep 2009
  2. 2.
    Bailey, J., Burd, E.: Tree-map visualisation for web accessibility. Computer Software and Applications Conference, COMPSAC’05, pp. 275–280 (2005)Google Scholar
  3. 3.
    Bickmore, T.W., Schilit, B.N.: Digestor: device-independent access to the World Wide Web. Computer Networks and ISDN Systems 29, 1075–1082 (1997)CrossRefGoogle Scholar
  4. 4.
    Buchanan, G., Farrant, S., Jones, M., Thimbleby, H.: Improving mobile internet usability. International Conference on World Wide Web, WWW’01, pp. 673–680 (2001)Google Scholar
  5. 5.
    Buyukkokten, O., Garcia-Molina, H., Paepcke, A., Winograd, T.: Power browser: efficient web browsing for PDAs. SIGCHI Conference on Human Factors in Computing Systems, CHI’00, pp. 430–437 (2000)Google Scholar
  6. 6.
    Chisholm, W., Vanderheiden, G., Jacobs, I.: Web content accessibility guidelines 1.0. W3C Web accessibility initiative. (1999). Accessed Sep 2009
  7. 7.
    Church, K., Smyth, B., Cotter, P., Bradley, K.: Mobile information access: a study of emerging search behavior on the mobile internet. ACM Trans. Web 1(1), article 4 (2007)Google Scholar
  8. 8.
    Coursaris, C.K., Kim, D.J.: A Research agenda for mobile usability. 26th Conference on Human Factors in Computing Systems, CHI’07, pp. 2345–2350 (2007)Google Scholar
  9. 9.
    Cui, Y., Roto, V.: How people use the web on mobile devices. International Conference on World Wide Web, WWW’08, pp. 905–914 (2008)Google Scholar
  10. 10.
    DeMarco, T.: Controlling Software Projects: Management, Measurement and Estimates. Prentice Hall, Upper Saddle River, NJ (1986)Google Scholar
  11. 11.
    Dujmovic, J.J.: Neural networks—concepts, applications, and implementations. In: Antognetti, P., Milutinovic, V. (eds.) Preferential Neural Networks, pp. 155–206. Prentice Hall, Upper Saddle River, NJ (1991)Google Scholar
  12. 12.
    Dujmovic, J.J.: A Method for evaluation and selection of complex hardware and software systems. 22nd International Computer Measurement Group Conference, pp. 368–378 (1996)Google Scholar
  13. 13.
    Garofalakis, J., Stefanis, V.: MokE: a tool for Mobile-ok evaluation of web content. International Cross-Disciplinary Conference on Web Accessibility, W4A’08, pp. 57–64 (2008)Google Scholar
  14. 14.
    Harper, S.: Mobile web: reinventing the wheel? ACM SIGACCESS Access. Comput. 90, 16–18 (2008)CrossRefGoogle Scholar
  15. 15.
    Hunter, J., Lagoze, C.: Combining RDF and XML schemas to enhance interoperability between metadata application profiles. International World Wide Web Conference, WWW’01, pp. 457–466 (2001)Google Scholar
  16. 16.
    ISO 9241-11: Ergonomic requirements for office work with visual display terminals (VDTs)—Part 11: Guidance on usability. International Organization of Standardization (1998)Google Scholar
  17. 17.
    International Telecommunication Union, ITU: Worldwide mobile cellular subscribers to reach 4 billion mark late 2008. (2008). Accessed Sep 2009
  18. 18.
    Jain, R.: The mobile web in developing countries. W3C Workshop on the Mobile Web in the Developing Countries (2006)Google Scholar
  19. 19.
    Kaikkonen, A., Roto, V.: Navigating in a mobile XHTML application. SIGCHI Conference on Human Factors in Computer Systems, CHI 2003, pp. 329–336 (2003)Google Scholar
  20. 20.
    Kelly, B., Sloan, D., Phipps, L., Petrie, H., Hamilton, F.: Forcing standardization or accommodating diversity?: a framework for applying the WCAG in the real world. International Cross-Disciplinary Workshop on Web Accessibility, W4A’05, pp. 46–54 (2005)Google Scholar
  21. 21.
    Klyne, G., Reynolds, F., Woodrow, F., Ohto, H., Hjelm, J., Butler, M., Tran, L.: Composite capability/preference profiles (CC/PP): structure and vocabularies 1.0. W3C Device Independence Working Group. (2004). Accessed Sep 2009
  22. 22.
    Leporini, B., Paternò, F.: Applying web usability criteria for vision-impaired users: does it really improve task performance? Int. J. Human-Comput. Interact. 24(1), 17–47 (2008)Google Scholar
  23. 23.
    Lewis, J.R.: IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. Int. J. Human-Comput. Interact. 7(1), 57–78 (1995)CrossRefGoogle Scholar
  24. 24.
    Mankoff, J., Dey, A., Batra, U., Moore, M.: Web accessibility for low bandwidth input. ACM SIGACCESS Conference on Computers and Accessibility, ASSETS’02, pp. 17–24 (2002)Google Scholar
  25. 25.
    Mich, L., Franch, M., Gaio, L.: Evaluating and designing web site quality. IEEE Multimed 10(1), 34–43 (2003)CrossRefGoogle Scholar
  26. 26.
    Nielsen, J., Tahir, M.: Homepage Usability: 50 Websites Deconstructed. New Riders Publishing, Indianapolis (2000)Google Scholar
  27. 27.
    Nielsen, J.: Mobile Web 2009 = Desktop Web 1998. Jakob Nielsen’s Alertbox. (2009). Accessed Sep 2009
  28. 28.
    Olsina, L., Rossi, G.: Measuring web application quality with WebQEM. IEEE Multimed. 9(4), 20–29. IEEE Computer Society Press (2002)Google Scholar
  29. 29.
    Owen, S., Rabin, J. W3C mobileOK Basic Tests 1.0. W3C Mobile Web Initiative. (2008). Accessed Sep 2009
  30. 30.
    Rabin, J., McCathieNevile, C.: Mobile web best practices 1.0. W3C Mobile Web Initiative. (2008). Accessed Sep 2009
  31. 31.
    Rohra, P.: Re: People with disabilities using mobile devices to interact with the Web. WAI-IG mailing list. (2008). Accessed Sep 2009
  32. 32.
    Roto, V.: Web browsing on mobile phones—characteristics of user experience. Dissertation, Helsinki University of Technology (2006)Google Scholar
  33. 33.
    Scheppe, K.: mobileOK Pro Tests Version 1. W3C Mobile Web Initiative (2008). Accessed Sep 2009
  34. 34.
    Shrestha, S.: Mobile web browsing: usability study. International Conference on Mobile Technology, Applications, and Systems, MC’07, pp. 187–194 (2007)Google Scholar
  35. 35.
    Sloan, D., Heath, A., Hamilton, F., Kelly, B., Petrie, H., Phipps, L.: Contextual web accessibility—maximizing the benefit of accessibility guidelines. International Cross-Disciplinary Workshop on Web Accessibility, W4A’06, pp. 121–131 (2006)Google Scholar
  36. 36.
    Sullivan, T., Matson, R.: Barriers to use: usability and content accessibility on the Web’s most popular sites. ACM Conference on Universal Usability, CUU’00, pp. 139–144 (2000)Google Scholar
  37. 37.
    Thatcher, J., Burks, M.R., Heilmann, C., Henry, S.L., Kirkpatrick, A., Lauke, P.H., Lawson, B., Regan, B., Rutter, R., Urban, M., Waddell, C.D.: Web Accessibility: Web Standards and Regulatory Compliance. Springer, New York (2006)Google Scholar
  38. 38.
    Trewin, S.: Physical usability and the mobile web. International Cross-Disciplinary Workshop on Web Accessibility, W4A’06, pp. 109-112 (2006)Google Scholar
  39. 39.
    Vigo, M., Arrue, M., Brajnik, G., Lomuscio, R., Abascal, J.: Quantitative metrics for measuring web accessibility. International Cross-Disciplinary Workshop on Web Accessibility, W4A’07, pp. 99–107 (2007)Google Scholar
  40. 40.
    Vigo, M., Kobsa, A., Arrue, M., Abascal, J.: User-tailored web accessibility evaluations. ACM Conference on Hypertext and Hypermedia, HYPERTEXT’07, pp. 95–104 (2007)Google Scholar
  41. 41.
    Vigo, M., Aizpurua, A., Arrue, M., Abascal, J.: Evaluating web accessibility for specific mobile devices. International Cross-Disciplinary Conference on Web Accessibility, W4A’08, pp. 65–72 (2008)Google Scholar
  42. 42.
    Vigo, M., Brajnik, G., Arrue, M., Abascal, J.: Tool independence for the web accessibility quantitative metric. Disabil. Rehabil. Assist. Technol. 4(4), 248–263 (2009)CrossRefGoogle Scholar
  43. 43.
    Vigo, M., Leporini, B., Paternò, F.: Enriching web information scent for blind users. ACM SIGACCESS Conference on Computers and Accessibility, ASSETS’09, pp. 123–130 (2009)Google Scholar
  44. 44.
    Yesilada, Y., Chuter, A., Henry, S.L.: Shared web experiences: barriers common to mobile device users and people with disabilities. W3C Web accessibility initiative. (2008). Accessed Sep 2009

Copyright information

© Springer-Verlag 2010

Authors and Affiliations

  • Markel Vigo
    • 1
    Email author
  • Amaia Aizpurua
    • 1
  • Myriam Arrue
    • 1
  • Julio Abascal
    • 1
  1. 1.Computer Science School, Department of Computer Architecture and TechnologyUniversity of the Basque CountryDonostiaSpain

Personalised recommendations