Can Evaluation Patterns Enable End Users to Evaluate the Quality of an e-learning System? An Exploratory Study

  • Carmelo Ardito
  • Rosa Lanzilotti
  • Marcin Sikorski
  • Igor Garnik
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8514)


This paper presents the results of an exploratory study whose main aim is to verify if the Pattern-Based (PB) inspection technique enables end users to perform reliable evaluation of e-learning systems in real work-related settings. The study involved 13 Polish and Italian participants, who did not have an HCI background, but used e-learning platforms for didactic and/or administrative purposes. The study revealed that the participants were able to effectively and efficiently apply the PB inspection technique with minimum effort. However, in some cases, participants complained that, in some cases, the technique appeared time demanding. This work provides some valuable suggestions to redesign the evaluation tools of the PB technique, in order to improve the focus on specific elements of the e-learning system and to streamline better the evaluation process.


usability inspection technique exploratory study 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Hornbæk, K., Frøkjær, E.: Comparing usability problems and redesign proposals as input to practical systems development. In: Proc. of CHI 2005, pp. 391–400. ACM (2005)Google Scholar
  2. 2.
    Steves, M.P., Morse, E., Gutwin, C., Greenberg, S.: A comparison of usage evaluation and inspection methods for assessing groupware usability. In: Proc. of ACM SIGGROUP Conference on Supporting Group Work, pp. 125–134. ACM (2001)Google Scholar
  3. 3.
    Tan, W., Dahai, L., Bishu, R.: Web evaluation: heuristic evaluation vs. usability testing. International Journal of Industrial Ergonomics 39(4), 621–627 (2009)CrossRefGoogle Scholar
  4. 4.
    Gerdt, P., Miraftabi, R., Tukiainen, M.: Evaluating educational software environments. In: Proc. of Computers in Education, pp. 675–676. IEEE Computer Society (2002)Google Scholar
  5. 5.
    Mendes, E., Hall, W., Harrison, R.: Applying metrics to the evaluation of educational hypermedia application. Journal of Universal Computer Science 4(4), 382–403 (1998)Google Scholar
  6. 6.
    Quinn, C.N., Alem, L., Eklund, J.A.: Pragmatic evaluation methodology for an assessment of learning effectiveness in instructional systems. In: Bewster, S., Cawsey, A., Cockton, G. (eds.) Human–Computer Interaction, vol. II, pp. 55–56. British Computer Society (1997)Google Scholar
  7. 7.
    Squires, D., Preece, J.: Predicting quality in educational software: evaluating for learning, usability, and the synergy between them. Interact. with Computers 11(5), 467–483 (1999)CrossRefGoogle Scholar
  8. 8.
    Jeffries, R., Desurvire, H.W.: Usability testing vs heuristic evaluation: was there a context? ACM SIGCHI Bulletin 24(4), 39–41 (1992)Google Scholar
  9. 9.
    Doubleday, A., Ryan, M., Springett, M., Sutcliffe, A.: A comparison of usability techniques for evaluating design. In: Proc. of DIS 1997, pp. 101–110. Springer, Heidelberg (1997)Google Scholar
  10. 10.
    Law, E.: Heuristic Evaluation. In: Proceedings of COST294-MAUSE International Workshop “Review, Report and Refine Usability Evaluation Methods (R3-UEM)”, pp. 61–63 (2007)Google Scholar
  11. 11.
    Lanzilotti, R., De Angeli, A., Ardito, C., Costabile, M.F.: Do patterns help novice evaluators? A comparative study. Intern. Journal of Human-Computer Studies 69, 52-69 (2011)Google Scholar
  12. 12.
    Matera, M., Costabile, M.F., Garzotto, F., Paolini, P.: SUE Inspection: an effective method for systematic usability evaluation of hypermedia. IEEE Transactions on Systems, Man and Cybernetics – Part A 32(1), 93–103 (2002)CrossRefGoogle Scholar
  13. 13.
    Bias, R.G., Mayhew, D.J.: Cost-justifying usability: An update for the Internet age, 2nd edn. Morgan Kaufmann Publishers, San Francisco (2005)Google Scholar
  14. 14.
    Hudson, J.: Beyond Usability to User Experience. In: Workshop UXEM at CHI 2008 (2008), (last access on January 31, 2012)
  15. 15.
    Lallemand, C.: Toward a closer integration of usability in software development: a study of usability inputs in a model-driven engineering process. In: Proc. of EICS 2011, pp. 299–302. ACM (2011)Google Scholar
  16. 16.
    Rosenbaum, S., Rohn, J.A., Humburg, J.: A toolkit for strategic usability: Results from workshops, panels, and surveys. In: Proc. of CHI 2000, pp. 337–344. ACM (2000)Google Scholar
  17. 17.
    Seffah, A., Donyaee, M., Kline, R.B., Padda, H.K.: Usability measurement and metrics: A consolidated model. Software Quality Journal 14(2), 159–178 (2006)CrossRefGoogle Scholar
  18. 18.
    Venturi, G.: Troost. J. 2004. Survey on the UCD integration in the industry. In: Proc. of NordiCHI 2004, pp. 449–452. ACM (2004)Google Scholar
  19. 19.
    Höök, K., Löwgren, J.: Strong concepts: Intermediate-level knowledge in interaction design research. ACM TOCHI 19(3), Article 23 (2012)Google Scholar
  20. 20.
    Ardito, C.: Usability of virtual reality systems. Thesis. University of Bari Aldo Moro, Italy (2002)Google Scholar
  21. 21.
    Sikorski, M., Garnik, I.: Towards Methodology for User Experience Measurement in on-line Services. In: Korczak, J., Dudycz, H., Dyczkowski, M. (eds.) Advanced Information Technologies for Management - AITM 2010, Research Papers of Wroclaw University of Economics, vol. (147) (2011)Google Scholar
  22. 22.
    Hertzum, M., Jacobsen, N.E.: The evaluator effect: a chilling fact about usability evaluation methods. Intern. Journal of Human–Computer Interaction 15(1), 183–204 (2003)CrossRefGoogle Scholar
  23. 23.
    Ardito, C., Costabile, M.F., De Angeli, A., Lanzilotti, R.: Systematic evaluation of e-learning systems: an experimental validation. In: Proc. of NordiCHI 2006, pp. 195–202. ACM (2006)Google Scholar
  24. 24.
    Nielsen, J., Landauer, T.K.: A mathematical model of the finding of usability problems. In: Proc. of INTERCHI 1993, pp. 296–313. ACM (1993)Google Scholar
  25. 25.
    Vredenburg, K., Mao, J.Y., Smith, P.W., Carey, T.: A survey of user-centered design practice. In: Proc. of CHI 2002, pp. 471–478. ACM (2002)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Carmelo Ardito
    • 1
  • Rosa Lanzilotti
    • 1
  • Marcin Sikorski
    • 2
  • Igor Garnik
    • 2
  1. 1.Computer Science DepartmentUniversity of Bari Aldo MoroItaly
  2. 2.Faculty of Management and EconomicsGdansk University of TechnologyPoland

Personalised recommendations