Evidence Based Design of Heuristics for Computer Assisted Assessment

  • Gavin Sim
  • Janet C. Read
  • Gilbert Cockton
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5726)


The use of heuristics for the evaluation of interfaces is a well studied area. Currently there appear to be two main research areas in relation to heuristics: the analysis of methods to improve the effectiveness of heuristic evaluations; and the development of new heuristic sets for novel and specialised domains. This paper proposes an evidence based design approach to the development of domain specific heuristics and shows how this method was applied within the context of computer assisted assessment. A corpus of usability problems was created through a series of student surveys, heuristic evaluations, and a review of the literature. This corpus was then used to synthesise a set of domain specific heuristics for evaluating CAA applications. The paper describes the process, and presents a new set of heuristics for evaluating CAA applications.


Heuristics usability computer assisted assessment 


  1. 1.
    Cockton, G., Woolrych, A., Hindmarch, M.: Reconditioned Merchandise: Extending Structured Report Formats in Usability Inspection. In: CHI 2004, Vienna (2004)Google Scholar
  2. 2.
    Coyle, C.L., et al.: Heuristic Evaluations at Bell Labs: Analyses of Evaluator Overlap and Group Sessions. In: CHI 2007. ACM Press, San Jose (2007)Google Scholar
  3. 3.
    Korhonen, H., Koivisto, E.M.: Playability Heuristics for Mobile Games. In: MobileHCI 2006. ACM, Helsinki (2006)Google Scholar
  4. 4.
    Paddison, C., Englefield, P.: Applying heuristics to accessibility inspections. Interacting with Computers 16(2), 507–521 (2004)CrossRefGoogle Scholar
  5. 5.
    Nielsen, J.: Enhancing the Explanatory Power of Usability Heuristics. In: Proceedings of the SIGCHI conference on Human factors in computing systems: celebrating interdependence. ACM, Boston (1994)Google Scholar
  6. 6.
    Ling, C., Salvendy, G.: Extension of Heuristic evaluation method: a review and reaapraisal. International Journal of Ergonomics and Human Factors 27(3), 179–197 (2005)Google Scholar
  7. 7.
    Gray, W.D., Salzman, M.C.: Damaged Merchandise? Journal of Human-Computer Interaction 13(4), 203–262 (1998)CrossRefGoogle Scholar
  8. 8.
    Somervell, J., McCrickard, D.: Better discount evaluation: illustrating how critical parameters support heuristic creation. Interacting with Computers 17(5), 592–612 (2005)CrossRefGoogle Scholar
  9. 9.
    Hartson, H.R., Andre, T.S., Williges, R.C.: Criteria for Evaluating Usability Evaluation Methods. International Journal Human Computer Interaction 15(1), 145–181 (2003)CrossRefGoogle Scholar
  10. 10.
    Squires, D., Preece, J.: Predicting quality in educational software: Evaluating for learning, usability and the synergy between them. Interacting with Computers 11, 467–483 (1999)CrossRefGoogle Scholar
  11. 11.
    Bull, J., McKenna, C.: Blueprint for computer-assisted assessment, p. 168. Loughborough University (2001)Google Scholar
  12. 12.
    Parlangeli, O., Marchigiani, E., Bagnara, S.: Multimedia systems in distance education: effects of usability on learning. Interacting with Computers 12(1), 37–49 (1999)CrossRefGoogle Scholar
  13. 13.
    Piguet, A., Peraya, D.: Creating web-integrated learning environments: An analysis of WebCT authoring tools in respect to usability. Australian Journal of Educational Technology 16(3), 302–314 (2000)Google Scholar
  14. 14.
    Berg, G.A.: Human-Computer Interaction (HCI) in Educational Environments: Implications of Understanding Computers as Media. Journal of Educational Multimedia and Hypermedia 9(4), 347–368 (2000)Google Scholar
  15. 15.
    Sim, G., Horton, M., Strong, S.: Interfaces for online assessment: friend or foe? In: 7th HCI Educators Workshop. LTSN, Preston (2004)Google Scholar
  16. 16.
    Davies, P.: There’s no confidence in multiple-choice testing. In: Sixth International Computer Assisted Assessment Conference, Loughborough (2002)Google Scholar
  17. 17.
    Nielsen, J., Mack, R.L.: Usability Inspection Methods. John Wiley & Sons, New York (1994)CrossRefGoogle Scholar
  18. 18.
    Desurvire, H., Caplan, M., Toth, J.A.: Using heuristics to Evaluate the Playability of Games. In: CHI 2004. ACM, Vienne (2004)Google Scholar
  19. 19.
    Jacobsen, N.E., John, B.E.: The evaluator effect in usability studies: problem detection and severity judgements. In: Proceeding of the Human Factors and Ergonomics Society 42nd Annual Meeting. HFES, Chicago (1998)Google Scholar
  20. 20.
    Nielsen, J.: Usability Engineering. Morgan Kaufmann, San Francisco (1994)zbMATHGoogle Scholar
  21. 21.
    Connell, I.W., Hammond, N.V.: Comparing usability evaluation principles with heuristics: problem instances versus problem types. In: Human-Computer Interaction - INTERACT 1999. IOS Press, Edinburgh (1999)Google Scholar
  22. 22.
    Embi, Z.C., Hussain, H.: Analysis of local and foreign edutainment products- and effort to implement the design framework for an edutainment environment in Malaysia. Journal of Computers in Mathematics and Science Teaching 24(1), 27–42 (2005)Google Scholar
  23. 23.
    Fernandes, G., Holmes, C.: Applying HCI to music related hardware. In: CHI 2002. ACM, Minneapolis (2002)Google Scholar
  24. 24.
    Sim, G., Holifield, P.: Computer Assisted Assessment: All those in favour tick here. In: World Conference on Educational Multimedia, Hypermedia and Telecommunications. AACE, Lugano (2004)Google Scholar
  25. 25.
    Sim, G., Holifield, P.: Piloting CAA:All aboard. In: 8th International Computer Assisted Assessment Conference, Loughborough (2004)Google Scholar
  26. 26.
    Sim, G., et al.: Heuristic Evaluations of Computer Assisted Assessment Environments. In: World Conference on Educational Multimedia, Hypermedia and Telecommunications. AACE, Vancouver (2007)Google Scholar
  27. 27.
    Conole, G., Warburton, B.: A review of computer-assisted assessment. ALT-J 13(1), 19–33 (2005)CrossRefGoogle Scholar
  28. 28.
    Sim, G., Holifield, P., Brown, M.: Implementation of computer assisted assessment: lessons from the literature. ALT-J 12(3), 215–229 (2004)CrossRefGoogle Scholar
  29. 29.
    Jeffries, R., Desurvire, H.: Usability Testing vs Heuristic Evaluation: Was there a contest? SIGCHI Bulletin 24(4), 39–41 (1992)CrossRefGoogle Scholar
  30. 30.
    Lavery, D., Cockton, G.: Representing Predicted and Actual Usability Problems. In: Workshop on Representation in Interactive Software Development. Queen Mary and Westfield College, University of London (1997)Google Scholar
  31. 31.
    Sim, G., Read, J.C., Holifield, P.: Evaluating the user experience in CAA Environments: What affects user satisfaction? In: 10th International Computer Assisted Assessment Conference, Loughborough (2006)Google Scholar
  32. 32.
    Nielsen, J.: Finding usability problems through heuristic evaluation. In: Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, Monterey (1992)Google Scholar

Copyright information

© IFIP International Federation for Information Processing 2009

Authors and Affiliations

  • Gavin Sim
    • 1
  • Janet C. Read
    • 1
  • Gilbert Cockton
    • 2
  1. 1.School of Computing Engineering and Physical SciencesUniversity of Central LancashirePrestonUK
  2. 2.Department of Computing, Engineering and TechnologyUniversity of Sunderland SunderlandUK

Personalised recommendations