Abstract
This article describes the development and assessment of a coding scheme for finding both usability and fun problems through observations of young children playing computer games during user tests. The proposed coding scheme is based on an existing list of breakdown indication types of the detailed video analysis method (DEVAN). This method was developed to detect usability problems in task-based products for adults. However, the new coding scheme for children’s computer games takes into account that in games, fun, in addition to usability, is an important factor and that children behave differently from adults. Therefore, the proposed coding scheme uses 8 of the 14 original breakdown indications and has 7 new indications. The article first discusses the development of the new coding scheme. Subsequently, the article describes the reliability assessment of the coding scheme. The anytwo agreement measure of 38.5% shows that thresholds for when certain user behavior is worth coding will be different for different evaluators. However, the any-two agreement of .92 for a fixed list of observation points shows that the distinction between the available codes is clear to most evaluators. Finally, a pilot study shows that training can increase any-two agreement considerably by decreasing the number of unique observations, in comparison with the number of agreed upon observations.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Barendregt, W., &Bekker, M. M. (2004). Towards a framework for design guidelines for young children’s computer games. In M. Rauterberg (Ed.),Proceedings of the IFIP ICEC Conference (pp. 365–376). Berlin: Springer.
Barendregt, W., Bekker, M. M., &Speerstra, M. (2003). Empirical evaluation of usability and fun in computer games for children. In M. Rauterberg, M. Menozzi, & J. Wesson (Eds.),Proceedings of the IFIP 8th International Conference on Human-Computer Interaction, (pp. 705–708). Amsterdam: IOS Press.
Cockton, G., &Lavery, D. (1999). A framework for usability problem extraction.In Proceedings of the IFIP 7th International Conference on Human-Computer Interaction, (pp. 344–352). Amsterdam: IOS Press.
Hertzum, M., &Jacobsen, N. E. (2001). The evaluator effect: A chilling fact about usability evaluation methods.International Journal of Human-Computer Interaction,13, 421–443.
Lavery, D., &Cockton, G. (1997). Representing predicted and actual usability problems. In H. Johnson, P. Johnson, & E. O’Neill (Eds.),Proceedings of the International Workshop on Representations in Interactive Software Development (pp. 97–108). London: University of London, Queen Mary and Westfield College.
Malmberg Uitgeverij (2003). Wereld in getallen, Groep 3: Het Pretpark [World in numbers, Group 3: The Theme Park] [Computer software]. Amsterdam: Author.
Malone, T. W., &Lepper, M. R. (1987). Making learning fun: A taxonomy of intrinsic motivations for learning. In R. E. Snow & M. J. Farr (Eds.),Aptitude, learning, and interaction III: Cognitive and affective process analysis (pp. 223–253). Hillsdale, NJ: Erlbaum.
MediaMix Benelux (2002). Regenboog, de mooiste vis van de zee [Rainbow, the most beautiful fish in the ocean] [Computer software]. Overijse, Belgium: Author.
Mindscape (2003). Robbie Konijn, Groep 3: Pret in de Wolken [Robbie Rabbit, Group 3: Fun in the Clouds] [Computer software]. Haarlem, The Netherlands: Author.
Noldus (2002). The Observer Pro [Computer software]. Wageningen, The Netherlands: Noldus.
Pagulayan, R. J., Keeker, K., Wixon, D., Romero, R., &Fuller, T. (2003). User-centered design in games. In J. Jacko & A. Sears (Eds.),Handbook for human-computer interaction in interactive systems (pp. 883–906). Mahwah, NJ: Erlbaum.
Rauterberg, M. (1995). About a framework for information and information processing of learning systems. InPreprints of Proceedings of “International Conference on Information System Concepts” (pp. 4.1–4.15). Marburg: Phillipps University.
Rouse, R., III (2005).Game design: Theory and practice (2nd ed.). Plano, TX: Wordware.
Vermeeren, A. P. O. S., den Bouwmeester, K., Aasman, J., &de Ridder, H. (2002). DEVAN: A detailed video analysis of user test data.Behaviour & Information Technology,21, 403–423.
Author information
Authors and Affiliations
Corresponding author
Additional information
This work was supported by a grant from the Innovation-Oriented Research Program Human-Machine Interaction of the Dutch government.
Rights and permissions
About this article
Cite this article
Barendregt, W., Bekker, M.M. Developing a coding scheme for detecting usability and fun problems in computer games for young children. Behavior Research Methods 38, 382–389 (2006). https://doi.org/10.3758/BF03192791
Received:
Accepted:
Issue Date:
DOI: https://doi.org/10.3758/BF03192791