A Framework for Usability Evaluation of Mobile Mixed Reality Games

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7522)


This research presents a framework that supports usability experts in determining which method to use when evaluating Mobile Mixed Reality Games (MMRGs). These are games that combine the real and virtual world by means of e.g. a smartphone and require the player to change their geographical location. As some different styles of MMRGs exist, e.g. running versus cunning or multiplayer versus single player, not every method is suitable for each style. The results of the methods are benchmarked against a heuristic evaluation and it is shown that using Instant Data Analysis (IDA), Diary, interaction logs combined with audio diary and retrospective think aloud combined with IDA perform statistically comparable, but that the latter is not favorable based on qualitative merits.


mixed reality augmented reality games usability evaluation 


  1. 1.
    Bier, E.A., Stone, M.C., Pier, K., Buxton, W., DeRose, T.D.: Toolglass and magic lenses. In: Proceedings of the 20th Annual Conference on Computer Graphics and Interactive Techniques - SIGGRAPH 1993, pp. 73–80. ACM Press, New York (September 1993)CrossRefGoogle Scholar
  2. 2.
    Desurvire, H., Caplan, M., Toth, J.A.: Using heuristics to evaluate the playability of games. In: Extended Abstracts of the 2004 Conference on Human Factors and Computing Systems - CHI 2004, pp. 1509–1512. ACM Press, New York (2004)CrossRefGoogle Scholar
  3. 3.
    Ericsson, K., Simon, H.: Protocol analysis: verbal reports as data. 1st paperb edn. The MIT Press, Cambridge (1985)Google Scholar
  4. 4.
    Gielkens, C.: A Framework for Usability Evaluation of Mobile Mixed Reality Games. Master, Utrecht University (2012),
  5. 5.
    Kjeldskov, J., Stage, J.: New techniques for usability evaluation of mobile systems. International Journal of Human-Computer Studies 60(5-6), 599–620 (2004)CrossRefGoogle Scholar
  6. 6.
    McCall, R., Wetzel, R., Löschner, J., Braun, A.: Using presence to evaluate an augmented reality location aware game. Personal and Ubiquitous Computing 15(1), 25–35 (2010)CrossRefGoogle Scholar
  7. 7.
    Milgram, P., Kishino, F.: A taxonomy of mixed reality visual displays. IEICE Transactions on Information and Systems E Series D 77(12), 1321–1321 (1994)Google Scholar
  8. 8.
    Nielsen, J.: Severity Ratings for Usability Problems (1995)Google Scholar
  9. 9.
    Pinelle, D., Wong, N., Stach, T.: Heuristic evaluation for games. In: Proceeding of the Twenty-Sixth Annual CHI Conference on Human Factors in Computing Systems - CHI 2008, pp. 1453–1462. ACM Press, New York (2008)Google Scholar
  10. 10.
    Pinelle, D., Wong, N., Stach, T., Gutwin, C.: Usability heuristics for networked multiplayer games. In: Proceedings of the ACM 2009 International Conference on Supporting Group Work - GROUP 2009, pp. 169–178. ACM Press, New York (2009)CrossRefGoogle Scholar
  11. 11.
    Sharp, H., Rogers, Y., Preece, J.: Interaction Design: Beyond Human-Computer Interaction. Wiley (2007)Google Scholar
  12. 12.
    Thomas, B., Close, B., Donoghue, J., Squires, J., Bondi, P.D., Piekarski, W.: First Person Indoor/Outdoor Augmented Reality Application: ARQuake. Personal and Ubiquitous Computing 6(1), 75–86 (2002)CrossRefGoogle Scholar
  13. 13.
    Wetzel, R., Blum, L., Broll, W., Oppermann, L.: Designing mobile augmented reality games. In: Furht, B. (ed.) Handbook of Augmented Reality, 1st edn., pp. 513–529. Springer (2011)Google Scholar
  14. 14.
    Wetzel, R., Blum, L., Feng, F., Oppermann, L., Straeubig, M.: Tidy City: A Location-based Game for City Exploration Based on User-created Content. In: Proceedings of Mensch & Computer 2011, Chemnitz, Germany, pp. 487–498 (2011)Google Scholar

Copyright information

© IFIP International Federation for Information Processing 2012

Authors and Affiliations

  1. 1.Utrecht UniversityUtrechtThe Netherlands
  2. 2.Fraunhofer FIT, Schloss BirlinghovenSankt AugustinGermany

Personalised recommendations