Skip to main content

A Framework for Usability Evaluation of Mobile Mixed Reality Games

  • Conference paper

Part of the Lecture Notes in Computer Science book series (LNISA,volume 7522)


This research presents a framework that supports usability experts in determining which method to use when evaluating Mobile Mixed Reality Games (MMRGs). These are games that combine the real and virtual world by means of e.g. a smartphone and require the player to change their geographical location. As some different styles of MMRGs exist, e.g. running versus cunning or multiplayer versus single player, not every method is suitable for each style. The results of the methods are benchmarked against a heuristic evaluation and it is shown that using Instant Data Analysis (IDA), Diary, interaction logs combined with audio diary and retrospective think aloud combined with IDA perform statistically comparable, but that the latter is not favorable based on qualitative merits.


  • mixed reality
  • augmented reality
  • games
  • usability evaluation


  1. Bier, E.A., Stone, M.C., Pier, K., Buxton, W., DeRose, T.D.: Toolglass and magic lenses. In: Proceedings of the 20th Annual Conference on Computer Graphics and Interactive Techniques - SIGGRAPH 1993, pp. 73–80. ACM Press, New York (September 1993)

    CrossRef  Google Scholar 

  2. Desurvire, H., Caplan, M., Toth, J.A.: Using heuristics to evaluate the playability of games. In: Extended Abstracts of the 2004 Conference on Human Factors and Computing Systems - CHI 2004, pp. 1509–1512. ACM Press, New York (2004)

    CrossRef  Google Scholar 

  3. Ericsson, K., Simon, H.: Protocol analysis: verbal reports as data. 1st paperb edn. The MIT Press, Cambridge (1985)

    Google Scholar 

  4. Gielkens, C.: A Framework for Usability Evaluation of Mobile Mixed Reality Games. Master, Utrecht University (2012),

  5. Kjeldskov, J., Stage, J.: New techniques for usability evaluation of mobile systems. International Journal of Human-Computer Studies 60(5-6), 599–620 (2004)

    CrossRef  Google Scholar 

  6. McCall, R., Wetzel, R., Löschner, J., Braun, A.: Using presence to evaluate an augmented reality location aware game. Personal and Ubiquitous Computing 15(1), 25–35 (2010)

    CrossRef  Google Scholar 

  7. Milgram, P., Kishino, F.: A taxonomy of mixed reality visual displays. IEICE Transactions on Information and Systems E Series D 77(12), 1321–1321 (1994)

    Google Scholar 

  8. Nielsen, J.: Severity Ratings for Usability Problems (1995)

    Google Scholar 

  9. Pinelle, D., Wong, N., Stach, T.: Heuristic evaluation for games. In: Proceeding of the Twenty-Sixth Annual CHI Conference on Human Factors in Computing Systems - CHI 2008, pp. 1453–1462. ACM Press, New York (2008)

    Google Scholar 

  10. Pinelle, D., Wong, N., Stach, T., Gutwin, C.: Usability heuristics for networked multiplayer games. In: Proceedings of the ACM 2009 International Conference on Supporting Group Work - GROUP 2009, pp. 169–178. ACM Press, New York (2009)

    CrossRef  Google Scholar 

  11. Sharp, H., Rogers, Y., Preece, J.: Interaction Design: Beyond Human-Computer Interaction. Wiley (2007)

    Google Scholar 

  12. Thomas, B., Close, B., Donoghue, J., Squires, J., Bondi, P.D., Piekarski, W.: First Person Indoor/Outdoor Augmented Reality Application: ARQuake. Personal and Ubiquitous Computing 6(1), 75–86 (2002)

    CrossRef  Google Scholar 

  13. Wetzel, R., Blum, L., Broll, W., Oppermann, L.: Designing mobile augmented reality games. In: Furht, B. (ed.) Handbook of Augmented Reality, 1st edn., pp. 513–529. Springer (2011)

    Google Scholar 

  14. Wetzel, R., Blum, L., Feng, F., Oppermann, L., Straeubig, M.: Tidy City: A Location-based Game for City Exploration Based on User-created Content. In: Proceedings of Mensch & Computer 2011, Chemnitz, Germany, pp. 487–498 (2011)

    Google Scholar 

Download references

Author information

Authors and Affiliations


Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2012 IFIP International Federation for Information Processing

About this paper

Cite this paper

Gielkens, C., Wetzel, R. (2012). A Framework for Usability Evaluation of Mobile Mixed Reality Games. In: Herrlich, M., Malaka, R., Masuch, M. (eds) Entertainment Computing - ICEC 2012. ICEC 2012. Lecture Notes in Computer Science, vol 7522. Springer, Berlin, Heidelberg.

Download citation

  • DOI:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-33541-9

  • Online ISBN: 978-3-642-33542-6

  • eBook Packages: Computer ScienceComputer Science (R0)