Awkward Annie: Game-Based Assessment of English Pragmatic Skills

  • G. Tanner JacksonEmail author
  • Lindsay Grace
  • Patricia Inglese
  • Jennifer Wain
  • Robert Hone
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10714)


This paper describes efforts related to the design, development, and evaluation of a game-based assessment called Awkward Annie. The Awkward Annie game focuses on English language pragmatic skills and represents a novel design that asks players to violate social norms by intentionally selecting the most inappropriate things to say to virtual colleagues. Awkward Annie was evaluated through two studies. Study 1 was a small-scale lab-based study with predominantly non-native English speakers (n = 36). Study 2 was a scaled up extension of the same design but with native English speakers on Amazon Mechanical Turk (n = 328). Both studies explored potential relations between user experience, understanding the play mechanics (i.e., being inappropriate), game performance, and English pragmatic ability. The studies also indicate cultural differences in players comfort with acting inappropriately. The results of these studies benefit both cross-cultural game design and the assessment of socio-pragmatics and pragmalinguistics. Additionally, this research advances the current state of assessment practices by investigating how games could potentially provide an alternative to traditional selected response testing.


Educational games English pragmatics User experience 


  1. 1.
    Belz, J.A.: The role of computer mediation in the instruction and development of L2 pragmatic ability. Ann. Rev. Appl. Linguist. 27, 45 (2007)CrossRefGoogle Scholar
  2. 2.
    Clyne, M.: Inter-Cultural Communication at Work: Cultural Values in Discourse. Cambridge University Press, Cambridge, UK (1994)Google Scholar
  3. 3.
    Cole, H., Griffiths, M.D.: Social interactions in massively multi-player online role-playing gamers. Cyber Psychol. Behav. 10(4), 575–583 (2007)CrossRefGoogle Scholar
  4. 4.
    Cornillie, F., Thorne, S.L., Desmet, P.: ReCALL special issue: digital games for language learning: challenges and opportunities. ReCALL 24(03), 243–256 (2012)CrossRefGoogle Scholar
  5. 5.
    Garris, R., Ahlers, R., Driskell, J.E.: Games, motivation, and learning: a research and practice model. Simul. Gaming 33(4), 441–467 (2002)CrossRefGoogle Scholar
  6. 6.
    Gee, J.P.: What video games have to teach us about learning and literacy. Comput. Entertain. 1(1), 20 (2003)CrossRefGoogle Scholar
  7. 7.
    Hoffman, E., John, M., Makany, T.: How do game design frameworks align with learning and assessment design frameworks? Paper presented at the annual meeting for the National Council for Measurement in Education, Philadelphia, PA (2014)Google Scholar
  8. 8.
    Ishihara, N., Cohen, A.D.: Teaching and Learning Pragmatics: Where Language and Culture Meet. Pearson, Harlow (2010)Google Scholar
  9. 9.
    Jackson, G.T., McNamara, D.S.: Motivation and performance in a game-based intelligent tutoring system. J. Educ. Psychol. 105, 1036–1049 (2013)CrossRefGoogle Scholar
  10. 10.
    Leech, G.N.: Principles of Pragmatics. Longman, London, England (1983)Google Scholar
  11. 11.
    Limberg, H.: Principles for pragmatics teaching: apologies in the EFL classroom. ELT J. 69(3), 275–285 (2015)CrossRefGoogle Scholar
  12. 12.
    McDonald, K.K., Hannafin, R.D.: Using web-based computer games to meet the demands of today’s high-stakes testing: a mixed method inquiry. J. Res. Technol. Educ. 35(4), 459–472 (2003)CrossRefGoogle Scholar
  13. 13.
    Pivec, M., Dziabenko, O., Schinnerl, I.: Game-based learning in universities and lifelong learning: “UniGame: social skills and knowledge training” game concept. J. Univ. Comput. Sci. 10(1), 14–26 (2004)Google Scholar
  14. 14.
    Riddiford, N., Joe, A.: Tracking the development of sociopragmatic skills. TESOL Q. 44(1), 195–205 (2010)CrossRefGoogle Scholar
  15. 15.
    Surface, E.A., Dierdorff, E.C., Watson, A.M.: Special operations language training software measurement of effectiveness study: Tactical Iraqi study final report. Special Operations Forces Language Office, Tampa, FL, USA (2007)Google Scholar
  16. 16.
    Taguchi, N.: Longitudinal study of higher-order inferential ability in L2 English. In: Byrnes, H., Ortega, L. (eds.) The Longitudinal Study of Advanced L2 Capacities. Lawrence Erlbaum, Mahwah (2008)Google Scholar
  17. 17.
    Timpe-Laughlin, V., Wain, J., Schmidgall, J.: Defining and operationalizing the construct of pragmatic ability: Review and recommendations. ETS Research Report Series. Princeton, NJ. Educational Testing Service (2015)Google Scholar
  18. 18.
    Timpe, V.: Assessing Intercultural Language Learning. Peter Lang, Frankfurt aM (2013a)Google Scholar
  19. 19.
    Timpe, V.: The difficulty with difficulty: the issue of determining task difficulty in TBLA. J. Linguist. Lang. Teach. 4(1), 13–28 (2013)Google Scholar
  20. 20.
    Yee, N.: Motivations for play in online games. Cyber Psychol. Behav. 9(6), 772–775 (2006)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • G. Tanner Jackson
    • 1
    Email author
  • Lindsay Grace
    • 2
  • Patricia Inglese
    • 1
  • Jennifer Wain
    • 1
  • Robert Hone
    • 2
  1. 1.Educational Testing ServicePrincetonUSA
  2. 2.American UniversityWashington, DCUSA

Personalised recommendations