Using Qualitative Data Analysis to Measure User Experience in a Serious Game for Premed Students

  • Marjorie A. ZielkeEmail author
  • Djakhangir Zakhidov
  • Daniel Jacob
  • Sean Lenox
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9740)


The University of Texas Transformation in Medical Education Portal (UT TIME Portal) is a game-based learning platform for select premed students, with a particular emphasis on communication and professionalism. In addition to quantitative data on system usage and user performance, the UT TIME Portal generates rich sets of qualitative data collected through discussion board posts and pre- and post- surveys. Using NVivo 10’s built-in tools, our team used this qualitative data to measure game experience outcomes in many ways by building and testing out hypotheses about our user experience design. The ability to tag, code and organize themes to then be analyzed in the context of quantitative data generated by the UT TIME Portal adds an important dimension to understanding the user experience and generates insights not possible to glean from quantitative data alone.


Qualitative analysis Discussion boards Nvivo Serious game Medical education Online learning portal Education and training Asynchronous access Asynchronous practice Adaptive learning Emergent learning systems Intelligent agents 



The research team would like to thank colleagues from the UT Southwestern Medical School and the students from the UT TIME program and UT Dallas for their support of this research.


  1. 1.
    Howell, T., Jr.: HHS: Number insured under Obamacare swells to 17.6 million. (2015, September 22). Retrieved January 4, 2016.
  2. 2.
    Susan, B.: New Physician Workforce Projections Show the Doctor Shortage Remains Significant. (2015, March 3). Retrieved January 6, 2016.
  3. 3. UT-PACT BA/MD Program - Health Professions Advising Center (HPAC) - The University of Texas at Dallas (2015). Accessed July 28, 2015
  4. 4.
    Cooke, M., Irby, D., Sullivan, W., Ludmerer, K.: American Medical Education 100 Years after the Flexner Report. The New England Journal of Medicine. Boston: Sep 28, 2006. vol. 355, Iss. 13, pp. 1339–44 (6 pp.)Google Scholar
  5. 5.
    Leape, L., Berwick, D., Clancy, C., Conway, J., Gluck, P., Guest, J., et al.: Transforming healthcare: a safety imperative. Qual. Saf. Health Care 18(6), 424–428 (2009). doi: 10.1136/qshc.2009.036954 CrossRefGoogle Scholar
  6. 6.
    Transformation In Medical Education (TIME): A multi-institutional initiative within the University of Texas system. (n.d.). Retrieved January 6, 2016.
  7. 7.
    Ke, F.: A qualitative meta-analysis of computer games as learning tools. Handb. Res. Effective Electron. Gaming Educ. 1, 1–32 (2009)CrossRefGoogle Scholar
  8. 8.
    Breuer, J.S., Bente, G.: Why so serious? On the relation of serious games and learning. Eludamos J. Comput. Game Cult. 4(1), 7–24 (2010)Google Scholar
  9. 9.
    Sitzmann, T.: A meta-analytic examination of the effectiveness of computer-based simulation games. Pers. Psychol. 64, 489–528 (2011). doi: 10.1111/j.1744-6570.2011.01190.x CrossRefGoogle Scholar
  10. 10.
    Rogers, T.B., Kuiper, N.A., Kirker, W.S.: Self-reference and the encoding of personal information. J. Personal. Soc. Psychol. 35(9), 677–688 (1977). doi: 10.1037/0022-3514.35.9.677 CrossRefGoogle Scholar
  11. 11.
    Vos, N., Van Der Meijden, H., Denessen, E.: Effects of constructing versus playing an educational game on student motivation and deep learning strategy use. Comput. Educ. 56(1), 127–137 (2011)CrossRefGoogle Scholar
  12. 12.
    Dalbello, M.: A genealogy of digital humanities. J. Documentation 67(3), 480–506 (2011)CrossRefGoogle Scholar
  13. 13.
    Smith, A.: U.S. Smartphone Use in 2015 (2015, April 01). Retrieved February 26, 2016.
  14. 14.
    Aggarwal, C.C.: Social network data analytics. Springer, New York (2011)CrossRefzbMATHGoogle Scholar
  15. 15.
    Gandomi, A., Haider, M.: Beyond the hype: Big data concepts, methods, and analytics. Int. J. Inf. Manag. 35(2), 137–144 (2015)CrossRefGoogle Scholar
  16. 16.
    Ochieng, A.P.: An analysis of the strengths and limitation of qualitative and quantitative research paradigms. In: Problems of Education in the 21stcentury, 13, 13–18 (2009). Retrieved January 20, 2016.
  17. 17.
    Taylor, C.: Descriptive and Inferential Statistics: How Do They Differ? (2014, December 16). Retrieved February 26, 2016.
  18. 18.
    Kazdin, A.: Research design in clinical psychology. Allyn and Bacon, Boston (2003)Google Scholar
  19. 19.
    Leech, N.L., Onwuegbuzie, A.J.: Beyond constant comparison qualitative data analysis: Using NVivo. School Psychol. Q. 26(1), 70 (2011)CrossRefGoogle Scholar
  20. 20.
    DeLyser, D., Sui, D.: Crossing the qualitative-quantitative divide II Inventive approaches to big data, mobile methods, and rhythmanalysis. Prog. Hum. Geogr. 37(2), 293–305 (2013)CrossRefGoogle Scholar
  21. 21.
    Van’t Riet, A., Berg, M., Hiddema, F., Sol, K.: Meeting patients’ needs with patient information systems: potential benefits of qualitative research methods. Int. J. Med. Inf. 64(1), 1–14 (2001)CrossRefGoogle Scholar
  22. 22.
    Woods, M., Paulus, T., Atkins, D.P., Macklin, R.: Advancing qualitative research using qualitative data analysis software (QDAS)? Reviewing potential versus practice in published studies using ATLAS. ti and NVivo, 1994–2013. Social Science Computer Review, 0894439315596311 (2015)Google Scholar
  23. 23.
    Wang, W., Chen, L., Thirunarayan, K., Sheth, A.P.: Harnessing twitter “big data” for automatic emotion identification. In: Privacy, Security, Risk and Trust (PASSAT), 2012 International Conference on and 2012 International Conference on Social Computing (SocialCom) (pp. 587–592). IEEE (2012, September)Google Scholar
  24. 24.
    Shaver, P., Schwartz, J., Kirson, D., O’connor, C.: Emotion knowledge: further exploration of a prototype approach. J. Pers. Soc. Psychol. 52(6), 1061 (1987)CrossRefGoogle Scholar
  25. 25.
    Connolly, T.M., Boyle, E.A., MacArthur, E., Hainey, T., Boyle, J.M.: A systematic literature review of empirical evidence on computer games and serious games. Comput. Educ. 59(2), 661–686 (2012)CrossRefGoogle Scholar
  26. 26.
    Zielke, M.A., Zakhidov, D., Jacob, D., Hardee, G.: Beyond fun and games: toward an adaptive and emergent learning platform for pre-med students with the UT TIME portal. In: IEEE SeGAH 2016 Conference Proceedings Paper presented at IEEE 2016 International Conference on Serious Games and Applications for Health, Orlando, Florida. In press (2016, M)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Marjorie A. Zielke
    • 1
    Email author
  • Djakhangir Zakhidov
    • 1
  • Daniel Jacob
    • 1
  • Sean Lenox
    • 1
  1. 1.The University of Texas at DallasRichardsonUSA

Personalised recommendations