Investigating the feasibility of using assessment and explanatory feedback in desktop virtual reality simulations

  • Guido MakranskyEmail author
  • Richard Mayer
  • Anne Nøremølle
  • Ainara Lopez Cordoba
  • Jakob Wandall
  • Mads Bonde
Research Article


There is great potential in making assessment and learning complementary. In this study, we investigated the feasibility of developing a desktop virtual reality (VR) laboratory simulation on the topic of genetics, with integrated assessment using multiple choice questions based on item response theory (IRT) and feedback based on the cognitive theory of multimedia learning. A pre-test post-test design was used to investigate three research questions related to: (1) students’ perceptions of assessment in the form of MC questions within the VR genetics simulation; (2) the fit of the MC questions to the assumptions of the partial credit model (PCM) within the framework of IRT; and (3) if there was a significant increase in intrinsic motivation, self-efficacy, and transfer from pre- to post-test after using the VR genetics simulation as a classroom learning activity. The sample consisted of 208 undergraduate students taking a medical genetics course. The results showed that assessment items in the form of gamified multiple-choice questions were perceived by 97% of the students to lead to higher levels of understanding, and only 8% thought that they made the simulation more boring. Items within a simulation were found to fit the PCM and the results showed that the sample had a small significant increase in intrinsic motivation and self-efficacy, and a large significant increase in transfer following the genetics simulation. It was possible to develop assessments for online educational material and retain the relevance and connectedness of informal assessment while simultaneously serving the communicative and credibility-based functions of formal assessment, which is a great challenge facing education today.


Simulations Desktop virtual reality Assessment Explanatory feedback Item response theory Cognitive theory of multimedia learning Retrieval practice 



This research was funded by Innovation fund Denmark.

Compliance with ethical standards

Conflict of interest

Mads Bonde is a co-founder of the simulation development company Labster that provided the simulation that was used in this study. Ainara Lopez Cordoba works at Labster. The remaining authors declare that they have no conflict of interest.

Informed consent

Ethical consent was obtained from all participants in accordance with the ethical regulations of the Health Research Ethics Committee in Denmark.


  1. Adesope, O. O., Trevisan, D. A., & Sundararajan, N. (2017). Rethinking the use of tests: A meta-analysis of practice testing. Review of Educational Research, 87(3), 659–701.CrossRefGoogle Scholar
  2. Ai-Lim Lee, E., Wong, K. W., & Fung, C. C. (2010). How does desktop virtual reality enhance learning outcomes? A structural equation modeling approach. Computers & Education, 55(4), 1424–1442. Scholar
  3. Almond, R. G., Mislevy, R. J., Steinberg, L., Yan, D., & Williamson, D. (2015). Bayesian networks in educational assessment. New York: Springer.CrossRefGoogle Scholar
  4. Andrich, D., Sheridan, B., & Luo, G. (2010). Rasch models for measurement: RUMM2030. Perth, Australia: RUMM Laboratory.Google Scholar
  5. Au, W. (2007). High-stakes testing and curricular control: A qualitative metasynthesis. Educational Researcher, 36(5), 258–267. Scholar
  6. Bangert-Drowns, R. L., Kulik, C.-L. C., Kulik, J. A., & Morgan, M. (1991). The instructional effect of feedback in test-like events. Review of Educational Research, 61(2), 213–238. Scholar
  7. Bayraktar, S. (2000). A meta-analysis on the effectiveness of computer-assisted instruction in science education. Journal of Research on Technology in Education, 34(2), 173–189. Scholar
  8. Black, P. J., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in Education, 5(1), 7–73.Google Scholar
  9. Bonde, M. T., Makransky, G., Wandall, J., Larsen, M. V., Morsing, M., Jarmer, H., et al. (2014). Improving biotechnology education through simulations and games. Nature Biotechnology, 32(7), 694–697. Scholar
  10. Boud, D. (1995). Assessment and learning: Contradictory or complementary? In P. Knight (Ed.), Assessment for learning in higher education (pp. 35–48). London: Kogan.Google Scholar
  11. Brown, P. C., Roediger, H. L., & McDaniel, M. A. (2014). Make it stick: The science of successful learning. Cambridge, MA: Harvard University Press.CrossRefGoogle Scholar
  12. Burbules, N. C. (2006). Rethinking the virtual. In J. Weiss, J. Nolan, J. Hunsinger, & P. Trifonas (Eds.), The international handbook of virtual learning environments (pp. 37–58). Dordrecht: Springer. Scholar
  13. Butler, A. C., Karpicke, J. D., & Roediger, H. L., III. (2008). Correcting a metacognitive error: feedback increases retention of low-confidence correct responses. Journal of Experimental Psychology. Learning, Memory, and Cognition, 34(4), 918.CrossRefGoogle Scholar
  14. Butler, A. C., & Roediger, H. L. (2008). Feedback enhances the positive effects and reduces the negative effects of multiple-choice testing. Memory & Cognition, 36(3), 604–616.CrossRefGoogle Scholar
  15. Cranney, J., Ahn, M., McKinnon, R., Morris, S., & Watts, K. (2009). The testing effect, collaborative learning, and retrieval-induced facilitation in a classroom setting. European Journal of Cognitive Psychology, 21, 919–940.CrossRefGoogle Scholar
  16. Cummings, J. J., & Bailenson, J. N. (2016). How immersive is enough? A meta-analysis of the effect of immersive technology on user presence. Media Psychology, 19(2), 272–309. Scholar
  17. Dantas, A. M., & Kemm, R. E. (2008). A blended approach to active learning in a physiology laboratory-based subject facilitated by an e-learning component. Advances in Physiology Education, 32, 65–75. Scholar
  18. De Jong, T., & Van Joolingen, W. R. (1998). Scientific discovery learning with computer simulations of conceptual domains. Review of Educational Research, 68(2), 179–201.CrossRefGoogle Scholar
  19. Deci, E., Eghrari, H., Patrick, B., & Leone, D. (1994). Facilitating internalization: the self-determination theory perspective. Journal of Personality, 62(1), 119–142.CrossRefGoogle Scholar
  20. DeVellis, R. F. (1991). Scale development: Theory and applications. Thousand Oaks, CA, USA: Sage.Google Scholar
  21. Dunlosky, J., Rawson, K. A., Marsh, E. J., Nathan, M. J., & Willingham, D. T. (2013). Improving students’ learning with effective learning techniques: Promising directions from cognitive and educational psychology. Psychological Science in the Public Interest, 14, 4–58.CrossRefGoogle Scholar
  22. Gardner, L., Sheridan, D., & White, D. (2002). A web-based learning and assessment system to support flexible education. Journal of Computer Assisted Learning, 18, 125–136.CrossRefGoogle Scholar
  23. Gerjets, P., & Kirschner, P. (2009). Learning from multimedia and hypermedia. In S. Ludvigsen, et al. (Eds.), Technology-enhanced learning (pp. 251–272). Berlin: Springer.CrossRefGoogle Scholar
  24. Groth-Marnat, G. (2000). Visions of clinical assessment: Then, now, and a brief history of the future. Journal of Clinical Psychology, 56(3), 349–365.;2-T.CrossRefGoogle Scholar
  25. Hattie, J. (2009). Visible learning: A synthesis of 800 + meta-analyses on achievement. London: Routledge.Google Scholar
  26. Johnson, C. I., & Priest, H. A. (2014). The feedback principle in multimedia learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (pp. 449–463). Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  27. Jones, N. (2018). The virtual lab: Can a simulated laboratory experience provide the same benefits for students as access to a real-world lab? Nature, 562, S5–S7.CrossRefGoogle Scholar
  28. Kapur, M. (2008). Productive failure. Cognition and instruction, 26(3), 379–424.CrossRefGoogle Scholar
  29. Khan, K. S., Davies, D. A., & Gupta, J. K. (2001). Formative self-assessment using multiple true-false questions on the Internet: feedback according to confidence about correct knowledge. Medical Teacher, 23, 158e163.CrossRefGoogle Scholar
  30. Kluger, A. N., & DeNisi, A. (1996). The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin, 119(2), 254–284. Scholar
  31. Kraiger, K., Ford, J. K., & Salas, E. (1993). Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. Journal of Applied Psychology, 78(2), 311–328. Scholar
  32. Kruglikova, I., Grantcharov, T. P., Drewes, A. M., & Funch-Jensen, P. (2010). The impact of constructive feedback on training in gastrointestinal endoscopy using high-fidelity virtual-reality simulation: a randomised controlled trial. Gut, 59(2), 181–185.CrossRefGoogle Scholar
  33. Labster. (2019). Labster—Cytogenetics lab. Retrieved January 16th 2019 from
  34. Larsen, D. P., Butler, A. C., & Roediger, H. L. (2009). Repeated testing improves long-term retention relative to repeated study: A randomized controlled trial. Medical Education, 43, 1174–1181.CrossRefGoogle Scholar
  35. Lee, E. A.-L., & Wong, K. W. (2014). Learning with desktop virtual reality: Low spatial ability learners are more positively affected. Computers & Education, 79, 49–58. Scholar
  36. Lee, E. A.-L., Wong, K. W., & Fung, C. C. (2010). How does desktop virtual reality enhance learning outcomes? A structural equation modeling approach. Computers & Education, 55(4), 1424–1442. Scholar
  37. Makransky, G., Bonde, M. T., Wulff, J. S. G., Wandall, J., Hood, M., Creed, P. A., et al. (2016). Simulation based virtual learning environment in medical genetics counseling: An example of bridging the gap between theory and practice in medical education. BMC Medical Education, 16, 98. Scholar
  38. Makransky, G., Borre-Gude, S., & Mayer, R. E. (2019a). Motivational and cognitive benefits of training in immersive virtual reality based on multiple assessments. Journal of Computer Assisted Learning. Scholar
  39. Makransky, G., & Lilleholt, L. (2018). A structural equation modeling investigation of the emotional value of immersive virtual reality in education. Educational Technology Research and Development, 66, 1141–1164.CrossRefGoogle Scholar
  40. Makransky, G., Lilleholt, L., & Aaby, A. (2017). Development and validation of the Multimodal Presence Scale for virtual reality environments: A confirmatory factor analysis and item response theory approach. Computers in Human Behavior, 72, 276–285. Scholar
  41. Makransky, G., Mayer, R. E., Veitch, N., Hood, M., Christensen, K. B., & Gadegaard, H. (2019b). Equivalence of using a desktop virtual reality science simulation at home and in class. PLoS ONE, 14(4), e0214944. Scholar
  42. Makransky, G., & Petersen, G. B. (2019). Investigating the process of learning with desktop virtual reality: A structural equation modeling approach. Computers & Education. Scholar
  43. Makransky, G., Schnohr, C., Torsheim, T., & Currie, C. (2014). Equating the HBSC family affluence scale across survey years: A method to account for item parameter drift using the Rasch model. Quality of Life Research, 23(10), 2899–2907. Scholar
  44. Makransky, G., Terkildsen, T. S., & Mayer, R. E. (2019c). Adding immersive virtual reality to a science lab simulation causes more presence but less learning. Learning and Instruction, 60, 225–236. Scholar
  45. Makransky, G., Wismer, P., & Mayer, R. E. (2018). A gender matching effect in learning with pedagogical agents in an immersive virtual reality science simulation. Journal of Computer Assisted Learning. Scholar
  46. Marcus, N., Ben-Naim, D., & Bain, M. (2011). Instructional support for teachers and guided feedback for students in an adaptive elearning environment. In Information Technology: New Generations (ITNG), 2011 Eighth International Conference on (pp. 626–631). IEEE.Google Scholar
  47. Masters, G. N. (1982). A rasch model for partial credit scoring. Psychometrika, 47(2), 149–174. Scholar
  48. Mayer, R. E. (2008). Learning and instruction (2nd ed.). Upper Saddle River, NJ: Pearson Merrill Prentice Hall.Google Scholar
  49. Mayer, R. E. (2009). Multimedia learning (2nd ed.). New York: Cambridge University Press.CrossRefGoogle Scholar
  50. Mayer, R. E. (2011). Applying the science of learning. Boston: Pearson.Google Scholar
  51. McDaniel, M. A., Agarwal, P. K., Huelser, B. J., McDermott, K. B., & Roediger, H. L. I. I. I. (2011). Test-enhanced learning in a middle school science classroom: The effects of quiz frequency and placement. Journal of Educational Psychology, 103(2), 399–414.CrossRefGoogle Scholar
  52. McDermott, K. B., Agarwal, P. K., D’Antonio, L., Roediger, H. L., & McDaniel, M. A. (2014). Both multiple-choice and short-answer quizzes enhance later exam performance in middle and high school classes. Journal of Experimental Psychology: Applied, 20, 3–21.Google Scholar
  53. McGaghie, W. C., Issenberg, S. B., Cohen, E. R., Barsuk, J. H., & Wayne, D. B. (2011). Does simulation-based medical education with deliberate practice yield better results than traditional clinical education? A meta-analytic comparative review of the evidence. Academic Medicine : Journal of the Association of American Medical Colleges, 86(6), 706–711. Scholar
  54. McGaghie, W. C., Issenberg, S. B., Petrusa, E. R., & Scalese, R. J. (2010). A critical review of simulation-based medical education research: 2003–2009. Medical Education, 44(1), 50–63. Scholar
  55. Merchant, Z., Goetz, E. T., Cifuentes, L., Keeney-Kennicutt, W., & Davis, T. J. (2014). Effectiveness of virtual reality-based instruction on students’ learning outcomes in K-12 and higher education: A meta-analysis. Computers & Education, 70, 29–40. Scholar
  56. Meyer, O. A., Omdahl, M. K., & Makransky, G. (2019). Investigating the effect of pre-training when learning through immersive virtual reality and video: A media and methods experiment. Computers & Education. Scholar
  57. Mislevy, R., J. (2016). Postmodern test theory. The gordon commission on the future of assessment in education. Retrieved from
  58. Moreno, R. (2004). Decreasing cognitive load for novice students: Effects of explanatory versus corrective feedback in discovery-based multimedia. Instructional Science, 32(1/2), 99–113. Scholar
  59. Moreno, R., & Valdez, A. (2005). Cognitive load and learning effects of having students organize pictures and words in multimedia environments: The role of student interactivity and feedback. Educational Technology Research and Development, 53(3), 35–45. Scholar
  60. National Research Council. (2011). Learning science through computer games and simulations. Washington: National Research Council.Google Scholar
  61. Nitko, A. J. (1996). Educational assessment of students. Des Moines, IA: Prentice-Hall Order Processing Center.Google Scholar
  62. Pallant, J. F., & Tennant, A. (2007). An introduction to the Rasch measurement model: An example using the Hospital Anxiety and Depression Scale (HADS). British Journal of Clinical Psychology, 46(1), 1–18. Scholar
  63. Pellegrino, J. W., Chudowsky, N., & Glaser, R. (Eds.). (2001). Knowing what students know: The science and design of educational assessment. Washington, DC: National Academies Press.Google Scholar
  64. Pellegrino, J. W., & Hilton, M. L. (2012). Education for life and work: Developing transferable knowledge and skills in the 21st century. Washington, DC: National Academies Press.Google Scholar
  65. Perkins, D. (1994). Do students understand understanding? Education Digest, 59(5), 21.Google Scholar
  66. Pintrich, P. R. R., Smith, D., Garcia, T., & McKeachie, W. (1991). A manual for the use of the motivated strategies for learning questionnaire (MSLQ). Ann Arbor, MI: University of Michigan.Google Scholar
  67. Polly, P., Marcus, N., Maguire, D., Belinson, Z., & Velan, G. M. (2014). Evaluation of an adaptive virtual laboratory environment using Western Blotting for diagnosis of disease. BMC Medical Education, 14(1), 222.CrossRefGoogle Scholar
  68. Richardson, M., Abraham, C., & Bond, R. (2012). Psychological correlates of university students’ academic performance: A systematic review and meta-analysis. Psychological Bulletin, 138(2), 353–387. Scholar
  69. Roediger, H. L., & Butler, A. C. (2011). The critical role of retrieval practice in long-term retention. Trends in Cognitive Sciences, 15(1), 20–27.CrossRefGoogle Scholar
  70. Roelle, J., & Berthold, K. (2017). Effects of incorporating retrieval into learning tasks: The complexity of the tasks matters. Learning and Instruction, 49, 142–156.CrossRefGoogle Scholar
  71. Ronen, M., & Eliahu, M. (2000). Simulation—a bridge between theory and reality: the case of electric circuits. Journal of Computer Assisted Learning, 16(1), 14–26.CrossRefGoogle Scholar
  72. Rummer, R., Schweppe, J., Scheiter, K., & Gerjets, P. (2008). Lernen mit Multimedia: die kognitiven Grundlagen des Modalitätseffekts. Psychologische Rundschau, 59(2), 98–107.Google Scholar
  73. Rutten, N., Van Joolingen, W. R., & Van Der Veen, J. T. (2012). The learning effects of computer simulations in science education. Computers & Education, 58(1), 136–153. Scholar
  74. Ryan, R. M., Deci, E. L. (2000). Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American Psychologist, 55 (1): 68–78. CiteSeerX10.1.1.529.4370.
  75. Ryan, R. M., & Deci, E. L. (2016). Facilitating and hindering motivation, learning, and well-being in schools: Research and observations from self-determination theory. In K. R. Wentzel & D. B. Miele (Eds.), Handbook of motivation at school (2nd ed., pp. 96–119). New York: Routledge.Google Scholar
  76. Sackett, P. R., Borneman, M. J., & Connelly, B. S. (2008). High-stakes testing in higher education and employment. American Psychologist, 64, 215–227. Scholar
  77. Sadler, D. R. (1998). Formative assessment: revisiting the territory. Assessment in Education, 5(1), 77e84.Google Scholar
  78. Schraw, G., Mayrath, M. C., ClarkeMidura, J., & Robinson, D. H. (Eds.). (2012). Technology based assessments for 21st century skills: Theoretical and practical implications from modern research. IAPGoogle Scholar
  79. Schunk, D. H., & DiBenedetto, M. K. (2016). Self-efficacy theory in education. In K. R. Wentzel & D. B. Miele (Eds.), Handbook of motivation at school (pp. 34–54). New York: Routledge.Google Scholar
  80. Shavelson, R. J., Young, D. B., Ayala, C. C., Brandon, P. R., Furtak, E. M., Ruiz-Primo, M. A., ··· & Yin, Y. (2008). On the impact of curriculum-embedded formative assessment on learning: A collaboration between curriculum and assessment developers. Applied Measurement in Education, 21(4), 295–314.Google Scholar
  81. Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153–189. Scholar
  82. Shute, V. J. (2011). Stealth assessment in computer-based games to support learning. Computer Games and Instruction, 55(2), 503–524.Google Scholar
  83. Shute, V. J., & Becker, B. J. (2010). Innovative assessment for the 21st century. New York, NY: Springer.CrossRefGoogle Scholar
  84. Shute, V. J., & Ke, F. (2012). Assessment in game-based learning. In D. Eseryel (Ed.), Assessment in game-based learning: Foundations, innovations, and perspectives (pp. 43–58). New York: Springer. Scholar
  85. Shute, V. J., Leighton, J. P., Jang, E. E., & Chu, M.-W. (2016). Advances in the science of assessment. Educational Assessment, 21(1), 34–59. Scholar
  86. Shute, V. J., Ventura, M., Bauer, M., & Zapata-Rivera, D. (2009). Melding the power of serious games and embedded assessment to monitor and foster learning. Serious Games: Mechanisms and Effects, 2, 295–321.Google Scholar
  87. Smith Jr., E. V. (2002). Detecting and evaluating the impact of multidimensionality using item fit statistics and principal component analysis of residuals. Journal of Applied Measurement, 3(2), 205–231. Retrieved from
  88. Stiggins, R. J., Arter, J. A., Chappuis, J., & Chappuis, S. (2004). Classroom assessment for student learning: doing it right–using it well. Portland: Assessment Training Institute.Google Scholar
  89. Strandbygaard, J., Bjerrum, F., Maagaard, M., Winkel, P., Larsen, C. R., Ringsted, C., ··· & Sorensen, J. L. (2013). Instructor feedback versus no instructor feedback on performance in a laparoscopic virtual reality simulator: a randomized trial. Annals of Surgery, 257(5), 839–844.Google Scholar
  90. Tennant, A., & Conaghan, P. G. (2007). The Rasch measurement model in rheumatology: What is it and why use it? When should it be applied, and what should one look for in a Rasch paper? Arthritis Care and Research, 57(8), 1358–1362. Scholar
  91. Thisgaard, M., & Makransky, G. (2017). Virtual learning simulations in high school: Effects on cognitive and non-cognitive outcomes and implications on the development of STEM academic and career choice. Frontiers in Psychology. Scholar
  92. Tsai, F.-H., Tsai, C.-C., & Lin, K.-Y. (2015). The evaluation of different gaming modes and feedback types on game-based formative assessment in an online learning environment. Computers & Education, 81, 259–269. Scholar
  93. Uner, O., & Roediger, H. L. (2018). The effect of question placement on learning from textbook chapters. Journal of Applied Research in Memory and Cognition, 7(1), 116–122.CrossRefGoogle Scholar
  94. Wigfield, A., Guthrie, J. T., Tonks, S., & Perencevich, K. C. (2004). Children’s motivation for reading: Domain specificity and instructional influences. Journal of Educational Research, 97(6), 299–309. Scholar
  95. Williams, J. R. (2008). The declaration of Helsinki and public health. Bulletin of the World Health Organization. Scholar
  96. Wilson, M., & Sloane, K. (2000). From principles to practice: An embedded assessment system. Applied Measurement in Education, 13(2), 181–208.CrossRefGoogle Scholar
  97. Zimmerman, B. J. (2000). Self-efficacy: An essential motive to learn. Contemporary Educational Psychology, 25(1), 82–91.CrossRefGoogle Scholar

Copyright information

© Association for Educational Communications and Technology 2019

Authors and Affiliations

  1. 1.Department of PsychologyUniversity of CopenhagenCopenhagenDenmark
  2. 2.Psychological and Brain SciencesUniversity of California Santa BarbaraSanta BarbaraUSA
  3. 3.Department of Cellular and Molecular MedicineUniversity of CopenhagenCopenhagenDenmark
  4. 4.LabsterCopenhagenDenmark
  5. 5.Department of EducationUniversity of AarhusCopenhagenDenmark
  6. 6.Department of Drug Design and PharmacologyUniversity of CopenhagenCopenhagenDenmark

Personalised recommendations