Simulation offers unique affordances over traditional training (e.g., remote access, mastery learning experiences, immediate feedback) relevant to teacher training in behavior management. This study describes a user-based evaluation of Interactive Virtual Training for Teachers (IVT-T). The study involved observing representative users (seven advanced education majors) perform benchmark tasks with the system, complete rating scales, and participate in interviews to evaluate the usability and instructional design quality of IVT-T. Global usability ratings based on established usability rating scales suggested IVT-T was adequately usable while observations of user performance and semi-structured interviews revealed design shortcomings that impeded effective user performance and informed ways to improve the interface. Observations of user performance, for example, identified 36% of usability problems related to learning, 19% = screen design, 17% = terminology; 3% = system capabilities and 25% = other problems. Cross analysis of user semi-structured interviews pointed to the system’s ability to convey believable, visually appealing, realistic characters and classrooms. More contextual cues, multiple challenging behaviors featured at the same time, and changes to the visual appearance of the classroom would enhance realism. Revisions made to enhance the usability and instructional design elements of IVT-T are discussed. In addition, implications for teacher educators and researchers involved in the development of instructional technologies are summarized along with the potential value of including simulation in teacher training for behavior management.
This is a preview of subscription content,to check access.
Access this article
Similar content being viewed by others
Alessi, S. M., & Trollip, S. R. (2001). Multimedia for learning: Methods and development. Boston, MA: Allyn & Bacon.
Annetta, L., Lamb, R., Minogue, J., Folta, E., Holmes, S., Vallett, D., et al. (2014). Safe science classrooms: Teacher training through serious educational games. Information Sciences, 264, 61–74.
Antonenko, P. D., Dawson, K., & Sahay, S. (2017). A framework for aligning needs, abilities and affordances to inform design and practice of educational technologies. British Journal of Educational Technology, 48(4), 916–927.
Aslan, S., & Reigeluth, C. M. (2016). Investigating “the coolest school in America:” How technology is used in a learner-centered school. Educational Technology Research and Development, 64(6), 1107–1133.
Atkins, M. S., Shernoff, E. S., Frazier, S. L., Schoenwald, S. K., Cappella, E., Marinez-Lora, A., et al. (2015). Re-designing community mental health services for urban children: Supporting schooling to promote mental health. Journal of Consulting and Clinical Psychology, 83(5), 839–852.
Atteberry, A., Loeb, S., & Wyckoff, J. (2017). Teacher churning: Reassignment rates and implications for student achievement. Educational Evaluation and Policy Analysis, 39(1), 3–30.
Badiee, F., & Kaufman, D. (2015). Design evaluation of a simulation for teacher education. Sage Open, 5(2), 1–10. https://doi.org/10.1177/2158244015592454.
Bangor, A., Kortum, P. T., & Miller, J. T. (2008). An empirical evaluation of the system usability scale. International Journal of Human-Computer Interaction, 24(6), 574–594.
Bellotti, F., Berta, R., & De Gloria, A. (2010). Designing effective serious games: Opportunities and challenges for research. International Journal of Emerging Technologies in Learning, 5(SI3), 22–35.
Beutler, L. E., & Harwood, T. M. (2004). Virtual reality in psychotherapy training. Journal of Clinical Psychology, 60(3), 317–330.
Bowman, D. A., Gabbard, J. L., & Hix, D. (2002). A survey of usability evaluation in virtual environments: classification and comparison of methods. Presence: Teleoperators & Virtual Environments, 11(4), 404–424.
Brooke, J. (1996). SUS-A quick and dirty usability scale. Usability Evaluation in Industry, 189(194), 4–7.
Brunsma, D. L. (2005). Uniforms in public schools: A decade of research and debate. Lanham, MD: Rowman & Littlefield Publishing Group.
Chin, J. P., Diehl, V. A., & Norman, K. L. (1988). Development of an instrument measuring user satisfaction of the human-computer interface. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 213–218). ACM.
Cooke, L. (2010). Assessing concurrent think-aloud protocol as a usability test method: A technical communication approach. IEEE Transactions on Professional Communication, 53(3), 202–215.
Dawson, M. R., & Lignugaris-Kraft, B. (2017). Meaningful practice: Generalizing foundation teaching skills from TLE TeachLivE™ to the classroom. Teacher Education and Special Education, 40(1), 26–50.
Dede, C. (2009). Immersive interfaces for engagement and learning. Science, 323(5910), 66–69. https://doi.org/10.1126/science.1167311.
Desimone, L. M., Porter, A. C., Garet, M. S., Yoon, K. S., & Birman, B. F. (2002). Effects of professional development on teachers’ instruction: Results from a three-year longitudinal study. Educational Evaluation and Policy Analysis, 24(2), 81–112.
Dieker, L., Hynes, M., Stapleton, C., & Hughes, C. (2007). Virtual classrooms: Star simulator. New Learning Technology SALT, 4, 1–22.
Dieker, L. A., Hughes, C. E., Hynes, M. C., & Straub, C. (2017). Using simulated virtual environments to improve teacher performance. Journal of the National Association for Professional Development Schools, 10(3), 62–81.
Drisko, J. W., & Maschi, T. (2015). Content analysis: Pocket guides to social work research methods. Oxford, NY: Oxford University Press.
Epstein, J. L., Sanders, M. G., Sheldon, S. B., Simon, B. S., Salinas, K. C., Jansorn, N. R., … & Williams, K. J. (2008). School, Family, and Community Partnerships: Youth Handbook for Action (3rd ed.). Thousand Oaks, CA: Corwin Press.
Evertson, C. M., & Weinstein, C. S. (Eds.). (2006). Handbook of classroom management: Research, practice, and contemporary issues. Mahwah, NJ: Lawrence Erlbaum.
Finstad, K. (2006). The system usability scale and non-native English speakers. Journal of Usability Studies, 1(4), 185–188.
Fletcher, J. D. (2009). The value of expertise and expert performance: A review of evidence from the military. In K. A. Ericsson (Ed.), Development of Professional Expertise (pp. 449–469). NY: Cambridge University Press.
Fonteyn, M. E., Vettese, M., Lancaster, D. R., & Bauer-Wu, S. (2008). Developing a codebook to guide content analysis of expressive writing transcripts. Applied Nursing Research, 21(3), 165–168.
Gabbard, J. L., Hix, D., & Swan, J. E. (1999). User-centered design and evaluation of virtual environments. IEEE Computer Graphics and Applications, 19(6), 51–59.
Gabbard, J. L., & Swan, J. E., II. (2008). Usability engineering for augmented reality: Employing user-based studies to inform design. IEEE Transactions on Visualization and Computer Graphics, 14(3), 513–525.
Graafland, M., Schraagen, J. M. C., Boermeester, M. A., Bemelman, W. A., & Schijven, M. P. (2015). Training situational awareness to reduce surgical errors in the operating room. British Journal of Surgery, 102(1), 16–23.
Gregory, S., & James, R. (2011). VirtualPREX: Open and Distance Learning for pre-service teachers. In Expanding Horizons-New Approaches to Open and Distance Learning. Presented at the 24th ICDE World Conference on Open & Distance Learning, Bali.
Gresalfi, M. S., & Barnes, J. (2016). Designing feedback in an immersive videogame: supporting student mathematical engagement. Educational Technology Research and Development, 64(1), 65–86.
Guarino, C. M., Santibañez, L., & Daley, G. A. (2006). Teacher recruitment and retention: A review of the recent empirical literature. Review of Educational Research, 76, 173–208.
Harper, B. D. & Norman, K. L. (1993). Improving User Satisfaction: The Questionnaire for User Interaction Satisfaction Version 5.5. In: Proceedings of the 1st Annual Mid-Atlantic Human Factors Conference (pp. 224–228), Virginia Beach, VA.
Harrati, N., Bouchrika, I., Tari, A., & Ladjailia, A. (2016). Exploring user satisfaction for e-learning systems via usage-based metrics and system usability scale analysis. Computers in Human Behavior, 61, 463–471.
Hartson, R., & Pyla, P. S. (2012). The UX book: Process and guidelines for ensuring a quality user experience. Waltham, MA: Elsevier.
Henry, D., Guerra, N., Huesmann, R., Tolan, P., VanAcker, R., & Eron, L. (2000). Normative influences on aggression in urban elementary school classrooms. American Journal of Community Psychology, 28(1), 59–81. https://doi.org/10.1023/A:1005142429725.
Hew, K. F., & Brush, T. (2007). Integrating technology into K-12 teaching and learning: Current knowledge gaps and recommendations for future research. Educational Technology Research and Development, 55(3), 223–252.
Hix, D., & Hartson, R. (1993). Developing user interfaces: Ensuring usability through product and process. New York, NY: Wiley & Sons.
Horswill, I., & Lisetti, C. L. (2011). On the simulation of human frailty. In BICA (pp. 146–150).
Ingersoll, R. M., & Strong, M. (2011). The impact of induction and mentoring programs for beginning teachers: A critical review of the research. Review of Educational Research, 81, 201–233.
Jaspers, M. W. (2009). A comparison of usability methods for testing interactive health technologies: Methodological aspects and empirical evidence. International Journal of Medical Informatics, 78(5), 340–353.
Kazdin, A. E. (2005). Parent management training. New York: Oxford University Press.
Kolb, D. A., Boyatzis, R. E., & Mainemelis, C. (2000). Experiential learning theory: Previous research and new directions. In R. J. Sternberg & L. E. Zhang (Eds.), Perspectives on cognitive, learning, and thinking styles (pp. 227–247). Mahwah, NJ: Lawrence Erlbaum.
Laine, T. H., Nygren, E., Dirin, A., & Suk, H. J. (2016). Science spots AR: A platform for science learning games with augmented reality. Educational Technology Research and Development, 64(3), 507–531.
Lieber, E. (2009). Mixing qualitative and quantitative methods: Insights into design and analysis issues. Journal of Ethnographic & Qualitative Research, 3(4), 218–227.
Lindsey, L., & Berger, N. (2009). Experiential approach to instruction. In C. M. Reigeluth & A. A. Carr-Chellman (Eds.), Instructional-design theories and models (Vol. 3, pp. 117–142). New York, NY: Routledge.
Ludwick, D. A., & Doucette, J. (2009). Adopting electronic medical records in primary care: Lessons learned from health information systems implementation experience in seven countries. International Journal of Medical Informatics, 78(1), 22–31.
Lugrin, J. L., Latoschik, M. E., Habel, M., Roth, D., Seufert, C., & Grafe, S. (2016). Breaking bad behaviors: A new tool for learning classroom management using virtual reality. Frontiers in ICT, 3, 1–26.
Mayhew, D. J. (1999). The usability engineering lifecycle: A practitioner's handbook for user interface design. San Francisco, CA: Morgan Kaufman.
McLaughlin, T., Smith, D., & Brown, I. A. (2010, June). A framework for evidence based visual style development for serious games. In Proceedings of the Fifth International Conference on the Foundations of Digital Games (pp. 132–138).
McLellan, E., MacQueen, K. M., & Neidig, J. L. (2003). Beyond the qualitative interview: Data preparation and transcription. Field Methods, 15(1), 63–84.
Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook (2nd ed.). Sage Publications, Inc.
Miles, M. B., & Huberman, A. M. (2019). Qualitative data analysis: A methods sourcebook (4th ed.). Thousand Oaks, CA: Sage.
Nielsen, J. (1993). Usability engineering. Boston, MA: Academic Press.
Ouellette, R. R., Frazier, S. L., Shernoff, E. S., Cappella, E., Mehta, T. G., Maríñez-Lorad, A., et al. (2018). Teacher job stress and satisfaction in urban schools: Disentangling individual, classroom, and organizational level influences. Behavior Therapy, 49, 494–508.
Owens, J. S., Allan, D. M., Hustus, C., & Erchul, W. P. (2018). Examining correlates of teacher receptivity to social influence strategies within a school consultation relationship. Psychology in the Schools, 55, 1041–1055. https://doi.org/10.1002/pits.22163.
Parlangeli, O., Marchigiani, E., & Bagnara, S. (1999). Multimedia systems in distance education: Effects of usability on learning. Interacting with Computers, 12(1), 37–49.
Pas, E. T., Johnson, S. R., Larson, K. E., Brandenburg, L., Church, R., & Bradshaw, C. P. (2016). Reducing behavior problems among students with autism spectrum disorder: Coaching teachers in a mixed-reality setting. Journal of Autism and Developmental Disorders, 46(12), 3640–3652.
Rayner, C., & Fluck, A. (2014). Pre-service teachers’ perceptions of SimSchool as preparation for inclusive education: A pilot study. Asia-Pacific Journal of Teacher Education, 42(3), 212–227.
Regalla, M., Hutchinson, C., Nutta, J., & Ashtari, N. (2016). Examining the impact of a simulation classroom experience on teacher candidates’ sense of efficacy in communicating with English learners. Journal of Technology and Teacher Education, 24(3), 337–367.
Richey, R., Klein, J., & Tracey, M. W. (2011). The instructional design knowledge base: Theory, research and practice. New York, NY: Routledge.
Saldaña, J. (2015). The coding manual for qualitative researchers. Washington, DC: Sage.
Schussler, D., Frank, J., Lee, T. K., & Mahfouz, J. (2017). Using virtual role-play to enhance teacher candidates’ skills in responding to bullying. Journal of Technology and Teacher Education, 25(1), 91–120.
Shernoff, E. S., & Kratochwill, T. R. (2007). Transporting an evidence-based classroom management program for preschoolers with disruptive behavior problems to a school: An analysis of implementation, outcomes, and contextual variables. School Psychology Quarterly, 22(3), 449–472. https://doi.org/10.1037/1045-3822.214.171.1249.
Shernoff, E. S., Mehta, T., Atkins, M. S., Torf, R., & Spencer, J. (2011). A qualitative study of the sources and impact of stress among urban teachers. School Mental Health, 3, 59–69. https://doi.org/10.1007/s12310-011-9051-z.
Shernoff, E. S., Lakind, D., Frazier, S. L., & Jakobsons, L. (2015). Coaching early career teachers in urban elementary schools: A mixed-method study. School Mental Health, 7, 6–20. https://doi.org/10.1007/s12310-014-9136-6.
Shernoff, E. S., Frazier, S. L., Marinez-Lora, A., Lakind, D., Atkins, M. S., Jakobsons, L., et al. (2016). Expanding the role of school psychologists to support early career teachers: A mixed-method study. School Psychology Review, 45, 226–249.
Shernoff, E. S., Frazier, S. L., Lisetti, C., Buche, C., Lunn, S., Brown, C., et al. (2018). Early career teacher professional development: Bridging simulation technology with evidence-based behavior management. Journal of Technology and Teacher Education, 26(2), 299–326.
Simonsen, B., Fairbanks, S., Briesch, A., Myers, D., & Sugai, G. (2008). Evidence-based practices in classroom management: Considerations for research to practice. Education & Treatment of Children, 31(3), 351–380. https://doi.org/10.1353/etc.0.0007.
Sitzmann, T. (2011). A meta-analytic examination of the instructional effectiveness of computer-based simulation games. Personnel Psychology, 64(2), 489–528.
Slaughter, L., Norman, K.L., Shneiderman, B. (1995, March) Assessing users' subjective satisfaction with the Information System for Youth Services (ISYS). In Proceedings of the Third Annual Mid-Atlantic Human Factors Conference. Blacksburg, VA: Virginia Tech.
Thomas, D. E., Bierman, K. L., & Conduct Problems Prevention Research Group. (2006). The impact of classroom aggression on the development of aggressive behavior problems in children. Development and Psychopathology, 18(2), 471–487.
Thompson, N., & McGill, T. J. (2017). Genetics with Jean: the design, development and evaluation of an affective tutoring system. Educational Technology Research and Development, 65(2), 279–299.
Tracey, M. W., Hutchinson, A., & Quinn, G. (2014). Instructional designers as reflective practitioners: Developing professional identity through reflection. Educational Technology Research & Development, 62(3), 315–334. https://doi.org/10.1007/s11423-014-9334-9.
Varier, D., Dumke, E., Abrams, L., Conklin, S., Barnes, J., & Hoover, N. (2017). Potential of one-to-one technologies in the classroom: Teachers and students weigh in. Educational Technology Research and Development, 65(4), 967–992. https://doi.org/10.1007/s11423-017-9509-2.
Verdú, E., Regueras, L. M., Gal, E., Castro, J. P., Verdú, M. J., & Kohen-Vacs, D. (2017). Integration of an intelligent tutoring system in a course of computer network design. Educational Technology Research and Development, 65, 653–677. https://doi.org/10.1007/s11423-016-9503-0.
Virzi, R. A. (1992). Refining the test phase of usability evaluation: How many subjects is enough? Human Factors, 34(4), 457–468.
Whyte, E. M., Smyth, J. M., & Scherf, K. S. (2015). Designing serious game interventions for individuals with autism. Journal of Autism and Developmental Disorders, 45(12), 3820–3831.
Xie, K., Kim, M. K., Cheng, S. L., & Luthy, N. C. (2017). Teacher professional development through digital content evaluation. Educational Technology Research and Development, 65(4), 1067–1111.
Zibit, M., & Gibson, D. (2005). simSchool: The game of teaching. Innovate: Journal of Online Education, 4, 1–7.
The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305A150166 to the Rutgers, the State University of New Jersey. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.
Conflict of interest
The authors declare that they have no conflict of interest.
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Shernoff, E.S., Von Schalscha, K., Gabbard, J.L. et al. Evaluating the usability and instructional design quality of Interactive Virtual Training for Teachers (IVT-T). Education Tech Research Dev 68, 3235–3262 (2020). https://doi.org/10.1007/s11423-020-09819-9