Abstract
Strong interpersonal communication skills (ICS) are critical for educational and career success, but effective and widely accessible training systems are not available. This paper describes a 2 × 2 × 2 experimental study of an online, educational simulation for practice with the ICS of active listening. The simulation was customized for women graduate students in the natural sciences and engineering. In such environments, where gender stereotyping is common, ICS can make the difference between continued progress and discouraging setbacks. The pedagogical effects of following three instructional support variables were investigated: (1) elaborative versus simple feedback (2) presence versus absence of a static image to accompany the content delivered aurally by a human pedagogical agent, and (3) presence versus absence of instructional hints. The four outcome measures were self-reported knowledge about, skill in applying, and self-efficacy with respect to active listening, along with the usability of the simulation itself. Participants in the study included N = 137 women in the natural sciences and engineering. Results showed that the instructional support variables were significantly related to the outcome measures of knowledge, skills, and usability, but not self-efficacy with respect to active listening. A three-way interaction among all three of the instructional support variables was found to be statistically significant for both the knowledge and skills outcome variables; for both, the highest scores were obtained by participants who were presented with elaborative feedback and neither pedagogical agent image nor hints. Also, participants who received elaborative feedback reported the simulation to have significantly greater usability than those who received simple feedback.
Similar content being viewed by others
References
Adams, W. K., Reid, S., Lemaster, R., McKagan, S. B., Perkins, K. K., Dubson, M., & Wieman, C. E. (2008). A study of educational simulations Part 1: Engagement and learning. Journal of Interactive Learning Research, 19(3), 397–419.
American Board of Medical Specialties. (2014). Standards for the ABMS program for maintenance of certification (MOC). Accessed 4 May 2015. http://www.abms.org/media/1109/standards-for-the-abms-program-for-moc-final.pdf.
Atkinson, R. K., Mayer, R. E., & Merrill, M. M. (2005). Fostering social agency in multimedia learning: Examining the impact of an animated agent’s voice. Contemporary Educational Psychology, 30, 117–139.
Ayres, P., & Sweller, J. (2005). The split-attention principle in multimedia learning. In R. Mayer (Ed.), Cambridge handbook of multimedia learning (pp. 135–146). New York: Cambridge University Press.
Azevedo, R., & Bernard, R. M. (1995). A meta-analysis of the effects of feedback in computer-based instruction. Journal of Educational Computing Research, 13(2), 111–127.
Bandura, A. (1977). Self-efficacy: Toward a unifying theory of behavioral change. Psychological Review, 84, 191–215.
Bangert-Drowns, R. L., Kulik, C. C., Kulik, J. A., & Morgan, M. T. (1991). The instructional effect of feedback in test-like events. Review of Educational Research, 61, 213–238.
Bateman, J., & Davies, D. (2011). Virtual patients: Are we in a new era? Academic Medicine, 86(2), 151.
Baylor, A. L., & Kim, S. (2009). Designing nonverbal communication for pedagogical agents: When less is more. Computers in Human Behavior, 25(2), 450–457.
Baylor, A. L., & Ryu, J. (2003). The effects of image and animation in enhancing pedagogical agent persona. Journal of Educational Computing Research, 28(4), 373–394.
Berkhof, M., Van Rijssen, H. J., Schellart, A. J., Anema, J. R., & Van der Beek, A. J. (2011). Effective training strategies for teaching communication skills to physicians: An overview of systematic reviews. Patient Education and Counseling, 84(2), 152–162.
Bernstein, B. L., & Russo, N. F. (2008). Explaining too few women in academic science and engineering careers: A psychosocial perspective. In M. Paludi (Ed.), Series on the psychology of women at work: Challenges and solutions for our female workforce. Vol 2: Obstacles and the identity juggle (pp. 1–33). Westport, CN: Praeger Press.
Bodemer, D., Ploetzner, R., Feuerlein, I., & Spada, H. (2004). The active integration of information during learning with dynamic and interactive visualisations. Learning and Instruction, 14(3), 325–341.
Borrego, M., & Newswander, L. K. (2010). Definitions of interdisciplinary research: Toward graduate-level interdisciplinary learning outcomes. The Review of Higher Education, 34(1), 61–84.
Boud, D. (2000). Sustainable assessment: Rethinking assessment for the learning society. Studies in Continuing Education, 22(2), 151–167.
Bransford, J. D., Brown, A. L., & Cocking, R. R. (1999). How people learn. Washington, DC: National Academy Press.
Brooke, J. (1996). SUS: A quick and dirty usability scale. In P. W. Jordan, B. Thomas, B. A. Weerdmeester, & I. L. McClelland (Eds.), Usability evaluation in industry. London: Taylor & Francis.
Bultas, M. W. (2011). Enhancing the pediatric undergraduate nursing curriculum through simulation. Journal of Pediatric Nursing, 26(3), 224–229.
Butow, P., Cockburn, J., Girgis, A., Bowman, D., Schofield, P., D’Este, C., & Tattersall, M. H. (2008). Increasing oncologists’ skills in eliciting and responding to emotional cues: Evaluation of a communication skills training program. Psycho-Oncology, 17(3), 209–218.
Campione, J. C., & Brown, A. L. (1987). Linking dynamic testing with school achievement. In C. S. Lidz (Ed.), dynamic testing (pp. 82–115). New York: Guilford.
Chan, D., & Schmitt, N. (1997). Video-based versus paper-and-pencil method of assessment in situational judgment tests: Subgroup differences in performance and face validity perceptions. Journal of Applied Psychology, 82, 143–159.
Chen, L. H. (2010). Web-based learning programs: Use by learners with various cognitive styles. Computers & Education, 54(4), 1028–1035.
Clark, R. E., & Feldon, D. F. (2005). Five common but questionable principles of multimedia learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (pp. 97–115). New York: Cambridge University Press.
Cook, D. A., Erwin, P. J., & Triola, M. M. (2010). Computerized virtual patients in health professions education: A systematic review and meta-analysis. Academic Medicine, 85, 1589–1602.
Cook, D. A., & Triola, M. M. (2009). Virtual patients: A critical literature review and proposed next steps. Medical Education, 43, 303–311.
Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334.
deJong, T. (2010). Cognitive load theory, educational research, and instructional design: some food for thought. Instructional Science, 38, 105–134.
Devito, J. A. (2007). The interpersonal communication book (11th ed.). New York: Pearson.
Feinstein, A. H., & Cannon, H. M. (2002). Constructs of simulation evaluation. Simulation & Gaming, 33(4), 425–440.
Ford, N., & Chen, S. Y. (2001). Matching/mismatching revisited: an empirical study of learning and teaching styles. British Journal of Educational Technology, 32(1), 5–22.
Frechette, C., & Moreno, R. (2010). The roles of animated pedagogical agents’ presence and nonverbal communication in multimedia learning environments. Media Psychology, 22(2), 61–72.
Gordon, T. (1975). P.E.T: Parent effectiveness training. New York: New American Library.
Green, S. B., & Salkind, N. J. (2014). Using SPSS for Windows and Macintosh: Analyzing and understanding data. Upper Saddle River, NJ: Pearson.
Guadagno, R. E., Blascovich, J., Bailenson, J. N., & Mccall, C. (2007). Virtual humans and persuasion: The effects of agency and behavioral realism. Media Psychology, 10(1), 1–22.
Gunter, R., & Stambach, A. (2005). Differences in men and women scientists’ perceptions of workplace climate. Journal of Women and Minorities in Science and Engineering, 11(1), 97–116.
Hargie, O. (2011). Skilled interpersonal communication: Theory, research, and practice. New York, NY: Routledge.
Hargie, O., Boohan, M., McCoy, M., & Murphy, P. (2010). Current trends in communication skills training in UK schools of medicine. Medical Teacher, 32, 385–391.
Hartman, H. (2002). Scaffolding and cooperative learning. Human learning and instruction (pp. 23–69). New York: City College, University of New York.
Hattie, J., & Gan, M. (2011). Instruction based on feedback. In R. E. Mayer & P. A. Alexander (Eds.), Handbook of research on learning and instruction (pp. 249–271). New York, NY: Routledge.
Hattie, J., & Timperley, H. (2007). The power of feedback. Review of educational research, 77(1), 81–112.
Heidig, S., & Clarebout, G. (2011). Do pedagogical agents make a difference to student motivation and learning? Educational Research Review, 6, 27–54.
Herrington, J., Reeves, T. C., Oliver, R., & Woo, Y. (2004). Designing authentic activities in web-based courses. Journal of Computing in Higher Education, 16(1), 3–29.
Hill, C. (2009). Helping skills: Facilitating exploration, insight and action (3rd ed.). Washington, DC: American Psychological Association.
Issenberg, S. B., McGaghie, W. C., Petrusa, E. R., Lee Gordon, D., & Scalese, R. J. (2005). Features and uses of high-fidelity medical simulations that lead to effective learning: A BEME systematic review. Medical Teacher, 27(1), 10–28.
Jaehnig, W., & Miller, M. L. (2007). Feedback type s in programmed instruction: A systematic review. The Psychological Record, 57(2), 219–232.
Johnson, C. I., & Priest, H. A. (2014). The feedback principle in multimedia learning. In R. E. Mayer (Ed.), Cambridge handbook of multimedia learning (2nd ed., pp. 449–463). New York: Cambridge University Press.
Kalyuga, S. (2007). Expertise reversal effect and its implications for learner-tailored instruction. Educational Psychology Review, 19, 509–539.
Kalyuga, S., Ayres, P., Chandler, P., & Sweller, J. (2003). The expertise reversal effect. Educational Psychologist, 38, 23–31.
Kennedy, G. E. (2004). Promoting cognition in multimedia interactivity research. Journal of Interactive Learning Research, 15(1), 43–61.
Knapp, M., & Hall, J. (2009). Nonverbal communication in human interaction (7th ed.). Boston, MA: Wadsworth.
Kraiger, K., Ford, J., & Salas, E. (1993). Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. Journal of Applied Psychology, 78(2), 311–328.
Krathwohl, D. R., Bloom, B. S., & Masia, B. B. (1964). Taxonomy of educational objectives: The classification of educational goals. Handbook II: The affective domain. New York: David McKay.
Kuk, K., Milentijević, I., Rančić, D., & Spalević, P. (2012). Pedagogical agent in multimedia interactive modules for learning–MIMLE. Expert Systems with Applications, 39(9), 8051–8058.
Kulhavy, R. W., White, M. T., Topp, B. W., Chan, A. L., & Adams, J. (1985). Feedback complexity and corrective efficiency. Contemporary Educational Psychology, 10(3), 285–291.
Lappalainen, P. (2009). Communication as part of the engineering skills set. European Journal of Engineering Education, 34(2), 123–129.
Levinson, W., Lesser, C. S., & Epstein, R. M. (2010). Developing physician communication skills for patient-centered care. Health Affairs, 29(7), 1310–1318.
Levitt, D. H. (2001). Active listening and counselor self-efficacy: Emphasis on one micro-skill in beginning counselor training. The Clinical Supervisor, 20(2), 101–115.
Lin, L., Atkinson, R. K., Christopherson, R. M., Joseph, S. S., & Harrison, C. J. (2013). Animated agents and learning: Does the type of verbal feedback they provide matter? Computers & Education, 67, 239–249.
Louwerse, M. M., Graesser, A. C., Lu, S., & Mitchell, H. H. (2005). Social cues in animated conversational agents. Applied Cognitive Psychology, 19, 693–704.
Mason, B. J., & Bruning, R. (2001). Providing feedback in computer-based instruction: What the research tells us. University of Nebraska–Lincoln: Center for Instructional Innovation. Retrieved from http://dwb.unl.edu/Edit/MB/MasonBruning.html.
Mautone, P. D., & Mayer, R. E. (2001). Signaling as a cognitive guide in multimedia learning. Journal of Educational Psychology, 93(2), 377.
Mayer, R. E. (2001). Multimedia learning. Cambridge, UK: Cambridge University Press.
Mayer, R. E. (2005). Principles for managing essential processing in multimedia learning: Segmenting, pretraining, and modality principles. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (pp. 169–182). New York: Cambridge University Press.
Mayer, R. E. (2009). Multimedia learning (2nd ed.). New York: Cambridge University Press.
Mayer, R. E. (2014). Principles based on social cues in multimedia learning: Personalization, voice, image, and embodiment principles. In R. E. Mayer (Ed.), Cambridge handbook of multimedia learning (2nd ed., pp. 345–368). New York: Cambridge University Press.
Mayer, R. E., & DaPra, C. S. (2012). An embodiment effect in computer-based learning with animated pedagogical agents. Journal of Experimental Psychology: Applied, 18(3), 239–252.
Mayer, R. E., Dow, G. T., & Mayer, S. (2003a). Multimedia learning in an interactive self-explaining environment: What works in the design of agent-based microworlds. Journal of Educational Psychology, 95, 806–812.
Mayer, R. E., Fennell, S., Farmer, L., & Campbell, J. (2004). A personalization effect in multimedia learning: Students learn better when words are in conversational style rather than formal style. Journal of Educational Psychology, 96(2), 389–395.
Mayer, R. E., & Fiorella, L. (2014). Principles for reducing extraneous processing in multimedia learning: Coherence, signaling, redundancy, spatial contiguity, and temporal contiguity principles. In R. E. Mayer (Ed.), Cambridge handbook of multimedia learning (2nd ed., pp. 279–315). New York: Cambridge University Press.
Mayer, R. E., & Moreno, R. (2003). Nine ways to reduce cognitive load in multimedia learning. Educational Psychologist, 38, 43–52.
Mayer, R. E., Sobko, K., & Mautone, P. (2003b). Social cues in multimedia learning: Role of speaker’s voice. Journal of Educational Psychology, 95(2), 419–425.
McGaghie, W. C. (1999). Simulation in professional competence assessment: Basic considerations. In A. Tekian, C. H. McGuire, & W. C. McGaghie (Eds.), Innovative simulations for assessing professional competence (pp. 7–22). Chicago: Department of Medical Education, University of Illinois at Chicago.
Moreno, R. (2004). Decreasing cognitive load for novice students: Effects of explanatory versus corrective feedback in discovery-based multimedia. Instructional Science, 32(1–2), 99–113.
Moreno, R., & Durán, R. (2004). Do multiple representations need explanations? The role of verbal guidance and individual differences in multimedia mathematics learning. Journal of Educational Psychology, 96(3), 492.
Moreno, R., & Mayer, R. E. (1999). Multimedia-supported metaphors for meaning making in mathematics. Cognition and instruction, 17(3), 215–248.
Moreno, R., & Mayer, R. E. (2005). Role of guidance, reflection, and interactivity in an agent-based multimedia game. Journal of Educational Psychology, 97(1), 117.
Moreno, R., & Mayer, R. E. (2007). Interactive multimodal learning environments. Educational Psychology Review, 19, 309–326.
Moreno, R., Mayer, R. E., Spires, H. A., & Lester, J. C. (2001). The case for social agency in computer-based teaching: Do students learn more deeply when they interact with animated pedagogical agents? Cognition and Instruction, 19(2), 177–213.
Motowidlo, S. J., Dunnette, M. D., & Carter, G. W. (1990). An alternative selection procedure: The low-fidelity simulation. Journal of Applied Psychology, 75, 640–647.
Narciss, S., & Huth, K. (2004). How to design informative tutoring feedback for multimedia learning. In H. M. Niegemann, D. Leutner, & R. Brunken (Eds.), Instructional design for multimedia learning (pp. 181–195). Munster, NY: Waxmann.
National Academy of Engineering. (2004). The engineer of 2020: Visions of engineering in the new century. Washington, DC: The National Academies Press.
National Association of Colleges and Employers. (2012). Job Outlook 2013. Bethlehem, PA: NACE. Downloaded 12 Aug 2013. http://www.unco.edu/careers/assets/documents/NACEJobOutlookNov2013.pdf.
National Research Council. (2012). Education for life and work: Developing transferable knowledge and skills in the 21st century. Washington, DC: The National Academies Press.
Paas, F., Tuovinen, J., van Merriënboer, J. J. G., & Darabi, A. (2005). A motivational perspective on the relation between mental effort and performance: Optimizing learner involvement in instruction. Educational Technology Research and Development, 53, 25–34.
Passow, J. H. (2012). Which ABET competencies do engineering graduates find most important in their work. Journal of Engineering Education, 101, 95–118.
Perkins, K., Adams, W., Dubson, M., Finkelstein, N., Reid, S., Wieman, C., & LeMaster, R. (2006). PhET: Interactive simulations for teaching and learning physics. The Physics Teacher, 44, 18.
Phillips, B. (1999). Reformulating dispute narratives through active listening. Conflict Resolution Quarterly, 17, 161–180.
Plass, J., Homer, B. D., & Hayward, E. O. (2009). Design factors for educationally effective animations and simulations. Journal of Computing in Higher Education, 21, 31–61.
Plass, J. L., Moreno, R., & Brünken, R. (2010). Cognitive load theory. Cambridge: Cambridge University Press.
Pridemore, D. R., & Klein, J. D. (1995). Control of practice and level of feedback in computer-based instruction. Contemporary Educational Psychology, 20, 444–450.
Pulakos, E. D., & Schmitt, N. (1996). An evaluation of two strategies for reducing adverse impact and their effects on criterion-related validity. Human Performance, 9, 241–258.
Reeves, B., & Nass, C. (1996). The media equation: How people treat computers, television, and new media like real people and places. Stanford, CA: CLSI Publications.
Renkl, A. (1997). Learning from worked-out examples: A study on individual differences. Cognitive Science, 21, 1–29.
Rogers, C. R. (1951). Client-centered therapy. Boston: Houghton-Mifflin.
Ronen, M., & Eliahu, M. (2000). Simulation—A bridge between theory and reality: The case of electric circuits. Journal of Computer Assisted learning, 16(1), 14–26.
Rosenberg-Kima, R. B., Baylor, A. L., Plant, E. A., & Doerr, C. E. (2008). Interface agents as social models for female students: The effects of agent visual presence and appearance on female students’ attitudes and beliefs. Computers in Human Behavior, 24(6), 2741–2756.
Russell, S., & Norvig, P. (1994). Artificial intelligence. Egnlewood Cliffs: Prentice-Hall.
Sadler, D. R. (1989). Formative assessment and the design of instructional systems. Usability of the ALS, 18(2), 119–144.
Schmitt, N., & Chan, D. (1998). Personnel selection: A theoretical approach. Thousand Oaks, CA: Sage.
Schworm, S., & Renkl, A. (2007). Learning argumentation skills through the use of prompts for self-explaining examples. Journal of Educational Psychology, 99, 285–296.
Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153–189.
Shute, V. J., Hansen, E. G., & Almond, R. G. (2007). An assessment for learning system called ACED: Designing for learning effectiveness and accessibility. In ETS Research Report No. RR-07-26, Princeton, NJ.
Smith, S., Hanson, J. L., Tewksbury, L. R., Christy, C., Talib, N. J., Harris, M. A., et al. (2007). Teaching patient communication skills to medical students: A review of randomized controlled trials. Evaluation and the Health Professions, 30(1), 3–21.
Strobel, J., Wang, J., Weber, N. R., & Dyehouse, M. (2013). The role of authenticity in design-based learning environments: The case of engineering education. Computers & Education, 64, 143–152.
Sweller, J. (1999). Instructional design in technical areas. Melbourne: ACER Press.
Sweller, J., van Merriënboer, J. J. G., & Paas, F. (1998). Cognitive architecture and instructional design. Educational Psychology Review: General, 10, 251–296.
Trenholm, S., & Jensen, A. (2004). Interpersonal communication. Oxford, UK: Oxford University Press.
Tullis, T., & Albert, B. (2008). Measuring the user experience: collecting, analyzing, and presenting usability metrics. Burlington, MA: Morgan Kaufman.
Van der Kleij, F. M., Feskens, R. C., & Eggen, T. J. (2015). Effects of feedback in a computer-based learning environment on students’ learning outcomes. A meta-analysis. Review of Educational Research,. doi:10.3102/0034654314564881.
Van der Kleij, F. M., Timmers, C. F., & Eggen, T. J. (2011). The effectiveness of methods for providing written feedback through a computer-based assessment for learning: A systematic review. CADMO, 19(1), 21–39.
van Merriënboer, J. J. G., & Sweller, J. (2005). Cognitive load theory and complex learning: Recent developments and future directions. Educational Psychology Review, 17, 147–177.
VanLehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R., Taylor, L., & Wintersgill, M. (2005). The Andes physics tutoring system: Lessons learned. International Journal of Artificial Intelligence in Education, 15(3), 147–204.
Weger, H, Jr, Bell, G. R. C., & Emmett, M. C. (2010). Active listening in peer-interviews: The influence of message paraphrasing on perceptions of listening skill. International Journal of Listening, 24, 34–49.
Weger, H, Jr, Bell, G. R. C., Minei, M. E., & Robinson, M. C. (2015). The relative effectiveness of active listening in initial interactions. International Journal of Listening, 28, 13–31.
Wieman, C. E., Adams, W. K., & Perkins, K. K. (2008). PhET: Simulations that enhance learning. Science, 322, 682–683.
Wilkins, K., Bernstein, B. L., Bekki, J. M., Harrison., C. J., & Atkinson, R. K. (2012) Development of the science technology engineering and mathematics—active listening skills assessment (STEM-ALSA). In Proceedings of the 2012 ASEE/IEEE Frontiers in Education Conference, pp. 1218–1223.
Wouters, P., Paas, F., & van Merriënboer, J. (2008). How to optimize learning from animated models: A review of guidelines based on cognitive load. Review of Educational Research, 78, 645–675.
Acknowledgments
The work reported here was supported by the National Science Foundation (NSF) Grant 0910384. Any opinions, findings, and conclusions and recommendations expressed in this report are those of the authors and do not necessarily reflect the views of the NSF. We gratefully acknowledge the contributions of Dr. Robert Atkinson in conceptualizing this study.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Bernstein, B.L., Bekki, J.M., Wilkins, K.G. et al. Analysis of instructional support elements for an online, educational simulation on active listening for women graduate students in science and engineering. J Comput High Educ 28, 136–171 (2016). https://doi.org/10.1007/s12528-016-9110-4
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12528-016-9110-4