Background

Social restrictions and health concerns tied to the COVID-19 pandemic forced medical schools around the world to shut down classrooms and use existing online communication tools, including multimedia resources and virtual platforms, to deliver their curricula online [1, 2]. In the immediate aftermath of this transition, much of the discussion about the potential impact of COVID-19 on medical education and on students’ experiences or academic performance occurred primarily in editorials or opinion articles that provided little or no supporting data [1, 3]. For example, anecdotal evidence indicated that learners experience increased fatigue and burnout from multiple hours on virtual platforms, colloquially termed “Zoom fatigue” in popular media [4, 5]. More recently, qualitative data has emerged that supports this idea that the sudden transition to remote learning coincides with feelings among medical students of exhaustion, social isolation and disengagement, and difficulty focusing, particularly with respect to synchronous, small-group sessions [6,7,8].

Despite these new reports, there remains scant quantitative evidence from the literature demonstrating the impact of remote learning on student performance in medical education. Here we examined the impact of transitioning to remote learning on preclinical medical student performance and learning outcomes by comparing summative assessment data for a medical neuroscience course before and after transitioning to remote learning. To help conceptualize the nature of the problem and frame the interpretation of data, we used the Community of Inquiry (CoI) theoretical framework, which represents a process of creating a deep and meaningful (collaborative-constructivist) online learning experience through the development of three interdependent elements: social, cognitive, and teaching presence [9]. The CoI framework has been used to evaluate the strengths and weaknesses in online learning and can predict student performance and satisfaction among novice online learners [10].

Activity

We evaluated the potential impact of transitioning to remote learning on exam performance among second year preclinical medical students at the University of Illinois College of Medicine Peoria (UICOMP) by comparing summative assessment data for a medical neuroscience course over a 2-year period (2019 and 2020), which corresponds to pre- and post-transition to remote learning, respectively. This study was part of a quality improvement project approved by the UICOMP Institutional Review Board (IRB#1148244–1).

UICOMP is a regional campus with an average class size of 65 students. The preclinical curriculum is systems-based, spanning the first 2 years (phase 1) of a 4-year medical education. The preclinical curriculum focuses on teaching core foundational sciences using an organ system–based organizational structure. Multiple interactive learning formats are used throughout phase 1, including case-based learning, team-based learning, and flipped classroom, in addition to lectures and asynchronous independent learning modules.

In response to the COVID-19 pandemic, UICOMP suspended all in-person sessions in March of 2020, transitioning to an entirely online delivery of preclinical curriculum. One course affected by this transition was the second-year, 10-week medical neuroscience course, which began in August of 2020. This is an integrated, multi-disciplinary course, which covers both neuroscience and behavioral sciences. In 2020, all interactive sessions (47% of total content) occurred synchronously over the Zoom™ virtual platform. In contrast, all interactive synchronous sessions in 2019 were conducted in-person (Table 1). Because our analysis focused on interactive synchronous sessions, we excluded lectures from 2019 with limited or no learner interactivity from the synchronous data from that year.

Table 1 Summary of UICOMP medical neuroscience preclinical curriculum before and after the transition to remote learning. Approximate proportions of total coursework delivered in asynchronous and synchronous formats are indicated by percentage
Full size table

A summative assessment (final exam) of ~110 items was administered at the conclusion of the course. The final exam represented 60% of a student’s final course grade and was non-compensatory, meaning students needed to pass the exam to pass the course. Only assessment items with an item discrimination value ≥ 0.2 were included in the analysis, which is the recommended threshold that ensures the measurement objective of the exam is attained [11]. Those items that met the criteria (approximately 30% and 40% of the 2019 and 2020 exams, respectively) were categorized by delivery modality and analyzed using Student’s t test to compare mean scores from 2019 to 2020. Based on the current multifaceted definition of construct validity as described in the Standards for Educational and Psychological Testing [11, 12], content (examination blueprint aligned to course learning objectives, delivery modality and USMLE topics); internal structure (item difficulty, discrimination, and test score reliability), and consequences (passing standards for the students) were all considered as part of the theoretical validity testing framework.

Results

To evaluate the potential impact of remote learning on student academic performance, we compared data from the 2019 and 2020 medical neuroscience course final exams. We found that overall assessment scores (which included all items, not only those that met study criteria) did not change significantly between years. However, when items were grouped by delivery modality, items which assessed content that was delivered via interactive, synchronous sessions had mean scores in 2020 (mean = 0.700, SEM ± 0.039, n = 22) that were significantly lower than 2019 (mean = 0.831, SEM ± 0.034, n = 24) by 13 points, t(44) = 2.55, p = 0.014. Performance on items assessing content delivered asynchronously between 2019 (mean = 0.673, SEM ± 0.055, n = 11) and 2020 (mean = 0.719, SEM ± 0.041, n = 25) showed no significant change t(34) = 0.64 p = ns (Fig. 1).

Fig. 1
figure 1

Comparison of student performance on medical neuroscience course summative final exam. Exam items with an item discrimination value of 0.2 or greater were categorized according to content delivery method (synchronous interactive or asynchronous) and analyzed using Student’s t test to compare mean scores from 2019 (in-person) synchronous (n = 24) and asynchronous (n = 11) and 2020 (remote) synchronous (n = 22) and asynchronous (n = 25). Each bar of the histogram corresponds to mean values (± SEM). *p < 0.05

Full size image

Furthermore, we analyzed exam performance from the preceding course according to the same methods. This 7-week course, which covers gastrointestinal and renal systems, occurs at the end of the first year of medical school and precedes the summer recess. Importantly, in 2020, this was the first course offered entirely online. The mode of delivery of the gastrointestinal/renal course was comparable to the neuroscience course, as shown in Table 1. In 2019, the course was 15% asynchronous and 85% synchronous, while in 2020 after transition to remote learning, the course was 50% asynchronous and 50% synchronous. Interestingly, performance on exam items assessing content delivered via interactive, synchronous sessions in the gastrointestinal/renal course showed no appreciable difference between years (t(80) = 0.85, p = ns), suggesting that the decline in performance for synchronous remote learning sessions was not immediately apparent following the transition to remote learning.

Discussion

Our findings suggest that after approximately 24 weeks of remote delivery of preclinical curriculum, summative assessment performance significantly declined for items linked to material taught in an interactive, synchronous format. Moreover, exam scores from a preceding course, which occurred at the outset of the COVID-19 pandemic and transition to remote learning, showed no equivalent decline in performance. These data contribute to a relatively new but growing body of work examining the impact of prolonged periods of exposure to synchronous activity via video conferencing technology on medical student academic performance.

The now widespread use of technology-enabled virtual delivery of synchronous content during the COVID-19 pandemic has been characterized as a potential logical extension of active learning in the classroom [13]. Medical education places a premium on synchronous, interactive learning, which allows students to participate in real-time discussions with peers and instructors. However, synchronous learning via virtual platforms changes the nature of both peer-peer interaction as well as learner-instructor interaction, severely limiting the face-to-face elements of communication and potentially disrupting the teaching and social presence elements of the CoI framework [9]. Additionally, mental exhaustion, social disengagement, and an inability to concentrate are all documented manifestations of prolonged exposure to synchronous content delivered using virtual platforms and have the potential to impact student performance and experiences [5,6,7, 14]. Our findings show that, for learners who transitioned from in-person medical education to remote delivery, the efficacy of synchronous learning delivered using virtual platforms potentially decreases over time.

In examining data in aggregate, we were limited in the ability to control for extenuating conditions that might have impacted student performance. For example, technology and connectivity issues, as well as faculty experience and comfort levels with remote teaching, can lead to variation in effectiveness of remote instruction [15, 16]. While UICOMP provided faculty and staff training opportunities and technical support throughout the pandemic, it is possible that not all course faculty and staff were completely prepared for the challenges of teaching synchronously online. Heightened anxiety among medical students over the economy and public health, and having to adapt to a new remote learning environment, can also impact students’ experiences with remote learning [6, 8, 17]. Finally, we acknowledge the scope of this study (one course, one institution, one metric of academic performance) as a limitation of the data reported here. Further research is needed to determine if these findings remain consistent over longer durations of remote learning and over additional internal and external metrics of academic performance.

Looking ahead, it will also be important to further examine whether remote delivery of preclinical curriculum disproportionately challenges certain subsets of students and whether factors such as age, socioeconomic status, and learning style correlate with successful learning outcomes following remote delivery [18]. While the data reported here do not parse out other aspects of student experience, for example, behavioral and psychological effects attributed to remote synchronous learning, these results do support the idea that sustained remote delivery of medical school preclinical curriculum can have a measurable impact on student learning outcomes.