Introduction

Many medical schools worldwide experienced an abrupt disruption from the traditional in-person teaching format and transitioned to an online approach in response to the COVID-19 pandemic [1,2,3,4,5]. This unavoidable and sudden change in instructional delivery influenced medical students in many ways and may push medical education to integrate more virtual teaching moving forward, generating a need to better understand the effect of online education on medical student educational outcomes.

While in-person instruction with lectures and workshops is the standard method of curriculum delivery for most medical schools, there is significant evidence to support the effectiveness of online learning as well. Many medical students view online learning positively and prefer the opportunity to grasp material at their own pace [6,7,8]. Students also perceive online learning to be an “enjoyable and effective” method to improve consultation skills [6, 7]. Educational outcomes, however, have not been shown to be significantly different when comparing hybrid instruction—defined as a combination of in-person instruction and online supplementation—with solely in-person instruction. While medical students view supplemental online learning as an excellent resource for their education, they do not consider it a replacement for traditional in-person instruction [6, 8]. Further studies on nursing and medical students have shown that while course satisfaction is consistently higher with hybrid instruction compared to traditional in-person instruction, there is often no significant difference between the two approaches on test scores [9,10,11]. Finally, hybrid learning has also been shown to be at least as effective or more effective with the integration of online instruction when compared to traditional in-person instruction alone [12].

While there are many studies comparing the educational outcomes of traditional in-person instruction to a hybrid approach, solely online instruction has not been studied because medical schools only began this type of instruction in response to the COVID-19 pandemic. Systematic review of multiple studies comparing online-only and traditional in-person instructions for teaching physical examination to healthcare students has shown no significant advantage of one over the other [13]. A study by Subramanian et al. evaluated the effect of online web-based modules in comparison with traditional didactic lectures on exam scores and retention 22 days later in 30 third-year medical students. They found no significant difference in pre-intervention scores but found significantly higher post-intervention scores and retention in the web-based group compared to the traditional didactic group [14]. Finally, a study conducted by Bettinger et al. collected data on the educational outcomes of undergraduate students at a large university for over 4 years of operation including over 230,000 students enrolled in more than 750 courses. Students usually take on average two-thirds of their classes online and one-third of their classes in person [15]. Online and in-person classes differed only by delivery format. The authors found that taking a course online reduces student grades by about one-third of a standard deviation, reduces student grades in future courses by one-eighth of a standard deviation, and reduces the probability of remaining enrolled a year later by over ten percentage points [15].

Beyond learning, online instruction may also impact medical student mental health. The transition from the school setting to home results in isolation, an increased use of email, and struggles with establishing boundaries between work and home, all of which affect the mental wellbeing of medical students [16]. Due to unique stigma surrounding mental health, medical students are already a vulnerable group regarding mental wellbeing compared to the general population, with higher reports of depression, suicidal ideation, and untreated mental illness [17]. In a global study of the mental health of medical students from 12 different countries, medical students were found to have high rates of mental health concerns, stress, burnout, and substance abuse [18].

Other factors specifically associated with the COVID-19 pandemic that may impact medical student mental wellbeing include disturbances caused by family affairs, fear of new viral strains, social isolation, increased media attention, and ambiguity about the state and outlook of the pandemic [19, 20]. COVID-19 social distancing mandates may exacerbate feelings of isolation in medical students and thus contribute to an overall poorer mental wellbeing in addition to pre-existing feelings of isolation given rigorous individual study in the preclinical years [16]. In a study of 234 medical students during COVID-19, both female and male medical students reported feelings of emotional detachment from friends, family, and fellow classmates and a decrease in overall work performance and study period. About a quarter of students also report feelings of disheartenment during quarantine [21].

In contrast to the reports above, a study by Bolatov et al. of medical students at Astana Medical University in Kazakhstan found improved mental health after transitioning from in-person to online learning in response to COVID-19. The study authors believed that the online-only curriculum gave students more independence and a more flexible learning schedule which may have positively impacted their mental health. Bolatov et al. also report the most pronounced symptoms of depression and anxiety were found amongst the students who experienced a decline in academic performance with the transition to the online-only format [22].

Up until the COVID-19 pandemic, online teaching methods in medical schools have been considered as a supplement to the in-person instruction. With the onset of the COVID-19 pandemic, medical schools were required to limit in-person learning sessions and transition to virtual teaching. While online-only instruction may have been the safest and only option at the time, many believe that this transition to online learning will transform medical education to integrate more virtual teaching in the future [3, 5, 6, 23]. Therefore, studying the effect of online instruction on medical education is vital to facilitate evidence-based decisions to implement online instruction in medical schools.

In the fall of 2020, the Lewis Katz School of Medicine (LKSOM) gave second-year preclinical medical students the option to attend case-based workshops either online or in person, and transitioned lectures online for all students. This transition created a natural experiment where a single class of students were taught the same course material via the same instructors with two different formats for course delivery: online only (online workshops and online lectures) versus hybrid (in-person workshops and online lectures). We examined this group of preclinical medical students to determine if there was a difference between the online-only and hybrid groups in educational performance as well as mental wellbeing. Parameters measured included grades, mental wellbeing, and subjective perception of education quality. Finally, we were also interested in categorizing the qualitative challenges faced by medical students in the COVID-19 pandemic.

Methods

Setting and Participants

The study took place at LKSOM. All second-year preclinical medical students in the class of 2023 were invited to participate in the survey. The survey was introduced by email as well as verbally at a lecture opening. Participation was voluntary, and the survey was taken anonymously online to encourage responses and honest reporting. The survey was distributed via email first at the end of the cardiology, pulmonary, and renal curriculum block (block 8) and then distributed again following a verbal announcement to the class. Participants were explained the purpose of the study but were not informed about the hypotheses.

The cardiology, pulmonary, and renal curriculum block (block 8) at LKSOM is taught via lectures and case-based workshops. All lectures were held live online for both the online-only and hybrid learning groups with faculty available to answer questions virtually. Lectures were also recorded so students could watch and review the lectures at their convenience. Case-based workshops were held two to three times a week. Students were divided into groups of five to six students to prepare the cases for workshop; the cases were identical for both the online-only and hybrid groups. Online-only students were grouped with other online-only students and met virtually, while hybrid students were grouped together and met in person. Faculty were available during the entire workshop both online and in-person to assist students with questions and clarify major learning objectives. Case-based workshops were student-led, and small groups from both the online-only and hybrid groups were selected to teach cases to the entire class (both online and hybrid) via live online broadcasting. Case-based workshops were not recorded to encourage students to engage and interact during the sessions.

Data Collection

We designed a descriptive cross-sectional study to assess the effect of online-only and hybrid learning on students, measured via an anonymous 24-question online survey: one question to generate a unique identifier, four questions to collect demographic information (i.e., gender, living situation), four questions related to online-only vs. hybrid group inclusion, five questions to assess grades, two questions to measure mental wellbeing, five questions to measure course satisfaction and interest, and one question each to assess social support, faculty support, and classmate support. Optional open-ended questions were provided throughout the survey to give students the option to elaborate on their responses. The survey was developed in collaboration by the study authors and reviewed through multiple rounds of revisions by our group to ensure survey questions addressed our study aims and used simple unbiased language. The survey was validated by face validation from faculty at the LKSOM (senior associate dean of education and associate dean for student affairs and medical humanities). A copy of the survey and responses is provided in Online Resource 1.

Self-reported course grades during the cardiology, pulmonology, and renal course block (block 8) were collected as a numerical score ranging from 0 to 100. The final course grade of block 8 is composed of two mid-course exams (25% and 35% of the final grade) and a final exam (40% of the final grade). All students from the online-only and hybrid groups took the same multiple-choice exams at the same time proctored in-person at the medical school. Self-reported mental wellbeing scores were collected with a single question to rate mental wellbeing during block 8 via a five-point Likert scale (from 1—unwell to 5—well) followed by a free-response question to allow for further commentary [24]. Additional data was collected to further investigate the potential impact of various factors on student wellbeing. These additional factors included social support (i.e., friends, family; from 1—unsupported to 5—supported), faculty support (from 1—unsupported to 5—supported), and classmate support (from 1—disconnected to 5—connected) which were also measured via individual questions by a five-point Likert scale [24]. Using the five-point Likert scale, we measured student satisfaction with the course (from 1—unsatisfied to 5—satisfied), feeling distracted or focused throughout the course (from 1—distracted to 5—focused), and general interest in course material (from 1—not interested at all to 5—very interested) through a question for each factor [24].

Data Analysis

Descriptive statistics were calculated for final course grades for the online-only group, the hybrid group, and the cohort as a whole. An unpaired t test was used to compare grades, mental wellbeing scores, and general satisfaction scores between the online-only group and hybrid groups [25]. In addition, to investigate the impact of various factors on mental wellbeing, Pearson correlation was calculated between mental wellbeing scores and quantitative factors such as course grade, course satisfaction, social support, faculty support, classmate support, and course interest. A comparison of categorical variables such as course mode and gender was performed using a chi-square test. The level of statistical significance was determined as a p value ≤ 0.05. All analyses were completed using R version 4.1.0. Post hoc power analysis was performed with G*Power 3.1.9.7 and is discussed in Online Resource 2 [26, 27].

Results

Of the 198 total students, we collected 61 responses (42.6% [N = 26] male, 55.7% [N = 34] female, 1.6% [N = 1] preferred not to specify). There was no significant difference in gender between the online-only and hybrid groups (p = 0.1801). 49.2% (N = 30) of the class opted into the online-only group, and the remaining 50.8% (N = 31) were in the hybrid group. Reasons for choosing the online-only instruction included convenience, safety, ease of focusing from home, outside obligations (i.e., childcare, family, work), and saving money. Reasons for choosing hybrid instruction included preference for asking questions in person and social engagement with classmates and faculty.

Final Course Grade

The overall mean final course grade of the cohort as a whole was 84.7 (SD = 6.88) with a range of 67.0 to 97.0. The online-only group had a significantly lower mean final course grade (mean: 83.0, SD = 6.71, range: 67.0–95.8) compared to the hybrid group (mean: 86.6, SD = 6.68, range: 70.0–97.0, p = 0.04293). Final course grade results are summarized in Fig. 1.

Fig. 1
figure 1

Final course grades in the online-only and hybrid learning groups. 

Full size image

In comparing the online-only and hybrid learning groups, there was a significantly lower mean final course grade in the online-only group (mean: 83.0, SD = 6.71, range: 67.0–95.8) compared to the hybrid group (mean: 86.6, SD = 6.68, range: 70.0–97.0, p = 0.04293)

Because our survey assessed differences in final course grades between the online-only and hybrid learning groups cross-sectionally, we were limited in assessing whether the difference in student grades was because of the method of instruction or due to sampling bias (i.e., individual aptitude for the particular course content or baseline difference in educational performance between individuals in the groups). For this reason, we analyzed whether students’ perceived grades had changed since transitioning to the online-only or hybrid model. Self-reports of whether the student’s grade had changed after moving from the traditional learning method to the online-only or hybrid model did not show a significant difference between the online-only and hybrid groups (p = 0.7101). Most students report their grades have not changed or are now lower since moving to the hybrid or online-only teaching model.

Mental Wellbeing

There was no significant difference in mental wellbeing scores between the online-only and hybrid groups (p = 0.6858). Moreover, there was no significant difference in social support (p = 0.1203), faculty support (p = 0.1888), and classmate support (p = 0.6889) between groups. Students reported the following reasons for poorer mental wellbeing: news stories (i.e., election results, police violence), pressure to be in person, USMLE Step 1 stress, social isolation, rigorous coursework, lack of guidance and support, finances, COVID-19-related shutdowns, family, lack of time and sleep, COVID-19 stress with the state of the pandemic, infecting loved ones, distractions with the home/work environment, difficulty accessing routine medical care for chronic and new medical conditions, and hospitalization.

While there were no significant group differences in mental wellbeing, social support, faculty support, or classmate support between the online-only and hybrid learning groups, as a post hoc analysis, the sample was stratified by their mode of instruction (online only and hybrid), and the relationships between mental wellbeing and factors such as grades, course satisfaction, social support, faculty support, class support, and course interest were investigated using the Pearson correlation test (Table 1). In the online-only group, there was a significant association between mental wellbeing and both faculty support and classmate support. In contrast, in the hybrid group, there was a significant association between mental wellbeing and faculty support but not with classmate support (Fig. 2). In the online group, there was also a significant association between mental wellbeing and grades, course satisfaction, social support, and course interest. In the hybrid group, there was a significant association between mental wellbeing and course satisfaction, and social support. Mental wellbeing was not significantly associated with final course grades or course interest.

Table 1 Mental wellbeing associations in the online-only and hybrid groups
Full size table
Fig. 2
figure 2

Association between mental wellbeing and faculty support or classmate support in the online-only vs. hybrid learning groups.

Full size image

In the online-only group (shown in blue), there was a significant association (p value < 0.05) between mental wellbeing and both faculty support (r = 0.475) and classmate support (r = 0.494). In contrast, in the hybrid group (shown in red), there was a significant association between mental wellbeing and faculty support (r = 0.485) but not with classmate support (r = 0.104)

Pearson correlation (r) is given between mental wellbeing scores and factors such as final course grade, course satisfaction, social support, faculty support, classmate support, and course interest. In the online-only group, there was a significant association (p value < 0.05) between mental wellbeing and grades, course satisfaction, social support, faculty support, classmate support, and course interest. In contrast, in the hybrid group, there was only a significant association between mental wellbeing and course satisfaction, social support, and faculty support.

Course Satisfaction

There was no significant difference in measures of course satisfaction (p = 0.9332) or course interest (p = 0.5214) between the online-only and hybrid groups.

Change in Mode of Instruction

Four students were found to have changed their mode of instruction from hybrid to online only from the previous curriculum block (block 7) to the current surveyed block (block 8). No students changed from the online-only to hybrid curriculum. All four students reported a decrease in their final course grades between the previous hybrid curriculum block to the online-only curriculum block. Reasons for switching to the online-only curriculum included convenience, difficulty viewing overhead monitors with glasses, and difficulty understanding in-person presenters wearing masks.

Discussion

The aim of the study is to investigate the effect of online-only vs. hybrid instructional delivery on medical students’ education as well as mental wellbeing. In addition, we were interested in medical student attitudes regarding the different teaching modalities and the various challenges faced by medical students in the setting of the COVID-19 pandemic.

Our group found a significant difference in grades between the hybrid and online-only groups with no significant difference in course satisfaction or course interest. This suggests online-only instruction may be less effective than hybrid instruction in academic performance when the only difference in curriculum is course delivery. This is consistent with the findings from a study by Bettinger et al. of over 230,000 undergraduate students in which taking a course online reduced student grades by about one-third of a standard deviation [15]. Given the only difference in curriculum between the groups was being in person or online for the case-based workshops, one explanation for this finding may be a deficiency of faculty support in the online-only group.

Mental wellbeing also differed between the two groups. While mental wellbeing was not significantly associated with classmate support in the hybrid group, mental wellbeing was significantly associated with faculty support. In contrast, mental wellbeing was significantly associated with both classmate support and faculty support in the online-only group (Fig. 2). We speculate that greater faculty support in the hybrid group may negate the necessity for support from classmates while in the online-only group, there may have been a higher reliance on classmates rather than faculty because of their physical disconnection from the school. This suggests online-only instruction may produce a barrier to student and faculty relationships. This faculty support may play a role in both supporting student mental health and contributing to the observed higher final course grades in the hybrid group compared to the online-only group. In-person workshops also may have facilitated discussion and higher-level engagement with course material in the hybrid group as well.

While our results suggest online teaching is not as effective as hybrid teaching, the online-only curriculum was created in reaction to the pandemic, so it was essentially a direct translation of the traditional didactic model broadcasted online. Therefore, it was not designed intentionally for online teaching and lacked many of the advantages that online teaching can offer such as simulation and application [28]. In fact, Subramanian et al. found that students performed better on post-intervention test scores and retention 22 days later when participating in web-based learning modules compared to the traditional lecture-based course delivery []. If the online curriculum had been designed for virtual teaching, perhaps we may have found different results. However, if the only difference in curriculum is instructional delivery, our results suggest some level of in-person instruction is more effective than a strictly online course.

Only four students in our study switched teaching modes, and all four moved from the hybrid model to the online-only model. All four students report a decrease in their grades after the switch, supporting the notion that the hybrid model may be more effective than the online-only model. Conversely, since only four students switched, the hybrid model may just be a better fit for their learning styles and the reason for switching from the hybrid to the online-only group may confound their grade outcome.

Our group found no significant difference in mental wellbeing ratings between the online-only and hybrid group. This suggests that the delivery of instruction did not play a significant role in the students’ overall wellbeing. Students report many reasons for poor mental wellbeing, and while some of these factors (such as lack of guidance and support, distracting home and work environments, and isolation) could be less severe in the hybrid group than the online group, many of the other reported factors affecting mental wellbeing (such as media reports, USMLE Step 1 stress, rigorous coursework, finances, family, lack of time and sleep, and COVID-19 related stressors) are independent of course delivery, which could explain why there was no significant difference in mental wellbeing between the online-only and hybrid groups. COVID-19 may also have impacted mental wellbeing by adding heightened feelings of isolation on top of the already high pressure of medical training, further interfering with interpersonal relationships with classmates and professors [29]. Unique stressors experienced by medical students during the COVID-19 pandemic in our study include COVID-19 anxiety and impact on loved ones, decreased access to routine medical care for chronic and new medical conditions, and hospitalizations. These examples highlight the extraordinary challenges faced by medical students during this pandemic.

Nationwide, medical schools have been forced to switch to online teaching with the onset of COVID-19, and because of this shift, there has been increased interest in understanding the effectiveness of online education and increased consideration for its future integration in didactic teaching. Previous studies have demonstrated improved knowledge retention for students in a hybrid teaching method over traditional in-person teaching methods; however, when comparing online learning to traditional learning, there are mixed reports of either no significant difference or improved performance [14]. It’s unclear from our study how online-only instruction alone compares to traditional in-person instruction, but—consistent with previous findings—our study indicates that the hybrid model seems to yield better retention and teaching of course material compared to the online-only model [15].

While the online-only model may have been the safest—and only—option for some medical schools [6, 23], we advise caution in force-fitting a traditional didactic curriculum of lectures and case-based workshops into an online-only format. While medical students may favor the online-only option because of the additional flexibility and convenience, this model may have lower educational outcomes—which may also disproportionately impact students of lower socioeconomic status [31, 32]. Some unique barriers associated with the online-only model of instruction include a secure Wi-Fi connection, continual course access, a safe and suitable learning environment, access to supplemental resources such as board exam preparatory courses, and reliable electronic learning tools. Based on our findings, we recommend a hybrid in-person and online method of instruction over a strict online translation of traditional didactic teaching as an effective method of content delivery for future medical education.

Limitations

Our study has a relatively small sample size (N = 61) and power (Online Resource 2). Therefore, there is a potential for sampling bias where participants in online-only and hybrid groups are not equivalent for comparing the study parameters, especially given self-selection for the teaching groups. Using a larger sample size and randomization of students into course delivery groups could minimize this effect. Moreover, the participants of our study included second-year medical students from a single institution. This situation provided a unique opportunity to perform a relatively controlled investigation (i.e., identical course content, instructors), but this small, localized setting may limit the generalizability and transferability of findings to other institutions. Finally, the self-reporting nature of the survey lends itself to subjective differences of the respondents such as individual differences in assessing experiences, recalling memory, prejudice, and misinformation. Future investigations with a larger randomized sample with the inclusion of broad student populations may be needed to confirm our findings.

Conclusion

When comparing the online-only instruction with the hybrid instruction, students who chose hybrid instruction had better educational outcomes. Mental wellbeing was not significantly associated with the online-only or hybrid delivery; however, students reported considerable impact on mental wellbeing from COVID-19-related stress.