Introduction

On March 17, 2020, the Association of American Medical Colleges (AAMC) published guidelines recommending the immediate removal of medical students from clinical settings, citing the safety of both patients and students in the midst of the COVID-19 pandemic [1]. On that date, the University of California San Diego School of Medicine (UCSD) suspended student involvement in all on-site clinical activities, necessitating the sudden and complete transition to remote learning with 8 weeks remaining in the academic year. This unprecedented change impacted the experience of medical students enrolled in core clinical clerkships.

Whereas clinical clerkships center around in-person patient care, the immediate transition to off-site learning in response to COVID-19 demanded that clerkship directors rapidly develop remote learning curricula to best meet the objectives of each clerkship. UCSD clerkships bolstered existing online case modules with additional content and new learning platforms. Didactic teaching and small group sessions were transitioned to online formats and often increased in frequency. Clerkships incorporated various additional components such as remote progress checks, quizzes, telehealth observation or participation, journaling reflections, or weekly hour logs (Supplemental Digital Appendix 1). Given the unprecedented nature of the remote curriculum and evolving requirements, the grading policy was made pass/fail in accordance with Liaison Committee on Medical Education guidelines [2].

This abrupt transition to remote learning presented an opportunity to evaluate the relative merits of remote learning for acquiring fundamental clinical knowledge and skills. In this study, we sought to describe the student experience with remote learning and to better understand the resources, structure, and content that best facilitate learning in this setting. This information may provide valuable insight for medical educators as they continue to navigate clinical learning during the COVID-19 pandemic.

Methods

Study Population

This study was conducted at the UCSD School of Medicine, an accredited allopathic medical school located in La Jolla, CA. Eligible participants included all medical students enrolled in third-year clinical clerkship rotations as of March 30, 2020. These clerkships included medicine, neurology, reproductive medicine, pediatrics, psychiatry, and surgery. The UCSD Institutional Review Board (IRB) approved this protocol as a quality improvement study.

Survey Design and Implementation

The full survey instrument is available online (Supplemental Digital Appendix 2). Students identified which clerkship they were on from March 30, 2020 to April 10, 2020, and branching logic then prompted the students to assess only the remote learning curriculum of that clerkship. The remaining survey items were uniform across all participants irrespective of clerkship. Students were asked to assess such aspects as resources, workload, structure, and communication. Students were also asked about their learning environment, social connectedness, and preparation for the future. Five open-ended items invited students to provide perspectives on telehealth experiences, strengths and gaps in the remote curriculum, and overall reflections about the remote learning experience.

The survey was administered to all eligible participants (n = 132) in April 2020 via an online survey administration software platform (Qualtrics, Provo, UT). Survey completion required approximately 15 min. The survey was anonymous, optional, and not linked to student evaluations.

Statistical Analyses

Descriptive statistics were generated using counts/frequencies. Statistical analyses were conducted in Microsoft Excel, version 15.30 (Microsoft Corporation, Redmond, WA). A total of 296 unique narrative responses to five open-ended items were analyzed and coded for thematic content by two independent graders (CC and BM), who used their content expertise to identify preliminary concepts. They iteratively reviewed comments to identify concepts and map each comment to thematic domains. Four domains were identified, with perfect correspondence between the two graders. These domains were also reviewed and approved by all co-authors. Any discrepancies in mapping of individual comments were reviewed by all co-authors, and final mapping was determined by majority consensus. Representative comments were extracted for illustration.

Results

Of 132 students invited to participate in the survey, 96 (73%) responded, with robust response rates across individual clerkships (64–85%).

Remote Learning Resources

All third-year clerkships offered synchronous case conferences or journal clubs, online/asynchronous case simulations (such as Aquifer; Lebanon, NH, USA), and online/asynchronous question banks (such as UWorld; Dallas, TX, USA). More than 80% of students in each clerkship used these resources (Table 1). There was variation in the other types of remote learning resources provided and the relative rates of utilization by students (Table 1). Rates of utilization were generally lower for more “passive” resources such as online videos/podcasts and online textbooks.

Table 1 Student utilization of remote learning resources in core clerkships among third-year medical students at the University of California San Diego School of Medicine, March–April 2020. Values indicate the number (percentage) of students in each clerkship who self-reported using a remote learning resource in each category. Clerkships often offered multiple resources per category; numbers reported in the table reflect those corresponding to the most frequently used resource in each category. “N/A” indicates that the clerkship did not offer any resources in that category
Full size table

More than 50% of students who used each remote learning resource agreed or strongly agreed that the resource was valuable (Fig. 1). The highest-rated resources were online question banks and evaluations/progress reviews, for which all students who used these resources either agreed or strongly agreed that they were valuable. Similarly, 92% of students who participated in telehealth experiences agreed or strongly agreed that they were valuable. Online textbooks had the lowest ratings, with only 50% (13/26) of students agreeing that they were useful, with the remaining half about equally divided between neutrality (6/26, 23%) and disagreement that they were valuable (7/26, 27%). When prompted to indicate any additional resources used, students listed a range of instructional videos, question banks, online flashcards, and various review texts.

Fig. 1
figure 1

Student perceptions regarding the value of various remote learning resources on core clinical clerkships, March–April 2020. Students were asked to rate their level of agreement with whether a specific remote learning resource was valuable for their medical education

Full size image

About three-quarters of respondents (69/94, 73%) felt that the number of remote learning resources incorporated in the clerkships was “about right.” About one-fifth (21/94, 22%) felt that there were too many resources, and only several (4/94, 4%) felt that there were too few. No students felt that there were “far too few” or “far too many” resources among any clerkship (Supplemental Digital Appendix 3).

Quantity and Structure of Remote Learning Curriculum

There was substantial variation in how much time students dedicated to remote learning during clerkships. More than a quarter of students (26/94, 28%) indicated spending more than 30 hours per week on remote learning (Fig. 2). The majority of students (77/93, 83%) felt that the quantity of required content was appropriate. One student (1/93, 1%) felt that there was too little content, and 15 students (15/93, 16%) felt that there was “too much” or “far too much” content (Supplemental Digital Appendix 3).

Fig. 2
figure 2

Distribution of hours spent engaged in the core clinical clerkships remote learning curricula, March–April 2020. Students were asked to respond to the question, “Approximately how many hours per week do you spend in total on clerkship engagement and studying?”

Full size image

Most students (81/94, 86%) felt that the degree of structure was appropriate. No students thought that the structure of required content was “far too rigid” or “far too flexible” (Supplemental Digital Appendix 3). The majority of students (79/94, 84%) felt that pass/fail was the fairest grading system for remote learning during this period.

Preparedness for Subsequent Stages of Training

Only about a quarter of students (25/91, 27%) felt prepared for the NBME shelf exam, though 40% (36/91) endorsed feeling “somewhat prepared” (Supplemental Digital Appendix 3). In contrast, over three-quarters of students felt either prepared (52%) or somewhat prepared (26%) for their final year of medical school despite the academic disruption (Supplemental Digital Appendix 3).

Living Environments and Connectedness

Third-year students generally reported sufficient access to technology and Internet connectivity for the purposes of remote learning. A minority (11/94, 12%) reported inadequate quiet study space, but most students agreed or strongly agreed (75/94, 80%) that their living arrangements were conducive to remote learning (Supplemental Digital Appendix 4). Most students (62%) agreed or strongly agreed that they still felt connected to the School of Medicine, and 57% still felt connected to classmates (Supplemental Digital Appendix 4).

Narrative Responses

  1. I.

    Structure

Many respondents reported favorable experiences with various aspects of curriculum flexibility. Nearly half (35/73, 48%) indicated that increased flexibility was the best aspect of remote studies (Table 2). Most students cited the ability to work at their own pace (Table 3, a) or to utilize resources best suited to their individual learning styles. Flexible scheduling allowed students to study during periods of the day they found most productive (b) and limited conflicts with other demands on students’ time. Students also noted that increased flexibility resulted in improved self-care, wellness, and life balance (c).

Table 2 Free-response themes: best and worst components of curricula, and aspects to continue in future curricula
Full size table
Table 3 Student quotations illustrating thematic content of free-response questions
Full size table

In contrast, some respondents cited challenges with lack of structure in the remote curriculum, including difficulties with time management and maintaining motivation to study (d, e). Many students desired outlines and timelines to guide study efforts. Students expressed mixed sentiments regarding the efficiency of remote learning. While many felt that self-paced learning and the elimination of commutes increased efficiency, others felt that distractions in the home environment or certain mandatory components (e.g., lengthy lectures) made their learning less efficient.

  1. II.

    Engagement and Isolation

Students identified decline in productive learning associated with prolonged online sessions, particularly with passive learning formats (f). Respondents frequently lauded interactive components as a means to combat digital fatigue and increase engagement (g, h). Students particularly valued opportunities to discuss real patient cases via virtual rounding, “chalk talks,” or telehealth (i, j). Online modules without active learning components were viewed less favorably. Despite concerns about isolation, some students nonetheless felt that the ability to participate remotely in certain learning activities was a distinct advantage, due to either increased availability of didactic opportunities (k) or the inherent convenience (l). Students appreciated expanded access to a variety of online learning resources made possible by institutional subscriptions, and many voiced hope that this access would be maintained in the future (m).

  1. III.

    Clinical Experience

A majority of respondents (38/63, 60%) identified loss of clinical experience as the biggest gap in remote clerkship curricula. Many students reflected that though some basic medical knowledge may have actually improved during this period, the loss of clinical experience was far more detrimental to gaining practical skills necessary to be good physicians (n–p). Students highly valued opportunities to participate in clinical activities through a variety of remote formats and cited both the educational value and the satisfaction of contributing to patient care (q, r). Those who participated in telehealth generally found the experience to provide good clinical learning and/or personal fulfillment (s). Some students did not miss the clinical experience nearly as much and were grateful for the additional study time afforded by its absence (t).

  1. IV.

    Anxiety

Anxiety and uncertainty were reflected in a variety of responses. General anxiety regarding the global pandemic or specific concerns about individual or family well-being pervaded the educational experience for many students (u, v). Uncertainty regarding the effects of academic disruption on plans for the final year of medical school and preparation for residency were also sources of anxiety (w). Disruption of normal routines and the additional stresses of stay-at-home orders made remote learning particularly challenging for some (x). Good communication between medical educators and students was felt by many students to be critical in maintaining an effective educational environment amidst the chaos (y, z).

Discussion

Emergence of the COVID-19 pandemic in the USA demanded that medical schools transition to remote formats within a matter of days [3]. Core clerkship directors were faced with the challenge of implementing novel means to teach clinical medicine to students who were unable to set foot on school campuses or hospital wards. Our survey results suggest that students found some approaches successful, while others were lacking. Our key findings were that (1) remote clinical education posed unique challenges, (2) opportunities exist for improving future remote curricula, and (3) educators should be aware of the substantial impact on medical students’ mental health and well-being incurred by this pandemic.

Unique Challenges of Remote Clinical Learning

Clinical education is particularly challenging to translate to a remote format, and there is little precedent to guide curriculum development in this regard. Recent studies have discussed the impact of the COVID-19 pandemic on clinical education but have notable limitations. Some works have focused on specific resources such as Zoom [4], Google Hangouts [5], or other videoconferencing tools [6], but without formative discussion of pedagogical issues. Others have described challenges [7,8,9,10], but did not include rigorous evaluation of student perspectives. Our results demonstrated that many third-year medical students identified loss of clinical experience as the biggest gap in remote clerkship curricula: students lost live experiences with patients and families and had few opportunities to observe and interact with faculty and residents outside of didactic environments. Students also missed the challenges of quick thinking and “learning under pressure” inherent to many aspects of clinical medicine. Many expressed fear of lacking practical knowledge and skills necessary for success in residency and future clinical practice. These results highlight the unique challenges of helping students develop skills in clinical practice derived from in-person interactions and physical exams, which are not as easily transferred to remote formats as classroom-based education.

Improving Remote Clinical Curricula

Although students acutely felt the loss of the traditional clinical experience, our study identified a few strategies that might help address this gap. Some approaches may merit incorporation into the “hybrid” strategies that develop as students slowly return to clinical venues.

One key strategy to improve clinical training if on-site activities remain limited is optimizing student participation in telehealth encounters. While telehealth has long been touted as an opportunity to improve efficiency and access of care [11,12,13,14] and has played a role in prior epidemics such as the Zika outbreak in 2016 [15], the expansion of telehealth in response to COVID-19 has been unprecedented [16,17,18]. Telehealth prominence will likely persist after this pandemic has resolved, and our results indicate that students wish to learn the fundamentals of telehealth and to participate in some capacity. This may prove more feasible for some specialties than others (psychiatry vs. surgery, e.g.). While there have been some studies regarding strategies for training medical students about communication in telehealth encounters [19], this is a relatively underdeveloped area of medical education and ripe for ongoing study and development of best practices.

We observed that the most favored remote learning tools were those that best engaged the learner. The three most highly valued types of resources in our survey involved active learning, while the three least valued were passive learning modalities (Fig. 1). Individual or small group sessions and interactive question banks may help reassure students of their progress, identify and address deficits, and avoid digital fatigue. Our findings also suggest that even when students return to the rich learning environment of clinics and wards, opportunities may be lost if students are not actively engaged in inquiry and direct patient care.

Remote learning provided clerkship students with an unprecedented level of flexibility. Students valued the ability to self-pace their learning and to participate in academic activities remotely in ways which were previously possible but rarely practical. Student participation in learning activities spread across multiple clinical sites or institutions may be less limited by geographic constraints that we previously realized. As students transition back to clinical responsibilities, inclusion of remote learning components into hybrid curricula may permit a degree of flexibility that proves valuable to both students and educators.

Student Well-being

Our survey illustrated that the pandemic and the transition to remote learning had considerable implications for medical student mental health and well-being. Numerous students expressed anxiety stemming from the transition to remote learning, uncertainty regarding academic and career futures, and the pandemic itself. Mental health issues and burnout have been increasingly recognized among both practicing physicians and medical students [20,21,22,23]. The clerkship year of medical school has been identified as a particularly vulnerable time, when students experience increased emotional exhaustion and decreased empathy [24, 25]. The finding that some students actually thrived despite being confined at home during a global pandemic also illustrates the degree to which clerkship demands may negatively impact well-being during “normal” times (Table 3, c). Maintaining flexibility in curricular structure may help students cope with external stresses and facilitate their ability to participate in self-care and wellness activities. Increased adoption of remote videoconferencing may enable clerkship directors to check in with students more frequently and easily than in-person meetings, especially if students are dispersed across different clinical sites. This strategy could also be applied to peer check-ins as a source of support [26]. Student mental health should be a principal consideration when re-evaluating clerkship curricula as we continue to move through and eventually out of the COVID-19 pandemic.

Limitations, Strengths, and Future Directions

This study was conducted at a single institution and may thus have limited generalizability. No demographic data were collected, so we could not analyze how factors such as age, gender, race, or socioeconomic status may have influenced student perceptions.

Strengths of the study included a robust overall response rate (> 70%) and a high number of narrative comments (296). Compared with prior studies about COVID-19 and medical education, which have been primarily narrative in nature and from the perspective of medical educators, this study obtained both quantitative and qualitative data from medical students, thereby offering a more comprehensive, student-centered, and data-driven perspective.

It will be interesting to learn in the months ahead whether objective assessments of medical knowledge and clinical reasoning demonstrate adverse effects, or conversely whether the ability to focus efforts on knowledge-based studies improves student performance on standardized testing. It will also be interesting to determine whether the changes in medical education resulting from this pandemic have effects on students’ choices of specialty, match results, and preparedness for residency training.

Conclusions

Strategic incorporation of select remote learning components into clinical clerkships may permit a favorable student experience even when opportunities for on-site participation are limited. While certain strategies and resources to optimize remote learning may prove successful, clinical experiences are essential in medical education. Access to remote platforms allows students to explore a variety of computer-based resources and utilize those best suited to their individual learning styles. Self-paced curricula should be sufficiently structured to guide students in meeting course objectives and core competencies. Remote videoconferencing appears most successful for active learning, though care should be taken to mitigate digital fatigue. Telehealth provides an opportunity for student engagement with patients and providers which is valued by students. As we reflect on the future of clinical education, we must be mindful to manage expectations of students and educators during these challenging times, and student wellness initiatives should be closely aligned with student learning.