Introduction

Controversies exist over the effect of the COVID-19 pandemic on medical education, oscillating between the positive additions of virtual learning and education innovation and the negative consequences of reduced educational time and decreased clinical volumes. It is difficult to predict the ultimate outcomes of these changes, but it is essential to study them to plan for future surges or unforeseeable events that may similarly influence educational programming.

Research describing the impact of the COVID-19 pandemic on trainee experiences and competencies is beginning to materialize [1,2,3,4,5,6]. Studies have indicated similar trends of decreased clinical exposures and training, regardless of specialty, including redeployments, cross-coverage for sick call, outpatient clinic cancellations, fewer inpatient shifts, and increased telemedicine encounters [1, 3, 6,7,8]. Decreased visits coupled with trainee redeployment from usual rotations have resulted in decreased clinical exposure [1,2,3]. Formal educational activities were also impacted due to shifts in clinical needs, social distancing regulations, and resident illnesses, requiring programs to quickly act to reformat their educational programs and utilize virtual tools in an attempt to preserve their existing curricula [1, 4, 9, 10]. Educational innovations for bedside teaching, small and large format didactics, and clinical skills training were often put into place to address the urgent need to adequately train residents and fellows. Many studies have been published that detail the implementation of specific educational innovations; however, owing to their rapid implementation and short period of study, many do not have data beyond feasibility measures [5, 11,12,13,14,15].

Few frameworks, if any, exist that explain how learning is affected by natural disasters or pandemics; though some learning theories suggest ways in which non-monotonic learning allows people to acquire new knowledge and skills to cope with new and surprising phenomena [16, 17]. We have seen a wealth of innovation in education throughout the pandemic, and as we look toward the future, educational experts have suggested many of these innovations that should outlast the pandemic [18]. Some studies have sought to evaluate learners’ and educators’ impressions on educational experiences, but these studies varied, with some reporting an overall negative impact on education and others highlighting the benefits of innovation in medicine and education [3, 5, 7, 9]. To the best of our knowledge, no studies have compared educator and trainee perceptions on the pandemic’s impact on medical training, including the specific effects on skills, productivity, teaching, and learning preferences.

As New York City was the epicenter of the pandemic in early 2020, we felt uniquely equipped to perform this type of evaluation since many graduate medical programs had to quickly adapt both clinical and educational program structure to accommodate high acuity and dense patient volumes [19,20,21]. Pediatrics was in a unique position early in the pandemic due to lower clinical volumes and the need to deploy staff to care for adults, while still needing to maintain a clinical footprint and effective educational training. This unique perspective and the innovations that came from it may be applicable to other specialties as we consider the future landscape of graduate medical education (GME).

In this multi-institutional study, we aimed to determine changes and trends in GME necessitated by the COVID-19 pandemic, including perceived changes to (1) educational, clinical, and academic activities, (2) clinical and teaching skills, and (3) effectiveness of educational innovations and desirability for future use, as assessed by pediatric trainees and faculty.

Methods

Setting and Participants

This study was conducted in June 2021, 15 months after the World Health Organization declared a global pandemic. In this cross-sectional study, we surveyed pediatric residents, fellows, and faculty members at three urban, academic, tertiary care centers in New York City. Montefiore Medical Center and its associated Children’s Hospital at Montefiore is located in the Bronx, New York. Mount Sinai Medical Center and its associated Kravis Children’s Hospital is located on the Upper East Side of Manhattan. Columbia University and its associated Morgan Stanley Children’s Hospital is located in Washington Heights, New York.

At the time of this study, each training program had approximately 60–75 residents and 35–85 fellows. All pediatric trainees at each institution were invited to participate via anonymous survey. A total of 428 trainees were invited to participate in the study (154 at Institution A, 149 at Institution B, and 125 at Institution C). Trainees were informed that their participation would not impact their evaluations in any way.

A similar but distinct survey was sent to faculty members who have a role in graduate medical education at each institution. Inclusion criteria for faculty were: 1. ACGME designated core faculty, 2. residency and/or fellowship program directors, 3. faculty who direct a resident or fellow rotation, and 4. faculty with a GME leadership role, e.g., education committee, Clinical Competency Committee. A total of 191 faculty members were invited to participate in the study based on these inclusion criteria (37 at Institution A, 66 at Institution B, and 88 at Institution C). In total, 619 trainees and faculty members were invited to participate.

Survey Tools

Two similar but distinct surveys were used, one for trainees (Supplement 1) and one for faculty (Supplement 2). Surveys utilized multiple choice and open-ended formats to assess perceptions about changes to graduate medical education, new educational innovations being employed, impact on skill sets and scholarly work, desires for sustainability, and respondent characteristics. Perceptions were measured on a 5-point Likert-like scale and questions regarding participating activities were assessed via descriptive multiple-choice questions. Survey questions were developed by the four co-investigators who are on faculty at each of the three institutions participating in this study. The innovations asked about in the surveys were informed by a thorough literature review as well as the experiences of the authors at each institution [1, 5, 12, 14, 15, 18]. The surveys were reviewed and piloted by trainees and faculty who are not primary investigators or enrolled in the study to assess clarity and face validity. All surveys were anonymous and included minimal identifying information.

Recruitment

Participants were emailed and asked to participate in the voluntary online surveys via Qualtrics. Email addresses of residents and fellows were obtained from the program directors of the three academic pediatric departments. Email addresses of faculty members were obtained from the office of the Vice Chair of Education at each institution. The invitations were sent to faculty and trainees from the investigators at each institution, ensuring there was no confusion about the voluntary nature of this survey and that it was not a program requirement. Three reminders were sent over the course of 1 month to increase response rate.

Data Analysis

Data were analyzed via descriptive summary statistics (e.g., percentages). Comparisons of trainee and faculty responses to survey items were made via chi-square analyses and with McNemar tests for paired data. Bivariate associations of respondent characteristics with responses were examined using chi-square tests. For variables having multiple response categories, post hoc pairwise comparisons were conducted to identify which specific category or categories of response differed significantly by group. Where indicated, categories were then collapsed into binary variables for further analysis. Open-ended responses were reviewed via thematic analysis. Two of the investigators on this study independently reviewed all qualitative data and coded for themes. Investigators compared and reconciled their independent coding and then independently reviewed and reconciled a second and final time.

Ethical Considerations

This study was deemed exempt by the Institutional Review Boards at Columbia University Irving Medical Center, Montefiore Medical Center, and Mount Sinai Medical Center.

Results

Demographics

Of the 194 respondents, 88 (45.4%) were faculty and 106 (54.6%) were trainees. There was a 31% overall response rate, 46% for faculty and 25% for trainees. There were similar response rates from each of the three institutions. The demographic characteristics of the respondents is displayed in Table 1. The majority of both faculty (75.3%) and trainees (78.7%) were female. Approximately half of the faculty held the rank of assistant professor (48.3%), with 29.2% associate professor and 20.2% full professor. There was representation from all specialties including academic general pediatrics (22.2%), hospital medicine (10.0%), emergency medicine (7.8%), intensive care (18.9%), and other subspecialties (41.1%). There was also a wide range of representation from different ages and number of years on faculty. For trainees, 14.8% were interns, 43.5% residents, and 39.8% fellows.

Table 1 Demographic characteristics of respondents
Full size table

Educational Time

The majority of respondents reported a decrease in the number of hours dedicated to formal educational activities (69.8%), inpatient clinical teaching (77.7%), and outpatient clinical teaching (77.8%). Trainees were more likely to report a decrease in hours dedicated to formal educational activities when compared to faculty (77.4% vs. 59.8%, p = 0.003), and inpatient clinical teaching (84.3% vs. 67.1%, p = 0.02); however, trainees and faculty similarly reported decreased hours in outpatient clinical teaching (77.7% vs. 77.9%, p = 1.0).

Feedback and Evaluation

Most respondents (51.9%) reported a decrease in feedback, with only 7.6% stating that feedback increased during the pandemic. Trainees were more likely to report a decrease in feedback they received compared to faculty reporting a decrease in feedback they provided (59.6% vs. 42.7%, p = 0.01).

Formal evaluation also decreased — 48.7% of trainees reported a decrease in formal evaluation; however, 64.6% of faculty felt that the amount of formal evaluation they provided stayed the same, and only 34.4% reported a decrease in formal evaluation. Additionally, despite the ACGME requirement, 59.5% of trainees reported that they were not evaluated via Observed Clinical Encounters (OCE).

Clinical and Teaching Skills

Regarding clinical skills (Fig. 1A), most trainees and faculty similarly perceived that trainee skills in physical examinations and oral presentations stayed the same. Skills in telemedicine were the only clinical skill both faculty and trainees perceived as increasing over the course of the pandemic. For teaching skills (Fig. 1B), the majority of respondents perceived that trainees’ clinical teaching, didactic presentations, and giving feedback stayed the same; however, trainees and faculty alike perceived that trainees’ virtual teaching skills improved. Trainees were more likely to perceive improvements in their clinical teaching skills when compared to faculty (21% vs. 9%, p < 0.01). Faculty had similar perceptions of their own teaching skills, with the majority reporting that their clinical teaching, didactic presentations, and feedback skills remained the same, while virtual teaching skills increased.

Fig. 1
figure 1

Changes in skills and productivity during the pandemic; A faculty and trainee perceptions on the changes in trainee clinical skills; B faculty and trainee perceptions on the changes in trainee teaching skills; C reported changes in trainee and faculty productivity; *p < 0.05

Full size image

Academic Productivity

Respondents also reported on the changes in their academic productivity. As depicted in Fig. 1C, faculty were more likely than trainees to report a decrease in conducting research, writing articles, manuscripts, or book chapters, and presenting at local/regional/national conferences. Approximately one-third of trainees actually reported an increase in conducting research and writing manuscripts during the pandemic.

Participation in Virtual Education

The vast majority of respondents (93.4%) participated in virtual education conferences most or all of the time (94.8% of trainees and 91.7% of faculty). In contrast, only 36.5% of faculty reported teaching virtually most or all of the time. Furthermore, only 4.2% of faculty reported extensive training in using virtual tools for education, and 42.7% of faculty had no training at all. Consistent with this lack of training, only 28.9% of faculty feel very comfortable educating trainees using virtual tools.

Table 2 depicts the differences in rates of participation among trainees and faculty both as learners and as educators. Trainees were more likely to participate in virtual precepting, didactic conferences, case-based conferences, journal clubs, mock codes, and use online question banks when compared with faculty. Faculty were more likely to participate as learners in virtual multi-institution conferences and use podcasts when compared with trainees. As educators, faculty had significantly less experience teaching with nearly every modality compared with the number of trainees who learned from those same modalities.

Table 2 Participation in virtual education as learners and educators
Full size table

Perceived Effectiveness of Virtual Teaching Modalities

Trainees reported the three most effective educational modalities during the pandemic respectively were: 1. online question banks, 2. podcasts, and 3. multi-institution conferences, while faculty thought the most effective were: 1. virtual grand rounds, 2. virtual journal clubs, and 3. online question banks. Trainees and faculty similarly found the least effective modality was virtual mock codes. Figure 2 depicts the differences between faculty and trainees who perceived the modalities as “very effective.” Faculty rated virtual grand rounds, virtual didactics, virtual case-based conferences, virtual journal club, and web-based modules significantly higher than trainees. Virtual multi-institution conferences were the only modality trainees rated significantly higher than faculty. There was no statistical significance when comparing faculty in different ranks. When comparing training levels, fellows rated virtual grand rounds, virtual didactics, virtual journal club, multi-institution conferences, and web-based modules significantly higher than residents.

Fig. 2
figure 2

Perceived effectiveness of innovative virtual teaching modalities by percentages of faculty vs. trainees who perceive each modality as “very effective.” Modalities are listed in order of perceived effectiveness from the perspective of trainees. Conf, conference; *p < 0.05

Full size image

Overall Impact

When asked about the overall impact of the COVID-19 pandemic on medical education, 75.5% of trainees reported a negative impact compared to 50% of faculty (p < 0.01). There was no statistical significance when comparing faculty with different ranks, subspecialty practices, primary learners, or deployment history. For trainees, fellows were significantly more likely to report a positive impact of the pandemic on medical education compared to residents. Table 3 outlines the many shared and distinct themes between faculty and trainees in the qualitative data on overall impact to medical education. The most relevant shared themes among trainees and faculty were the benefits of virtual education including improved access, attendance, and collaboration, the increased opportunities for feedback and skill-building with telemedicine, and the decrease in clinical learning experiences. Faculty-specific themes included challenges to workflows, a systemic lack of focus on education, and increased empathy for trainees. Trainee-specific themes included not only improved work/life balance and increased support but also increased burnout, moral injury, and concerns about future competence.

Table 3 Shared and distinct themes regarding the impact of the pandemic on medical education
Full size table

Future Directions

When asked to choose the top three educational modalities they would like to continue, 60.3% of trainees chose virtual didactic conferences, 56.9% chose virtual grand rounds, and 40.5% chose virtual case-based conferences. Of the faculty, 83.3% chose virtual grand rounds, 62.5% chose virtual didactic conferences, and 38.5% chose virtual multi-institution conferences. Faculty were significantly more likely to choose virtual grand rounds than trainees (83.3% vs. 56.9%, p < 0.01) and multi-institutional conferences (38.5% vs. 25.9%, p = 0.04). Trainees were more likely to choose online question banks than faculty (27.6% vs. 12.5%, p < 0.01). Within the trainee cohort, fellows were more likely than residents to choose virtual didactics (74.4% vs. 50.8%, p = 0.02) and virtual multi-institution conferences (48.8% vs. 12.7%, p < 0.01). There were no statistically significant differences among faculty with different ranks.

The majority of participants, both as learners and as teachers, prefer future educational sessions to be a combination of in-person with the option for virtual attendance, whereas only 6% of participants desire entirely in-person sessions. Though the percentages are small, faculty were more likely to prefer teaching entirely in-person compared to trainees’ preference for learning in-person (11.5% vs. 6.1%, p = 0.01). Zero faculty members prefer teaching entirely virtual, yet 14% prefer their own educational sessions to be entirely virtual. Regardless of modality, 100% of faculty prefer to teach live (in-person or virtual), yet 22% of trainees and 23% of faculty prefer their educational sessions be pre-recorded for on-demand viewing.

In the analysis of the qualitative data, there were many shared and distinct themes among faculty and trainees regarding educational changes and innovations they would like to continue in their educational programs (Table 3). The most common response was to continue virtual educational modalities, specifically grand rounds, didactic sessions, and multi-institutional conferences. Many providers desire a hybrid approach with opportunities for both in-person and virtual learning, with in-person sessions reserved for more interactive or experiential learning. Faculty and trainees expressed the need for dedicated educational time, both for teaching and learning. Many faculty specifically noted the need for increased faculty development, particularly for telemedicine and the use of technology for teaching. Trainees highlighted the need to move away from standard didactics in medical education, shifting toward more interactive, case-based modalities, with asynchronous components.

Discussion

Historically, there have been unprecedented events such as Hurricane Katrina, Hurricane Sandy, and other natural disasters/events that caused short-term but significant changes to medical education. Very few studies exist that outline the impact of these changes, and none provides evidence-based planning for strategic changes that minimize disruption to medical training [22,23,24]. If we have learned anything from these historical events and the many surges of the current pandemic, it is that we cannot predict external factors that can alter healthcare and education. Instead, we must study how shifts in practice affect both educators and trainees and be prepared to implement innovations and strategies that best serve our learners and the healthcare system.

This study revealed the major shifts that occurred in both clinical learning experiences and formal educational opportunities. Our findings align with the limited data already published that show decreases in clinical exposures and massive shifts to online and virtual learning [1, 4, 6, 9]. Despite decreases in educational experiences and concerns regarding trainee competencies [1, 4], most perceived their skills in physical examination, oral presentations, clinical teaching, didactic presentations, and providing feedback to have stayed the same. Many perceived that their skills in virtual teaching and telemedicine even improved over the course of the pandemic. Interestingly, faculty and trainee perceptions of trainee skills overall seemed to align, which may provide credence to what is often considered subjective assessment [25]. Additionally, this suggests that the changes incurred from the pandemic may not have been as damaging to trainee competencies as one may have feared, though further assessment into actual skillsets and skill retention over time is necessary to substantiate this.

Another important finding of this study is that most trainees and faculty participated in virtual learning, but very few educators actually taught virtually. Only 4% had extensive training and less than one-third were very comfortable using virtual teaching tools. There is a need to train all faculty in virtual tools for education; otherwise, the burden may fall on the trainees themselves or on the small group of educators who feel confident with these virtual modalities. Programs may also consider using asynchronous techniques that may require fewer educators to teach many learners. For example, podcasts and question banks were considered very effective by both faculty and trainees and require minimal effort on the educator once developed. Utilizing existing podcasts or question banks could alleviate some of the burden on educators to develop their own material and would offer education styles that appeal to many learners [26].

Our study found that most participants were interested in continuing virtual education, which is in contrast to some studies that suggested learners would prefer not to continue virtual learning in the future [5]. Based on the data from our study, we recommend continuing some level of virtual education, particularly when it best suits the content. For example, educators may use virtual modalities for lectures that have no interactive elements and in-person for simulations, procedural training, and small group discussions. Additionally, we recommend that all educators have training on using virtual tools for education. Even if used infrequently, this will allow educators to pivot to virtual education more seamlessly in the case of another surge or widespread event. Educational programs should capitalize on the innovations that were effective (e.g., virtual grand rounds, multi-institution conferences, didactics, online question banks, and podcasts) and eliminate those that were least effective (e.g., virtual mock codes, virtual rounding, and virtual precepting). This would enable program directors to ensure that in-person rounding, precepting, and simulation may be the priority for education during a surge and enable more flexibility for the didactic and grand round sessions to remain virtual.

We found that perceptions of educational innovations and their effectiveness did not always align between faculty and trainees, so it will be important to consider who the learners are when planning an educational session. From a programmatic standpoint, leadership should consider trainee perspectives when designing curricula and making changes to educational programs. There were some differences between faculty and trainees regarding learning style preferences; however, most differences shown in the qualitative themes were the trainees’ heightened emphasis on work/life balance, burnout, camaraderie, and support. An emerging focus in medical education literature is the learning climate, and our results show the importance of addressing all aspects of the learning environment for trainees [27,28,29].

Some of the negative themes raised that must be addressed by programs include decreases in clinical learning experiences and formal educational opportunities, challenges with workflows, trainee fears about future competence, lack of community-building, burnout, and moral injury. Program directors should also focus on sustaining the positive themes such as increased access and collaboration through virtual learning modalities, improvements in work/life balance particularly for trainees, increased empathy of faculty for trainees, and increased support and solidarity during significant events such as the COVID-19 pandemic. Notably, while most faculty reported decreases in their academic productivity, one-third of trainees increased their research and manuscript-writing during the pandemic. Programs should consider ways to foster that growth in scholarship as clinical responsibilities resume. Finally, we recommend including trainees in discussions about changes to their educational programs as we saw many themes in trainee responses that faculty may not have considered.

Our study is limited in that we had low response rates, particularly by trainees. While this may introduce non-response bias, the large sample size and similar response rates among each institution provide sufficient sampling to allow for generalizability. Another limitation is that we only addressed perceived outcomes. Future efforts should seek to assess more objective outcomes such as skills via observed patient encounters or simulation, percentage of trainees with a job or fellowship at the time of graduation, scoring on board exams, or other competency-based assessments. We did not analyze data on each institution separately, as every institution had varying responses to the pandemic, and we aimed to focus more on general trends to inform more universal programmatic changes. Additionally, we intentionally only conducted this study at children’s hospitals since pediatrics had a unique perspective early on in the pandemic due to lower clinical volumes and deployments, allowing for greater focus on educational innovation while combating challenges prevalent among all specialties. We believe these findings are applicable to all fields who similarly need to find solutions for changing clinical landscapes. Finally, we chose to limit this study to a very specific geographic region, as this region was hard hit in the first wave of the pandemic, and these institutions had to pivot quickly to adapt to the new reality of COVID-19, owing to the need to balance rapid implementation with high acuity and volume relative to the rest of the country. Reproducing this study would make these results more generalizable; however, we believe insights from these New York City institutions, particularly educational innovations and future interests related to virtual learning, are applicable to GME on a broad scale.

Conclusion

The COVID-19 pandemic led to massive shifts in healthcare and medical education, resulting in decreased formal education and clinical learning experiences but advances in innovative tools for education. Our study showed that most faculty and trainees desire a hybrid learning model, highlighting increased access, attendance, and collaboration through virtual modalities. As the pandemic wanes and efforts to return to pre-pandemic teaching methods pervade, programs should consider ways to incorporate successful innovations and improve ineffective ones, including all stakeholders in decision-making.