Introduction

The impact of the COVID-19 pandemic on medical education is ongoing. Widespread suspension of in-person medical student instruction has required rapid restructuring of existing rotations to minimize interruption of students’ progress toward graduation. Clinical rotations at many institutions have traditionally relied heavily on student shadowing of physicians and other care providers, with much ad-lib teaching arising in the course of clinical work. The task of developing remote course modules to substitute for apprenticeship-style educational experience has posed an unprecedented challenge but also forced innovation, requiring medical educators to integrate newer technologies into a fresh approach to meeting the learning and assessment requirements of medical student clerkships.

Radiology is inherently technology-oriented. As medical imaging has become nearly universally digital, systems for remote medical image transfer have become commonplace and allow patient images to be anonymized and compiled into teaching files; some offer robust image manipulation and may integrate quiz-writing tools. Examples of such image transfer tools include the following:

There are also numerous online radiology educational resources already in existence, many developed by medical associations, and universities. All of these factors facilitate the construction of meaningful, engaging remote learning opportunities for medical students unable to attend in-person radiology rotations.

The Association of American Medical Colleges (AAMC) issued a guidance document on March 17, 2020, recommending the limitation of medical student participation in direct patient contact activities [1]. Our university system suspended all student clinical assignments on March 19, 2020, upon which the Director of Clinical Curriculum in the Office of Medical Education surveyed existing course directors to recruit faculty for the immediate development of new electives, to be conducted in “virtual” online format [2]. Our Department of Radiology was able to rapidly develop several remote rotations for the School of Medicine. We present our experience in implementing a remote learning module in Trauma and Emergency Radiology (hereafter referred to as “Emergency Radiology”), including student feedback on the course, and discuss implications for the design of future radiology electives, which may be improved by integrating the guided independent learning approach of the remote course with the best features of our traditional clerkship.

Materials and methods

Traditional rotation

The Emergency Radiology elective at our large urban teaching hospital has been previously conducted as a traditional in-person rotation. Medical students participate in clinical work in the imaging suites and radiology reading rooms that are dedicated to the Emergency Department and Level 1 Trauma Center. The 4-week long elective is offered to third and fourth year medical students, with no clinical prerequisites. On-site students participate in relevant weekly interdepartmental conferences and have access to daily resident conferences. Only limited online resources, principally pre-recorded institutional lectures, have been specifically recommended for review during the traditional rotation. Evaluation of student performance has relied primarily on subjective assessment of engagement by faculty working with each student (varying based on shift assignments), and grading of a case presentation submitted at completion of rotation. The course director(s) met with rotating students at the beginning, middle, and end of the elective, to set expectations, and give as well as receive feedback.

Scope of course revision

Conversion of the traditional apprenticeship-style rotation to an entirely remote format necessitated fundamental restructuring of course content and objectives. Direct interaction between students and faculty became limited to scheduled online meetings, which were significantly constrained by staff schedules, as the Trauma and Emergency Imaging service remained active despite curtailment of elective care throughout our medical system. Students on our virtual elective did not interact with radiology residents, who were also removed from clinical duties during this time period. Substantial self-directed learning resources were required to substitute for full time attendance in the radiology workrooms. We drew upon a variety of institutional and publicly available resources to amass sufficient content for the four-week remote elective, which was offered for three sequential rotation blocks spanning 30 March 2020 through 19 June 2020.

Remote elective format

The restructured remote elective in Emergency Radiology relied heavily on blended learning, combining online educational endeavors with live face-to-face instruction, and utilized the flipped classroom mode of instruction, a reversed delivery teaching approach in which class time, traditionally dedicated to didactic presentation of new material, is instead used to solidify, clarify and more deeply explore concepts already introduced through pre-class assignments [3]. At the start of the course, students were issued a written course overview summarizing content, goals, and objectives for the elective, detailing weekly assignments, and specifying the online meeting schedule.

A brief introductory module on general radiology concepts was followed by sequential units dedicated to emergency imaging of the head/face, neck/spine, chest, and abdomen/pelvis. Students were directed to review a variety of publicly available online radiology education resources covering a broad range of topics including appropriate utilization and ordering of imaging studies, radiological anatomy, and interpretation of common imaging studies (reinforced during the interactive sessions), and to review selected publicly available unknown cases. Students were also assigned a group of unknown imaging cases from our institution through the browser-based Pacsbin platform (Orion Medical Technologies, Baltimore, MD, pacsbin.com). Answers to a corresponding question set were to be submitted prior to an end-of-week group meeting at which the cases and questions were discussed. These sessions also included short case-based didactic lectures pertinent to each week’s content. At the final group meeting, students gave short PowerPoint presentations on selected topics of interest in trauma and emergency radiology.

After the first rotation, an additional interactive unknown case review session was added to each week’s schedule, during which faculty guided students through select imaging cases. These sessions provided additional opportunities for students to explore their developing expertise, ask questions about assigned materials, and share feedback on their progress.

Pacsbin assessments

The course directors created weekly unknown case sets by uploading anonymized images from the institutional picture archiving and communication system (PACS) to Pacsbin. Pacsbin supports Digital Imaging and Communications in Medicine (DICOM) standard format, emulates the PACS workstation interface by providing common image manipulation tools, and allows presentation of accompanying questions (Fig. 1). Student answers can either trigger immediate feedback, or be submitted for instructor review, as in our course. At the beginning of the rotation, students were provided with written instructions on the creation of a free Pacsbin account (required to access the weekly assessments) and the use of the various simulated workstation features (such as scroll, pan, zoom, and window/level adjustment). The selected cases included both traumatic and non-traumatic emergency conditions, and a variety of imaging modalities (Table 1).

Fig. 1
figure 1

Representative case from a Pacsbin assessment

Full size image
Table 1 Pacsbin assessment diagnoses
Full size table

Interactive sessions

Course directors and other necessary faculty were issued licenses for our institutional “Business” level Zoom account (Zoom Video Communications, San Jose, CA, zoom.com), allowing secure meetings of essentially unlimited duration (up to 24 h), with up to 300 participants. This platform was primarily accessed by camera/microphone/speaker-equipped desktop or laptop computer workstation, which facilitated screen-sharing by faculty, or by tablet; utilization of the application on smartphone for radiology education is hampered by the smaller screen size. Access to one of these devices, with a compatible operating system, browser software, and internet access, was a technical prerequisite for participation in the Zoom meetings.

Course evaluation

At the end of each virtual rotation, students were asked to complete an online course evaluation on the SurveyMonkey platform (SurveyMonkey Inc., San Mateo, CA, surveymonkey.com). All responses were completely anonymous, and survey submission was not a course requirement.

The evaluation forms rated the assigned educational resources, pre/post-rotation student confidence in ordering and interpreting imaging studies, overall course quality, and student likelihood to recommend the elective to others. Surveys also queried student interest in a radiology elective prior to the pandemic, relative preference for an in-person rotation, and perception of online course adequacy as an alternative to the traditional rotation. These assessments were quantified using five-point Likert scales. The survey also asked students to estimate time spent on various course components, and rate appropriateness of course workload on a three-point scale. A copy of the full survey can be found in supplementary file 1.

Consultation with our Institutional Review Board (IRB) determined that this investigation did not constitute human research and did not require IRB review.

Results

Course enrollment

The virtual Emergency Radiology elective was taken by a total of 13 students during the three sessions. Earlier the same academic year, prior to the pandemic, only a single student had completed our dedicated four-week elective, although 29 students had spent 1 to 2 weeks each on our service as part of the general Diagnostic Radiology course. On-site rotations resumed in July 2020; however, due to continued efforts to maintain social distancing in the reading rooms, an online version of our elective was again offered in July 2020. No students enrolled in the Emergency Radiology course at that time.

Survey results

Almost all of the online elective students returned completed course evaluations (92%, 12/13 students), which were solicited at the end of each rotation. Survey results rating overall course quality, pre/post-course confidence levels, appropriateness of workload, and student likelihood to recommend the course to others are presented in Table 2.

Table 2 Emergency radiology virtual rotation evaluation results
Full size table

Course quality and student satisfaction

Students were asked to assess their level of agreement to several statements using a 5-point Likert scale (with 5 indicating strong agreement). Students strongly agreed that the material they learned during the course was clinically relevant, accessible, and engaging (weighted average = 4.92/5, n = 12). Self-reported confidence in both study ordering and image interpretation increased at the conclusion of the course (image ordering: pre-course/post-course average = 3.08/5 and 4.17/5, respectively, n = 12; image interpretation: pre-course/post-course average = 2.25/5 and 4.08/5, respectively, n = 12). The majority of respondents (92%, 11/12 students) reported that they were very likely to recommend the elective to others. Despite uniformly positive ratings for the remote elective, 60% (3/5) of the students who reported experience with both formats indicated a preference for a hybrid course combining elements of both, while 40% (2/5) would still prefer the traditional rotation. Although five students reported previously having taken an in-person radiology elective, only four had indicated that the virtual course was not their first radiology elective, suggesting an erroneous response to one of those items.

Open-ended comments for the online course were extremely positive. One student reported that they enjoyed the elective and found it applicable for someone planning on going into emergency medicine or will be caring for trauma patients. Another commented that the online elective was applicable to many specialties outside of radiology.

Educational resources

Students rated individual resources on a 5-point Likert scale, with 5 being “excellent.” The Zoom meetings were the highest-rated online course component (5/5, n = 12), followed by the unknown case assignments (4.83/5, n = 12), and the end-of-rotation student presentations (4.42/5; n = 12). Ratings for the remaining online resources ranged from “average” to “excellent,” none being rated “below average” or “poor” (average range = 3.25–4.5). Although not a requirement, students could opt to attend daily virtual resident conferences via Zoom. Attendance at these lectures was not formally monitored. The resident lectures were the lowest-rated of all listed resources, receiving an overall average rating of 3.25/5 (n = 8). It should be noted that aside from the Zoom sessions, Pacsbin assessments, and end-of-rotation presentations, there was a variable response rate regarding other resources (n = 8–11). A list of the assigned resources can be found in the online supplemental materials.

Open-ended responses gave further insight into students’ opinions about various course components. With regard to the weekly interactive sessions, one student enjoyed engaging with instructors and other students. The combination of self-directed learning and interactive sessions was also appreciated, with one student appreciating the flexibility to learn independently and subsequently discuss the material with faculty and peers. There were also many positive comments about the Pacsbin cases. Students preferred being able to scroll through images over viewing static images and found the quizzes fun and helpful. One student reported wanting to see as many cases as possible.

Course workload

Most students reported spending 1–3 hours/week on the unknown case assignments, which required submission of completed question sets before each meeting at the end of the week. The Zoom meetings themselves averaged 1.5 h each. Time spent reviewing the other recommended online resources varied, with 58% of students spending less than 5 h/week, 25% spending 5–7 h/week, and only 17% spending more than 7 h/week. Despite the number and variety of assigned resources, no student rated course workload as “too much;” most students said the workload was “just right” (75%, 9/12 students), while a few felt it was “too little” (25%, 3/12 students).

Student recommendations

Student recommendations for course improvement included reducing the number of assigned resources, incorporating additional interactive sessions, and providing more specific guidance about how to approach unknown cases. In response to feedback submitted after the first virtual Emergency Radiology rotation, we instituted several changes, including adding a mid-week interactive Zoom session each week, during which faculty demonstrated search patterns for relevant modalities and guided students through a set of unknown cases, introducing and reinforcing the skills required for independent student completion of the assigned Pacsbin assessments. We also made adjustments to the online resources specified for review and distributed structured report templates to highlight important structures and findings for each anatomic region and modality.

Discussion

Course enrollment

Although many students elect to spend one week with the Emergency Imaging service during the on-site General Radiology rotation, the dedicated on-site 4-week Emergency Radiology elective typically has limited enrollment. The substantial increase in online student enrollment for this course during the pandemic is likely explained by the abrupt decrease in University course offerings. The majority of students who enrolled in this online course (75%, 9/12 students) had already planned to take a radiology elective, prior to the cancellation of in-person rotations. Perhaps these students saw the online course as a viable option, particularly when other options were limited. This is further supported by the lack of course enrollment once on-site medical student rotations resumed.

Course structure and resources

Self-directed learning

This online course relied heavily on self-directed learning, with students assigned to review a combination of online modules, short didactic videos, and publicly available unknown cases, which were either scrollable stacks or selected static images. Research has demonstrated that adult students can be very successful at independent learning [4] and that many prefer to replace traditional didactic lectures with podcasts, websites, simulations, interactive tutorials, and internet-based educational games [5, 6]. While individual student preference for format and specific assigned material varied, our survey results indicate that students were able to find personally effective material for each unit. Students in this course appreciated the flexibility of self-directed learning, which, when combined with regular interactive faculty sessions, has been shown to be similarly effective when compared to a traditional radiology rotation [7].

Flipped-classroom sessions

Weekly online small group meetings were reserved for interactive learning activities and review of the weekly Pacsbin assessments. The incorporation of these flipped-classroom elements shifts the educator’s role towards that of a facilitator, encouraging problem solving, rather than imparting didactic knowledge [3, 8, 9]. While these methods require substantial preparation on the instructor’s part, they are efficient, resulting in high-yield educational impact with less instructional time [3]. Despite the limited time available for our weekly small group meetings, students universally rated these sessions as excellent and reported that the material was relevant, accessible, and engaging. The unknown case conferences and Pacsbin assessments aided in solidifying the information acquired during self-directed study.

Pacsbin assessments and case conferences

The Pacsbin assessments were a key component of the course, providing students with the opportunity to independently problem-solve. Curating specific cases can ensure that students will be exposed to critical imaging diagnoses, something often left to chance in the reading room. The use of virtual workstations is of value in creating an immersive experience [10, 11]. Previous work has shown that students prefer an interactive image viewer to static images and that this improves student understanding [11]; our survey data support this. Although not evaluated in this study, virtual workstations have also been shown to influence a student’s consideration of radiology as a career [11].

The weekly Pacsbin assessments were untimed and not formally graded, serving as formative assessments, a “low-stakes’ method intended to stimulate learning and provide feedback [12]. The assessments were “open-book;” students could utilize any resources to answer the assigned questions. Many of the cases were intentionally challenging, going beyond the scope of the assigned materials. The goal of these assignments was to encourage each student to individually seek out and critically appraise additional references in order to arrive at a diagnosis or answer a particular question. The Pacsbin assessments were popular with the students, rated as a very useful educational resource, second only to the weekly small group sessions.

Unknown case conferences were conducted on the third day of each unit, when the material for each unit was fairly new. These case conferences incorporated the Socratic method, with a line of questioning used to help students arrive at the correct answers, in hopes of fostering critical thinking skills [13]. When done in a supportive manner, this style of teaching has proven to be a preferred and effective style for third and fourth year medical students learning radiology [14].

A small majority of students (67%, 8/12 students) opted to attend daily resident conferences. They were the lowest-rated of the resources listed in the survey, presumably because the material presented was too advanced for students new to radiology.

Limitations

Although we had a high survey response rate (92%, 12/13 students), the sample size was small due to the limited time period during which on-site student rotations were canceled. As formal pre/post-rotation assessments were not performed, our conclusions rely solely on survey data, a subjective measure limited by recall bias.

Learning resources were rated by a variable number of students. The universally rated resources were the small group sessions, Pacsbin assessments, and end-of-course presentations, all of which were course requirements. The remaining resources had variable response rates; the reasons for which were not assessed. It is possible that some resources may not have been conducive to a student’s learning style or may have presented technical barriers. This survey also did not specifically inquire as to additional resources students may have used, although it is known through informal discussion that some students did seek these out.

The lack of direct immersion in the radiology work environment limits the virtual rotation in ways for which it is difficult to compensate. One lost experience is simply exposure to the “feel” of a radiologist’s workday, invaluable for the student considering the specialty as a career. Another is witnessing the many factors that impact the quality of a patient’s imaging workup, from the adequacy and appropriateness of the study indication, to the availability of clinical consultation, to patient stability and ability to tolerate the demands of a particular modality. Imaging acutely ill and injured patients requires expedient decision making, the flexibility to adapt to often unexpected changes in patient status, as well as rapidly evolving differential diagnoses. Close communication between a patient’s primary clinical team and various members of the radiology service is required to provide timely and appropriate diagnostic services. Although we did not emulate these important facets of our specialty in our precipitously designed online elective, there are potential avenues to deepen the remote experience. Virtual read-out sessions, implemented at some institutions to allow ongoing clinical work by radiology residents while maintaining social distancing, could be expanded to include medical students [15]. It can be imagined that robust interactive patient care simulation software might also meet some of these goals.

Despite the many challenges posed by online medical student education, our experience shows that this format can be well-received by students, and benefits from the blending of self-directed study with instructor-led learning.

New opportunities

As medical students re-enter the hospital and resume in-person radiology courses in many parts of the country, it is a perfect time for educators to reflect on the different engaging teaching strategies and educational techniques that were necessary during the pandemic, and incorporate some of the lessons learned.

Both written materials and online content may already be assigned to students enrolled in radiology courses. Re-evaluating learning resources to ensure that they are well-suited to today's students and their learning styles would be beneficial, given student preference for short videos and interactive materials. Such formats might be favored over more traditional references such as textbooks, which have become less popular in recent years [16].

Assuming students engage in self-study, there is less of a need for didactic lectures. Faculty can more effectively help students apply their newly acquired knowledge through unknown case conferences utilizing curated medical-student-level scrollable cases. Such learning activities could easily be incorporated into an on-site radiology course.

Using the faculty-developed Pacsbin assessments as a model, similar student-created assessments could also be developed. This would serve as both an engaging learning opportunity for the student authors, while also creating an enduring teaching file for students, residents, and faculty in the future. This could be very effectively accomplished through student collaboration with residents and attendings, who could aid in case selection and review. The SPACS cases assigned in our course are an example of a similar initiative [17].

Additional opportunities to increase medical student engagement abound. In the reading room, implementations such as the “look ahead” technique give students the opportunity to independently review current imaging studies in real-time and present findings to an attending, which this has been shown to increase interest, engagement, and educational value [18]. The innovative incorporation of radiology “field trips” coupled with reflective practices can be used to further engage students, whereby students observe various radiology studies and procedures, and subsequently answer questions about what they observed [19]. Involving students in gathering relevant clinical history and physical exam findings, searching for relevant literature, and protocolling imaging studies can also increase student engagement [20].

Conclusion

Diagnostic radiology lends itself particularly well to online education. The success of our online radiology course highlights the value of self-directed education, blended learning, and flipped-classroom techniques. Many elements of this Emergency Radiology elective could be modified to suit other online radiology courses. Additionally, application of the principles used during the development of an online course can be instructive as students and educators return to the reading room for on-site rotations and may lead to a more active and meaningful learning experience for medical students.