Abstract
The COVID-19 pandemic catalyzed the integration of a virtual education curriculum to support radiation oncologists in training. We report outcomes from Radiation Oncology Virtual Education Rotation (ROVER) 2.0, a supplementary virtual educational curriculum created for radiation oncology residents globally. A prospective cohort of residents completed surveys before and after the live virtual webinar sessions (pre- and post-surveys, respectively). Live sessions were structured as complex gray-zone cases across various core disease sites. Resident demographics and responses were summarized using means, standard deviations, and proportions. Nine ROVER sessions were held from October 2020 to June 2021. A total of 1487 registered residents completed the pre-survey, of which 786 attended the live case discussion and 223 completed post-surveys. A total of 479 unique radiation oncology residents (of which 95, n = 19.8%, were international attendees) from 147 institutions (national, n = 81, 55.1%; international, n = 66, 44.9%) participated in the sessions. There was similar participation across post-graduate year (PGY) 2 through 5 (range n = 86 to n = 105). Of the 122 unique resident post-surveys, nearly all reported learning through the virtual structure as “very easy” or “easy” (97.5%, n = 119). A majority rated the ROVER 2.0 educational sessions to be “valuable or “very valuable” (99.2%, n = 121), and the panelists-attendee interaction as “appropriate” (97.5%, n = 119). Virtual live didactics aimed at radiation oncology residents are feasible. These results suggest that the adoption of the ROVER 2.0 curricula may help improve radiation oncology resident education.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
Introduction
With the COVID-19 pandemic and mandated social distancing, disruption in traditional in-person teaching models led to the creation of alternate virtual educational platforms in residency training [1,2,3,4,5,6,7,8]. Radiation Oncology Virtual Education Rotation (ROVER) is a novel virtual education platform developed to support radiation oncology education for medical students and residents. ROVER was initially implemented in 2020 during the COVID-19 pandemic, when medical student clerkships were on hold, to improve medical student interest and knowledge in radiation oncology [9]. Initially, these virtual didactics were intended to serve as a temporary replacement for in-person classroom teaching; however, virtual learning offers unique advantages as a supplement to in-person teaching even in the post-pandemic setting [1, 4,5,6, 8]. In the previous meta-analysis, technology-enhanced stimulation in health profession education has shown to be effective in outcomes for knowledge, skills, and behavior [10].
Recent publications have emphasized the wide variation and potential deficiencies in the curricula of radiation oncology residency programs [11,12,13], with a desire for greater resource sharing between institutions, particularly during the pandemic [11]. As ROVER was found to be feasibly implemented, well attended, and well received, ROVER 2.0 was launched to enhance the education of radiation oncology residents. Live sessions were led by invited faculty from diverse institutions across the USA. ROVER 2.0 sessions focused on difficult cases to stimulate thoughtful discussion and emphasize practice variation in a virtual format, which facilitated increased connectivity. We report here the results of our multi-institutional case-based virtual sessions for radiation oncology residents.
Methods
ROVER 2.0 Design
ROVER 2.0 comprised a series of virtual educational panels tailored for residents. Each session covered gray-zone cases across disease sites representative of core rotations in a typical residency program (Table 1). Faculty members presented complex case variations and areas of clinical equipoise to stimulate discussion around institutional practice differences in radiation oncology.
Zoom (Zoom Video Communications, San Jose, CA), a cloud-based video conferencing tool, was used to host the 1-h live sessions (12–1 PM PST). All sessions were password protected and required attendee registration. Zoom features, including poll questions and live chat Q&A, were used to facilitate engagement between the faculty and attendees. Each session was moderated by three to four radiation oncology faculty from different radiation oncology academic programs across the country; faculty were deliberately invited from different geographical areas to promote speaker diversity (Northeast, n = 4; Midwest, n = 7; South, n = 9; West, n = 9). Sessions were advertised on Twitter (Twitter Inc., San Francisco, CA) and distributed to mailing lists of groups including Association of Residents in Radiation Oncology (ARRO), American College of Radiation Oncology (ACRO), and Association of Radiation Oncology Program Coordinators (AROPC), with information to sign up along with fliers. The calendar invite reminder feature was included in the registration to encourage live attendance; however, all sessions were recorded and uploaded to our website archive for asynchronous viewing (https://www.radoncvirtual.com/past-rover-sessions).
Data Collection and Statistical Analysis
This study was exempted by our institutional review board. Resident pre- and post-session surveys (Supplementary Fig. 1) were collected using Research Electronic Data Capture (RedCap), a web-based application for capturing clinical research data hosted by our institution. Although pre- and post-survey results were linked by a respondent, responses were anonymized and aggregated. Residents had to complete the pre-session survey as part of the registration process to receive the Zoom link and password.
Pre- and post-session survey responses were summarized using mean, standard deviations (SDs), and proportions. Responder vs. non-responder characteristics were compared using the Chi-square test, Fisher’s exact test, and Mann–Whitney U test. Inductive thematic analysis was performed by a single user (author NS) to determine the common feedback themes [12]. Statistical analyses were performed using SAS software (version 9.4; SAS Institute, Cary, NC).
Results
Resident Characteristics
Nine ROVER 2.0 sessions were held from October 2020 to June 2021. A total of 786 registered residents attended the live case discussion, and 223 completed post-surveys. There were 479 unique radiation oncology resident attendees (of which 95, n = 19.8%, were international attendees) from 147 institutions (national, n = 81, 55.1%; international, n = 66, 44.9%) (Table 2). The five most represented institutions in the USA were Northwestern University in Illinois (n = 15), Beaumont Hospital in Michigan (n = 14), Johns Hopkins University in Maryland (n = 14), Stanford University in California (n = 13), and University of Miami in Florida (n = 12). International resident attendees (n = 95, 19.8%) were most commonly from institutions in Asia (n = 50, 52.6%), North America (excluding the USA, including the Caribbean; n = 18, 18.9%), Africa (n = 12, 12.6%), and Europe (n = 12, 12.6.%). Table 2 characteristics were also evaluated by international vs. national resident attendees (Supplementary Table 1).
Among all resident attendees, slightly over half were male (55.9%, n = 268), and representation was similar across PGY-2 through 5 (range of n = 86 to n = 105), with low representation from PGY-1 (6.9%, n = 33) (Table 2). Most residents (69.9%, n = 335) reported having mock oral examinations through their home programs, either yearly (32.8%, n = 110), every rotation (31.9%, n = 107), or rotation dependent (28.1%, n = 94). Residents reported an average of 5.5 h (± 4.4 SD) per week dedicated for didactic education by their home programs. Mock oral frequency was the only significant factor between post-survey responders vs. non-responders (p = 0.0022), with post-respondents reporting a higher value and still finding the ROVER sessions valuable. There were no other significant systematic differences between responders vs. non-responders. Approximately, one-third of the residents reported fewer didactics (25.3%, n = 121), less faculty engagement in didactics (33.4%, n = 160), and less access to faculty (31.5%, n = 151) due to the COVID-19 pandemic. Over half of residents reported that faculty engagement in teaching sessions remained unchanged (55.7%, n = 267), while a third (33.4%, n = 160) reported less overall faculty engagement due to COVID-19. Similarly, most residents reported access to faculty being unchanged (60.8%, n = 291), while almost one-third (31.5%, n = 151) reported less access to faculty due to the pandemic.
Interest in ROVER 2.0 and Virtual Learning
The top three reasons for signing up for sessions were (1) opportunity to hear from a diverse panel of experts (78.1%, n = 374), (2) interest in a new virtual learning platform (39.5%, n = 189), and (3) reduced didactic training due to COVID-19 (12.1%, n = 58). Most residents reported being interested in discussions centered on the following topics during the sessions (non-exclusive): treatment standard of care and existing evidence (70.8%, n = 339), contouring and treatment planning (70.4%, n = 337), case variations (70.0%, n = 335), emerging treatment options (70.0%, n = 335), and case work-up, including imaging review and staging (61.4%, n = 294). Over one-third of residents (38.0%, n = 182) viewed virtual education platforms as equal to in-person learning, while a similar proportion (36.7%, n = 176) reported in-person learning as superior but recognized the need for virtual platforms due to the pandemic.
ROVER 2.0 Perceived Effectiveness
Pre-session, most residents rated their expertise in a session’s disease site as “beginner” (34.0%, n = 506) or “competent” (37.5%, n = 558) (Fig. 1A). Of post-session respondents, 74.8% (n = 167) “agreed” or “strongly agreed” with the statement that they feel more confident in treating the specific disease site because of the session (Fig. 1B). A total of 122 unique residents completed 223 post-session surveys. Most residents (97.5%, n = 119) found learning through the virtual session format to be “very easy” or “easy.” The majority of residents rated the sessions as “valuable” or “very valuable” (66.4%, n = 81, and 32.8%, n = 40, respectively). Nearly all residents (99.0%, n = 121) were satisfied and would recommend the session to a fellow resident. There was no significant difference between post-survey responses for US (n = 31) vs. international unique residents (n = 74).
There were three central themes in resident feedback: (1) reduced number of cases for more in-depth discussions, (2) interactive polls, and (3) effective panelist discussions (Table 3). First, reducing the number of cases to 2–3 per session would allow attendees to learn about clinical variation in depth from each panelist. Residents appreciated the integration of polls throughout the sessions to enhance engagement and panelist-attendee interaction. Residents also enjoyed the effective format with multi-institutional panelists shedding light on practice variation across institutions. Most residents (97.5%, n = 119) rated the level of interaction between panelists and attendees as “appropriate.” Finally, in terms of content feedback, 40.2% of residents (n = 49) wanted more time to be spent on discussion of treatment standard of care and supportive evidence, followed by 37.7% (n = 46) interested in dedicating more time to contouring and treatment planning. There were no distinct themes pertaining to different disease subsites.
Discussion
Overall, the multi-institutional ROVER 2.0 program of live virtual case-based sessions was well attended and well received among radiation oncology residents. Notably, the sessions were attended by 479 unique residents from 147 unique institutions globally, with almost 20% representation from international residents (most commonly in Asia). There are an estimated 773 radiation oncology residents and 91 ACGME-accredited programs in the USA in 2020–2021 [14], thus indicating a high turnout rate for the ROVER 2.0 sessions. Additionally, there was a similar mix of all post-graduate levels (with the exception of PGY-1), suggesting the material was perceived as useful for all levels of training; the low PGY-1 participation is not unexpected given PGY-1 s have not yet started at their radiation oncology training program.
Importantly, over half (56–69%) of residents did not report a pandemic-related change in access to faculty, faculty engagement during didactics, or hours of the resident didactic curriculum with an average time of 5.5 h per week, similar to a prior study [11]. Despite the fact that in-person educational opportunities were unchanged for the majority of respondents, the live virtual sessions were still highly ranked and subjectively effective: Nearly all residents would recommend the session to a fellow resident and found the sessions to be valuable or very valuable, and the majority (84%) felt that the session improved their ability to treat the disease site in question. This finding emphasizes that virtual learning provides unique benefits that can supplement standard in-person learning. A common feedback point revealed a critical advantage of this platform: effective multi-institutional panelist discussion (Table 3), with an appreciation for discussion of practice variation and differing treatment approaches (a request of 70% of respondents on the pre-survey). Radiation oncology exhibits strong practice variation between geographies and institutions [15,16,17], partly related to differences in available technologies, but even beyond that based on training, varying adoption of new studies [17], practice setting (i.e., academic versus private practice [16]), and inherent clinical equipoise for many topics in our field. Thus, our panelists, which were deliberately constructed with an eye to speaker diversity, allowed for thoughtful discussion of varying treatment approaches. By removing typical barriers such as travel, virtual learning allows for multi-institutional (or even global) collaboration and sharing of knowledge/resources in a convenient and cost-effective manner, similarly reflected in other survey studies [1, 8, 18,19,20,21]. To this end, the use of gray-zone cases was particularly useful given there was no definitive “correct” answer, thus allowing for thoughtful discussion of different treatment approaches.
A potential concern of added virtual education is that residents may experience burn-out and fatigue with virtual presentations, especially if their didactics were unchanged during COVID. However, approximately 80% found the timing convenient (deliberately scheduled 12–1 PM PST, with strict time constraints), sessions were monthly (and avoided summer months and early in the academic year when residents are adjusting), and once again, nearly all respondents viewed the sessions as valuable despite the majority having unchanged didactics. It should also be emphasized that approximately one-quarter to one-third (25–33%) of residents reported a negative impact of the pandemic on either their dedicated didactic time, faculty engagement during didactics, or faculty access; thus, the virtual sessions may have unfortunately served more than a supplementary role for some.
Even pre-pandemic, there have been concerns regarding the wide variation of radiation oncology curricula between institutions [11,12,13]. The radiation oncology residency curriculum in the USA is not standardized, unlike the European Society for Radiotherapy and Oncology (ESTRO) Core Curriculum [20] or the European Society for Medical Oncology and American Society of Clinical Oncology (ESMO-ASCO) global core curriculum for medical oncology [21]. There have been recent successful attempts to standardize medical student radiation oncology clerkships by ROECSG [22]. A recently published national survey of program directors highlighted the need to similarly standardize the radiation oncology residency curricula [11, 23]. Notably, in this survey, no programs shared identical curricula content. Additionally, some institutions averaged as low as 20% faculty presence during didactics, and almost one-third of residents reported disruption of didactics by clinical responsibilities. Finally, there was a shared desire for increased faculty involvement in didactics, updated didactic content, introduction of novel evidence-based treatment techniques, and sharing of resources nationally across programs, particularly among small programs with limited resources. Thus, virtual learning can potentially improve and supplement the quality of programs with limited expert faculty or technology by providing high-quality didactics to cover these topics (i.e., brachytherapy, pediatric radiation oncology, lymphoma, proton therapy, CyberKnife/GammaKnife).
An invaluable advantage of traditional in-person didactics is the interactive atmosphere and comradery [1, 7, 8]; the hands-on nature of experiential learning is also hard to simulate virtually [1]. Thus, several studies have shown a desire to return to in-person learning but continuing supplemental virtual learning given the unique benefits as addressed above. A potential benefit of the virtual interactive atmosphere is that participants may feel more comfortable raising questions because of the relative anonymity and dismantling of the physical classroom space [1, 24]. Finally, the interaction was also enhanced by obtaining feedback from registrants on how to improve further sessions; in-person didactics would similarly benefit from obtaining and acting on resident feedback to enhance the learning experience.
We acknowledge several limitations of this study. Most notably, there may be selection bias in residents who participated in the surveys, such that those who are more favorable to virtual learning participated. The post-survey results with the low response rates may be especially impacted by this bias, an inherent limitation of any such survey study. Our pre- and post-surveys also captured subjective evaluations for the sessions rather than evaluating whether the sessions were objectively beneficial in improving resident understanding (alluding to Kirkpatrick’s evaluation of hierarchy in learning, which distinguishes between the perception of knowledge versus actual processing and acquisition of that knowledge [25]). Given the complexity of these sessions, which focused on gray-zone cases with no single correct answer, and discussions regarding subjective practice variation, subjective knowledge assessment and student evaluation of teaching were felt to be more reasonable metrics than objective quizzes. These metrics were also fitting, given the intention of this study was to evaluate the feasibility and reception of these nuanced virtual sessions, which are the first of their kind for radiation oncology residents. That being said, implementation of a more objective evaluation, such as a brief pre- and post-survey quiz, would be warranted to better characterize the effectiveness of the program. Finally, we only acquired the feedback from participants of the live session; registrants who did not attend the live session but viewed the recording at a later date were not quantified nor surveyed. On that note, an added benefit of virtual didactics is, in fact, the ability to record and archive sessions for future viewings, for registrants who could not attend due to scheduling constraints, or even for future residents (rather than repeating the live webinar annually).
Because live sessions were recorded and archived, and due to residents’ desire for more nuanced topics under-represented in their core curricula, such as safety/quality improvement, palliative care, and interviewing/job search [11, 13], sessions for ROVER 2.0 2021–2022 will expand to include the aforementioned topics, as well as other specialized topics like brachytherapy and advanced treatment modalities (i.e., proton therapy, stereotactic body radiotherapy [SBRT]). Resident feedback was vital in this first iteration of ROVER 2.0 to structuring the future 2021–2022 sessions.
Conclusion
Based on our results, virtual radiation oncology teaching is a viable tool for global resident learning that should continue to have a role post-pandemic. Approximately 99% of the residents reported this virtual education platform as valuable, with one-third reporting decreased faculty engagement due to COVID-19. Thus, the continued implementation of a virtual radiation oncology curriculum will serve as an adjunct to traditional resident education for learners globally.
Data Availability
Research data are stored in an institutional repository and will be shared upon request to the corresponding author.
References
Malik M, Valiyaveettil D, Joseph D (2021) Optimizing e-learning in oncology during the COVID-19 pandemic and beyond. Radiat Oncol J 39(1):1–7. https://doi.org/10.3857/roj.2020.00710
Tan LT, Tanderup K, Nappa A et al (2021) Impact of transitioning to an online course – a report from the ESTRO gyn teaching course. Clin Transl Radiat Oncol 29:85–92. https://doi.org/10.1016/j.ctro.2021.06.001
Chao TN, Frost AS, Newman JG (2020) Interactive virtual surgical education during COVID-19 and beyond. Acad Med 11:e9. https://doi.org/10.1097/ACM.0000000000003609
Albert TJ, Bradley J, Starks H et al (2022) Internal medicine residents’ perceptions of virtual morning report: a multicenter survey. J Gen Intern Med 37(6):1422–1428. https://doi.org/10.1007/s11606-021-06963-7
Essilfie AA, Hurley ET, Strauss EJ, Alaia MJ (2020) Resident, fellow, and attending perception of e-learning during the COVID-19 pandemic and implications on future orthopaedic education. J Am Acad Orthop Surg 28(19):e860–e864. https://doi.org/10.5435/JAAOS-D-20-00579
Larocque N, Shenoy-Bhangle A, Brook A et al (2021) Resident experiences with virtual radiology learning during the COVID-19 pandemic. Acad Radiol 28(5):704–710. https://doi.org/10.1016/j.acra.2021.02.006
Theodorou CM, Joshi A, Chahine AA et al (2021) Multi-institutional collaborative surgery education didactics: virtual adaptations during a global pandemic. J Surg Educ 78(4):1340–1344. https://doi.org/10.1016/j.jsurg.2020.12.013
Franklin G, Martin C, Ruszaj et al (2021) How the COVID-19 pandemic impacted medical education during the last year of medical school: a class survey. Life (Basel) 11(4). https://doi.org/10.3390/life11040294
Kahn JM, Sandhu N, Eyben RV et al (2021) Radiation Oncology Virtual Education Rotation (ROVER) for medical students. Int J Radiat Oncol Biol Phys 111(1):29–35. https://doi.org/10.1016/j.ijrobp.2021.03.057
Cook D, Hatala R, Brydges R et al (2011) Technology-enhanced simulation for health professions education: a systematic review and meta-analysis. JAMA 306(9):978–88. https://doi.org/10.1001/jama.2011.1234
Abrams MJ, Golden DW, Huang GC (2021) A call for reform: Variability and insufficiency in radiation oncology resident didactics-a brief report and national survey of program directors. J Cancer Educ 18:1–4. https://doi.org/10.1007/s13187-021-02080-5
Nowell LS et al (2017) Thematic analysis: striving to meet the trustworthiness criteria. Int J Qual Methods. https://doi.org/10.1177/1609406917733847
Spraker MB, Nyflot M, Hendrickson K et al (2017) A survey of residents’ experience with patient safety and quality improvement concepts in radiation oncology. Pract Radiat Oncol 7(4):e253–e259. https://doi.org/10.1016/j.prro.2016.11.008
Accreditation Council for Graduate Medical Education (ACGME) Data Resource Book 2020-2021. 401 North Michigan Avenue, Suite 2000, Chicago, Illinois 60611 53. https://www.acgme.org/globalassets/pfassets/publicationsbooks/2020-2021_acgme_databook_document.pdf
Valle LF, Chu FI, Kundu P et al (2021) National variation in the delivery of radiation oncology procedures in the non-facility-based setting. Cancer Med 10(14):4734–4742. https://doi.org/10.1002/cam4.4028
Kim AE, Wang GM, Waite KA et al (2021) Cross-sectional survey of patients, caregivers, and physicians on diagnosis and treatment of brain metastases. Neurooncol Pract 8(6):662–673. https://doi.org/10.1093/nop/npab042
Jagsi R, Falchook AD, Hendrix LH et al (2014) Adoption of hypofractionated radiation therapy for breast cancer after publication of randomized trials. Int J Radiat Oncol Biol Phys 90(5):1001–1009. https://doi.org/10.1016/j.ijrobp.2014.09.032
Chick RC, Clifton GT, Peace KM et al (2020) Using technology to maintain the education of residents during the COVID-19 pandemic. J Surg Educ 77(4):729–732. https://doi.org/10.1016/j.jsurg.2020.03.018
Cook A, Salle JL, Reid J et al (2005) Prospective evaluation of remote, interactive videoconferencing to enhance urology resident education: the genitourinary teleteaching initiative. J Urol 174(5):1958–1960. https://doi.org/10.1097/01.ju.0000177483.65528.40
Benstead K, Lara PC, Andreopoulos et al (2019) Recommended ESTRO core curriculum for radiation oncology/radiotherapy 4th edition. Radiother Oncol 141:1–4. https://doi.org/10.1016/j.radonc.2019.08.013
Hansen HH, Bajorin DF, Muss HB et al (2004) Recommendations for a global core curriculum in medical oncology. Ann Oncol 15(11):1603–1612. https://doi.org/10.1093/annonc/mdh447
Golden DW, Braunstein S, Jimenez R et al (2016) Multi-Institutional implementation and evaluation of a curriculum for the medical student clerkship in radiation oncology. J Am Coll Radiol 13(2):203–209. https://doi.org/10.1016/j.jacr.2015.06.036
Abrams M, Golden DW, Huang GC (2020) Radiation oncology resident didactic variability: a national program director survey. [Abstract]. Am Soc Radiat Oncol Ann Meet
Kanneganti A, Lim KM, Chan G et al (2020) Pedagogy in a pandemic – COVID-19 and virtual continuing medical education (vCME) in obstetrics and gynecology. Acta Obstet Gynecol Scand 99(6):692–695. https://doi.org/10.1111/aogs.13885
Yardley S, Dornan T (2012) Kirkpatrick’s levels and education ‘evidence.’ Med Educ 46(1):97–106. https://doi.org/10.1111/j.1365-2923.2011.04076.x
Acknowledgements
We thank all the faculty members (Dr. Abhishek Solanki, Dr. Arya Amini, Dr. Beth Beadle, Dr. Chelsea Pinnix, Dr. Chris Hallemeier, Dr. Christina Henson, Dr. Daniel Chang, Dr. Emma Fields, Dr. Emma Holliday, Dr. Jerry Jaboin, Dr. Joanna Yang, Dr. John Lucas, Dr. Jona Hattangadi-Gluth, Dr. Jonathan Tward, Dr. Kara Romano, Dr. Kimberly Corbin, Dr. Kiran Kumar, Dr. Kristin Higgins, Dr. Lindsay Burt, Dr. Mitchell Kamrava, Dr. Neha Vapiwala, Dr. Nikhil Joshi, Dr. Paul Nguyen, Dr. Puneeth Iyengar, Dr. Rachel Jimenez, Dr. Rachel Rabinovitch, Dr. Sean Sachdev, Dr. Stephanie Terezakis, and Dr. Suzanne Evans) for their contribution to our virtual webinars. We also thank the residents who joined us virtually to attend the multi-institutional teaching sessions.
Author information
Authors and Affiliations
Corresponding authors
Ethics declarations
Conflict of Interest
The authors declare no competing interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Sandhu, N.K., Rahimy, E., Hutten, R. et al. Radiation Oncology Virtual Education Rotation (ROVER) 2.0 for Residents: Implementation and Outcomes. J Canc Educ 38, 977–984 (2023). https://doi.org/10.1007/s13187-022-02216-1
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s13187-022-02216-1