Introduction

Until March 2020, most medical students and physicians had limited experience with video calls in a professional capacity. The Covid-19 pandemic, however, catapulted the match process into the age of virtual communication, as limiting face-to-face interaction and interstate travel became paramount. In May 2020, the Coalition for Physician Accountability, a cross-organizational group consisting of multiple medical education organizations, including the Association of American Medical Colleges (AAMC), American Medical Association (AMA), Accreditation Council for Graduate Medical Education (ACGME), and others, strongly recommended virtual interviewing for the 2021 Match cycle [1]. In response, residency programs across the country pivoted to create a recruitment season that embraced technology.

There has been much speculation as to the potential benefits and disadvantages of virtual interviews for surgical residency programs and applicants. Areas of particular interest have included technological concerns, financial and scheduling advantages, effects on diversity initiatives, and the impact on match results. In this article, we examine the general expectations and concerns that programs and applicants expressed heading into this unusual match season, expound on how our general surgery residency program adapted to virtual interviewing, and discuss further avenues of research to improve the virtual interview process moving forward.

Technological Aspects of Setting Up Virtual Interviews

While virtual interviews are relatively new to the field of medicine, the business world has been conducting them for years, and has published advice for interviewees on how to succeed in a virtual setting [2]. Using these recommendations as a backbone, program directors, residents, and medical education organizations compiled lists of tips to help programs and applicants make the most of their interviews [3,4,5,6]. Some common suggestions are for candidates to dress appropriately for the formality of the interview, maintain the camera at eye level, use adequate lighting, keep the phone or computer plugged into the charger throughout the interview, turn off device notifications to avoid being interrupted, ensure a reliable internet connection, and keep the background neutral and clutter-free [2,3,4,5,6].

When choosing a platform on which to host our program’s virtual interviews, it became clear that we would need the ability to use breakout rooms to accommodate having a Q&A session with our residents in addition to the multiple simultaneous interview rooms. We felt breakout rooms were vital to the success of our interview days to foster informal, spontaneous interactions to allow the candidates to assess the culture of the program. We ultimately settled on using our university subscription for Zoom™. There are many important factors to consider when choosing a platform. A secure platform is necessary not only to keep hackers out of the interview session, but also to ensure applicants do not unintentionally interrupt each other’s interviews. Ensuring the interview session is password protected and using a virtual waiting room can prevent intrusion from hackers and other interviewees [4]. It is also helpful for both interviewers and interviewees to be receive a notification 2–5 min prior to the end of each interview to ensure applicants have adequate time to get their questions answered. We found it beneficial to have the interview automatically end at the allotted time in order to stay on schedule. Understanding the benefits and limitation of each platform is essential in constructing an effective virtual interview day.

Adequate technological support and maintaining communication with both interviewers and interviewees are paramount for running a successful virtual interview. Multiple sources suggest emailing applicants and faculty in advance with information regarding the platform being used to host the interview, the schedule for the day, and the contact information of the program coordinator in case of last-minute technological problems [4, 7]. Applicants should also be advised to test out the platform ahead of time and to familiarize themselves with it [7].

Prior to the start of our interview season, our residency held a test day where medical students and preliminary residents from our institution were interviewed by our core faculty. The students, residents, and faculty were asked to provide feedback about our virtual interview setup, and our faculty in turn provided feedback to the interviewees about their backgrounds and professional appearance. Our program chose to hold a virtual meeting with faculty the night before interviews to help them test the software and to go over the schedule for the following day. We also provided faculty with the phone numbers of their interviewees in case they got disconnected during their interviews. We did make adjustments throughout the interview season based on feedback from faculty, residents, and applicants. We found this flexibility to be an essential part of our success, and our interview days went very smoothly.

Although the Zoom™ platform worked well for the formal interview sessions, we have discussed that in future we would like to use more creative online platforms to improve the interview process. Our program had beta tested an online virtual world called Sophya™ [8] for previous resident events. This platform, much like massive multiplayer online games, allows each person to create an online avatar and to travel around a virtual space while interacting with the avatars of other users; in this case, residents and faculty. In the Sophya™ platform, walking close to another avatar or group of avatars will bring up a video chat window for each person to allow one-on-one or group conversation. It is also possible to create locked private video chats within the group, which would be conducive to an interview environment. This platform would allow applicants to easily interact with multiple different people and create their own breakout rooms. To our knowledge, this has not yet been tried, but it could improve the applicant’s interview day experience by making interactions feel more natural.

Organizing the Interview Day

Even before the interview season commenced, there was particular concern that the awkwardness of virtual communication would significantly limit the effectiveness of pre-interview social events. Conversation in virtual social events may die down quickly, so it was recommended that programs set an end time in advance to help keep applicants engaged and prevent extended lulls in conversation [4, 9]. Other recommendations for a successful virtual social included maintaining a good ratio of applicants to trainees, providing a dress code, and letting applicants know ahead of time whether they could eat and drink during the event. Breakout rooms were also found to be crucial, as they allow applicants to interact with residents more easily, and also observe how residents interact with each other [4, 9]. Many recommended using room themes to improve the flow of discourse within these breakout rooms. Themes are pre-set topics that can help drive the conversation and prevent applicants from having to hear the same question answered multiple times in different breakout rooms. Good examples of themes include “what is it like to live or commute in the area” and “what do the residents do in their free time” [4, 9]. We found switching the rooms roughly every 15 min helped keep the conversation flowing while still providing enough time for applicants to have their questions answered. Additionally, after receiving feedback from applicants that they wanted more opportunities to get to know each other, we began implementing a 30-min applicant-only session prior to the start of the social, which was very well received.

Because virtual interview days were generally shorter than they would have been in person [10], we split our days into separate morning and afternoon sessions. All applicants joined us initially for grand rounds and a presentation from the program director, after which the afternoon group was able to log off until their session began later. This dual session received positive feedback from applicants and faculty.

Some programs recommended decreasing the length of each individual interview due to decreased attention span in a virtual setting [11], however, others warned that longer interviews were needed for applicants to adequately assess the program [12]. After trialing 30-min and 20-min interviews, we ultimately settled on each candidate having three 25-min interviews with a paired faculty member and resident, as well as shorter “speed dating” style interviews with the Program Director, Associate Program Director, and Chairperson. We found that this approach allowed adequate time for interviewers and applicants to assess each other without running out of topics of conversation.

Between interviews, our applicants were put in a breakout room with 2–5 residents who were available to answer questions. The resident room was not compulsory, and applicants were invited to turn off their camera and take a break between their interviews if they preferred. We received feedback from multiple applicants that having the resident room available was helpful and gave them great insight into our program’s culture.

Financial Advantages and Convenience of Virtual Interviews

Interview season is infamous for being a financial drain on medical students, who generally have significant levels of debt and no income. One of the most obvious benefits of virtual interviews is that no money needs to be spent on travel, accommodations, or food. Most surveys show virtual interviews save applicants roughly $500–800 per program [12,13,14,15,16,17]. Applicants also miss fewer clinical days when they do not need to travel, which is particularly important for non-medical student applicants, such as preliminary residents interviewing for categorical positions [14, 15]. Therefore, virtual interviews are not only financially beneficial to applicants, but also prevent disruptions to their learning and may level the playing field for non-student applicants. These benefits are not confined only to applicants. Programs also benefit financially from virtual interviews. Whereas our program usually spends approximately $10,000–15,000 on event space and food for interviews, this year we spent $0. Additionally, because virtual interview days are less time consuming, there may be a decrease in lost productivity by surgical faculty [10, 14, 16].

Increased Number of Applications Per Program

One of the major concerns going into the 2020–2021 interview season was that the decreased cost and increased ease of interviewing, as well as fears that the unusualness of this application cycle would affect candidates’ abilities to match, would lead to applicants applying to and interviewing at significantly more programs as backups [18, 19]. Preliminary data from the AAMC confirm that surgical programs received an average of 1,045.68 applications for the 2021 match cycle, increased from 887.57 in 2020 [20]. This led to concerns that the most competitive candidates could potentially fill up interview spots at an increased number of programs, thus causing other qualified but slightly less competitive candidates to be left without interviews. Programs therefore had legitimate concerns that by interviewing these highly competitive candidates they could end up with unmatched spots [21]. Fortunately, preliminary data from a subset of over 60 surgery residency programs showed that these fears appear to be unfounded, as they did not find an increase in the number of interviews received by the top candidates [22]. General Surgery programs also had to be aware of the increase in applicants for integrated programs in Vascular Surgery, Plastic Surgery, and Cardiothoracic Surgery. Many of these applicants applied to General Surgery Programs as a backup, presumably with the General Surgery Programs ending up lower on their rank lists. The dual applicants further confounded the potential for not filling the categorical spots in a General Surgery Program. As part of the National Resident Matching Program (NRMP) rules, programs are not allowed to ask candidates to disclose when they are applying to different tracks, which meant that programs had to be discerning and holistic in their approach when deciding which candidates to offer an interview [23]. Luckily, despite initial concerns, NRMP data showed that categorical general surgery programs did not seem to have a more difficult time matching applicants this year than previously (99.7% both years). Applicants to categorical general surgery programs did have a slightly lower match rate this year (53.7% compared to 56.4% in 2020), but this decrease is likely a continuation of recent trends [17, 24].

Although it did not appear to affect program match rates, potential solutions to the problem of applicants accepting an increasing number of interviews have been discussed at length in the recent literature. Popular suggestions include limiting the number of interviews applicants are allowed to accept and introducing preference signaling [19]. Preference signaling is a process in which applicants indicate to a pre-specified number of programs that they are particularly interested in that program. Usually, signaling is done naturally (i.e., choosing to do an away rotation, or attending a second look). With constraints in place limiting applicants’ abilities to naturally signal their interest, an artificial signaling system could help programs ensure they are interviewing and ranking applicants who are genuinely interested [9, 11, 25]. A third suggestion is for programs to simply interview more people [11], however, most programs agree this would be an unnecessary hassle with minimal reward, particularly, if the other safeguards are put in place [9].

Institutionalized Discrimination in the Virtual Age

With increasing understanding of the importance of diversity in medicine, there has been speculation as to how changes to the interview process would affect socioeconomic, gender, and racial diversity in hiring. While some hoped that the changes would act as an equalizer, others quickly pointed out that the remote interview process could exacerbate established biases and introduce new ones.

Although there is not yet information on match results by gender, a survey of ENT applicants in 2020 suggested that females may be more negatively affected by virtual interviewing than males. In this survey, 70.0% of female applicants felt less confident about their ability to match because of the switch to virtual interviewing, compared to only 47.1% of male applicants [18]. To continue to promote gender equity, it is imperative that future research seek to determine the reasons behind this difference and find ways to address them.

Another concern expressed by applicants was that programs would be more likely to favor those who already had a connection to the program. Although faculty were hopeful that the uniqueness of this season would spark a more holistic review of each candidate, most applicants tended to suspect that programs would instead place more weight on test scores, pedigree, degree type, and connections [18, 19]. Fortunately, cursory data from the NRMP showed that degree type did not appear to carry more weight than in the previous years, as Doctor of Osteopathic Medicine (DO) students actually had a slightly increased rate of matching into a categorical general surgery position compared to last year (62.13% from 59.4%). Additionally, graduate applicants appear to have had a slightly higher match rate this year than previously. This may support that programs did in fact perform more holistic reviews of candidates. However, concerns that international medical graduates (IMGs) would particularly suffer in this cycle did appear to be supported by preliminary NRMP match data, which showed IMGs fared slightly worse than in 2020 [17, 24]. It was postulated that IMGs were disproportionately affected by the limited capacity for away rotations, which are usually essential for networking and helping them establish themselves as competitive candidates [9].

One way virtual interviews may promote diversity in hiring is that the decreased cost of interviewing could help level the playing field for applicants without stable finances. In the past, these students would have been unable to attend as many interviews as their peers, who may be getting financial support from family [11]. There is, however, concern that students who are unable to pay for high-speed internet or expensive camera and audio equipment will make a poorer impression on a virtual interview, an issue that they would not have in person [26]. Additionally, applicants with children, particularly single parents of young children, are at a significant disadvantage interviewing virtually during a pandemic. It is generally recommended that applicants who cannot control background noise perform their interviews from the medical school library [26]. However, with remote learning and pandemic-related issues in obtaining childcare, asking an applicant to separate themselves from their children may not be straightforward. Instead of putting the onus on applicants to protect themselves from discrimination, programs should encourage interviewers to be cognizant of their own biases and should support the applicants who would be affected by them. Of note, there is a paucity of data published on the effects of virtual interviews on ableist hiring biases, but this is an important topic that will require further research. Similarly, it is of vital importance that future studies investigate how virtual interviews impact racial hiring biases in order to improve equitability.

Assessing Programs and Applicants in a Virtual Setting

One of the most daunting aspects of virtual interviewing is the difficulty of making an informed decision based on only a few hours of video calls. For applicants, resident morale and the culture of the program are two of the most important factors for determining rank list order [27]. Many are therefore concerned they will not be able to adequately assess programs without having the opportunity to interact with residents and faculty [14, 15, 18]. Because of this, virtual social events with trainees are considered an absolute necessity during a virtual interview season [4, 9, 11].

Over the course of an in-person interview day, applicants interact with multiple different faculty members and residents in a less formal manner between interviews as well as at lunch. During a virtual interview day, however, applicants may only interact with the faculty and residents who formally interview them. With a decreased quantity of interactions, applicants are more likely to assume that a single interaction reflects more widely on the program as a whole, a phenomenon known as signal distortion [7]. Signal distortion may also affect how programs evaluate interviewees, particularly because of the high priority placed on interpersonal skills. The NRMP Program Director Survey from 2020 found that program directors cited interpersonal skills, interactions with faculty during interview and visit, interactions with housestaff during interview and visit, and feedback from current residents among the top 6 most important factors in ranking applicants [28]. Therefore, each interaction between a resident or faculty member and an applicant will carry more weight than it would during an in-person interview day, which could drastically affect how applicants rank programs and vice versa.

Additionally, programs are concerned about their ability to be able to adequately portray themselves to applicants [11]. To combat this, it is recommended that programs have an updated program website and active social media presence, as this helps applicants better understand a program’s culture and whether it would be a good fit for them [4, 11, 29]. Savvy programs were even able to use social media analytics to measure the impact of their program on platforms like Twitter™ and Instagram™. Programs that utilized creative videos and hashtags to improve their visibility in the virtual world found that these initiatives were received positively [30]. Applicants also overwhelmingly agree that programs should have a virtual Q&A session with the program director and provide virtual tours of the facility and surrounding area to help them better assess culture and fit [7, 11, 18, 31].

Interestingly, some of our preliminary residents who interviewed virtually for categorical positions said they felt virtual interviews were sufficient for adequately assessing a program. This may be because they already had a strong understanding of how a program should function, so could focus on assessing only the aspects that mattered most to them. In general, our preliminary residents said they drastically preferred the virtual interviews to in-person interviews because of decreased cost and travel time. Therefore, in the future, this group in particular may benefit from being offered the ability to interview virtually.

Concerns for the Future

The Covid-19 pandemic has necessitated many changes to both the match process and surgical education over the past year, and the effects of these changes on resident recruitment remain to be seen. Some suspect that, as a result, there may be increased attrition over the next few years. Because of the difficulties in assessing program culture over a virtual platform, there may be an increased number of residents who find that their new program is not a good fit for them [9]. Given that applicants feel social media is an important factor in understanding a program’s culture [11, 29], it will be interesting to see whether there was an application bias towards programs with a more sophisticated online presence, and whether applicants who matched these programs feel like the program was adequately portrayed.

Additionally, there is a concern that curriculum changes necessitated by Covid-19 may leave graduating medical students ill-prepared for their intern year, thereby increasing burnout. Away rotations in 2020 were largely postponed except in extenuating circumstances [1], which may have limited the number and breadth of surgical rotations students were able to experience. A survey of surgical residents found that those who did not do surgical sub-internships or take overnight call as medical students were more likely to report feeling underprepared at the start of residency. Even more concerning, this was significantly associated with increased risk of burnout [32]. This suggests that the difficulties of the 2021 match cohort are not over. Incoming residents may need more support and guidance than in the previous years, and their faculty and co-residents should be encouraged to assist them and be vigilant for signs of burnout.

Conclusion

While virtual interviewing was done out of necessity this year, there appear to be many benefits, particularly, with regard to overall convenience and decreased financial strain. While there are many potential disadvantages to virtual interviews, none of them appear insurmountable. Now that the infrastructure is in place, programs should consider how best to use virtual interviewing in the future. Many different possibilities have been proposed, such as offering virtual interviews to applicants already familiar with the program (i.e., from an away rotation) or who have already completed an intern year at another program. It has also been suggested that programs could offer a round of preliminary interviews virtually before inviting a subset of applicants for in-person interviews to help keep costs down [12]. However, more research is needed on systemic bias in virtual interviewing to mitigate inequalities within the hiring processes prior to making any permanent changes. The 2020–2021 interview season was a challenging one for all involved, but the lessons learned over the past year have the potential to revolutionize future residency hiring practices.