To the Editor:

The 2020–2021 Residency Recruitment Season introduced virtual interviews as a recommendation of the Coalition Work Group due to the COVID-19 pandemic. These guidelines were extended to the 2021–2022 recruitment season [1]. We designed a post-interview day study to identify benefits, limitations, and areas for improvement for an all-virtual recruitment season by obtaining applicants’ perspectives on these changes. A review of similar studies and the AAMC website noted that for virtual interviewing to remain a viable option, robust information needs to be available, including instructions for applicants on what to expect during their interview day [1,2,3]. Future recruitment seasons may benefit from hearing applicants’ perspectives, as organizations consider when and if to return to some form of in-person interviews.

To get a sense of applicants’ experience during the 2020–2021 Recruitment Season, we sent a web-based 15-question post-interview survey to all applicants that interviewed with Mount Sinai Beth Israel Psychiatry Residency Program, with consent implied upon completion. Fifty-six responses were obtained from 170 applicants before the rank list closure date. The survey included questions in Likert Scale, multiple-choice, and “yes or no” formats. Survey responses were auto-populated into a Google Forms document with no identifiable data. The hospital’s Institutional Review Board granted exempt status.

The 2019 National Residency Matching Program (NRMP) Applicant Survey Report, which looked back at the mainly in-person 2018–2019 recruitment season, identified the top 3 categories listed for importance in ranking a program as Overall Goodness of Fit (89%, 4.8/5), Interview Day Experience (82%, 4.6/5), and Desired Geographic Location (77%, 4.6/5) [4]. Through our survey, we hoped to understand how these categories might be affected through virtual interaction. Even though respondents stated a future preference for in-person interviews (35.1%; N = 20) or a combination of in-person and virtual interviews (43.9; N = 25), one of the most important findings of our survey was respondents’ belief in their ability to make a solid decision when ranking during the 2020–2021virtual recruitment season (N = 48; 84%).

“Fit,” the highest-ranked item in decision-making before the virtual season, is an often poorly defined term in existing literature and carries a risk of bias [3]. Instead of fit, we asked if respondents got a sense of the program’s culture from virtual interactions. Responses were mixed ranging from positive — “perfect” (N = 5; 9%), “very well” (N = 22; 39%), “well” (N = 13; 23%) — to negative — “somewhat well” (N = 15; 26.3%), “not at all” (N = 2; 3.5%). These results highlight the limitations of virtual interviews in showcasing the culture of a program.

In relation to the interview day experience [4], our results suggest that meeting the program director (N = 46; 82%) and engaging with residents in a social setting (N = 40; 71%) had a significant influence on respondents’ ranking decisions. These results and those of a similar article by Bates et al. [2] suggest that programs should not disregard the meaningfulness of socialization with potential colleagues and facetime with prospective leaders in structuring their recruitment season.

The third most important item to applicants before the virtual recruitment season, ranking in the desired geographic area [4], was constrained by the cost of travel. Virtual recruitment models created more equitable geographic access among applicants. Geary et al. noted that interview day attendance rates increased with fewer cancellations or no-shows [5]. Our respondents reported being able to apply to more programs because of the virtual interface (N = 37; 64%). One possible explanation is the potential reduction in economic burden for applicants. In prior years, the cost of interviewing at multiple programs — $1,000–$11,580 US dollars per applicant — was a limiting factor to some [3, 5]. The elimination of these expenses potentially leveled the playing field for those limited by economic constraints, geographic location, or clinical duties.

Despite increased equity in some aspects, respondents experienced challenges in other ways. Among them, finding an appropriate interview setting (n = 12; 20%), avoiding background noise (N = 12; 20%), having adequate internet connectivity (N = 9; 14%), and challenges with time zone differences (N = 5; 9%). One concern was how these challenges impacted applicants’ ability to portray their personality, but most of our respondents (N = 33; 58%) reported their personality was very well or perfectly portrayed. Only 2% (N = 2) answered that their personality was not portrayed at all.

Results also indicated inequalities in the resources each respondent had available, with about half of the respondents (N = 33; 57%) being offered an appropriate interview space with internet connectivity by their medical school. The heterogeneity of responses indicates that individual residency programs must work to avoid biased ranking based on these differences. Unconscious biases are a growing concern in graduate medical education and should be considered when structuring future recruitment seasons [3]. The AAMC has already designed guidelines and workshops to help tackle this problem [1].

This study carries its limitations. Given the small sample size and association with a single institution and specialty, its generalizability is limited. However, our results concurred with broader studies in different specialties [2, 5]. Because the survey was sent before the rank list closure date, a lower response rate may be due to fear of how participation might affect their ranking status and may have resulted in biased answers regardless of the confidentiality stated.

Recruitment in psychiatry may be specifically affected by virtual interviewing, as psychiatry as a field is predicated on the personal interaction of two people. Regardless of the platform used, ensuring an unhindered interview, where applicants and interviewers can interact without confounders, is critical in the selection process. The AAMC strongly recommends that programs collect data on their recruitment experience to develop evidence-based guidelines for future reference [1]. With further refinements, these changes to the recruitment process can be a solution to many past and current residency recruitment issues as we seek to mitigate the risk of conscious and unconscious bias and continue to provide equitable and well-informed choices to both applicants and residency programs.