Abstract
Purpose
To examine the characteristics between virtual multiple mini-interview (vMMI) and in-person interviews (ipMMI) in regard to difference in performance between applicant-reported gender identity and racial groups.
Methods
Retrospective multiple mini-interview (MMI) data from two vMMI interview cycles (2021 and 2022) consisting of 627 applicants and four ipMMI cycles (2017–2020) consisting of 2248 applicants. Comparisons were made between applicant subgroups including reported gender (male and female) and minority status (URiM and non-URiM). A three-way analysis of variance (ANOVA) was conducted to examine the effects of gender, URiM status, and interview modality (in-person vs. virtual) on MMI scores.
Results
There were no overall significant differences between annual ipMMI and vMMI scores. A significant main effect of gender was observed, with females scoring higher than males overall. An interaction between gender and URiM status was also found. Although not statistically significant, when the MMI was virtual, URiM applicants on average scored higher than non-URiM applicants. In both the ipMMI and vMMI, URiM males tended to score lower than their non-URiM counterparts, though this difference was not statistically significant. URiM females tended to score higher than non-URiM females during the vMMI, and this difference was statistically significant.
Conclusions
The switch to vMMI shows that there are no overall significant differences between the in-person and virtual formats; however, the finding that female URiM’s better performance in the virtual setting is novel. The cause of this finding is unknown but most likely reflects the complex interaction between race and gender. This insight requires future study and builds on the evidence that the MMI is an admissions tool to mitigate bias.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
Introduction
An interview is a required part of nearly all medical school admissions processes. With a few exceptions, prior to the COVID-19 pandemic, most interviews were conducted in person and consisted of either an unstructured one-on-one interview or a structured interview such as the multiple mini-interview (MMI). To protect both applicants and medical school staff during the COVID-19 pandemic, all medical school interviews went to a virtual format, starting either mid-cycle in 2019–2020 or at the beginning of the 2020–2021 admissions cycle[1, 2]. The transition required either the use of existing video conferencing technologies (e.g., Zoom, Microsoft Teams), the creation of an in-house system, or the purchase of a virtual interview system from a third-party vendor. Virtual interviews for medical school admissions were not unknown prior to the COVID-19 pandemic. A study published in 2018 placed the number of virtual interviews at US allopathic medical schools at 10% [3]. Virtual interviews for graduate medical education (i.e., residency and fellowship) were already being used by many programs prior to the pandemic [4]. In May of 2022, the Association of American Medical Colleges (AAMC) recommended the continuation of virtual residency interviews [5]. Several studies performed by residency and fellowship programs reported satisfaction with the virtual interview format [6,7,8]; however, some surveys found that in-person fellowship interviews were preferred by a majority applicants since they thought they could better evaluate a program’s facilities and meet faculty [9, 10].
Despite the associated costs and extra time required to learn, create, and/or purchase a system, several advantages seemed to emerge, for both the applicant and the medical school. Personnel performing interviews enjoyed greater flexibility, and there were possible cost savings for the institution [11, 12]. For the applicant, costs associated with travel, lodging, and time off from work decreased significantly or disappeared entirely. Lowering the financial barriers is believed to provide a more equitable medical school admissions process and greater access to medical school programs, including those that may have been out of reach for some applicants [13]. Decreasing the environmental impact of travel have also been seen as a reason to conduct virtual interviews [14, 15]. For these reasons, the AAMC has recommended that medical schools use virtual interviews with an optional in-person visit after acceptance [16].
Despite the potential advantages for the applicant, there were concerns about virtual interviews. One of the most concerning was exacerbating existing implicit and explicit biases that disproportionately affect applicants have been unrepresented in medicine (URiM) [17,18,19]. There were also concerns for new sources of potential bias including decreased access to adequate computer resources and Internet access, the difficulty in some facial recognition software recognizing darker skin, and the potential negative effect on interviewers observing applicants’ personal home or work environments seen through the video feed [17, 20].
Few studies in the pre-COVID-19 era directly compared virtual and in-person medical school interviews, and only one used virtual MMI (vMMI) as the interview format [21]. The University of New Mexico School of Medicine described their experience using a video-based platform in the pre-COVID-19 era but used a semi-structured interview format [3]. Several medical schools, both in and outside the USA, have published their experience of switching from in-person MMI (ipMMI) to vMMI, and none of them commented on the differences between the URiM applicants and the rest of the interview pool [11, 22,23,24,25,26,27]. Cork looked at Zoom vMMI and ipMMI and found the delivery of the MMI had comparable scoring [24]. Hammond and colleagues compared a synchronous vMMI using Zoom and their pre-COVID ipMMI for pharmacy school admissions and found that the vMMI was a reliable alternative to the ipMMI [28]. These forementioned studies did not specifically investigate race or gender as potential differences in the vMMI and ipMMI performance. This lack of comparison between vMMI and ipMMI performance across demographic factors is surprising given well-established literature on interview and rater bias and several articles describing how bias may manifest differently in the virtual setting [17, 18, 29,30,31]. This study describes a completely automated, synchronous MMI system using the same rater user interface for both vMMI and ipMMI with special attention paid to differences between different populations of interviewees.
Methods
The vMMI was developed using .NET (.NET Foundation, Richmond, WA) and integration with Microsoft Teams web version (Microsoft Corporation, Richmond, WA). An in-house developed student information system, MedOneStop, was used as an admissions interface for applicants, admissions staff, and raters. Both the vMMI and ipMMI had applicants rotate through eight stations, each with a different rater (interviewer) (Fig. 1). The same bank of MMI scenarios were used in both the ipMMI and vMMI. Scenarios that required interaction between two applicants were not used in the vMMI. Actors, either members of the community or medical students, were used in the ipMMI, but during the vMMI, the rater assumed the actor’s role. The raters were recruited primarily from the faculty, staff, and medical students at the University of Cincinnati College of Medicine (UCCOM) and Cincinnati Children’s Hospital Medical Center. First-time raters were required to attend a 1-hour online training session in addition to a general admission’s training session.
There were usually two vMMI sessions per interview day. To avoid common pitfalls such as blocked pop-ups, camera, and microphone issues, applicants and raters were required to complete a computer and video systems check within 24 hours of a vMMI. Raters were instructed to enter “Rater” as their name for anonymity. A photo of the student was provided on-screen for applicant reference and identity verification. The raters were blinded to the applicant’s American Medical College Application Service (AMCAS) and UCCOM secondary applications.
Each group (raters and applicants) met in separate initial Teams meetings with the admissions staff prior to the start of the vMMI to verify attendance and system operation. After the initial meeting and the official time began, applicants were shown their first station scenario for a set period (3 min). At this point, the applicant’s and rater’s browser would automatically open a Teams window for the interview to begin (6 min). At the end of the interview, the windows closed automatically, and the applicant was shown their next scenario (3 min). During this 3-min interval, the raters had the option to evaluate and assign a score to the applicant. After this period, the next station interview automatically began. The cycle continued until all applicants had rotated through eight stations. The process was automatic and could not be stopped for any one applicant. If at any point during a virtual interview an applicant or a rater experienced technical issues, they could click a button to message for help from IT support.
The ipMMI took place in the UCCOM’s Simulation Center, and typically two MMI sessions were done per interview day. After meeting with the admissions staff, applicants waited outside an individual simulation room door. They had 3 min to review the MMI scenario that was attached to the door. Guided by prompts from an overhead intercom, the applicant entered the room, and the interview began (6 min). At the end of 6 min, the applicant was prompted to leave the room and move to the next door where they had 3 min to read the next MMI scenario. The process was automatic and could not be stopped for any one applicant.
The same MedOneStop rater interface was used in the vMMI and the ipMMI. Both MMI modalities were cored using a Likert scale with 5 being the highest score and 1 being the lowest. The raters were trained to score towards the mean. The raters had the option to evaluate and score the applicant during the three-minute interval or within 24 hours from the start of the interview. The rater’s scores were adjusted based on their overall scoring over the course of the interview cycle.
Virtual adjusted annual MMI scores from medical school applicants in the 2021 and 2022 admissions cycles (N = 627) were compared with ipMMI scores from 2017 to 2020 (N = 2248) (Table 1). Additional comparisons were made between applicant subgroups including reported gender applicant-reported gender identity (male and female) and minority status (URiM and non-URiM). We used the AAMC definition of URiM as “any U.S. citizen or permanent resident who self-identified as one or more of the following race/ethnicity categories (alone or in combination with any other race/ethnicity category): American Indian or Alaska Native; Black or African American; Hispanic, Latino, or of Spanish Origin; or Native Hawaiian or Other Pacific Islander” [32]. The applicants select their race/ethnicity when completing the American Medical College Application Service® (AMCAS®) application.
Comparisons were made between applicant subgroups including reported gender (male and female) and minority status (URiM and non-URiM). To analyze the data, we conducted a three-way analysis of variance (ANOVA) with gender, URiM status, and application year group (in-person vs. virtual) as independent variables and the adjusted MMI score as the dependent variable. This approach allowed us to examine main effects and interactions between these factors. While Likert scale data are ordinal, the large sample size and the tendency for scores to be distributed around the mean justify the use of parametric statistics in this case. Levene’s test was used to assess the homogeneity of variances across groups. The UCCOM institution review board approved this research (2021–1032), including waiver of consent and the minimal risk to participants.
Results
The three-way ANOVA revealed several significant findings regarding the effects of gender, URiM status, and application year group on MMI scores (Tables 2 and 3). Examining main effects, we found a significant influence of gender (F(3, 3663) = 25.707, p < 0.001), with females (M = 0.066, SD = 0.417) scoring higher on average than males (M = − 0.066, SD = 0.428). However, neither URiM status (F(1, 3663) = 0.063, p = 0.802) nor application year group (F(1, 3663) = 0.139, p = 0.710) showed significant main effects, indicating no overall differences between URiM and non-URiM applicants or between in-person and virtual MMI scores.
We observed a significant interaction between gender and URiM status (F(1, 3663) = 4.667, p = 0.031). URiM females tended to score higher (M = 0.087, SD = 0.434) compared with non-URiM females (M = 0.060, SD = 0.411), while URiM males tended to score lower (M = − 0.096, SD = 0.452) compared with non-URiM males (M = − 0.056, SD = 0.420).
We found no significant interactions between gender and application year group (F(2, 3663) = 0.087, p = 0.917) or between URiM status and application year group (F(1, 3663) = 0.155, p = 0.694). Additionally, the three-way interaction between gender, URiM status, and application year group was not significant (F(1, 3663) = 0.570, p = 0.450).
It’s worth noting that Levene’s test indicated unequal variances across groups (F(11, 3663) = 1.990, p = 0.026). While this suggests some caution in interpreting the results, the large sample size and the robustness of ANOVA to moderate violations of homogeneity of variance lend credibility to our findings.
Discussion
Virtual interviews were being used extensively in job recruitment before the COVID-19 pandemic. A 2015 survey done by FutureStep, a recruitment process outsourcing and professional search company, found that 75% of companies used real time (synchronous) video interviews[33]. Despite the ubiquitous utilization of virtual interviews since the start of the COVID-19 pandemic, there is very little literature examining the role of bias specifically in virtual interviews. The AAMC introduced the Standardized Video Interview (SVI) as an online, asynchronous video interview designed to holistically assess applicants’ proficiency in two of the Accreditation Council for Graduate Medical Education (ACGME) competencies of interpersonal/communication skills and professionalism in the emergency medicine field [34]. After 4 years, the AAMC discontinued the SVI due to the lack of interest and the challenge of scaling the SVI to the multiple residencies [35]. A review of the data showed that SVI scores slightly favored Black applicants compared with white applicants and found no difference between Latino and Asian applicants compared to white applicants [36]. While female applicants did better than male applicants, there was no mention about how Black female applicants did as a separate category.
Our analysis revealed a significant main effect of gender on MMI performance, with females consistently outperforming males across both in-person and virtual formats. This finding aligns with previous research [17, 19, 37] and warrants further investigation into the underlying factors contributing to this gender difference in MMI performance.
The significant interaction between gender and URiM status provides new insights into the complexities of demographic factors in MMI performance. Notably, URiM females performed particularly well in the virtual MMI format, outscoring their non-URiM counterparts. This finding challenges concerns that virtual interviews might disadvantage URiM applicants and suggests that the virtual format may actually benefit certain demographic groups.
Contrary to some concerns raised in the literature, we found no significant main effect or interactions involving the application year group (in-person vs. virtual). This suggests that the transition to virtual MMIs did not systematically advantage or disadvantage any particular group, supporting the validity of virtual MMIs as an alternative to in-person interviews.
To our knowledge, this is the first study looking at within group and between group differences of a synchronous, virtual, and in-person MMI and how the two modalities affect applicant scoring between demographic groups such as race and gender. Female applicants performing better than their male counterparts in both MMI and non-MMI in-person interviews is well documented in the literature [38,39,40]. Other studies have reported that URiM status did not affect MMI performance [38, 41].
The results of the SVI and this study both showed that Black applicants did better than their white counterparts in these video interviews, although the SVI was asynchronous and our vMMI is synchronous. The underlying reason for this finding is uncertain. There is no literature delving into the reasons why Black females would have better performance in a virtual interview. It should be noted that in a large multiple national study of in-person MMI and traditional interviews, 4993 interviewees who underwent 7516 interviews, there was no association with URiM status and interview performance; however, these interviews were conducted in-person [38].
Several limitations should be considered when interpreting these results. First, while we used parametric statistics (ANOVA) with Likert scale data, this approach is justified by our large sample size and the tendency for scores to distribute around the mean. However, future studies might consider alternative statistical approaches specifically designed for ordinal data.
Second, Levene’s test indicated unequal variances across groups. While ANOVA is generally robust to moderate violations of homogeneity of variance, especially with large and balanced samples, this should be considered when interpreting the results.
Finally, this study was conducted at a single institution, which may limit the generalizability of our findings. Multi-institutional studies would be valuable to confirm these results across different settings and populations.
Conclusions
The transformation from in-person multiple mini-interviews (ipMMI) to virtual multiple mini-interviews (vMMI) in medical school admissions amid the COVID-19 pandemic offered an unanticipated opportunity to examine biases and differences within a complex evaluation framework. Our study, conducted at the University of Cincinnati College of Medicine, stands as the first to describe and compare both vMMI and ipMMI modalities, with an emphasis on their effect on different demographic groups. Our study demonstrates that the transition from in-person to virtual MMIs did not introduce significant biases or alter performance patterns across demographic groups. The consistent gender effect and the complex interaction between gender and URiM status highlight the need for continued research into factors influencing MMI performance. These findings support the use of virtual MMIs as a viable alternative to in-person interviews, potentially offering a more accessible and equitable admissions process. Future research should focus on understanding the underlying mechanisms of these demographic differences and exploring ways to ensure fair assessment across all applicant groups.
Data Availability
The data that support the findings of this study are not openly available due to reasons of sensitivity and privacy of the interviewees and raters.
References
Weiner, S. Shortened MCAT® exams, extended AMCAS® deadlines: how the pandemic has upended medical school admissions. March 14, 2023]; Available from: https://www.aamc.org/news-insights/shortened-mcat-exams-extended-amcas-deadlines-how-pandemic-has-upended-medical-school-admissions.
The Coalition for Physician Accountability’s Work Group on Medical Students in the Class of 2021 Moving Across Institutions for Post Graduate Training. Final Report and Recommendations for Medical Education Institutions of LCME-Accredited, U.S. Osteopathic, and Non-U.S. Medical School Applicants 2020 March 10, 2023]; Available from: https://www.aamc.org/system/files/2020-05/covid19_Final_Recommendations_Executive%20Summary_Final_05112020.pdf.
Ballejos MP, et al. An equivalence study of interview platform: does videoconference technology impact medical school acceptance rates of different groups? Adv Health Sci Educ Theory Pract. 2018;23(3):601–10.
Selvam R, et al. Video-based interviewing in medicine: a scoping review. Syst Rev. 2022;11(1):94.
Association of American Medical Colleges. AAMC Interview Guidance for the 2022–2023 Residency Cycle. February 16, 2023]; Available from: https://www.aamc.org/about-us/mission-areas/medical-education/aamc-interview-guidance-2022-2023-residency-cycle.
Keister DM, et al. The impact of virtual interviews on recruitment and implicit bias. Fam Med. 2022;54(10):769–75.
Ponterio JM, Keslar M, Lakhi NA. The virtual interview format for fellowship recruitment in obstetrics and gynecology: a nationwide survey of program directors. Med Educ Online. 2022;27(1):2054304.
Moran SK, et al. Should radiology residency interviews remain virtual? Results of a multi-institutional survey inform the debate. Acad Radiol. 2022;29(10):1595–607.
Besagar S, Patel S, Vajzovic L, Scott AW, Ji X, Chen Q, Finn A. Applicant perceptions of in-person versus virtual interview format for surgical retina fellowship. J Academic Ophthalmol. 2023;15(02):e271-5.
Tanguilig G, Kraeutler MJ, Mulcahey MK. Virtual interviews are not adequate replacements for in-person interviews for orthopaedic sports medicine fellowship program directors and applicants. Arthrosc Sports Med Rehabil. 2024;6(1):100835.
Carroll Turpin MA, et al. Rapid transition to a virtual multiple mini-interview admissions process: a new medical school’s experience during the COVID-19 pandemic. Acad Med. 2021;96(8):1152–5.
Edje L, et al. Using Skype as an alternative for residency selection interviews. J Grad Med Educ. 2013;5(3):503–5.
Millo L, Ho N, Ubel PA. The cost of applying to medical school - a barrier to diversifying the profession. N Engl J Med. 2019;381(16):1505–8.
Hampshire K, Shirley H, Teherani A. Interview without harm: reimagining medical training’s financially and environmentally costly interview practices. Acad Med. 2023;98(2):171–4.
Domingo A, et al. The carbon footprint and cost of virtual residency interviews. J Grad Med Educ. 2023;15(1):112–6.
Association of American Medical Colleges. Interviews in UME: where do we go from here? [cited 2023 August, 9 2023]; Available from: https://www.aamc.org/about-us/mission-areas/medical-education/interviews-ume-where-do-we-go-here.
Marbin J, Hutchinson YV, Schaeffer S. Avoiding the virtual pitfall: identifying and mitigating biases in graduate medical education videoconference interviews. Acad Med. 2021;96(8):1120–4.
Otugo O, et al. Bias in recruitment: a focus on virtual interviews and holistic review to advance diversity. AEM Educ Train. 2021;5(Suppl 1):S135–9.
Douglas A, Hendrix J. Black medical student considerations in the era of virtual interviews. Ann Surg. 2021;274(2):232–3.
Fuchs JW, Youmans QR. Mitigating bias in the era of virtual residency and fellowship interviews. J Grad Med Educ. 2020;12(6):674–7.
Tiller D, et al. Internet-based multiple mini-interviews for candidate selection for graduate entry programmes. Med Educ. 2013;47(8):801–10.
Eveland AP, et al. The virtues of the virtual medical school interview. Med Educ Online. 2021;26(1):1992820.
Kok KY, et al. Conducting multiple mini-interviews in the midst of COVID-19 pandemic. Med Educ Online. 2021;26(1):1891610.
Cork SC. A model for online delivery of multiple mini interviews. J Med Educ Curric Dev. 2023;10:23821205231183876.
Domes T, et al. Conducting a synchronous virtual multiple mini-interview using Webex for medical school admissions. Can Med Educ J. 2021;12(6):120–2.
Abraham Z, Melro C, Burm S. ‘Click, i guess i’m done’: applicants’ and assessors’ experiences transitioning to a virtual multiple mini interview format. Perspect Med Educ. 2023;12(1):594–602.
Sabesan V, et al. Effects of candidates’ demographics and evaluation of the virtual multiple mini interview (vMMI) as a tool for selection into paediatric training in Queensland. Med Teach. 2023;45(10):1148–54.
Hammond S, McLaughlin JE, Cox WC. Validity evidence for a virtual multiple mini interview at a pharmacy program. BMC Med Educ. 2023;23(1):551.
Cooke EA, et al. Virtual Interviewing for Radiology Residency: Pluses, Minuses, and A Review of the Literature. Acad Radiol. 2023;30(4):603–16.
Flores LE, et al. Three (more) tips to reduce socioeconomic bias in virtual anesthesiology interviews. Can J Anaesth. 2021;68(8):1296–7.
Pourmand A, et al. Feasibility and Usability of Tele-interview for Medical Residency Interview. West J Emerg Med. 2018;19(1):80–6.
Association of American Medical Colleges. FACTS Glossary June 30, 2024]; Available from: https://www.aamc.org/data-reports/students-residents/data/facts-glossary.
Maurer, R. Use of video for recruiting continues to grow. 2015 [cited 2023 February 10]; Available from: https://www.shrm.org/resourcesandtools/hr-topics/talent-acquisition/pages/use-video-recruiting-grow.aspx.
Bird SB, et al. Innovation in residency selection: the AAMC standardized video interview. Acad Med. 2019;94(10):1489–97.
Gallahue FE, et al. The AAMC standardized video interview: lessons learned from the residency selection process. Acad Med. 2020;95(11):1639–42.
Hopson LR, et al. The AAMC standardized video interview and the electronic standardized letter of evaluation in emergency medicine: a comparison of performance characteristics. Acad Med. 2019;94(10):1513–21.
Nwora C, Allred DB, Verduzco-Gutierrez M. Mitigating bias in virtual interviews for applicants who are underrepresented in medicine. J Natl Med Assoc. 2021;113(1):74–6.
Henderson MC, et al. Medical school applicant characteristics associated with performance in multiple mini-interviews versus traditional interviews: a multi-institutional study. Acad Med. 2018;93(7):1029–34.
Kennedy AB, et al. More than their test scores: redefining success with multiple mini-interviews. Med Sci Educ. 2020;30(3):1049–60.
Ross M, et al. Are female applicants rated higher than males on the multiple mini-interview? Findings From the University of Calgary. Acad Med. 2017;92(6):841–6.
Jerant A, et al. How medical school applicant race, ethnicity, and socioeconomic status relate to multiple mini-interview-based admissions outcomes: findings from one medical school. Acad Med. 2015;90(12):1667–74.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interest
The authors declare no competing interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Grier, D.D., Turner, L., Prichard, T.J. et al. Virtual and In-Person Multiple Mini-interviews: A Comparison of Two Modalities in Regard to Bias. Med.Sci.Educ. (2024). https://doi.org/10.1007/s40670-024-02142-5
Accepted:
Published:
DOI: https://doi.org/10.1007/s40670-024-02142-5