Introduction

In 2014, of the nearly 1000 applicants for orthopaedic residency programs, approximately 30% failed to secure a position in the match. Needless to say, the competition is intense. This disparity between numbers of applicants and positions available lends itself to the often stressful and challenging process of selecting future orthopaedic surgeons from an increasingly outstanding pool of medical students. Annually, orthopaedic surgery program directors and faculty across institutions seek to cull from bulky applications those qualities that would translate into enthusiastic residents, top-notch clinicians, and productive researchers. The importance of this selection process has perhaps never seemed greater as our field is increasingly hearing a call to action to increase the visibility of our treatment algorithms, quantify their impact on patient outcomes, and architect new metrics for quality of patient care. In this regard, the medical students we are selecting now are those who will help navigate the future of our specialty and ensure continued growth and successful adaptation.

In 2002, Bernstein et al. queried 109 program directors across the country as to which applicant characteristics they felt were most important and the top three answers were an applicant’s away rotation at their institution, United States Medical Licensing Examination (USMLE) Step 1 score, and medical school ranking [5]. Although these traits might still be ranked highly by most program directors today as well, their association with success as an orthopaedic resident has not consistently been borne out by the existing data. In some studies, associations have been found between resident performance and USMLE Step 1 scores, Alpha Omega Alpha (AOA) medical honor society induction, and third- and fourth-year medical school clerkship grades, but these have not been found to be significant in other followup studies [4, 11, 13, 14, 17, 31]. Furthermore, the preresidency criteria in some studies have been found to be associated with faculty evaluations only and, in others, to American Board of Orthopaedic Surgery (ABOS) Part I scores [9, 16, 25]. To date, there is no clear and reproducible consensus in the literature as to which factors show the greatest correlation with respect to specific outcome measures of future performance as an orthopaedic resident. Thus, the criteria used in the selection process continue to be many, poorly understood as to their predictive utility, and challenging to apply to programs across the board [32].

The variability in findings among these studies reflects the idea that although some key associations have been identified, further work is required to either confirm or negate these findings from within different populations of residents. Our goal, therefore, was to contribute to current and available data in this regard by analyzing well-maintained data available to us for 10 resident classes. We sought to investigate which preresidency academic and clinical achievements correlated with resident success based on previously established outcome criteria and our institution’s attending rank system. In particular, we asked which preresidency selection criteria is associated or is correlated with (1) objective outcome measures of ABOS I and Orthopaedics In-Training Exam (OITE) scores; and (2) subjective ratings of global evaluations and faculty rankings.

Materials and Methods

After obtaining institutional review board approval, we retrospectively reviewed charts for preresidency and residency data of all residents over the past 10 years (2001–2010) who matched at our institution for the orthopaedic surgical residency program. Individuals who completed the 5-year training are the participants in this study. Exclusion criteria included individuals who did not complete the 5-year program to full term. Data were gathered by an author of the study (TR) who was not familiar with any of the residents evaluated. A total of 61 orthopaedic residents were enrolled at our program over 10 years. One resident was excluded from analysis because the full 5-year term for the program was not completed. This left a total of 60 residents analyzed in this study, including 56 men and four women.

Input Measures: Preresidency Selection Factors

Preresidency selection factors were chosen for each of the 60 residents consistent with previously published studies [4, 9, 15, 16, 25, 26]. Selection criteria included: (1) USMLE Step 1 score; (2) USMLE Step 2 score; (3) Medical College Admission Test (MCAT) score; (4) number of clerkship honors; (5) number of letters of recommendation; (6) number of away rotations; (7) AOA membership; (8) completed subinternship at our institution; and (9) number of publications/abstracts. Five residents were foreign medical graduates and therefore were excluded from analysis involving MCAT scores, AOA membership, and clerkship honors. Only published papers and presented abstracts were counted as a “publication.” The number of published papers/abstracts was taken from Electron Residency Application Service as reported by the applicants.

Outcome Measures: Residency Performance

The objective outcome measures used in this study were ABOS Part I scores and OITE scores when taken during the fifth and last year of the residency. The subjective outcome measures were the global evaluation scores as filled out by faculty and faculty ranking of the residents within each residency class (rankings ranging from 1 to 6). During each year of residency, multiple faculty members are asked to fill out global evaluations to rate the quality and performance of residents. The global evaluations assess core competencies including (1) interpersonal and communication skills; (2) medical knowledge; (3) patient care and procedural skills; (4) teaching skills; and (5) personal appearance. These evaluations encompass many parameters of the core competencies set forth by the Accreditation Council for Graduate Medical Education. For each category in the global evaluations, faculty members rated the residents using a 5-point scale with 5 being the best score and 1 being the worst. We chose to analyze global evaluations given during the last 6 months of residency to accurately gauge the final outcome of each resident’s performance. We felt that this time period encompassed the culmination of the skills and performance level attained just before entering practice or fellowship. Faculty scores were averaged to provide a single rating for each competency per resident, and the competency scores in the different sections were averaged together for each resident to calculate one “total average” score. In other words, each resident had a score for each of the five competencies as well as a “total average” score, which was simply the average of the five competency scores. The other subject outcome measure was faculty rankings. Five senior faculty members who had worked with each of the 60 residents who completed their orthopaedic surgical residencies at our institution were asked to rank each resident within his or her class. These faculty members were asked to rank residents within each class using a subjective aggregate of their medical knowledge, patient care skills, communication skills, and professionalism. Residents were assigned a ranking from 1 (best in class) to 6 ranking (with 6 being the total number of residents within a given class year). An average rank was calculated for each of the residents to use this score for correlation analysis. The residency director and the department chair (both of whom would be familiar with the preresidency criteria of the residents) did not participate in the faculty rankings. Rater reliability was assessed yielding an intraclass correlation coefficient of 0.62 for single measures and 0.89 for average measures.

Statistical Analysis

Statistical analysis was performed with IBM SPSS software (Armonk, NY, USA) at our institution. For the objective outcome measures, bivariate linear correlation analysis was used to determine if any correlations exist between the objective measures and preresidency criteria. Point biserial correlation analysis was performed to analyze correlations between objective measures with AOA membership and completion of subinternship at our institution because those input measures are dichotomous. Mann-Whitney tests were also used to determine whether AOA membership had any associations with the subjective outcome measures; rank-biserial correlation coefficients were calculated when appropriate. Probability values were computed for each variable for all available data and a p value of < 0.05 was considered significant.

Results

Objective Measures

We found that USMLE Step 2 scores, MCAT scores, and number of honors in clerkships had positive linear correlations with ABOS Part I scores (Fig. 1). USMLE Step 2 scores and number of honors in clerkships had moderate positive linear correlations with ABOS I scores (r = 0.55, p < 0.001 and r = 0.45, p < 0.001, respectively). MCAT scores had a weaker positive linear correlation with ABOS I scores (r = 0.36, p = 0.008). No other preresidency criteria correlated with ABOS Part I scores (Table 1).

Fig. 1A–C
figure 1

The following three correlation graphs depict the significant correlations between different preresidency criteria (as presented on the x-axis) and ABOS Part I scores (y-axis). Pertinent r values and p values are shown for each graph. (A) USMLE Step 2 and (B) number of honors in clerkships moderately correlated with ABOS Part I scores, whereas (C) MCAT scores showed a weaker correlation.

Full size image
Table 1 Preresidency criteria and their correlations with American Board of Orthopaedic Surgery Part I scores
Full size table

For OITE score of the senior year, only weak positive correlations were found with USMLE Step 2 and number of honors in clerkships (r = 0.29, p = 0.02 and r = 0.35, p = 0.009, respectively; Fig. 2). No other preresidency criteria correlated with OTIE scores (Table 2).

Fig. 2A–B
figure 2

The following two correlation graphs depict the significant correlations between preresidency criteria on the x-axis ([A] USMLE Step 2 and [B] number of honors in clerkships) and OITE scores (y-axis). USMLE Step 2 and number of honors in clerkships were weakly correlated with OITE scores.

Full size image
Table 2 Preresidency criteria and their correlations with Orthopaedic In-Training Exam scores
Full size table

Subjective Measures

Higher total average global evaluation scores were associated with AOA membership. Residents with AOA had a median score of 3.94 (range, 3.45–4.28) as compared with a median score of 3.61 (range, 2.95–4.24; p = 0.005). A strong correlation was found between AOA membership and total average global evaluation scores with a with a rank-biserial correlation coefficient of r = 0.60 (Table 3). As shown in Table 3, AOA membership was associated with higher scores within each competency or subsection of the global evaluation with the most significant correlation being within the “interpersonal and communication skills” subsection (r = 0.67, p = 0.002). No other preresidency criteria correlated with the global evaluations (Table 4).

Table 3 Alpha Omega Alpha (AOA) membership was associated with higher scores on the global evaluation as determined by Mann-Whitney test*
Full size table
Table 4 Preresidency criteria and their correlations with global evaluation scores
Full size table

The other subjective outcome measure, faculty rankings, did not correlate with any preresidency criteria (Table 5).

Table 5 Preresidency criteria and their correlations with faculty rankings
Full size table

Discussion

Matching into an orthopaedic residency has been an exceedingly competitive task year after year. Program directors and selection committees have the daunting task of reading or skimming through hundreds of applicants’ selection criteria every year, including but not limited to USMLE Step 1, USMLE Step 2, AOA membership, grades in medical school, research activities, and letters of recommendation; the selection criteria used, or the way they are prioritized, may vary across different programs. Based on these criteria, programs must then rank the applicants based on whom they believe will be best fitting and most successful in their program. Finding which selection criteria are best correlated or associated with resident success can expedite the process and may provide some helpful information for those involved in selecting applicants. The assessment of preresidency selection criteria and success as a resident has been investigated within several specialties, generating many different results and conclusions as will be discussed [1–3, 6, 8, 10, 12, 15, 18–24, 27, 28, 30, 31]. Our study attempts to elucidate any selection criteria that may be predictive for orthopaedic resident success (in objective and subjective measures) because there is no standardized way of choosing applicants and every program seems to have its own preference of carrying out the process. Our specific aims were to take the nine preresidency selection criteria as described in the Materials and Methods section and analyze whether they correlate with (1) objective measures of resident performance (ABOS Part I and OITE); and (2) subjective measures of resident performance (global evaluation scores and faculty rankings). For the objective measures, we found that ABOS Part I scores had moderate positive linear correlations with USMLE Step 2 and honors in clerkships (r = 0.55, p < 0.001 and r = 0.45, p < 0.001, respectively) and a weak positive linear correlation with MCAT scores (r = 0.36, p = 0.008; Fig. 1). The other objective measure, namely, OITE scores, had only weak positive linear correlations with USMLE Step 2 scores and honors in clerkships (r = 0.29, p = 0.02 and r = 0.35, p = 0.009, respectively; Fig. 2). For the subjective measures, AOA membership was associated with higher total global evaluation scores (p = 0.005) as well as higher scores within each section (Table 1).

Limitations of this study include missing data points for the five foreign graduates as explained in the Materials and Methods section. These missing data decreased our sample size for the respective bivariate correlation analysis involving these missing variables. In general, our sample size of 60 residents may have led to missing some small but potentially meaningful differences in our data. Although surgical skills are a component of the “patient care and procedural skills” competency on the global evaluations, there was no assessment criteria that were specific to surgical skills alone. Rating of surgical skills also played a role in the subjective criteria of faculty ranking. Another consideration is that this study was only examining the residents in our program, which is a relatively large, academic-based orthopaedic residency program in a Level I trauma center located in an urban setting. Different types of residency programs (location/setting, academic/community, size of program) may impact residency success or may appeal to a certain type of applicant. Lastly, our faculty ranking system only ranked the residents within each graduating class from 1 (best in class) to 6. This system lacked year-to-year consistency because, for example, the first ranked resident in one graduating class cannot be reliably compared with the first ranked resident in another class.

In summary, our study found that in terms of objective outcomes, USMLE Step 2, and honors in clerkships moderately correlated with ABOS Part I scores, whereas MCAT scores weakly correlated with ABOS Part I scores. USMLE Step 2 and honors in clerkships also weakly correlated with OITE scores. Some studies have found the number of honors received in clinical clerkships and AOA membership to have a positive correlation with faculty evaluations; they did not find a correlation with OITE or ABOS scores [25]. Other studies have found positive correlations with USMLE Step I and ABOS Part I scores as well as USMLE Step 2 scores and performance as residents [9, 16]. In contrast, other analysis of preresidency selection factors including USMLE scores, AOA membership, research publications, age at entering residency, marital status, and medical school affiliation showed that only an applicant’s USMLE Step 1 score and marital status correlated with the OITE scores [26]. In the largest study to date, Spitzer et al. in a retrospective review of 147 residents found that preresidency selection factors such as medical school rank and scores on the USMLE Part 1 had a positive correlation with higher scores on the OITE [29]. Turner et al. [31] looked at predictive value of a composite scoring tool using weighted scores for medical school clerkship grades, AOA status, and USMLE Step 1 scores, among other criteria, and found it to be predictive of ABOS written and oral examinations as well as OITE scores (p < 0.001). They also found that USMLE Step 1, medical school clerkship scores, and AOA membership were associated with OITE results, but no correlations were found with USMLE Step 2 scores. In our study, the correlation between USMLE Step 2 scores with ABOS Part I scores is an interesting observation because often times interviews for residency are conducted without program directors having results for the USMLE Step 2 available if the test had not been taken before the interview. This highlights the potential importance of the USMLE Step 2 as an indicator for success in objective measures during residency. As compared with most other studies, our findings show that USMLE Step 2 correlates more strongly with objective measures of resident success (as compared with USMLE Step 1 versus objective measures) and suggests that program directors should request scores for this examination before granting an interview for Step 2 to be used as a residency selection criteria among others.

In summary, our study found that in terms of subjective outcomes, AOA membership was associated with higher total global evaluation scores and higher scores within each competency of the evaluation. The strongest correlation was with the section of “interpersonal and communication skills.” There have been several studies that attempt to correlate preresidency criteria with subjective outcome measures and these studies have produced variable results with some studies failing to show any correlation [7]. Spitzer et al. evaluate subjective outcomes in a manner similar to our study, which took into account patient care skills, medical knowledge, surgical skills, interpersonal relationships, and system-based practices; however, they did not find any positive correlations. In one of the first studies examining an association between preresidency selection criteria and subjective performance as an orthopaedic resident, Dirschl et al. [18] found that the number of honors grades in the third- and fourth-year medical school core clinical clerkships correlated with higher faculty ratings, which include sections on overall, cognitive, affective, and psychomotor skills. In our study, honors in clerkships did not correlate with subjective outcomes but only correlated with objective outcomes; only AOA membership was associated with higher global evaluation scores. Our faculty rankings failed to show any correlations with any of the preresidency criteria.

As orthopaedic surgery continues to draw highly successfully candidates and evolves as a competitive field for the match process, distinguishing optimal prospective candidates may continue to become more difficult. Our study has provided a basis to continue to improve on our own selection criteria, and each institution should consider analyzing their interview and selection process because this yearly ritual is far from standardized. Future studies are still needed to build on our results. Overall, we found that honors in clerkships, USMLE Step 2, and AOA membership were the preresidency selection criteria that correlated best with outcome measures of resident performance. As commonly perceived, USMLE Step 1 scores are used in some programs to screen applicants for a minimum score and that score may be used by programs when considering ranking applicants for the match. Our results suggest that more consideration be given to USMLE Step 2 scores, clerkship honor grades, and AOA membership because these factors correlated best with our measures of resident performance. This has implications for the residency application process because many applicants do not have a Step 2 score at the time of the interview; our data support the use of this score as a valuable tool for program directors.