Introduction

Postgraduate training in surgery continues to be a rigorous, challenging, and competitive period in the career of a surgeon. In 2022, 353 training programs with 9299 trainees produced 1394 graduates. This represents about 6% of the total GME trainees in the US. Overall, between 2017 and 2022, the number of surgical trainees has increased by 9.7% [1•]. Amongst these programs, there is considerable heterogeneity in location, type and size of hospital, and clinical environment. As prospective trainees (including medical students and preliminary residents) consider which programs are suitable for them, there are several resources (both public and private) available that can provide useful data and information. Included in this review are accrediting, certifying, and specialty-specific organizations, along with crowd-sourced collaboratives.

Information from Accrediting Organizations

Accreditation of training programs has been carried out by several different organizations over the last century [2]. In 1914, the American Medical Association’s Council on Medical Education first set standards for hospital internship programs, which included 603 hospitals and 3095 positions. In 1920, this Council became the Council on Medical Education and Hospitals (CMEH). In 1927, the CMEH published the first list of hospitals approved for residency training. In 1939, the American College of Surgeons (ACS) established its own criteria and approval process for graduate training in surgery [3, 4]. These two systems operated in parallel until 1950 when the Conference Committee on Graduate Training in Surgery, a collaboration among the American Board of Surgery, the ACS, and the American Medical Association (AMA) was established [5]. An important milestone in GME financing came in 1965 when Medicare and Medicaid were created and began to fund GME positions throughout the US. Currently, over $15 billion in federal funding supports residency training [6]. The Accreditation Council for Graduate Medical Education (ACGME) was established as a council within the AMA in 1981. In 2000, with consensus from the academic GME community that an independent organization was required to accredit training programs, the ACGME (in its current form) was founded in Chicago, IL as an independent, non-governmental, non-profit organization. In 2014, the ACGME integrated osteopathic programs into its accreditation structure [7]. Within the ACGME, the Surgery Residency Review Committee (RRC) is responsible for accreditation of all US general surgery training programs. Members of the RRC are solicited from ACGME member organizations (AMA, ABS, AOA, ACS), the surgical community, and include one public and one resident member [8].

The ACGME has a number of evaluation tools to measure residency program quality. Publicly available information includes historical and current data regarding program accreditation. Privately available information is provided to program leadership to identify areas for improvement and weaknesses in curricula specifically in relation to achieving and maintaining the requirements set forth by ACGME.

Letter of Notification

The mission of the RRC in Surgery is to ensure that training programs meet accreditation standards. Annually, the Surgery RRC reviews each program’s site visit data, certification examination pass rates, faculty and resident surveys, and Accreditation Data System (ADS) updates which includes major changes, faculty roster, faculty scholarly activity, and case log data [9]. Based on available information provided by the program, the RC generates a Letter of Notification (LoN) that includes the program’s accreditation status, resident complement, citations, areas for improvement, and commendation, if awarded. This information is given directly to programs and their Sponsoring Institution to help improve the program’s and/or Sponsoring Institution’s ability to meet ACGME’s requirements. Citations are given when a program is felt to not be in substantial compliance with those requirements. Areas for Improvement/Concerning Trends are assigned when data from the current or previous reviews shows a potential opportunity for not meeting requirements. ACGME recommends using RC-granted citations and RC-determined Areas for Improvement with program responses during each program evaluation committee (PEC) meeting and in the Annual Program Evaluation to assess quality and improve a program’s curricula [10].

Accreditation Status

Accreditation Status is another evaluation tool that is publicly accessible from the ACGME’s website [11]. National comparative data is available in the Data Resource Book (DRB) and shows larger scale data in relation to specialty accreditation status, attrition, program leadership turnover, among other data fields [1•]. The DRB lacks granular data at the program level, however.

Regardless of your data source, having a clear understanding of the timeline for seeking accreditation as well as the different accreditation statuses is vital when interpreting a program’s quality. Detailed information on accreditation status and process can be found in the ACGME policies and procedures [12••]. Table 1 summarizes the various options for accreditation status.

Table 1 Options for accreditation status
Full size table

Continued Accreditation indicates substantial compliance with the ACGME requirements. Programs with Continued Accreditation without Outcomes are felt to be in substantial compliance, however they lack the data pertaining to graduates. For example, they have not yet had a graduate attempt the certification process.

Probationary Accreditation warrants annual review and can be given two years in a row after which a program transitions to Continued Accreditation with or without Warning or Withdrawal of Accreditation. “Special Circumstances” implies egregious non-compliance or a tragic loss of resources.

ACGME Faculty and Resident Surveys

Each year, core faculty and trainees complete a survey administered by the ACGME [13•]. Programs are privately given the results and are encouraged to review them in their Program Evaluation Committees to make improvements in areas where non-compliance may be at risk. Topics covered in the survey are Resources, Professionalism, Patient Safety and Teamwork, Faculty Teaching and Supervision, Evaluation, Educational Content, Diversity and Inclusion, Clinical Experience, and Education. This evaluation tool is critical to program improvement as it is a primary method of insight of the trainee- and core faculty-perception of the educational environment. Characteristics pertaining to program culture such as being able to raise concerns without fear of retaliation, whether trainees are pressured to work more than 80 h, trainee abuse and harassment are all available in this survey. ACGME faculty and resident surveys are anonymous and their responses do not delineate PGY level, gender, race, ethnicity, age, or any other identifying characteristics of the respondents. However, one area of weakness of this survey is the exclusion of other teaching faculty. As previously mentioned, core faculty are the only faculty afforded the opportunity to respond to the annual survey, creating a potential source of bias. Responses are grouped and reported by program and are compared to national averages. Resident and core faculty’s overall evaluation and overall opinion of the program are provided with a comparison to the national mean. The report depicts compliance or non-compliance in all categories at a glance compared to the national mean.

Although this survey data can be valuable, it is paramount for residents and core faculty alike to know and understand the anonymity of the survey results. Both anonymity and accuracy have been brought to question in previous studies [14]. Sticca, et al. found that one in seven residents falsely answered questions regarding work hours and 1 in 5 residents didn’t understand the work hour-related question [15]. The ACGME has since redesigned the annual resident survey and approved new program requirements [16]. The validity of the updated survey has not been examined. Nevertheless, since the survey’s results are directly linked to accreditation decisions, the responses may not be accurate, especially in smaller programs where residents may feel their responses can be easily identified. While potentially useful, this information is not publicly available.

Case Logs

Case log data is generated by trainees’ routinely entering their operative experience in an ACGME-provided portal. Program level data is available privately and can be used in several ways. On an individual level, it is helpful to identify case log habits to identify operative experience through a resident’s training. Additionally, case log data can be used by program leadership to evaluate rotational experience when evaluating a program’s block diagram of rotations. Publicly available data is specialty-specific and is reported at a national level [11]. Programs can create a similar report and compare the experience of their graduating residents to the national means to identify areas of opportunity or excellence when evaluating rotation options for their PGY-1 through 4 levels. This is located under the “Reports” tab in ADS [17].

A program’s ability to achieve defined category minimums as well as total case requirements is highly valued by trainees, faculty, program leadership, the RRC, as well as applicants. Upon graduation, trainees’ operative experience is electronically transmitted to their relevant certifying boards as part of their certification process. In general, programs are required for their trainees to achieve a minimum of the following: [18]

  • 250 total cases by end of PGY 2 with 200 being in defined category, exposure codes, or endoscopy (50 non-defined category cases can be counted toward this)

  • 850 operative procedures in the defined minimum category with 200 of these completed during a trainee’s chief year

One limitation of utilizing case log data is the potential inaccuracy of reporting [19, 20]. Current Procedural Terminology (CPT) codes that correspond to defined category minimums vs. total cases are not widely available and these can vary by program. Total cases include exposure codes, first assist cases, and all CPT codes that do and do not count towards a trainee’s defined category minimums. Additionally, lack of experience of surgical residents with billing and coding can cause erroneous reporting of case logs if CPT codes for cases scrubbed are not widely available or discussed between the faculty and the trainees [20]. Additional technology leveraging interfaces with hospitals’ electronic medical records and operative evaluations may eventually provide more accurate data collected in real time [21,22,23].

Information from Certifying Organizations

The American Board of Surgery (ABS) was incorporated in 1937 in Philadelphia as an autonomous, non-profit entity [24]. Its charter was in accordance with the Advisory Board of Medical Specialties (the precursor of the modern-day American Board of Medical Specialties that includes 24 disciplines).

The mission of the ABS is to serve “the public and the specialty of surgery by providing leadership in surgical education and practice, by promoting excellence through rigorous evaluation and examination, and by promoting the highest standards for professionalism, lifelong learning, and the continuous certification of surgeons in practice” [25••].

The ABS generates data which can be used to gauge the educational strength of a training program. While annual ABS In-Service Training Exam (ABSITE) data is not publicly available, information on programs’ success in the Qualifying and Certifying examinations is [26]. Annually, ABSITE results are reported only to Program Directors (PDs), who commonly share the reports with individual trainees. Because the ABS does not retain ABSITE score reports, only examinees and their respective programs are aware of performance. The ABS describes the exam as “one factor of many that should be considered when assessing a resident’s performance” [27]. Using the ABSITE as one factor to assess performance is further supported by work done by Ray, et al. who showed that scores do not correlate with favorable evaluations of clinical performance of residents [28]. PDs commonly use the exam to ensure that residents are progressing well with their medical knowledge. Since scores are reported as percentiles, individual trainees can gauge how they are doing relative to other trainees nationally.

PDs have access to individual score reports for each trainee, as well as a program summary report. Individual score reports contain data such as candidate standard score, percent correct, and candidate percentile within each training level. The report also includes a plot of subtest standard scores as well as topic areas for incorrect answers, which trainees can use to refine their medical knowledge acquisition.

Program-wide ABSITE performance can also be used by PDs to take a critical view of their educational curriculum and make necessary changes. PDs can also use the test result, when taken in context with a holistic review of trainee performance, to determine the appropriateness of candidate probation or promotion. ABSITE results may also be requested by fellowship programs during their application process. Scores are also commonly requested by surgery residency training programs when attempting to fill open training positions. Once again, since score data is not stored by the ABS, this information is not available to prospective applicants, or by the public to judge the educational curriculum of a particular residency training program.

The ABS provides historic success rates on the written Qualifying Examination (QE) and oral Certifying Examination (CE) [29]. The ABS also publishes an annual report on individual residency program performance on its certification examinations. The report shows the most recent three-year average pass rate of first-time examinees on both the QE and CE for all ACGME accredited General Surgery Residency training programs. The ABS states that the report is “meant for the use of General Surgery Residency training programs, and for medical students who are interested in surgical residences.” The report includes the total number of test takers for the QE and CE, and the overall pass rate. Because the public can use this information to compare success rates for graduates of these programs, PDs view this as a high-stakes document.

Annually, PDs receive reports from the ABS which provides a program summary of performance on ABS examinations. This report shows individual candidate performance over a three-year period. They are also able to view their program’s average score for individual content categories, including abdomen, alimentary tract, breast endocrine and soft tissue, critical care, surgical specialties, perioperative care, and general surgical knowledge. PDs also receive an individualized program summary of performance over the past three years. This gives a percentile for the program on both the QE and CE first-time pass rate. This data is extremely useful to PDs to compare their programs nationally. This can be used to make changes to how the program may choose to prepare its trainees for either the QE or CE.

Resources for Medical Student Applicants to Surgical Programs

Medical students utilize a multitude of resources when evaluating residency programs. Traditional modes of obtaining information for residency programs include consulting faculty mentors, residents, other medical students, and residency programs themselves [30,31,32]. Additionally, the Association of American Medical Colleges provides a Residency Explorer Tool that uses data from the Electronic Residency Application Service (ERAS), the National Resident Matching Program (NRMP), as well as the National Board of Medical Examiners (NBME), and National Board of Osteopathic Medical Examiners (NBOME) to display characteristics of specific training programs and specialties [33]. Although these sources are still utilized by students in selecting residency programs, a significant shift has been made over the past few decades towards online information gathering [34, 35]. Social media and internet-based resources have emerged as popular modalities for evaluating programs [36, 37]. Current research is limited and specialty-specific but increasingly shows the most popular resources utilized are Doximity, Reddit, Instagram, and Twitter [38].

Doximity has emerged as one of the most popular social media websites utilized by medical students when evaluating residency programs [39]. Its aim is to connect healthcare workers, including medical students, and keep them current on news relating to their fields. After users create profiles, a feed is generated based on their affiliations and interests. One unique feature allows medical students to explore residency programs and apply various filters to organize them. Users can sort programs based on a variety of characteristics such as program reputation, size, and research output. Other tools featured on the website allow students to see the average USMLE and COMLEX scores achieved by matched applicants.

Another popular online forum is Reddit. Medical students can browse and subscribe to topic-specific pages to obtain information about various subjects, including residency programs. Several studies cite the reliance on Reddit by medical students to obtain information about programs [40]. Data are varied about the weight placed on online forums by applicants to evaluate programs [34]. Although medical students report that these forums have less weight than other modalities of information gathering, they remain an important tool [34].

Other social media websites such as Instagram and Twitter have been cited as tools utilized by medical students to evaluate residency programs’ culture [41]. Social fabric and camaraderie are becoming increasingly important factors for medical students applying to residency programs. Because of this, programs are well advised to maintain vibrant social media profiles.

Finally, several non-profit online resources have emerged as important tools utilized by medical students. These include podcasts, crowd-sourced Google Documents, and Texas STAR [42, 43]. All these modalities of information were created either by students, residents, or faculty to disseminate information about residency programs. Data are limited and specialty-specific.

Resources for Preliminary Residents Seeking Categorical Positions

Non-designated general surgery (GS) preliminary residents face unique challenges in completing their training. Fortunately, there are several pathways available to them, including restarting from PGY-1 in GS or any other specialty, continuing their training as categorical residents, completing a second year as preliminary residents, or pursuing a research degree while seeking a suitable training position.

Program-related factors have been shown to be associated with non-designated GS preliminary resident success in matching into GS or surgical subspecialities [44, 45]. Preliminary residents in their second post-graduate year are more likely to match into general surgery than in their first post-graduate year [44]. Programs should advertise whether they have a second preliminary year and professional development time for their program director. Both of these program-related factors have been associated with primary success of non-designated GS preliminary residents matching into surgical subspecialties [45].

To find open positions, non-designated GS preliminary residents can utilize several resources. The Association of Program Directors in Surgery’s (APDS) website advertises open training positions, including both preliminary and categorical GS positions, as well as research fellowship positions [43]. Additionally, the American Medical Association (AMA) and the AAMC offer services to help members find open positions and details about programs in all specialties. Specifically, the AMA offers FREIDA, and the AAMC offers Find A Resident, both of which require membership [45, 46].

The ERAS and NRMP are invaluable tools for preliminary residents, since they can apply to PGY-1 positions in any specialty or to advanced programs like radiology, but must compete with newly graduating physicians [47, 48]. The Match process follows an annual cycle and includes a chance to apply to unfilled positions during the Supplemental Offer and Acceptance Program (SOAP) during Match week. Participating in ERAS and the NRMP requires fees, proportional to the number of programs applied to and ranked.

Finally, there are also several third-party websites that require paid accounts [49,50,51,52]. Some information may also be available on social media and by word of mouth. However, they may be less reliable.

Harnessing Data-Driven Innovation for Surgical Education through EQIP

The demand for high-quality surgeons persists, and therefore the need for robust and comprehensive evaluation methods for surgical training programs becomes increasingly paramount. The APDS recognized this imperative and responded by spearheading the creation of the Educational Quality Improvement Program (EQIP). EQIP represents a paradigm shift in surgical education, empowering PDs to assess their programs, identify areas for improvement, and foster data-driven innovation to cultivate the next generation of exceptional surgeons. Currently, EQIP is not publicly available data and is seen only by program leadership, including program directors and administrators.

Unlike traditional regulatory frameworks, EQIP is designed as a collaborative and collegial platform, emphasizing continuous learning and improvement rather than punitive measures. The program acknowledges the significance of maintaining confidentiality and data security, and to that end, has partnered with the Society for Improving Medical Professional Learning/Procedural Learning and Safety Collaborative (SIMPL/PLSC) to establish a secure data platform. This commitment to data privacy instills confidence in participating programs, encouraging them to share valuable insights without hesitation.

At its core, EQIP enables general surgery residency programs to collect and analyze a comprehensive set of data points for their graduating residents. This includes many of the data points mentioned above (case log data, accreditation status, ACGME citations). A user-friendly, secure data interface captures essential information such as the residents' prior experiences as medical students, their progression through the surgical training program, and their individual performance. Once the data collection phase concludes, PDs gain access to a password-protected EQIP website, housing filterable report cards that facilitate benchmarking against a composite of other EQIP programs for each data element collected. This feature empowers PDs to gain a nuanced understanding of their program's strengths and weaknesses, thus facilitating targeted improvements.

The EQIP site extends its utility further by providing performance data for each participating program. By employing analytic equations that consider the success rates of the ABS QE and CE, EQIP generates an observed to expected ratio (O:E) ratio of performance. This metric serves as a powerful tool for PDs to assess the overall effectiveness of their training programs in preparing residents for these critical examinations. Armed with this information, PDs can focus their efforts on areas that require additional attention, leading to better outcomes for their trainees.

Beyond the filterable report cards and O:E reports, EQIP boasts additional potential applications that can revolutionize surgical education. The program has the capacity to facilitate the establishment of learning collaboratives, allowing smaller subsets of programs with shared characteristics, such as size, geography, or program type, to pool their data. By integrating additional data elements, these collaboratives can collectively examine outcomes and compare their performance against other EQIP participating centers. This cooperative approach fosters a culture of knowledge sharing and mutual growth, transcending individual program boundaries to advance the field as a whole.

Moreover, EQIP has the potential to develop best practices by identifying characteristics of top-performing centers. By analyzing their data, EQIP can uncover key factors and strategies that contribute to exceptional training outcomes. These best practices can then be disseminated across the surgical education community, empowering programs to adopt evidence-based approaches that have proven successful.

Conclusion

It is an exciting and dynamic period in surgical training. Increasing access to data, both public and private, is allowing educators to apply quality-improvement methodology to their training programs. In turn, applicants to training programs have increasing expectations of transparency surrounding training programs’ educational environments and outcomes in proficiency and autonomy. Eventually, as more and more data are applied to undergraduate and graduate medical education, surgeons’ clinical outcomes in independent practice can further be integrated into educational databases to establish a clear association between the quality of training and patient care.