Abstract
Purpose of Review
Measuring quality of surgical training programs is fundamental in recruitment, program improvement, and accreditation. There are several resources available that can provide useful data and information for programs, medical students, and preliminary residents to evaluate program quality. The purpose of this review is to summarize both public and private data.
Recent Findings
Privately available data largely consist of Accreditation Council for Graduate Medical Education (ACGME) resident and faculty surveys, Letter of Notification, and Case Log reports as well as the American Board of Surgery (ABS) In-Training Exam scores. Publicly available data include program accreditation status, and ABS Qualifying and Certifying Exam first-time pass rates. Medical students and preliminary residents can use resources such as social media platforms and internet-based sources to gain more information. Lastly, program leadership can drive innovation into their curricula utilizing platforms such as the Educational Quality Improvement Program (EQIP).
Summary
Increasing access to data is allowing educators to apply quality-improvement methodology to their training programs. Utilizing all available data will allow high-quality continuous improvement in the curricula development of surgical training programs.
Similar content being viewed by others
Explore related subjects
Find the latest articles, discoveries, and news in related topics.Avoid common mistakes on your manuscript.
Introduction
Postgraduate training in surgery continues to be a rigorous, challenging, and competitive period in the career of a surgeon. In 2022, 353 training programs with 9299 trainees produced 1394 graduates. This represents about 6% of the total GME trainees in the US. Overall, between 2017 and 2022, the number of surgical trainees has increased by 9.7% [1•]. Amongst these programs, there is considerable heterogeneity in location, type and size of hospital, and clinical environment. As prospective trainees (including medical students and preliminary residents) consider which programs are suitable for them, there are several resources (both public and private) available that can provide useful data and information. Included in this review are accrediting, certifying, and specialty-specific organizations, along with crowd-sourced collaboratives.
Information from Accrediting Organizations
Accreditation of training programs has been carried out by several different organizations over the last century [2]. In 1914, the American Medical Association’s Council on Medical Education first set standards for hospital internship programs, which included 603 hospitals and 3095 positions. In 1920, this Council became the Council on Medical Education and Hospitals (CMEH). In 1927, the CMEH published the first list of hospitals approved for residency training. In 1939, the American College of Surgeons (ACS) established its own criteria and approval process for graduate training in surgery [3, 4]. These two systems operated in parallel until 1950 when the Conference Committee on Graduate Training in Surgery, a collaboration among the American Board of Surgery, the ACS, and the American Medical Association (AMA) was established [5]. An important milestone in GME financing came in 1965 when Medicare and Medicaid were created and began to fund GME positions throughout the US. Currently, over $15 billion in federal funding supports residency training [6]. The Accreditation Council for Graduate Medical Education (ACGME) was established as a council within the AMA in 1981. In 2000, with consensus from the academic GME community that an independent organization was required to accredit training programs, the ACGME (in its current form) was founded in Chicago, IL as an independent, non-governmental, non-profit organization. In 2014, the ACGME integrated osteopathic programs into its accreditation structure [7]. Within the ACGME, the Surgery Residency Review Committee (RRC) is responsible for accreditation of all US general surgery training programs. Members of the RRC are solicited from ACGME member organizations (AMA, ABS, AOA, ACS), the surgical community, and include one public and one resident member [8].
The ACGME has a number of evaluation tools to measure residency program quality. Publicly available information includes historical and current data regarding program accreditation. Privately available information is provided to program leadership to identify areas for improvement and weaknesses in curricula specifically in relation to achieving and maintaining the requirements set forth by ACGME.
Letter of Notification
The mission of the RRC in Surgery is to ensure that training programs meet accreditation standards. Annually, the Surgery RRC reviews each program’s site visit data, certification examination pass rates, faculty and resident surveys, and Accreditation Data System (ADS) updates which includes major changes, faculty roster, faculty scholarly activity, and case log data [9]. Based on available information provided by the program, the RC generates a Letter of Notification (LoN) that includes the program’s accreditation status, resident complement, citations, areas for improvement, and commendation, if awarded. This information is given directly to programs and their Sponsoring Institution to help improve the program’s and/or Sponsoring Institution’s ability to meet ACGME’s requirements. Citations are given when a program is felt to not be in substantial compliance with those requirements. Areas for Improvement/Concerning Trends are assigned when data from the current or previous reviews shows a potential opportunity for not meeting requirements. ACGME recommends using RC-granted citations and RC-determined Areas for Improvement with program responses during each program evaluation committee (PEC) meeting and in the Annual Program Evaluation to assess quality and improve a program’s curricula [10].
Accreditation Status
Accreditation Status is another evaluation tool that is publicly accessible from the ACGME’s website [11]. National comparative data is available in the Data Resource Book (DRB) and shows larger scale data in relation to specialty accreditation status, attrition, program leadership turnover, among other data fields [1•]. The DRB lacks granular data at the program level, however.
Regardless of your data source, having a clear understanding of the timeline for seeking accreditation as well as the different accreditation statuses is vital when interpreting a program’s quality. Detailed information on accreditation status and process can be found in the ACGME policies and procedures [12••]. Table 1 summarizes the various options for accreditation status.
Continued Accreditation indicates substantial compliance with the ACGME requirements. Programs with Continued Accreditation without Outcomes are felt to be in substantial compliance, however they lack the data pertaining to graduates. For example, they have not yet had a graduate attempt the certification process.
Probationary Accreditation warrants annual review and can be given two years in a row after which a program transitions to Continued Accreditation with or without Warning or Withdrawal of Accreditation. “Special Circumstances” implies egregious non-compliance or a tragic loss of resources.
ACGME Faculty and Resident Surveys
Each year, core faculty and trainees complete a survey administered by the ACGME [13•]. Programs are privately given the results and are encouraged to review them in their Program Evaluation Committees to make improvements in areas where non-compliance may be at risk. Topics covered in the survey are Resources, Professionalism, Patient Safety and Teamwork, Faculty Teaching and Supervision, Evaluation, Educational Content, Diversity and Inclusion, Clinical Experience, and Education. This evaluation tool is critical to program improvement as it is a primary method of insight of the trainee- and core faculty-perception of the educational environment. Characteristics pertaining to program culture such as being able to raise concerns without fear of retaliation, whether trainees are pressured to work more than 80 h, trainee abuse and harassment are all available in this survey. ACGME faculty and resident surveys are anonymous and their responses do not delineate PGY level, gender, race, ethnicity, age, or any other identifying characteristics of the respondents. However, one area of weakness of this survey is the exclusion of other teaching faculty. As previously mentioned, core faculty are the only faculty afforded the opportunity to respond to the annual survey, creating a potential source of bias. Responses are grouped and reported by program and are compared to national averages. Resident and core faculty’s overall evaluation and overall opinion of the program are provided with a comparison to the national mean. The report depicts compliance or non-compliance in all categories at a glance compared to the national mean.
Although this survey data can be valuable, it is paramount for residents and core faculty alike to know and understand the anonymity of the survey results. Both anonymity and accuracy have been brought to question in previous studies [14]. Sticca, et al. found that one in seven residents falsely answered questions regarding work hours and 1 in 5 residents didn’t understand the work hour-related question [15]. The ACGME has since redesigned the annual resident survey and approved new program requirements [16]. The validity of the updated survey has not been examined. Nevertheless, since the survey’s results are directly linked to accreditation decisions, the responses may not be accurate, especially in smaller programs where residents may feel their responses can be easily identified. While potentially useful, this information is not publicly available.
Case Logs
Case log data is generated by trainees’ routinely entering their operative experience in an ACGME-provided portal. Program level data is available privately and can be used in several ways. On an individual level, it is helpful to identify case log habits to identify operative experience through a resident’s training. Additionally, case log data can be used by program leadership to evaluate rotational experience when evaluating a program’s block diagram of rotations. Publicly available data is specialty-specific and is reported at a national level [11]. Programs can create a similar report and compare the experience of their graduating residents to the national means to identify areas of opportunity or excellence when evaluating rotation options for their PGY-1 through 4 levels. This is located under the “Reports” tab in ADS [17].
A program’s ability to achieve defined category minimums as well as total case requirements is highly valued by trainees, faculty, program leadership, the RRC, as well as applicants. Upon graduation, trainees’ operative experience is electronically transmitted to their relevant certifying boards as part of their certification process. In general, programs are required for their trainees to achieve a minimum of the following: [18]
-
250 total cases by end of PGY 2 with 200 being in defined category, exposure codes, or endoscopy (50 non-defined category cases can be counted toward this)
-
850 operative procedures in the defined minimum category with 200 of these completed during a trainee’s chief year
One limitation of utilizing case log data is the potential inaccuracy of reporting [19, 20]. Current Procedural Terminology (CPT) codes that correspond to defined category minimums vs. total cases are not widely available and these can vary by program. Total cases include exposure codes, first assist cases, and all CPT codes that do and do not count towards a trainee’s defined category minimums. Additionally, lack of experience of surgical residents with billing and coding can cause erroneous reporting of case logs if CPT codes for cases scrubbed are not widely available or discussed between the faculty and the trainees [20]. Additional technology leveraging interfaces with hospitals’ electronic medical records and operative evaluations may eventually provide more accurate data collected in real time [21,22,23].
Information from Certifying Organizations
The American Board of Surgery (ABS) was incorporated in 1937 in Philadelphia as an autonomous, non-profit entity [24]. Its charter was in accordance with the Advisory Board of Medical Specialties (the precursor of the modern-day American Board of Medical Specialties that includes 24 disciplines).
The mission of the ABS is to serve “the public and the specialty of surgery by providing leadership in surgical education and practice, by promoting excellence through rigorous evaluation and examination, and by promoting the highest standards for professionalism, lifelong learning, and the continuous certification of surgeons in practice” [25••].
The ABS generates data which can be used to gauge the educational strength of a training program. While annual ABS In-Service Training Exam (ABSITE) data is not publicly available, information on programs’ success in the Qualifying and Certifying examinations is [26]. Annually, ABSITE results are reported only to Program Directors (PDs), who commonly share the reports with individual trainees. Because the ABS does not retain ABSITE score reports, only examinees and their respective programs are aware of performance. The ABS describes the exam as “one factor of many that should be considered when assessing a resident’s performance” [27]. Using the ABSITE as one factor to assess performance is further supported by work done by Ray, et al. who showed that scores do not correlate with favorable evaluations of clinical performance of residents [28]. PDs commonly use the exam to ensure that residents are progressing well with their medical knowledge. Since scores are reported as percentiles, individual trainees can gauge how they are doing relative to other trainees nationally.
PDs have access to individual score reports for each trainee, as well as a program summary report. Individual score reports contain data such as candidate standard score, percent correct, and candidate percentile within each training level. The report also includes a plot of subtest standard scores as well as topic areas for incorrect answers, which trainees can use to refine their medical knowledge acquisition.
Program-wide ABSITE performance can also be used by PDs to take a critical view of their educational curriculum and make necessary changes. PDs can also use the test result, when taken in context with a holistic review of trainee performance, to determine the appropriateness of candidate probation or promotion. ABSITE results may also be requested by fellowship programs during their application process. Scores are also commonly requested by surgery residency training programs when attempting to fill open training positions. Once again, since score data is not stored by the ABS, this information is not available to prospective applicants, or by the public to judge the educational curriculum of a particular residency training program.
The ABS provides historic success rates on the written Qualifying Examination (QE) and oral Certifying Examination (CE) [29]. The ABS also publishes an annual report on individual residency program performance on its certification examinations. The report shows the most recent three-year average pass rate of first-time examinees on both the QE and CE for all ACGME accredited General Surgery Residency training programs. The ABS states that the report is “meant for the use of General Surgery Residency training programs, and for medical students who are interested in surgical residences.” The report includes the total number of test takers for the QE and CE, and the overall pass rate. Because the public can use this information to compare success rates for graduates of these programs, PDs view this as a high-stakes document.
Annually, PDs receive reports from the ABS which provides a program summary of performance on ABS examinations. This report shows individual candidate performance over a three-year period. They are also able to view their program’s average score for individual content categories, including abdomen, alimentary tract, breast endocrine and soft tissue, critical care, surgical specialties, perioperative care, and general surgical knowledge. PDs also receive an individualized program summary of performance over the past three years. This gives a percentile for the program on both the QE and CE first-time pass rate. This data is extremely useful to PDs to compare their programs nationally. This can be used to make changes to how the program may choose to prepare its trainees for either the QE or CE.
Resources for Medical Student Applicants to Surgical Programs
Medical students utilize a multitude of resources when evaluating residency programs. Traditional modes of obtaining information for residency programs include consulting faculty mentors, residents, other medical students, and residency programs themselves [30,31,32]. Additionally, the Association of American Medical Colleges provides a Residency Explorer Tool that uses data from the Electronic Residency Application Service (ERAS), the National Resident Matching Program (NRMP), as well as the National Board of Medical Examiners (NBME), and National Board of Osteopathic Medical Examiners (NBOME) to display characteristics of specific training programs and specialties [33]. Although these sources are still utilized by students in selecting residency programs, a significant shift has been made over the past few decades towards online information gathering [34, 35]. Social media and internet-based resources have emerged as popular modalities for evaluating programs [36, 37]. Current research is limited and specialty-specific but increasingly shows the most popular resources utilized are Doximity, Reddit, Instagram, and Twitter [38].
Doximity has emerged as one of the most popular social media websites utilized by medical students when evaluating residency programs [39]. Its aim is to connect healthcare workers, including medical students, and keep them current on news relating to their fields. After users create profiles, a feed is generated based on their affiliations and interests. One unique feature allows medical students to explore residency programs and apply various filters to organize them. Users can sort programs based on a variety of characteristics such as program reputation, size, and research output. Other tools featured on the website allow students to see the average USMLE and COMLEX scores achieved by matched applicants.
Another popular online forum is Reddit. Medical students can browse and subscribe to topic-specific pages to obtain information about various subjects, including residency programs. Several studies cite the reliance on Reddit by medical students to obtain information about programs [40]. Data are varied about the weight placed on online forums by applicants to evaluate programs [34]. Although medical students report that these forums have less weight than other modalities of information gathering, they remain an important tool [34].
Other social media websites such as Instagram and Twitter have been cited as tools utilized by medical students to evaluate residency programs’ culture [41]. Social fabric and camaraderie are becoming increasingly important factors for medical students applying to residency programs. Because of this, programs are well advised to maintain vibrant social media profiles.
Finally, several non-profit online resources have emerged as important tools utilized by medical students. These include podcasts, crowd-sourced Google Documents, and Texas STAR [42, 43]. All these modalities of information were created either by students, residents, or faculty to disseminate information about residency programs. Data are limited and specialty-specific.
Resources for Preliminary Residents Seeking Categorical Positions
Non-designated general surgery (GS) preliminary residents face unique challenges in completing their training. Fortunately, there are several pathways available to them, including restarting from PGY-1 in GS or any other specialty, continuing their training as categorical residents, completing a second year as preliminary residents, or pursuing a research degree while seeking a suitable training position.
Program-related factors have been shown to be associated with non-designated GS preliminary resident success in matching into GS or surgical subspecialities [44, 45]. Preliminary residents in their second post-graduate year are more likely to match into general surgery than in their first post-graduate year [44]. Programs should advertise whether they have a second preliminary year and professional development time for their program director. Both of these program-related factors have been associated with primary success of non-designated GS preliminary residents matching into surgical subspecialties [45].
To find open positions, non-designated GS preliminary residents can utilize several resources. The Association of Program Directors in Surgery’s (APDS) website advertises open training positions, including both preliminary and categorical GS positions, as well as research fellowship positions [43]. Additionally, the American Medical Association (AMA) and the AAMC offer services to help members find open positions and details about programs in all specialties. Specifically, the AMA offers FREIDA, and the AAMC offers Find A Resident, both of which require membership [45, 46].
The ERAS and NRMP are invaluable tools for preliminary residents, since they can apply to PGY-1 positions in any specialty or to advanced programs like radiology, but must compete with newly graduating physicians [47, 48]. The Match process follows an annual cycle and includes a chance to apply to unfilled positions during the Supplemental Offer and Acceptance Program (SOAP) during Match week. Participating in ERAS and the NRMP requires fees, proportional to the number of programs applied to and ranked.
Finally, there are also several third-party websites that require paid accounts [49,50,51,52]. Some information may also be available on social media and by word of mouth. However, they may be less reliable.
Harnessing Data-Driven Innovation for Surgical Education through EQIP
The demand for high-quality surgeons persists, and therefore the need for robust and comprehensive evaluation methods for surgical training programs becomes increasingly paramount. The APDS recognized this imperative and responded by spearheading the creation of the Educational Quality Improvement Program (EQIP). EQIP represents a paradigm shift in surgical education, empowering PDs to assess their programs, identify areas for improvement, and foster data-driven innovation to cultivate the next generation of exceptional surgeons. Currently, EQIP is not publicly available data and is seen only by program leadership, including program directors and administrators.
Unlike traditional regulatory frameworks, EQIP is designed as a collaborative and collegial platform, emphasizing continuous learning and improvement rather than punitive measures. The program acknowledges the significance of maintaining confidentiality and data security, and to that end, has partnered with the Society for Improving Medical Professional Learning/Procedural Learning and Safety Collaborative (SIMPL/PLSC) to establish a secure data platform. This commitment to data privacy instills confidence in participating programs, encouraging them to share valuable insights without hesitation.
At its core, EQIP enables general surgery residency programs to collect and analyze a comprehensive set of data points for their graduating residents. This includes many of the data points mentioned above (case log data, accreditation status, ACGME citations). A user-friendly, secure data interface captures essential information such as the residents' prior experiences as medical students, their progression through the surgical training program, and their individual performance. Once the data collection phase concludes, PDs gain access to a password-protected EQIP website, housing filterable report cards that facilitate benchmarking against a composite of other EQIP programs for each data element collected. This feature empowers PDs to gain a nuanced understanding of their program's strengths and weaknesses, thus facilitating targeted improvements.
The EQIP site extends its utility further by providing performance data for each participating program. By employing analytic equations that consider the success rates of the ABS QE and CE, EQIP generates an observed to expected ratio (O:E) ratio of performance. This metric serves as a powerful tool for PDs to assess the overall effectiveness of their training programs in preparing residents for these critical examinations. Armed with this information, PDs can focus their efforts on areas that require additional attention, leading to better outcomes for their trainees.
Beyond the filterable report cards and O:E reports, EQIP boasts additional potential applications that can revolutionize surgical education. The program has the capacity to facilitate the establishment of learning collaboratives, allowing smaller subsets of programs with shared characteristics, such as size, geography, or program type, to pool their data. By integrating additional data elements, these collaboratives can collectively examine outcomes and compare their performance against other EQIP participating centers. This cooperative approach fosters a culture of knowledge sharing and mutual growth, transcending individual program boundaries to advance the field as a whole.
Moreover, EQIP has the potential to develop best practices by identifying characteristics of top-performing centers. By analyzing their data, EQIP can uncover key factors and strategies that contribute to exceptional training outcomes. These best practices can then be disseminated across the surgical education community, empowering programs to adopt evidence-based approaches that have proven successful.
Conclusion
It is an exciting and dynamic period in surgical training. Increasing access to data, both public and private, is allowing educators to apply quality-improvement methodology to their training programs. In turn, applicants to training programs have increasing expectations of transparency surrounding training programs’ educational environments and outcomes in proficiency and autonomy. Eventually, as more and more data are applied to undergraduate and graduate medical education, surgeons’ clinical outcomes in independent practice can further be integrated into educational databases to establish a clear association between the quality of training and patient care.
References
Papers of particular interest, published recently, have been highlighted as: •Of importance, ••Of major importance
•ACGME. 2022. ACGME Data Resource Book. ACGME. Accessed August 11, 2023. https://www.acgme.org/globalassets/pfassets/publicationsbooks/2021-2022_acgme__databook_document.pdf This reference provides access to national level publicly available data regarding program performance
Swanson AG. The genesis of the Coordinating Council on Medical Education and the Liaison Committee on Graduate Medical Education. Bull N Y Acad Med. 1974;50(11):1216–21.
MacEachern MT. Criteria for graduate training for surgery and a manual of graduate training for surgery. Bull Am Coll Surg. 1939;24:6–13.
Potts JR 3rd. Surgical program accreditation and case logs: What is the meaning of the minima? J Am Coll Surg. 2019;229(4):431–5. https://doi.org/10.1016/j.jamcollsurg.2019.05.019.
Holden WD. The Conference Committee on Graduate Education in Surgery. Surgery. 1968;64(2):506–8.
Committee on the Governance and Financing of Graduate Medical Education; Board on Health Care Services; Institute of Medicine. Graduate Medical Education That Meets the Nation's Health Needs. Eden J, Berwick D, Wilensky G, editors. Washington (DC): National Academies Press (US); 2014 Sep 30.
Nasca TJ. March 2014. NascaLetteracgme-aoa-aacomagreementMarch204. ACGME. Accessed August 11, 2023 https://www.acgme.org/What-We-Do/Accreditation/Single-GME-Accreditation-System
ACGME. 2023. Committees and Members Selection Process. ACGME. Accessed August 11, 2023. https://www.acgme.org/About-Us/Committees-and-Members-Selection-Process
Potts JR 3rd, Lipsett PA, Matthews JB. The Challenges of Program Accreditation Decisions in 2021 for the ACGME Review Committee for Surgery. J Surg Educ. 2021;78(2):394–9. https://doi.org/10.1016/j.jsurg.2020.08.023.
ACGME. 2017. High-Value Data Suggested for Use in Program Evaluation and Improvement. Accessed August 11, 2023. https://www.acgme.org/globalassets/PDFs/SelfStudy/HighValueDataSSandAPE.pdf
ACGME. 2023. Accreditation Council for Graduate Medical Education – Public. ACGME. Accessed August 11, 2023. https://apps.acgme.org/ads/Public/Reports/Report/26
••ACGME. 2021. Accreditation Council for Graduate Medical Education Policies and Procedures. Accessed August 11, 2023. https://www.acgme.org/globalassets/ab_acgmepoliciesprocedures.pdf this website provides detailed information regarding the policies and procedures for all programs for the ACGME.
•ACGME. November 2019. ACGME Faculty Survey Program FAQ. Accessed August 11, 2023. https://www.acgme.org/globalassets/ACGME_Faculty_Survey-Program_FAQs.pdf this FAQ document provides insight regarding the ACGME faculty survey. Specifically, it delineates answers to most commonly asked questions regarding timing, participation, anonymity, and use of data.
Fahy BN, Todd SR, Paukert JL, Johnson ML, Bass BL. How accurate is the Accreditation Council for Graduate Medical Education (ACGME) Resident survey? Comparison between ACGME and in-house GME survey. J Surg Educ. 2010;67(6):387–92. https://doi.org/10.1016/j.jsurg.2010.06.003.
Sticca RP, Macgregor JM, Szlabick RE. Is the Accreditation Council for Graduate Medical Education (ACGME) Resident/Fellow survey, a valid tool to assess general surgery residency programs compliance with work hours regulations? J Surg Educ. 2010;67(6):406–11. https://doi.org/10.1016/j.jsurg.2010.09.007.
Artino AR Jr, Malloy K, Miller RS, Kirk LM, Brigham TP. A redesign of the ACGME Resident/Fellow survey through expert reviews and cognitive interviews. J Grad Med Educ. 2023;15(1):67–73. https://doi.org/10.4300/JGME-D-22-00437.1.
Review Committee for Surgery. 2019. Defined Category Minimum Numbers for General Surgery Residents and Credit Role. ACGME. Accessed August 11, 2023. https://www.acgme.org/globalassets/definedcategoryminimumnumbersforgeneralsurgeryresidentsandcreditrole.pdf
Naik ND, Abbott EF, Aho JM, Pandian TK, Thiels CA, Heller SF, et al. The ACGME case log system may not accurately represent operative experience among general surgery interns. J Surg Educ. 2017;74(6):e106–10. https://doi.org/10.1016/j.jsurg.2017.09.032.
Balla F, Garwe T, Motghare P, Stamile T, Kim J, Mahnken H, et al. Evaluating coding accuracy in general surgery residents’ accreditation council for graduate medical education procedural case logs. J Surg Educ. 2016;73(6):e59–63. https://doi.org/10.1016/j.jsurg.2016.07.017.
Meholick AL, Jesneck JL, Thanawala RM, Seymour NE. Use of a secure web-based data management platform to track resident operative performance and program educational quality over time. J Surg Educ. 2020;77(6):e187–95. https://doi.org/10.1016/j.jsurg.2020.05.022.
Thanawala R, Jesneck J, Shelton J, Rhee R, Seymour NE. Overcoming systems factors in case logging with artificial intelligence tools. J Surg Educ. 2022;79(4):1024–30. https://doi.org/10.1016/j.jsurg.2022.01.013.
Abbott KL, Krumm AE, Clark MJ, Kendrick DE, Kelley JK, George BC. Representativeness of workplace-based operative performance assessments for resident operative experience. J Surg Educ. 2022;79(3):769–74. https://doi.org/10.1016/j.jsurg.2021.12.010.
American Board of Surgery. 2023. About Us. ABS. Accessed August 11, 2023. http://www.absurgery.org/default.jsp?abouthome
American Board of Surgery. 2023. Promoting Surgical Excellence Since 1937. ABS. Accessed August 11, 2023. http://www.absurgery.org/xfer/BookletofInfo-Surgery.pdf
••American Board of Surgery. 2023. General Surgery Residency Pass Rates. ABS. Accessed August 11, 2023. https://www.absurgery.org/default.jsp?prog_passreport this weblink provides access to three-year summaries reporting first-time pass rates on the American Board of Surgery Qualifying and Certifying
American Board of Surgery. 2023. ABS In-Training Examination. ABS. Accessed August 11, 2023. https://www.absurgery.org/default.jsp?certabsite
American Board of Surgery. 2023. Public Data. ABS. Accessed August 11, 2023. https://www.absurgery.org/default.jsp?publicdata
Ray JJ, Sznol JA, Teisch LF, Meizoso JP, Allen CJ, Namias N, Pizano LR, Sleeman D, Spector SA, Schulman CI. Association between American board of surgery in-training examination scores and resident performance. JAMA Surg. 2016;151(1):26–31. https://doi.org/10.1001/jamasurg.2015.3088.
Kutikov A, Morgan TM, Resnick MJ. The impact of residency match information disseminated by a third-party website. J Surg Educ. 2009;66(4):212–5. https://doi.org/10.1016/j.jsurg.2009.03.034.
Schlitzkus LL, Schenarts KD, Schenarts PJ. Is your residency program ready for Generation Y? J Surg Educ. 2010;67(2):108–11. https://doi.org/10.1016/j.jsurg.2010.03.004.
Schenarts PJ. Now arriving: surgical trainees from Generation Z. J Surg Educ. 2020;77(2):246–53. https://doi.org/10.1016/j.jsurg.2019.09.004.
American Association of Medical Colleges. 2023. Residency Explorer Tool. AAMC. Accessed August 11, 2023. https://www.residencyexplorer.org/
Sinno S, Mehta K, Squitieri L, Ranganathan K, Koeckert MS, Patel A, et al. Residency characteristics that matter most to plastic surgery applicants: a multi-institutional analysis and review of the literature. Ann Plast Surg. 2015;74(6):713–7. https://doi.org/10.1097/sap.0000000000000511.
Luk L, Maher MD, Desperito E, Weintraub JL, Amin S, Ayyala RS. Evaluating factors and resources affecting ranking of diagnostic radiology residency programs by Medical Students in 2016–2017. Acad Radiol. 2018;25(10):1344–52. https://doi.org/10.1016/j.acra.2018.05.001.
Butler A, Berke C, Zareef U, Tawfik AM, Varghese B, Frias G, et al. Social media and the orthopaedic surgery residency application process. Cureus. 2022;14(4): e23933. https://doi.org/10.7759/cureus.23933.
Fuller CC, Deckey DG, Brinkman JC, Tummala SV, Lu PG, Mishra N, et al. General surgery residency applicants’ perspective on social media as a recruiting tool. J Surg Educ. 2022;79(6):1334–41. https://doi.org/10.1016/j.jsurg.2022.06.003.
Renew JR, Ladlie B, Gorlin A, Long T. The impact of social media on anesthesia resident recruitment. J Educ Perioper Med. 2019;21(1):E632.
Rolston AM, Hartley SE, Khandelwal S, Christner JG, Cheng DF, Caty RM, et al. Effect of doximity residency rankings on residency Applicants’ program choices. West J Emerg Med. 2015;16(6):889–93. https://doi.org/10.5811/westjem.2015.8.27343.
Mackey C, Feldman J, Peng C, Way DP, Messman A. How do emergency medicine applicants evaluate residency programs in the post-COVID-19 era? AEM Educ Train. 2022;6(6): e10805. https://doi.org/10.1002/aet2.10805.
Checketts JX, Hunt T, Checketts BR, Scott JT, Johnson M, Boose M, et al. Analysis of Social Media Perceptions Among Orthopaedic Surgery Residency Applicants and Social Media Use by Residency Programs During the 2020 to 2021 Cycle. JB JS Open Access. 2021;. doi: https://doi.org/10.2106/jbjs.Oa.21.00083.
Stoehr JR, Hamidian Jahromi A, Gosain AK. Doctority: Plastic surgery-an informative podcast for applicants to learn about plastic surgery training programs. Plast Reconstr Surg Glob Open. 2021;9(7): e3717. https://doi.org/10.1097/gox.0000000000003717.
Yong TM, Austin DC, Molloy IB, Torchia MT, Coe MP. Online information and mentorship: perspectives from orthopaedic surgery residency applicants. J Am Acad Orthop Surg. 2021;29(14):616–23. https://doi.org/10.5435/jaaos-d-20-00512.
Association of Program Directors in Surgery. 2023. Open Training Positions. APDS. Accessed August 11, 2023. https://apds.org/education-careers/open-positions/
Hardy WA, Litts CS Jr, White JA. Career outcomes of nondesignated preliminary surgery residents: a systematic review and meta-analysis. J Surg Educ. 2023;80(12):1773–80. https://doi.org/10.1016/j.jsurg.2023.08.008.
Ritter KA, Anand RJ, Beard K, Edelman DA, Huth L, Namm J, Hope W, Allen S, Joshi A, Hildreth A, Tu C, Seavey CN, Vilchez V, Lipman J. Impact of surgery program characteristics on fate of non-designated preliminary surgery interns. J Surg Educ. 2020;77(6):e11–9. https://doi.org/10.1016/j.jsurg.2020.06.001.
American Medical Association. 2023. FREIDA Vacant Position Listings. AMA. Accessed August 11, 2023. https://freida.ama-assn.org/vacant-position
American Association of Medical Colleges. 2023. Find a Resident. AAMC. Accessed August 11, 2023. https://systems.aamc.org/findaresident/
American Association of Medical Colleges. 2023. Applying to Residencies with ERAS. AAMC. Accessed August 11, 2023. https://students-residents.aamc.org/applying-residencies-eras/applying-residencies-eras
National Residency Match Program. 2023. Fair, Equitable, Transparent, and Reliable. NRMP. Accessed August 11, 2023. https://www.nrmp.org/
Resident Swap, Inc. 2023. Resident Swap Overview. Resident Swap. Accessed August 11, 2023. https://residentswap.org
Outside match. 2023. Open Residency and Fellowship Spots. Outside Match. Accessed August 11, 2023. https://outsidematch.com/
Open Positions Residencies and Fellowships. 2023. Open Medical Residency Positions and Fellowships. Open Positions Residencies and Fellowships. Accessed August 11, 2023. https://www.openresidencypositions.com/
Funding
Open access funding provided by Rowan University. The authors did not receive support from any organization for the submitted work.
Author information
Authors and Affiliations
Contributions
KCS, RJA, DR, EH, GY, and AJ all participated in writing the manuscript. AJ created the table and AJ, EH, and KCS reviewed and made edits to the manuscript prior to submission. All authors reviewed the final manuscript.
Corresponding author
Ethics declarations
Conflict of interest
The authors have nothing to disclose.
Human and Animal Rights Informed Consent
No animal or human subjects by the authors were used in this study.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Conrad-Schnetz, K., Anand, R.J., Relles, D. et al. Determination of Quality in Training Programs Based on Outcomes and Data. Curr Surg Rep 12, 138–145 (2024). https://doi.org/10.1007/s40137-024-00400-4
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s40137-024-00400-4