CORR® Curriculum — Orthopaedic Education: Operative Assessment and the ACGME Milestones: Time for Change
Orthopaedic medical education is in the midst of a major paradigm shift. Where advancement in orthopaedic residency previously lacked specific content and achievement requirements, the recent implementation of the Milestones program as outlined by the Accreditation Council on Graduate Medical Education (ACGME) now provides a detailed framework for determining residency knowledge within specific core competencies . Residency education remains a polarizing issue, and just what constitutes proper oversight for resident education is controversial. In order to make good choices about the future of residency education reform, it is important to understand how we got to this point.
Orthopaedic surgery education (as well as the specialty) gradually evolved in the United States during the first three decades of the twentieth century. The influential 19th-century surgeon, William S. Halsted MD, initiated a residency program based on his observations of German education while Professor of Surgery at Johns Hopkins University School of Medicine. This residency program started with an internship role (of variable length) followed by an assistant resident position (6 years), and house surgeon (2 years). Residents progressed at their own rate, particularly as interns, with no guarantee of continuation in the program. Because of the small size of his program, Halsted knew his graduates well and did not graduate them until he felt they were ready. His mentoring also promoted academic surgeons who developed other specialties, including orthopaedic surgery .
Formal orthopaedic postgraduate education gradually developed during the 1920s and 1930s, but rapidly expanded after World War II. As the interest in this specialty developed, so too did the perceived need for more specialists [2, 8, 11, 13, 14]. In 1946, there were 79 “training services” with 238 residents, which expanded to 257 “training services” and 787 resident slots by 1949 . Speed  and Street  presented their ideal residency environment for adult orthopaedic surgery as consisting of 3 years, with junior and senior residents. Ideally, in the last 6 months of training, the resident would be able run a service with autonomy in clinic and in the operating room. Under such conditions, a surgeon-in-training could demonstrate competence and the ability to practice independently. Because the residencies were small (two to three per year group), resident capabilities were well known to the attending faculty members. As a result, faculty could attest to a graduate’s performance first-hand, knowing the education provided in the program as well as the capabilities of the graduate. As programs have expanded, with a greater number of residents evaluated by a greater number of faculty members, this capability has been lost.
Catalysts for Change
Often cited as the catalyst for reform, the Institute of Medicine’s (IOM) 1999 report, To Err is Human, had a major social impact on residency education by highlighting—in painstaking detail—preventable medical errors and examples of resident fatigue . But even before the explosive IOM report, there were highly-publicized warning signs that signaled the need for reform.
In 1978, Stephen Bergman, a psychiatrist writing under the pseudonym Samuel Shem MD, published The House of God, which outlined major residency education issues including lack of supervision, autonomy of residents without appropriate oversight, and subsequent resident burnout . Further concerns regarding unsupervised resident education were brought forward in the medico-legal trial regarding the death of an 18-year-old woman named Libby Zion, which raised questions about the role of overworked resident physicians in that case, and more generally, about the lack of supervision for interns and residents causing or leading to medical errors. Both of these events, coupled with the IOM report, were highly publicized both in lay media and medical journals, and served as the basis for residency education reform.
The ACGME Responds
Supervising institutions in the United States responded, based on the premise that either these medical organizations could make changes to internally police their training programs, or outside political powers would impose changes for them.
We have discussed the new ACGME changes in this space before [4, 5]. How effective have these changes been for both improving patient safety and resident education? Residency work-hour restrictions have been in place for more than 10 years, and while proponents of the present work-hour restrictions believe that patient safety, education, and quality of life have improved, evidence supporting these benefits has been largely anecdotal [6, 10]. While quality of life has been reported by resident surveys to be improved, better patient safety due to work-hour restrictions has not been demonstrated in any conclusive manner. Additionally, objectively written examinations measuring knowledge have not improved. A more recent change, prohibiting the first-year resident (postgraduate-year 1, or PGY1) from being in the hospital more than 16 hours, thus eliminating 24-hour call shifts, and having a supervisor present at all times, has been questioned. The ACGME has allowed some general surgery programs to allow PGY1 residents to take 24-hour call to determine if this can be done safely.
The ACGME’s Milestones program, defined by the ACGME as “competency-based developmental outcomes (eg, knowledge, skills, attitudes, and performance) that can be demonstrated progressively by residents and fellows from the beginning of their education through graduation to the unsupervised practice of their specialties”, is not without its detractors [1, 4]. Opponents believe the increased oversight and regulations leads to poorer preparation for independent practice and decreased surgical skill level at graduation demanding additional years in training .
Moving forward, one of the priorities of the ACGME should be to demonstrate the efficacy of the current restrictions. Depending on the evidence, the ACGME should be willing to change the present restrictions and publicly promote their efforts as a patient safety and quality measure.
The Time Is Now
Whether the metrics assessed by the ACGME’s Milestones program represent appropriate benchmarks for progress among orthopaedic surgery trainees remains to be seen. Are the 16 chosen milestones  representative of the core orthopaedic knowledge base? Some of the milestones procedures may need to be modified. For example, procedures for the pediatric septic hip are not nearly as common as knee arthroscopy, so assessments of this procedure for every resident may not be possible.
While some of the present endpoints assessed by the program may ultimately be removed or modified after this year’s ACGME review, we already know that the tools we use to assess these milestones are weak. The inability to assess the desired benchmarks both in terms of knowledge and surgical skills is a real criticism that the ACGME needs to address. The time is now to reflect and improve.
The ACGME should sponsor research in developing valid assessment tools for each of the topics to help justify the milestones, as well as to provide better quality assessment of resident’s competence. A program specifying a minimum number of procedures for a resident to demonstrate competence of a particular surgical procedure should be established in order to validate the use of such assessments. The American Board of Orthopaedic Surgery has already sponsored similar research pertaining to the intern surgical skills curriculum.
Educators of orthopedic residents ultimately share the same goal as the ACGME—advancing competent and ethical orthopaedic surgeons who are well-trained in all common orthopaedic procedures. In order to accomplish this goal, we must develop metrics for measurement of competence. We must verify that the metrics assessed by the milestones project represent appropriate benchmarks for progress among orthopaedic surgery trainees. We must provide proper supervision of trainees to ensure patient safety.
Yes, further research is needed, but change is already here. Orthopaedic educators are now required to measure and document competence. As orthopaedic educators, we must develop and validate techniques for doing this. It is our challenge to make meaningful assessments of our residents, and objectively measure competence, particularly in surgical skills.