A few years ago, I suggested in this column that core competencies for residents as defined by the Accreditation Council for Graduate Medical Education (ACGME) were “overrated” . I was correctly criticized by the commentators then—so much so, that if I were granted the Last Word now, I would simply say, “Dr. Chehade, Dr. Pinney, and Dr. Black: I was wrong and you are right.” Thus, it is with a touch of humility (induced if only temporarily by that memory), that I suggest that we rethink another of the ACGME requirements.
Well, here it goes: The ACGME requirement for resident research can be improved.
According to the ACGME , residents must either participate in sponsored research, prepare an article for a peer-reviewed publication, present research at a regional or national meeting, or participate in a structured literature review of an important topic.
Overall, these are reasonable means of attaining the worthwhile goal the ACGME describes, namely, the advancement of “residents’ knowledge of the basic principles of research, including how research is conducted, evaluated, explained to patients, and applied to patient care” . But there is room to improve the system. Consider these limitations of each of the four activities the ACGME identifies:
Participating in Sponsored Research
Participation in sponsored research surely can give the resident an appreciation of how research is conducted. Yet the requirement speaks specifically about “participation” and limits the domain to “sponsored”, that is to say, funded, research. These words suggests that the resident will be a small cog in a big machine—something more than a test-tube washer, no doubt, but not necessarily a full participant, either.
Submitting a Manuscript to a Journal
Submitting a manuscript to a journal requires not only participating in research but completing it. Nevertheless, this standard may push residents towards less-meaningful projects. That’s because journals favor positive results . Resident researchers, mindful of this bias, would tend to engage in projects likely to turn up positive results—like a reporting on a series of cases. We have enough of those low-evidence, retrospective studies already.
Presenting at a Meeting
Presenting at a meeting can be a worthwhile experience. The resident will not only have a chance to share his or her work, but the opportunity to see the work of other researchers. That could be a mixed bag, however, as the science at meetings is just not the best. Bhandari and colleagues  have shown that two-thirds of orthopaedic meeting abstracts “were not followed by publication of a full-text paper … [the] quality of reporting in abstracts proved inadequate, and inconsistencies between the final published paper and the original abstract occurred frequently.” Further, to the extent that meetings seek flashy results, this standard may induce residents to draw unwarranted inferences from their research data, to make the submission more “meeting worthy.”
(On a personal level, I can’t criticize national meetings too harshly; one of the best weeks of my life was February 15-22, 1993 at the meetings of the Orthopaedic Research Society [ORS] and the American Academy of Orthopaedic Surgeons. I was in San Francisco, on an expense account, away from on-call duties, and the Grateful Dead was playing three nights at the Oakland Coliseum. Of course, in retrospect, that trip to California might not have been the best use of time and money).
Conducting a Systematic Review
The creation of a systematic review is harder to criticize, as it offers at least three concrete benefits: The resident will learn the material in question; the process will teach the resident broadly-applicable skills in assessing published papers; and the end product, the review itself, will likely be useful to the medical community. Indeed, there is little not to like here, except perhaps one thing—that creating a systematic review can be a little boring. The process of reviewing a body of literature systematically is inherently algorithmic; it’s not hard to imagine a computer performing the same task. When performing a systematic review, creativity is not only not rewarded, it would be punished. And what fun is that?
Given all of the above, I think the ACGME requirement can be enhanced if a fifth criterion were added: Namely, that the standard would be met if the resident were to prepare a grant proposal.
For one, composing a grant proposal would require more than just “participation”; the resident must be actively engaged. Asking a good question and formulating methods to answer it promotes heightened awareness of the methods of valid science . The resident is likewise freed from the tyranny of positive results (as at this stage of the process, no results are needed at all—at least officially). There are also meetings just for grant writers [4, 8], so there is the opportunity for travel here, too. Because money is on the line, grants are typically evaluated far more rigorously than abstracts submitted to meetings. Last, because a good grant proposal requires a thorough examination of what is known already, the creation of a systematic literature review is almost inherent to the process. And there is still one more benefit—the resident may actually be awarded the grant.
So I hope CORR ® will bring back Dr. Chehade, Dr. Pinney and Dr. Black to comment. They got me last time. This time, I think, I am right … or at least a little less wrong.
Mellick J. Chehade PhD, MBBS, FRACS, FAOrthA
Associate Professor and Orthopaedic Trauma Consultant
The University of Adelaide
This time around, Dr. Bernstein uses his shovel to turn the topsoil rather than dig a hole for himself. Indeed, he makes some good points. “Participation” is such an open ticket for a varied level of actual engagement in the research. There are definite limitations and traps in what can be achieved using any of the proposed research options. Adding other options such as preparing a grant proposal is reasonable, but arguably no more a guarantee of success than the others if not done with sufficient input and reflection by the resident. For any of the options, how can we ensure that the contribution to the work from the resident is adequate and that they have become research-competent?
This is where closer inspection of the ACGME Program Requirements is required. It has not put all its eggs in the one basket. The document also states that:
“Resident education must include instruction in experimental design, hypothesis testing, and other current research methods, as well as participation in clinical or basic research” and that the “sponsoring institution and program should allocate adequate educational resources to facilitate resident involvement in scholarly activities” .
Even more importantly, the document also details faculty requirements (Section II.B.5) and includes maintaining “an environment of inquiry and scholarship with an active research component.” Additionally, “(f)aculty should encourage and support residents in scholarly activities, and the requirement for organized clinical discussions, rounds, journal clubs, and conferences, to ensure a structure for scholarly activity” .
In other words, the resident requires the right environment to support a solid grounding in research that includes quality supervision and mentorship. I believe all successful researchers can point to one or more key faculty member(s) who inspired them, and provided opportunities and guidance to pursue research interests. This is where the real challenge lies—not only in having an academic faculty, but in ensuring that they are well trained and afforded adequate time and space to provide quality supervision and mentorship to residents. This requires an alignment of institutional funding and policy that recognizes the value that good education brings to the health system which is sadly missing from most service delivery models of care .
Importantly, education in research should not be seen as something to be applied to research alone. It is the foundation for the same critical-thinking processes we need for sound clinical reasoning and a reflective approach to practice. Whether it is research data or the history and examination findings from a patient, it is evidence that needs to be accessed, interpreted and evaluated.
If nothing else, Dr. Bernstein has successfully managed to draw attention to the role of research and education in clinical practice, and the need to reflect both on the how and the why. It is, however, easy to get caught up in the finer details and miss the bigger picture.
As an aside, I had a photo taken with Carlos Santana at a music store across the road from an ORS meeting in San Francisco, CA, USA in 2001.
Stephen Pinney MD, MEd, FRCS(C)
Chief, Foot and Ankle Service
San Francisco Orthopaedic Residency Program, St. Mary’s Medical Center
Yes, Dr. Bernstein, I believe you are right this time. The development and submission of a formal research grant proposal should be accepted as a “fifth criterion” for meeting the ACGME residency requirements under the broad “Medical Knowledge” competency—and more specifically the “Basic Principles of Research” subsection. Completing a grant application would achieve the goals of this competency, namely ensuring that graduating residents have a solid understanding of how good research is conducted, evaluated, explained to patients, and applied to patient care. Furthermore, achieving this competency would not be subjective—either the grant has been completed and submitted, or it has not.
Dr. Bernstein advocates this change for its own sake—that the research requirement is inherently improved with this variation. I agree, but a meta-point is also worth making: The process of defining, implementing, and evaluating requirements is strengthened when proposals like this are entertained. Clearly, the lesson from genetics is that adaptation is necessary for long-term survival. I would argue this applies (in its own small way) to the ACGME competencies, that they must be adaptable to succeed in the long term.
Accepting the submission of a grant proposal as a means of achieving the basic research competency would represent a change to the ACGME core competencies. As such, it is worth analyzing the nature of this type of change as it has applicability to how we can think about future changes to the ACGME competencies and to the practice of medicine in general.
This change is incremental. It looks at the existing ways in which the basic research competency is being achieved and argues, effectively I believe, that the ACGME should also accept a similar research activity (writing a grant) as evidence of achievement of this competency. There is much to be gained over time with such incremental change. This type of step-wise change is the basis for many common quality and process improvement initiatives. Lean methodology, Six Sigma, total quality management, and other quality improvement strategies have proven effective in other industries and are now being introduced to the healthcare setting with the goal of transforming care delivery over time. It may be that there are other incremental additions that could be added as a means of achieving the basic research (and other) core competencies. We should be open to other ways in which achieving these competencies can be demonstrated, provided that the claim of competency can be validated.
In addition to being open to incremental change, perhaps we should also be open to radically different disruptive ways of demonstrating competency. Rather than starting with existing methods of achieving a competency and working forward with an incremental addition, could we not start with the competency itself and work backwards? Using this framework any valid way of demonstrating competency in understanding of basic research could be considered evidence of achieving competency. A resident actively participates in building a blog devoted to assessing the scientific validity of published research studies. Another resident creates an interactive educational module explaining the scientific method and the various pitfalls in performing research and analyzing research results. Could these types of endeavors if done well not provide ample evidence of competency in basic research?
Perhaps as a profession we are not yet ready to accept this type of free-form evidence that a competency or end-goal has been achieved. However, this type of unencumbered thinking will be increasingly common in healthcare. It is the basis for design thinking—a strategy that focuses on the user experience and the end goal, then works backwards. This trend to focus on the desired final product of a process and think creatively about how it can be achieved is coming whether we like it or not. As such we might consider some increased flexibility in what we accept as evidence for the achievement of the competencies that have been established for our profession.
Kevin P. Black MD
Professor and C. McCollister Evarts Chairman, Department of Orthopaedics and Rehabilitation
Penn State Milton S. Hershey Medical Center
I don’t always agree with Dr. Bernstein, as evidenced in my previous response to his commentary regarding the “overrated” core competencies . But he is a thoughtful and committed educator who is always ready to challenge dogma and the requirements that we are generally too eager to accept.
His suggestion that an additional option, specifically that a resident take a leadership role in writing a grant, is quite reasonable. The resident would develop a hypothesis, review the evidence, and propose a scientific methodology to answer the question. When compared to the other options, it seems to have the potential to achieve the desired goal of advancing resident knowledge through the basic principles of research.
But that’s the easy part. I’m a little surprised he didn’t go further and question the value and intent of the ACGME “Resident Research Requirement”. Dr. Bernstein quotes Section IV.B.1 of the ACGME Program Requirements for Orthopaedic Surgery: “The curriculum must advance residents’ knowledge of the basic principles of research, including how research is conducted, evaluated, explained to patients, and applied to patient care” . But it goes on to state that residents must participate in clinical or basic research. In Section IV.B.2, it adds, “Residents should participate in scholarly activity.” What I find myself questioning is the mandate that residents must participate in clinical or basic research. There are numerous aspects of the resident scholarly activity requirement that I fully support—from the need for residents to learn the principles of experimental design, to hypothesis testing, and research methodology. But do residents really need to participate in a research project to accomplish this? I don’t think so.
Research is critical to the advancement of musculoskeletal care and part of the mission of every academic orthopaedic department in the country. We have more than 200 active IRB projects in my department at present. But meaningful participation in a clinical or basic science research project takes a considerable amount of time and pulls residents away from their other patient-care learning opportunities. Why not make participation in clinical or basic science research optional? Research projects are an essential experience for those considering academic careers, which only a minority of our graduates choose to do.
Finally, I would like to point out what I believe to be an inconsistency in the ACGME requirements related to resident research and the Orthopaedic Surgery Milestone Project, a joint initiative between the ACGME and the American Board of Orthopaedic Surgery. Being a primary author/presenter of original research is identified as a Level 5 medical knowledge milestone and described as an aspirational goal that few, exceptional residents will achieve. This does not seem to be aligned with the resident research requirement.
So, Joe, I think your suggestion is a good one. I’m just a little disappointed that you didn’t go a little further in challenging the scholarly activity requirement. Or, maybe I’m the one who is wrong this time.
ACGME Program Requirements for Graduate Medical Education in Orthopaedic Surgery. Available at: https://www.acgme.org/Portals/0/PFAssets/ProgramRequirements/260_orthopaedic_surgery_2016.pdf. Accessed April 24, 2017.
Bernstein J. Not the Last Word: The ACGME core competencies are overrated. Clin Orthop Relat Res. 2014;472:2932–2937.
Bhandari M, Devereaux PJ, Guyatt GH, Cook DJ, Swiontkowski MF, Sprague S, Schemitsch EH. An observational study of orthopaedic abstracts and subsequent full-text publications. J Bone Joint Surg Am. 2002;84A:615–621.
Bone and Joint Initiative. Call for applications. Available at: https://www.usbji.org/programs/yii/call-for-applications. April 24, 2017.
Brand RA. Writing for Clinical Orthopaedics and Related Research. Clin Orthop Relat Res. 2008;466:239–247.
Chehade MJ, Gill TK, Kopansky-Giles D, Schuwirth L, Karnon J, McLiesh P, Alleyne J, Woolf AD. Building multidisciplinary health workforce capacity to support the implementation of integrated, people-centered models of care for musculoskeletal health. Best Pract Res Clin Rheumatol. 2016;30:559–584.
Emerson GB, Warme WJ, Wolf FM, Heckman JD, Brand RA, Leopold SS. Testing for the presence of positive-outcome bias in peer review: a randomized controlled trial. Arch Intern Med. 2010;170:1934–1939.
Orthopaedic Research Society. ORS/OREF/AAOS Grant Writing. Available at: http://www.ors.org/orsorefaaos-grant-writing/. Accessed April 24, 2017.
A note from the Editor-in-Chief:
We are pleased to present to readers of Clinical Orthopaedics and Related Research ® the next Not the Last Word. The goal of this section is to explore timely and controversial issues that affect how orthopaedic surgery is taught, learned, and practiced. We welcome reader feedback on all of our columns and articles; please send your comments to firstname.lastname@example.org.
The author certifies that neither he, nor any members of his immediate family, have any commercial associations (such as consultancies, stock ownership, equity interest, patent/licensing arrangements, etc.) that might pose a conflict of interest in connection with the submitted article.
All ICMJE Conflict of Interest Forms for authors and Clinical Orthopaedics and Related Research ® editors and board members are on file with the publication and can be viewed on request.
The opinions expressed are those of the writers, and do not reflect the opinion or policy of Clinical Orthopaedics and Related Research ® or The Association of Bone and Joint Surgeons®.
About this article
Cite this article
Bernstein, J. Not the Last Word: Rethinking the Resident Research Requirement. Clin Orthop Relat Res 475, 1948–1953 (2017). https://doi.org/10.1007/s11999-017-5392-8
- Resident Research
- Accreditation Council For Graduate Medical Education (ACGME)
- ACGME Requirement
- ACGME Competencies
- Analyzing Research Results