Supervising the Supervisors—Procedural Training and Supervision in Internal Medicine Residency
- First Online:
- Cite this article as:
- Mourad, M., Kohlwes, J., Maselli, J. et al. J GEN INTERN MED (2010) 25: 351. doi:10.1007/s11606-009-1226-z
- 645 Views
At teaching hospitals, bedside procedures (paracentesis, thoracentesis, lumbar puncture, arthrocentesis and central venous catheter insertion) are performed by junior residents and supervised by senior peers. Residents’ perceptions about supervision or how often peer supervision produces unsafe clinical situations are unknown.
To examine the experience and practice patterns of residents performing bedside procedures.
DESIGN AND PARTICIPANTS
Cross-sectional e-mail survey of 653 internal medicine (IM) residents at seven California teaching hospitals.
Surveys asked questions in three areas: (1) resident experience performing procedures: numbers of procedures performed and whether they received other (e.g., simulator) training; (2) resident comfort performing and supervising procedures; (3) resident reports of their current level of supervision doing procedures, experience with complications as well as perceptions of factors that may have contributed to complications.
Three hundred sixty-seven (56%) of the residents responded. Most PGY1 residents had performed fewer than five of any of the procedures, but most PGY-3 residents had performed at least ten by the end of their training. Resident comfort for each procedure increased with the number of procedures performed (p < 0.001). Although residents reported that peer supervision happened often, they also reported high rates of supervising a procedure before feeling comfortable with proper technique. The majority of residents (64%) reported at least one complication and did not feel supervision would have prevented complications, even though many reported complications represented technique- or preparation-related problems.
Residents report low levels of comfort and experience with procedures, and frequently report supervising prior to feeling comfortable. Our findings suggest a need to examine best practices for procedural supervision of trainees.
KEY WORDSmedical student and residency educationmedical educationcurriculum/program evaluationsupervision
Given the variability in the types and numbers of procedures performed by internists in practice, the American Board of Internal Medicine (ABIM) rescinded numeric procedural requirements in July 2007 in favor of residents being able to “recognize indications, contraindications and manage complications” of selected bedside procedures1. ABIM’s change in policy was aimed to limit the practice of “learning by doing,” and encourage the the competency-based education and assessment now recommended by the ACGME2,3.
However, at many of the nation’s academic hospitals, bedside invasive procedures, such as paracentesis, thoracentesis, lumbar puncture and central venous catheter (CVC) insertion, continue to be performed by trainees with widely variable experience and skill4. Evidence suggests more procedural experience is associated with reduced complications5–8, perhaps leading graduating Internal Medicine residents to report feeling comfortable performing inpatient bedside procedures9–11. However, few articles have examined residents’ experiences and perceptions around supervision during the years of their training, or how often peer-to-peer supervision may produce unsafe (or at least uncomfortable) clinical situations.
To better understand residents’ experiences with learning bedside procedures, we carried out a cross-sectional survey of medicine residents in a network of training hospitals in California. The aim of this study was to examine trainees’ perceptions of procedural training during their residency, comfort performing procedures, their perceptions of supervision as well as potential contributing factors to reported complications.
Sites and subjects
We surveyed trainees at seven teaching institutions participating in the Medical Education Research Network (MERN) of California. MERN is comprised of nine internal medicine residency programs from SGIM’s California and Hawaii Region, who meet annually to collaborate on educational research initiatives. MERN institutions include one community-based (Kaiser Permanente Medical Center-Oakland) and six university-based teaching institutions [Stanford, University of California (UC) Irvine, UC Los Angeles, UC San Diego, UC San Francisco and University of Southern California]. Participating MERN sites had ABIM board pass rates slightly above national averages, with MERN institutions averaging a 96% pass rate (91–100%) compared to a national average of 92%12. At the time of our survey, no MERN site had a didactic procedure service, and only three programs provided systems to track procedure completion and supervision. However, use of these systems was not required, and participation was low as a result. For example, at one site only 32% of all IM residents used the system in the year of our study. This study was reviewed and approved by the IRB and Committee on Human Research in each participating institution. Using rosters provided by the sites, we identified 653 first, second or third year Internal Medicine residents. We excluded preliminary residents and medicine-pediatrics residents.
The survey used in this study was a refined version of a survey we piloted at UCSF in 2007. Our 2007 survey was developed by the study authors including the associate program director responsible for meeting residency core competencies. The survey asked basic demographic information in addition to questions regarding procedure frequency, resident reported comfort and experience with complications for paracentesis, thoracentesis, lumbar puncture and CVC insertion. These procedures were selected from the procedures listed in the ABIM’s Policies and Procedures for Certification1 based on their frequency and risk for complications2,13–16. We enhanced our 2007 survey by adding questions focused on the following areas: (1) experience with performance of these procedures on a simulator; (2) comfort performing and supervising these procedures; (3) whether the procedures were supervised and how supervision was perceived; (4) complications, if any, and whether the resident felt there were any obvious contributing causes. Complications were defined by those most commonly described in the literature14–17.
All survey questions underwent a formal review for content validity by the UCSF Department of Medical Education. After the formal review, clarity and comprehension of questions were finalized by pretesting among a group of attending physicians and residents as well as the investigative team. (See online Appendix.)
Our survey was distributed via e-mail invitation at the end of the 2008 academic year, and responses between May 2008 and July 2008 were included. Each survey e-mail was accompanied by a note describing the purposes of the study and that results would remain anonymous and be used for research purposes only. Up to three reminder e-mails were sent to non-respondents, with an option to decline to receive additional automated reminders with each e-mail.
Participants were specifically asked to reflect on procedures performed in non-emergent situations and to estimate the total number of common procedures [paracenteses, thoracenteses, lumbar punctures, femoral central venous catheter insertions and internal jugular (IJ)/subclavian CVC insertions] they had performed during training.
We first characterized survey respondents using simple univariate and bivariate statistics, which we used to determine frequencies of responses to questions in our major domains. In order to replicate previous studies, we determined a ‘comfort threshold,’ defined as the reported case volumes where three-fourths of residents agreed to being comfortable performing a procedure independently2,18,19.
We used multivariate logistic regression methods to determine factors associated with reaching the ‘comfort threshold.’ Potential factors chosen from our demographic survey and entered into models manually included: trainee site, gender, career aspiration in an invasive subspecialty, residency year, report of a complication, experience on a simulator and increasing numbers of procedures performed. Factors were retained in the model if they remained associated with the outcome of interest at p < 0.05 or lower. All statistical tests were performed using the SAS statistical application program (release 9.2).
Respondent Characteristics (n = 367)
Resident year, n (%)
Training site, n (%)
Kaiser Permanente Medical Center, Oakland
UC Los Angeles
UC San Diego
UC San Francisco
University Southern California
Response rate, n (%)
Kaiser Permanente Medical Center, Oakland
UC Los Angeles
UC San Diego
UC San Francisco
University Southern California
Male, n (%)
Career aspirations, n (%)
Satisfied with supervision for paracentesis/thoracentesis/lumbar puncture, n (%)
Satisfied with supervision for central venous catheter insertion, n (%)
Median Procedure Volume by Resident Year
PGY1 median (range) n = 126
PGY2 median (range) n = 119
PGY3 median (range) n = 122
Comfort Performing Procedures
In multivariable models adjusting for PGY year, gender, training site and whether the trainee was planning a procedural specialty, the number of procedures remained strongly independently associated with meeting the comfort threshold. The adjusted odds ratio for each incremental procedure yielding comfort was 2.48 (95% CI: 1.81, 3.40) for paracentesis, 2.31 (95% CI: 1.85, 2.87) for thoracentesis, 2.34 (95% CI: 1.79, 3.05) for lumbar puncture, 2.31 (95% CI: 1.80, 2.96) for femoral CVCs and 1.90 (95% CI: 1.65, 2.20) for IJ or subclavian CVCs, all significant at the p < 0.05 level. Neither pursuit of invasive subspecialty career, nor use of a simulator, nor report of a complication was a significant predictor of comfort. Resident year had only a very limited impact, being only associated with increased comfort for PGY-3s performing femoral CVCs (OR 3.55, CI 1.01–12.53). Interestingly, male gender was associated with a two- to three-fold statistically significant increase in comfort for all procedures except paracentesis (OR 2.60–2.78). Site-specific variations in comfort were also noted, though these differences were based on small numbers of procedures reported within each subgroup at each site.
Supervision Practices of Residents
Number of residents who reported supervising before feeling comfortable with procedure performance
n = 126 (%)
n = 119 (%)
n = 122 (%)
n = 367 (%)
Internal jugular CVC
The majority (64%) of residents’ reported a complication with a procedure (defined as arterial puncture, arterial hematoma, cerebrospinal fluid leak requiring intervention, epidural hematoma, guide wire/catheter tip embolism, hemorrhage, hemothorax, inadequate analgesia, organ puncture, pneumothorax, venous hematoma). Of the 439 complications reported, 140 (32%) were instances of inadequate analgesia, 109 (25%) arterial puncture, 69 (16%) venous hematoma, 49 (11%) pneumothorax, 30 (7%) arterial hematoma, 18 (4%) CSF leak requiring intervention, 7 (2%) each of hemothorax and hemorrhage, and ≤4 (≤1%) each of bowel perforation, epidural hematoma, guide wire embolism and death. Eighty-eight percent of these residents felt that the complication could have been avoided.
When asked to select factors that would have prevented complications, 89% of residents felt that that quality of supervision had no role in avoiding the complications. The most commonly reported factors that could have avoided complications were better analgesia (20%), better patient positioning (15%), use of ultrasound (11%), better training in proper procedure technique (10%) and better characterization of anatomy (10%). Those who reported a complication had no differences in their comfort performing or supervising a procedure.
In this multicenter, cross-sectional survey, residents’ procedural experience and comfort were low in early and mid-level trainees and higher in more senior trainees. However, many residents reported supervising procedures they felt uncomfortable performing, and most reported involvement in at least one procedure with a complication. Residents felt that most complications would not have been prevented with improved supervision, even though many reported complications represented issues stemming from inadequate training or supervision (e.g., positioning, use of ultrasound).
While the vast majority of respondents were satisfied with the level of supervision, a large number of the supervisors in our study admitted to supervising prior to being comfortable with procedure technique. It is possible that difficulties providing criticism of peers and lack of qualified role models may underlie the disconnect between satisfaction with peer supervision and ‘premature’ supervisory roles. All institutions lacked formal competency assessment to ensure residents had mastery of procedure technique before attempting independent performance and supervision; lack of competency-based training would also potentially exacerbate ‘premature’ supervisory roles. Finally, peer-to-peer supervision and lack of qualified supervisors may have also limited residents’ level of insight into the causes of complications. Even though residents reporting experiencing complications amenable to supervision (in which case the supervisor could assist in overcoming the challenge) or training (where knowledge and self-efficacy could be maximized prior to the procedure), they rarely selected inadequate supervision or training as a reason for a complication21. Learning primarily from inexperienced peers, residents may have a poor understanding of the role of supervision not only in stepwise instruction, but also in anticipating and troubleshooting potential complications.
Our findings are particularly important given national trends that suggest Internal Medicine residents are performing fewer bedside procedures22–24. The number and variety of bedside procedures performed by board-certified internists are falling as well13. As a result, Internal Medicine residency programs are facing a lack of qualified trainees and supervisors for invasive bedside procedures. A first step in rectifying this shortage will be to rethink the traditional clinical practice of “learning by doing.” To this end, MERN participating sites have begun employing results from this survey to justify systems and support to provide increased procedural training. All seven institutions have web-based applications to track procedure completion, though three report suboptimal compliance by residents. Four institutions now require that a supervising resident have completed a requisite number of supervised procedures prior to independent performance or supervising others. Five institutions have incorporated simulation in their resident procedures training (both high and low fidelity), and two institutions have instituted a didactic procedure service at one of their hospital sites.
Simulation technology provides potentially useful approaches to ensure that residents become comfortable performing and supervising common procedures before performance in clinical settings25,26. Recent literature suggests that simulation not only aids in the development of the clinical knowledge and skills necessary for safe performance27,28, but also translates into measurable improvements in patient care29,30. At places with less established simulation technology, procedure services can also provide residents with the didactic teaching, opportunities for deliberate practice and supervision during procedures. Use of procedure services for resident education has been shown to increase resident comfort and self-perceived knowledge with a low rate of complications23,31,32.
This study has several limitations. As a cross-sectional survey, our data are a snapshot of a population of trainees who were at the end of a year of training and were therefore at a standard point, but who were asked to recall their experiences. A major limitation of our study is that resident’s procedure experience is determined by self-report alone. Residents who are comfortable with a procedure might overestimate the number of procedures they performed, whereas those uncomfortable may be more likely to minimize their experience. Existing tools available at the time of the study to track procedure completion provided limited and potentially unreliable results. Additionally, because we could not observe residents directly, our survey used reported level of comfort as a surrogate for competency. Competency refers to a resident’s ability to safely prepare for, perform and navigate the complications of a procedure, whereas comfort is likely mitigated by performer characteristics and may not correlate with observed performance33. The finding that women were less comfortable for most procedures might possibly reflect a published gender difference in comfort with technically invasive procedures rather than any difference in ability, and demonstrates the limitations of self-assessment34–36. Our survey cannot account for differences between responders and non-responders in procedural experience and comfort, and is likely subject to non-response bias. Though responders did not differ from non-responders in resident year and gender, plausibly those who were more experienced in procedures or more satisfied with their procedural experience and training may have not responded. Additionally our response rate is highly variable by site, leaving several sites underrepresented. We studied comfort threshold as a measure of a required procedure volume. While our threshold was chosen based on previous studies, in our data the cutoff left one-fourth of participants feeling uncomfortable. Our study was performed primarily at university-based teaching centers and may not represent the experiences at community-based teaching hospitals, or procedures performed by non-trainees. Lastly, as a cross-sectional study, we cannot establish a causal link between procedure volume and reported comfort performing procedures.
Even as certification requirements change, residents continue to face situations in which they are expected to perform these procedures supervised by inexperienced, uncomfortable and perhaps under-qualified residents. At the institutions we surveyed, many residents are still performing procedures, supervising procedures prior to being comfortable performing them and report a high frequency of complications. Advances in simulation and the advent of didactic procedure services offer options to residency programs seeking to provide effective procedural education and increase patient safety.
No special acknowledgements
Conflict of Interest