INTRODUCTION

Given the variability in the types and numbers of procedures performed by internists in practice, the American Board of Internal Medicine (ABIM) rescinded numeric procedural requirements in July 2007 in favor of residents being able to “recognize indications, contraindications and manage complications” of selected bedside procedures1. ABIM’s change in policy was aimed to limit the practice of “learning by doing,” and encourage the the competency-based education and assessment now recommended by the ACGME2,3.

However, at many of the nation’s academic hospitals, bedside invasive procedures, such as paracentesis, thoracentesis, lumbar puncture and central venous catheter (CVC) insertion, continue to be performed by trainees with widely variable experience and skill4. Evidence suggests more procedural experience is associated with reduced complications5–8, perhaps leading graduating Internal Medicine residents to report feeling comfortable performing inpatient bedside procedures9–11. However, few articles have examined residents’ experiences and perceptions around supervision during the years of their training, or how often peer-to-peer supervision may produce unsafe (or at least uncomfortable) clinical situations.

To better understand residents’ experiences with learning bedside procedures, we carried out a cross-sectional survey of medicine residents in a network of training hospitals in California. The aim of this study was to examine trainees’ perceptions of procedural training during their residency, comfort performing procedures, their perceptions of supervision as well as potential contributing factors to reported complications.

METHODS

Sites and subjects

We surveyed trainees at seven teaching institutions participating in the Medical Education Research Network (MERN) of California. MERN is comprised of nine internal medicine residency programs from SGIM’s California and Hawaii Region, who meet annually to collaborate on educational research initiatives. MERN institutions include one community-based (Kaiser Permanente Medical Center-Oakland) and six university-based teaching institutions [Stanford, University of California (UC) Irvine, UC Los Angeles, UC San Diego, UC San Francisco and University of Southern California]. Participating MERN sites had ABIM board pass rates slightly above national averages, with MERN institutions averaging a 96% pass rate (91–100%) compared to a national average of 92%12. At the time of our survey, no MERN site had a didactic procedure service, and only three programs provided systems to track procedure completion and supervision. However, use of these systems was not required, and participation was low as a result. For example, at one site only 32% of all IM residents used the system in the year of our study. This study was reviewed and approved by the IRB and Committee on Human Research in each participating institution. Using rosters provided by the sites, we identified 653 first, second or third year Internal Medicine residents. We excluded preliminary residents and medicine-pediatrics residents.

Survey development

The survey used in this study was a refined version of a survey we piloted at UCSF in 2007. Our 2007 survey was developed by the study authors including the associate program director responsible for meeting residency core competencies. The survey asked basic demographic information in addition to questions regarding procedure frequency, resident reported comfort and experience with complications for paracentesis, thoracentesis, lumbar puncture and CVC insertion. These procedures were selected from the procedures listed in the ABIM’s Policies and Procedures for Certification 1 based on their frequency and risk for complications2,13–16. We enhanced our 2007 survey by adding questions focused on the following areas: (1) experience with performance of these procedures on a simulator; (2) comfort performing and supervising these procedures; (3) whether the procedures were supervised and how supervision was perceived; (4) complications, if any, and whether the resident felt there were any obvious contributing causes. Complications were defined by those most commonly described in the literature14–17.

All survey questions underwent a formal review for content validity by the UCSF Department of Medical Education. After the formal review, clarity and comprehension of questions were finalized by pretesting among a group of attending physicians and residents as well as the investigative team. (See online Appendix.)

Survey process

Our survey was distributed via e-mail invitation at the end of the 2008 academic year, and responses between May 2008 and July 2008 were included. Each survey e-mail was accompanied by a note describing the purposes of the study and that results would remain anonymous and be used for research purposes only. Up to three reminder e-mails were sent to non-respondents, with an option to decline to receive additional automated reminders with each e-mail.

Participants were specifically asked to reflect on procedures performed in non-emergent situations and to estimate the total number of common procedures [paracenteses, thoracenteses, lumbar punctures, femoral central venous catheter insertions and internal jugular (IJ)/subclavian CVC insertions] they had performed during training.

Data Analyses

We first characterized survey respondents using simple univariate and bivariate statistics, which we used to determine frequencies of responses to questions in our major domains. In order to replicate previous studies, we determined a ‘comfort threshold,’ defined as the reported case volumes where three-fourths of residents agreed to being comfortable performing a procedure independently2,18,19.

We used multivariate logistic regression methods to determine factors associated with reaching the ‘comfort threshold.’ Potential factors chosen from our demographic survey and entered into models manually included: trainee site, gender, career aspiration in an invasive subspecialty, residency year, report of a complication, experience on a simulator and increasing numbers of procedures performed. Factors were retained in the model if they remained associated with the outcome of interest at p < 0.05 or lower. All statistical tests were performed using the SAS statistical application program (release 9.2).

RESULTS

Respondent Characteristics

Our survey response rate was 56% (367/653) and ranged from 31–72% across sites. Respondents were evenly distributed across residency year; differences in gender and career aspirations were similar to national residency profiles20. Responders and non-responders did not differ significantly in gender or residency year, but response rate by site was a significant source of responder variation (Table 1).

Table 1 Respondent Characteristics (n = 367)
Full size table

Procedure Volume

The median number of procedures reported by each resident increased with each successive postgraduate year. Most residents had performed fewer than five of each procedure at the end of the PGY-1 year, but had performed at least ten for nearly every procedure by the end of their training. Thoracentesis was consistently the least often performed procedure. The range of experience for each procedure was wide, demonstrating a variable experience of residents even within the same program. Residents reported little experience with procedure simulation, with only 4% of residents having experience with simulation for paracentesis, thoracentesis and lumbar puncture, and 21% having experience with CVC insertion simulation (Table 2).

Table 2 Median Procedure Volume by Resident Year
Full size table

Comfort Performing Procedures

Resident comfort increased with the number of procedures performed (p < 0.001, Fig. 1a). More than three-fourths of residents reported being comfortable (our defined “comfort threshold”) with paracentesis, lumbar puncture and femoral CVCs after having performed three to four, thoracentesis after having performed five to six and IJ and subclavian CVCs after seven to nine. Nearly all residents were comfortable after having performed ten procedures. The residency year at which trainees surpass the comfort threshold is seen in Figure 1b.

Figure 1
figure 1

Resident comfort with procedure performance. (a) Percentage of residents reporting being “comfortable” or “very comfortable” performing each procedure based on numbers of procedures performed. (b) Percentage of residents reporting being “comfortable” or “very comfortable” performing each procedure by resident year.

Full size image
Figure 2
figure 2

Resident comfort with procedure supervision. (a) Percentage of residents reporting being “comfortable” or “very comfortable” supervising each procedure based on numbers of procedures performed. (b) Percentage of residents reporting being “comfortable” or “very comfortable” supervising each procedure by resident year.

Full size image

In multivariable models adjusting for PGY year, gender, training site and whether the trainee was planning a procedural specialty, the number of procedures remained strongly independently associated with meeting the comfort threshold. The adjusted odds ratio for each incremental procedure yielding comfort was 2.48 (95% CI: 1.81, 3.40) for paracentesis, 2.31 (95% CI: 1.85, 2.87) for thoracentesis, 2.34 (95% CI: 1.79, 3.05) for lumbar puncture, 2.31 (95% CI: 1.80, 2.96) for femoral CVCs and 1.90 (95% CI: 1.65, 2.20) for IJ or subclavian CVCs, all significant at the p < 0.05 level. Neither pursuit of invasive subspecialty career, nor use of a simulator, nor report of a complication was a significant predictor of comfort. Resident year had only a very limited impact, being only associated with increased comfort for PGY-3s performing femoral CVCs (OR 3.55, CI 1.01–12.53). Interestingly, male gender was associated with a two- to three-fold statistically significant increase in comfort for all procedures except paracentesis (OR 2.60–2.78). Site-specific variations in comfort were also noted, though these differences were based on small numbers of procedures reported within each subgroup at each site.

Procedure Supervision

Ninety-eight percent of residents reported peer supervision for or independent performance of paracentesis, thoracentesis and lumbar puncture, while 70% reported peer supervision or independent performance of CVC insertion. By the end of PGY-2 year, more than 60% of residents had supervised thoracencenteses and CVC insertion, and more than 80% had supervised paracentesis and lumbar puncture. By the end of PGY-3 year, more than 80% of residents had supervised all procedures. Satisfaction with the levels of supervision was high for all procedures and did not vary significantly by reported supervisor or by resident year. Despite the large number of residents reporting that their responsibilities include supervision of procedures, comfort supervising was only achieved by the end of the PGY-2 year for lumbar puncture and femoral CVCs, and by the end of the PGY-3 year for thoracentesis and IJ/subclavian CVCs, the latter procedures being those with the highest complication rates14. A sizable proportion of residents reported supervising a procedure prior to feeling comfortable with the procedure themselves (Fig. 2, Table 3).

Table 3 Supervision Practices of Residents
Full size table

Complications

The majority (64%) of residents’ reported a complication with a procedure (defined as arterial puncture, arterial hematoma, cerebrospinal fluid leak requiring intervention, epidural hematoma, guide wire/catheter tip embolism, hemorrhage, hemothorax, inadequate analgesia, organ puncture, pneumothorax, venous hematoma). Of the 439 complications reported, 140 (32%) were instances of inadequate analgesia, 109 (25%) arterial puncture, 69 (16%) venous hematoma, 49 (11%) pneumothorax, 30 (7%) arterial hematoma, 18 (4%) CSF leak requiring intervention, 7 (2%) each of hemothorax and hemorrhage, and ≤4 (≤1%) each of bowel perforation, epidural hematoma, guide wire embolism and death. Eighty-eight percent of these residents felt that the complication could have been avoided.

When asked to select factors that would have prevented complications, 89% of residents felt that that quality of supervision had no role in avoiding the complications. The most commonly reported factors that could have avoided complications were better analgesia (20%), better patient positioning (15%), use of ultrasound (11%), better training in proper procedure technique (10%) and better characterization of anatomy (10%). Those who reported a complication had no differences in their comfort performing or supervising a procedure.

DISCUSSION

In this multicenter, cross-sectional survey, residents’ procedural experience and comfort were low in early and mid-level trainees and higher in more senior trainees. However, many residents reported supervising procedures they felt uncomfortable performing, and most reported involvement in at least one procedure with a complication. Residents felt that most complications would not have been prevented with improved supervision, even though many reported complications represented issues stemming from inadequate training or supervision (e.g., positioning, use of ultrasound).

While the vast majority of respondents were satisfied with the level of supervision, a large number of the supervisors in our study admitted to supervising prior to being comfortable with procedure technique. It is possible that difficulties providing criticism of peers and lack of qualified role models may underlie the disconnect between satisfaction with peer supervision and ‘premature’ supervisory roles. All institutions lacked formal competency assessment to ensure residents had mastery of procedure technique before attempting independent performance and supervision; lack of competency-based training would also potentially exacerbate ‘premature’ supervisory roles. Finally, peer-to-peer supervision and lack of qualified supervisors may have also limited residents’ level of insight into the causes of complications. Even though residents reporting experiencing complications amenable to supervision (in which case the supervisor could assist in overcoming the challenge) or training (where knowledge and self-efficacy could be maximized prior to the procedure), they rarely selected inadequate supervision or training as a reason for a complication21. Learning primarily from inexperienced peers, residents may have a poor understanding of the role of supervision not only in stepwise instruction, but also in anticipating and troubleshooting potential complications.

Our findings are particularly important given national trends that suggest Internal Medicine residents are performing fewer bedside procedures22–24. The number and variety of bedside procedures performed by board-certified internists are falling as well13. As a result, Internal Medicine residency programs are facing a lack of qualified trainees and supervisors for invasive bedside procedures. A first step in rectifying this shortage will be to rethink the traditional clinical practice of “learning by doing.” To this end, MERN participating sites have begun employing results from this survey to justify systems and support to provide increased procedural training. All seven institutions have web-based applications to track procedure completion, though three report suboptimal compliance by residents. Four institutions now require that a supervising resident have completed a requisite number of supervised procedures prior to independent performance or supervising others. Five institutions have incorporated simulation in their resident procedures training (both high and low fidelity), and two institutions have instituted a didactic procedure service at one of their hospital sites.

Simulation technology provides potentially useful approaches to ensure that residents become comfortable performing and supervising common procedures before performance in clinical settings25,26. Recent literature suggests that simulation not only aids in the development of the clinical knowledge and skills necessary for safe performance27,28, but also translates into measurable improvements in patient care29,30. At places with less established simulation technology, procedure services can also provide residents with the didactic teaching, opportunities for deliberate practice and supervision during procedures. Use of procedure services for resident education has been shown to increase resident comfort and self-perceived knowledge with a low rate of complications23,31,32.

This study has several limitations. As a cross-sectional survey, our data are a snapshot of a population of trainees who were at the end of a year of training and were therefore at a standard point, but who were asked to recall their experiences. A major limitation of our study is that resident’s procedure experience is determined by self-report alone. Residents who are comfortable with a procedure might overestimate the number of procedures they performed, whereas those uncomfortable may be more likely to minimize their experience. Existing tools available at the time of the study to track procedure completion provided limited and potentially unreliable results. Additionally, because we could not observe residents directly, our survey used reported level of comfort as a surrogate for competency. Competency refers to a resident’s ability to safely prepare for, perform and navigate the complications of a procedure, whereas comfort is likely mitigated by performer characteristics and may not correlate with observed performance33. The finding that women were less comfortable for most procedures might possibly reflect a published gender difference in comfort with technically invasive procedures rather than any difference in ability, and demonstrates the limitations of self-assessment34–36. Our survey cannot account for differences between responders and non-responders in procedural experience and comfort, and is likely subject to non-response bias. Though responders did not differ from non-responders in resident year and gender, plausibly those who were more experienced in procedures or more satisfied with their procedural experience and training may have not responded. Additionally our response rate is highly variable by site, leaving several sites underrepresented. We studied comfort threshold as a measure of a required procedure volume. While our threshold was chosen based on previous studies, in our data the cutoff left one-fourth of participants feeling uncomfortable. Our study was performed primarily at university-based teaching centers and may not represent the experiences at community-based teaching hospitals, or procedures performed by non-trainees. Lastly, as a cross-sectional study, we cannot establish a causal link between procedure volume and reported comfort performing procedures.

CONCLUSION

Even as certification requirements change, residents continue to face situations in which they are expected to perform these procedures supervised by inexperienced, uncomfortable and perhaps under-qualified residents. At the institutions we surveyed, many residents are still performing procedures, supervising procedures prior to being comfortable performing them and report a high frequency of complications. Advances in simulation and the advent of didactic procedure services offer options to residency programs seeking to provide effective procedural education and increase patient safety.