INTRODUCTION

The American Board of Internal Medicine (ABIM) instituted initial certification and maintenance of certification (MOC) examinations in support of its mission “To enhance the quality of health care by certifying internists and subspecialists who demonstrate the knowledge, skills and attitudes essential for excellent patient care.”1 Evidence has accumulated demonstrating that care provided by certified internists, compared to those who are not certified, is associated with better processes and outcomes,2 including mortality.3,4 While associations between certification and better care are fairly widely accepted, fewer studies have assessed associations of care with MOC.5,6,–7 One factor in the controversy over MOC in general, and the MOC examination in particular, has been skepticism about evidence of impact on patient care.8,9,10,11,12,–13

State medical board disciplinary action is an important outcome that matters to patients; it is directly related to quality of patient care and, in many cases, patient safety.14 Higher performance on the ABIM Internal Medicine (IM) initial certification examination is strongly correlated with decreased incidence of state medical board disciplinary actions.15 Physicians certified by ABIM have a five-fold decreased incidence of disciplinary actions compared to those who start IM residency, but fail to become certified.16 Among disciplined physicians, the distribution of severity of actions is less severe in certified compared to never certified physicians.16

Since IM MOC examinations are required at 10-year intervals for physicians initially certified after 1989,2 enough time has passed to accumulate sizable cohorts of physicians required to take the MOC examination to remain certified. Based on findings associated with initial certification, we hypothesized that those who pass the IM MOC examination within 10 years of initial certification would have lower rates of disciplinary actions compared to those who do not take or do not pass the examination. Specifically, we sought to address four primary hypotheses:

  1. 1.

    There is an inverse association between passing the MOC examination and the risk of disciplinary actions.

  2. 2.

    The trends in hazard rates for disciplinary actions over time are lower for those passing the MOC examination within 10 years of initial certification compared to those who do not pass within this time frame.

  3. 3.

    The risk of disciplinary action against physicians declines as scores on the MOC examination increase.

  4. 4.

    Disciplinary actions are proportionally less severe among those passing the MOC examination compared to those who do not pass within 10 years of initial certification.

METHODS

Study Population

General internists were eligible for inclusion if they passed the ABIM IM certification examination between 1/1/1990 and 12/31/2003 (n = 47,971). We did not include anyone ever certified in an IM subspecialty. Because our focus was the MOC examination and certificates were issued for 10 years during the study period, participants’ start dates in the study began 10 years following their initial certification year. Participants were followed until 12/31/2015. Using data available in the ABIM database, we excluded anyone believed to practice outside the USA (n = 1376), older than 75 years when passing the initial certification examination (n = 2), known to be dead, retired, or in restitution for behaviors not associated with medical licensure (n = 202). Because a disciplined license precludes taking the MOC examination, we excluded those disciplined prior to their study start date (n = 991) to avoid overestimating potential associations with the MOC examination.

Measurements

The primary outcome measure was time (years) to disciplinary action. The Federation of State Medical Boards (FSMB) provided ABIM information about physicians subjected to medical license-related disciplinary actions through the Disciplinary Action Notification System which provides all sanction records on all physicians in all state medical licensing boards to the certifying boards.17 The primary independent variable was passing the IM MOC examination by the individual’s study start date (initial certification date plus ten full calendar years). Individuals passing the IM MOC examination after their study start date (i.e., the eleventh year after initial certification or later) were defined as having not passed during the study period.

Our full model included variables pertaining to training, certification, physician demographics, and state regulatory environments in which they practiced. All variables were determined a priori and selected to maintain consistency with previous studies on disciplinary action that included similar variables.15,16 Training and certification covariates included years since the first attempt at the IM initial certification examination (a proxy for years from training), year of passing the IM initial certification examination (categorized into five discrete cohorts to account for policy changes over time: 1990–1992, 1993–1995, 1996–1998, 1999–2001, and 2002–2003), and attempts to pass the IM initial certification examination (one vs. more than one).

Demographic variables included sex, age at passing the IM initial certification examination (30 years and younger vs. 31 years and older), birth country (USA vs. international), medical school country (US/Canadian vs. international), and geographic region (Midwest, Northeast, South, and West). Separate variables were created for the four combinations of birth country by medical school country because previous studies suggested different risk profiles for these groups.15

Two indices related to state regulatory environments were created. The first was a state disciplinary rate index for which higher values represented higher disciplinary actions per 1000 physicians in each state and the District of Columbia.18 The second index reflected variability in state CME requirements to maintain licensure such that higher values represented greater CME requirements.19 Both indices were converted to Z-scores with means of zero and unit standard deviations across the 50 states and Washington DC (see Table 1).

Table 1 Demographic and Training Characteristics of Internal Medicine Physicians, Initially Certified by ABIM 1990–2003, by Status of Passing the IM MOC Exam (N = 45,400)
Full size table

Similar to prior studies,16 disciplinary actions were categorized as very severe if they resulted in loss of licensure, moderately severe if they resulted in license restrictions or probation, and less severe if they resulted only in reprimands or administrative actions (e.g., fines).

Statistical Analysis

An historical cohort study was designed to compare risks of disciplinary actions over time for physicians passing the IM MOC examination versus those who did not pass within 10 years of initial certification. We used Cox regression to compare disciplinary hazard rates between groups in the previously specified full model (see Table 1). We used accelerated failure time (AFT) regression to produce Kaplan-Meier estimates of cumulative proportions of disciplinary actions over time controlling for all study covariates. Goodness-of-fit indices and visual inspection indicated that we met the assumptions of each model. We graphically compared cumulative proportions of disciplinary actions over time among those who did not take the MOC examination within 10 years of initial certification with those who took and failed the MOC examination in the study period and found them to be similar (Fig. 1), so we retained the primary comparison of those passing the MOC exam in the 10-year requirement window with those who did not pass either because they did not take, or took and failed the examination. Numbers and rates of disciplinary actions among subgroups of those who did not pass the IM MOC examination within 10 years were tabulated. Person-years in the study began from the study start date and continued until a disciplinary action occurred or until the study end date of 12/31/2015. Multiple imputation via the fully conditional specification model was used to replace missing values in covariates, assuming data were missing at random (see Table 1).

Fig. 1
figure 1

Kaplan-Meier estimates of cumulative percentage of disciplines by IM MOC examination status within 10 years of initial certification, adjusted for study covariates. The Kaplan-Meier cumulative percentage estimates with pointwise confidence intervals are depicted in the graph. At the end of year 10, the two upper curves represent physicians who did not take an IM MOC examination (green, n = 13,653) or who attempted but did not pass (red, n = 3569), while the lower curve represents physicians who passed an IM MOC examination (blue, n = 28,178) within 10 years of initial certification.

Full size image

To correlate rates of disciplinary action with performance on first attempt IM MOC examination scores, we used exam Z-scores categorized into six, standard deviation, intervals. Within each interval, we calculated disciplinary actions per 10,000 person-years. We graphed disciplinary rates by Z-score intervals and used Kendall’s tau-b coefficient to gauge the magnitude of association.

We addressed the fourth hypothesis by using the Cochran-Armitage trend test to contrast the proportion of least severe to most severe disciplinary actions by MOC examination status among disciplined physicians.

To assess the possibility that observed associations of disciplinary actions with the IM MOC examination might be related to differences in baseline professionalism, we conducted a post-hoc sensitivity analysis incorporating residency program director ratings of professionalism into the full Cox regression model for all cohorts with complete ratings.

To assess whether initial certification examination score might be a better predictor of discipline than initial certification examination attempts, we conducted a post-hoc sensitivity analysis replacing initial certification examination attempts with the first attempt initial certification examination score in the full model.

P values were two-tailed. Cox regression coefficients were adjusted for multiple comparisons using a Holm stepdown correction. Analysis was conducted using SAS software version 9.4 (SAS Institute Inc., Cary, NC).

The Institutional Review Board of the University of Pennsylvania approved the study.

RESULTS

Demographic characteristics of physicians passing and not passing the IM MOC examination within 10 years of initial certification are displayed in Table 1.

Historical Cohort Study

Adjusted hazard ratios for discipline are displayed in Table 2. Among physicians who did not pass the IM MOC examination within 10 years of initial certification, the risk was more than double than that of those who did pass the examination (adjusted HR 2.09 (95% CI 1.83, 2.39)). Physicians who did not pass the IM initial certification examination on the first attempt had a 35% higher risk of discipline (adjusted HR 1.35 (95% CI 1.14, 1.60)) than those passing on the first attempt.

Table 2 Cox Regression Analysis Predicting Hazard Risk of Discipline, by Person-Years in Study (N = 45,400)
Full size table

Hazard ratios for discipline also differed by demographic characteristics and were higher for physicians who were male versus female (adjusted HR 1.95 (1.67, 2.27)), older than 30 years versus 30 years or younger at training completion (adjusted HR 1.27 (1.08, 1.50)), living in the South (adjusted HR 1.78 (1.49, 2.14)) versus Northeast, and practicing in states with higher disciplinary rates (adjusted HR 1.13 (1.05, 1.22)). Increased state CME requirements were not associated with significant differences in hazard ratios for discipline (adjusted HR 1.02, (0.94, 1.09)).

Numbers and rates of disciplinary actions among subgroups of individuals not passing the IM MOC examination within 10 years of initial certification are shown in Table 3.

Table 3 Numbers and Rates of Disciplinary Actions among Subgroups of Physicians Who did and did not Pass the IM MOC Examination within 10 Years of Initial Certification
Full size table

MOC Examination Score

We found an inverse relationship between IM MOC examination score and rates of disciplinary actions. (Fig. 2, Kendall’s tau-b coefficient = − 0.98 for trend, p < 0.001.)

Fig. 2
figure 2

Rates of disciplinary actions per 10,000 person-years (with 95% CIs), by MOC exam Z-score, Fall, 2004–Spring, 2015 (N = 34,700). MOC scores from November, 2004 to May, 2015 were used for the score analysis to keep the format and scale of the examination consistent over time because of a change in MOC exam format beginning in the Fall of 2004. Lower Z-score means lower MOC examination score.

Full size image

Severity of Discipline

Among disciplined physicians, those passing the IM MOC examination within ten years of initial certification had a higher proportion of less severe disciplinary actions and a lower proportion of very severe actions than those who did not pass (45.0 vs 34.1% and 20.3 vs. 24.7%, respectively, Cochran-Armitage trend test statistic Z = 3.01, P = 0.003).

Sensitivity Analyses

Not passing the IM MOC examination within 10 years of initial certification remained independently associated with higher disciplinary rates in both sensitivity analyses: specifically, controlling for baseline professionalism as measured by adding residency program director professionalism ratings to our full model (adjusted HR 2.1, 95% CI (1.80, 2.44)) and replacing initial certification examination attempts with initial certification examination score in the full model (adjusted HR 2.05, 95% CI (1.78, 2.35)).

DISCUSSION

Among physicians who had been certified for at least 10 years, there was a significant association between not passing the IM MOC examination and state medical board disciplinary actions; specifically, not passing the IM MOC examination within 10 years of initial certification was associated with more than twice the rate of disciplinary actions over time, compared to those who passed the examination. In addition, there was a “dose response” among those who took the IM MOC examination, with higher scores associated with significantly lower disciplinary rates. Finally, disciplinary actions were proportionally less severe among those who passed the IM MOC examination. These findings are similar to associations related to initial certification.15,16 That there is a differential association of disciplinary actions between physicians who did and did not pass the IM MOC examination is important information for patients and health care providers since it provides a marker for physicians who are less likely to have these actions.

About 2% of physicians in this population were disciplined, yet the absolute number receiving disciplinary actions was sizable (n = 949). Given that general internists may have panel sizes of 1000–3000 patients,20,21 the numbers of patients potentially cared for by physicians in this population with disciplinary actions could be conservatively estimated to be many hundreds of thousands to a few million. The large number of patients potentially affected further highlights the relevance of MOC to informing the public’s choice of physicians.22,23

In 2005, Papadakis et al. published the seminal work demonstrating association between professionalism in medical school and subsequent disciplinary actions quantified using Population Attributable Risk (PAR)24 at 26%.25 In comparison, the PAR, or percentage of total disciplinary actions in this population that can be attributed to not having passed the IM MOC exam is 35% (95% CI 28 to 39%).

Some have suggested the association between certification and disciplinary actions may relate more to the professionalism of those certified, compared to those who are not certified, than to medical knowledge.15,25,26 However, our sensitivity analysis suggested passing the IM MOC examination within 10 years of initial certification was associated with reduced disciplinary rates independent of prior professionalism ratings in residency. While it is true that residency ratings of core competencies are inter-correlated,27 our data add to growing evidence suggesting the amount of medical knowledge a physician has acquired, and maintains, is itself associated with better performance and care independent of other acts of professionalism.28,29,–30 This is particularly evident in the “dose response” of examination scores now demonstrated in the United States Medical Licensing Examination,31 certification15 and MOC (Fig. 2); i.e., it is not just passing vs failing that is associated with fewer disciplinary actions; rather, those with demonstrably more medical knowledge (i.e., higher scores) have significantly fewer disciplinary actions than those with lower scores distributed in a score-dependent gradient. Similarly, those who require more than one attempt to pass the IM initial certification examination are at greater risk for disciplinary actions (Table 2). Finally, those who fail the IM MOC examination have similarly higher disciplinary rates than those who do not take the IM MOC examination within 10 years of initial certification (Fig. 1). If it were merely an “act of professionalism” to complete the MOC process, then those who take but fail the examination should have disciplinary rates closer to those who take and pass the examination. However, these data demonstrate the opposite; supporting the hypothesis that more medical knowledge is associated with fewer disciplinary actions.

A close examination of published causes of disciplinary actions may reveal a rationale for the association between increased medical knowledge and fewer disciplinary actions. While it is true that Morrison et al. found in their large state sample of causes for discipline that most disciplinary actions might be categorized as lapses in professionalism, the single largest category was “negligence or incompetence” accounting for 34% of the causes of discipline.14 It is possible that physicians staying more current in knowledge and practice may well be more likely to recognize and provide appropriate care, thus avoiding actions considered as “negligence or incompetence.”

Some critics of MOC have suggested that CME is sufficient demonstration of staying current in knowledge and practice.13,32,33,–34 However, our study included differential CME amounts required by state medical licensing boards and found no difference in disciplinary rates associated with the amount of CME required for licensure, suggesting that completing CME alone does not reduce the risk of disciplinary action. By contrast, passing the IM MOC examination was the single strongest factor associated with fewer disciplinary actions in our analysis of this population.

Our study has limitations. First, this is a population-based cohort study, a strong study design for prognosis of outcomes like disciplinary actions, yet it is an observational study, and thus cannot determine causality. Second, while we did adjust for known and potentially associated variables, it is possible that factors for which we did not account may affect the risk of disciplinary actions. For example, our study may underestimate the association of not passing the IM MOC examination and disciplinary actions because some licensed physicians may have left the practice of medicine and, therefore, were no longer at risk for disciplinary actions. Published rates of attrition of general internists from the practice of medicine in the last decade were between 1735 and 23%36; whereas, there were only 16% in our population who never took the IM MOC examination, and this group had significantly higher rates of disciplinary actions than those who took and passed the MOC examination within 10 years (Table 3), indicating many were still practicing medicine. Insofar, as attrition from the practice of medicine may have occurred for some who never took the IM MOC examination, these already higher rates of disciplinary action would be underestimates of the true association. Also, we did not have a measure of patient volume per physician for our cohort. While some studies have demonstrated an association between patient volume and patient outcomes,3,4 others have not.37,38,39,–40 Even in those that did demonstrate an association, the magnitude of association was small; so, patient volume metrics, if added to our model, would be unlikely to change significantly the association of passing the IM MOC examination with disciplinary actions. Third, our study population included only general internists, the largest group of physicians certified. Future studies could assess similar associations in subspecialists. Fourth, our study excluded physicians disciplined within the first 10 years after initial certification, so our results are most generalizable to physicians who similarly were not disciplined in the first 10 years after initial certification, but this includes more than 98% of all physicians initially eligible for inclusion in our study. Fifth, we did not assess associations of disciplinary actions for physicians who were certified prior to 1990 because these physicians are not required to take the MOC examination to maintain their “time-unlimited” or “grandfathered” certificates.41 This may warrant future study given our finding that increased age is associated with increased risk of disciplinary actions.

In conclusion, this study provides important evidence to support the concept that passing a periodic assessment of medical knowledge among practicing physicians is associated with enhancing the quality of healthcare.