Some have questioned whether successful performance in the American Board of Internal Medicine (ABIM) Maintenance of Certification (MOC) program is meaningful. The association of the ABIM Internal Medicine (IM) MOC examination with state medical board disciplinary actions is unknown.
To assess risk of disciplinary actions among general internists who did and did not pass the MOC examination within 10 years of initial certification.
Historical population cohort study.
The population of internists certified in internal medicine, but not a subspecialty, from 1990 through 2003 (n = 47,971).
ABIM IM MOC examination.
General internal medicine in the USA.
The primary outcome measure was time to disciplinary action assessed in association with whether the physician passed the ABIM IM MOC examination within 10 years of initial certification, adjusted for training, certification, demographic, and regulatory variables including state medical board Continuing Medical Education (CME) requirements.
The risk for discipline among physicians who did not pass the IM MOC examination within the 10 year requirement window was more than double than that of those who did pass the examination (adjusted HR 2.09; 95% CI, 1.83 to 2.39). Disciplinary actions did not vary by state CME requirements (adjusted HR 1.02; 95% CI, 0.94 to 1.16), but declined with increasing MOC examination scores (Kendall’s tau-b coefficient = − 0.98 for trend, p < 0.001). Among disciplined physicians, actions were less severe among those passing the IM MOC examination within the 10-year requirement window than among those who did not pass the examination.
Passing a periodic assessment of medical knowledge is associated with decreased state medical board disciplinary actions, an important quality outcome of relevance to patients and the profession.
The American Board of Internal Medicine (ABIM) instituted initial certification and maintenance of certification (MOC) examinations in support of its mission “To enhance the quality of health care by certifying internists and subspecialists who demonstrate the knowledge, skills and attitudes essential for excellent patient care.”1 Evidence has accumulated demonstrating that care provided by certified internists, compared to those who are not certified, is associated with better processes and outcomes,2 including mortality.3,4 While associations between certification and better care are fairly widely accepted, fewer studies have assessed associations of care with MOC.5,6,–7 One factor in the controversy over MOC in general, and the MOC examination in particular, has been skepticism about evidence of impact on patient care.8,9,10,11,12,–13
State medical board disciplinary action is an important outcome that matters to patients; it is directly related to quality of patient care and, in many cases, patient safety.14 Higher performance on the ABIM Internal Medicine (IM) initial certification examination is strongly correlated with decreased incidence of state medical board disciplinary actions.15 Physicians certified by ABIM have a five-fold decreased incidence of disciplinary actions compared to those who start IM residency, but fail to become certified.16 Among disciplined physicians, the distribution of severity of actions is less severe in certified compared to never certified physicians.16
Since IM MOC examinations are required at 10-year intervals for physicians initially certified after 1989,2 enough time has passed to accumulate sizable cohorts of physicians required to take the MOC examination to remain certified. Based on findings associated with initial certification, we hypothesized that those who pass the IM MOC examination within 10 years of initial certification would have lower rates of disciplinary actions compared to those who do not take or do not pass the examination. Specifically, we sought to address four primary hypotheses:
There is an inverse association between passing the MOC examination and the risk of disciplinary actions.
The trends in hazard rates for disciplinary actions over time are lower for those passing the MOC examination within 10 years of initial certification compared to those who do not pass within this time frame.
The risk of disciplinary action against physicians declines as scores on the MOC examination increase.
Disciplinary actions are proportionally less severe among those passing the MOC examination compared to those who do not pass within 10 years of initial certification.
General internists were eligible for inclusion if they passed the ABIM IM certification examination between 1/1/1990 and 12/31/2003 (n = 47,971). We did not include anyone ever certified in an IM subspecialty. Because our focus was the MOC examination and certificates were issued for 10 years during the study period, participants’ start dates in the study began 10 years following their initial certification year. Participants were followed until 12/31/2015. Using data available in the ABIM database, we excluded anyone believed to practice outside the USA (n = 1376), older than 75 years when passing the initial certification examination (n = 2), known to be dead, retired, or in restitution for behaviors not associated with medical licensure (n = 202). Because a disciplined license precludes taking the MOC examination, we excluded those disciplined prior to their study start date (n = 991) to avoid overestimating potential associations with the MOC examination.
The primary outcome measure was time (years) to disciplinary action. The Federation of State Medical Boards (FSMB) provided ABIM information about physicians subjected to medical license-related disciplinary actions through the Disciplinary Action Notification System which provides all sanction records on all physicians in all state medical licensing boards to the certifying boards.17 The primary independent variable was passing the IM MOC examination by the individual’s study start date (initial certification date plus ten full calendar years). Individuals passing the IM MOC examination after their study start date (i.e., the eleventh year after initial certification or later) were defined as having not passed during the study period.
Our full model included variables pertaining to training, certification, physician demographics, and state regulatory environments in which they practiced. All variables were determined a priori and selected to maintain consistency with previous studies on disciplinary action that included similar variables.15,16 Training and certification covariates included years since the first attempt at the IM initial certification examination (a proxy for years from training), year of passing the IM initial certification examination (categorized into five discrete cohorts to account for policy changes over time: 1990–1992, 1993–1995, 1996–1998, 1999–2001, and 2002–2003), and attempts to pass the IM initial certification examination (one vs. more than one).
Demographic variables included sex, age at passing the IM initial certification examination (30 years and younger vs. 31 years and older), birth country (USA vs. international), medical school country (US/Canadian vs. international), and geographic region (Midwest, Northeast, South, and West). Separate variables were created for the four combinations of birth country by medical school country because previous studies suggested different risk profiles for these groups.15
Two indices related to state regulatory environments were created. The first was a state disciplinary rate index for which higher values represented higher disciplinary actions per 1000 physicians in each state and the District of Columbia.18 The second index reflected variability in state CME requirements to maintain licensure such that higher values represented greater CME requirements.19 Both indices were converted to Z-scores with means of zero and unit standard deviations across the 50 states and Washington DC (see Table 1).
Similar to prior studies,16 disciplinary actions were categorized as very severe if they resulted in loss of licensure, moderately severe if they resulted in license restrictions or probation, and less severe if they resulted only in reprimands or administrative actions (e.g., fines).
An historical cohort study was designed to compare risks of disciplinary actions over time for physicians passing the IM MOC examination versus those who did not pass within 10 years of initial certification. We used Cox regression to compare disciplinary hazard rates between groups in the previously specified full model (see Table 1). We used accelerated failure time (AFT) regression to produce Kaplan-Meier estimates of cumulative proportions of disciplinary actions over time controlling for all study covariates. Goodness-of-fit indices and visual inspection indicated that we met the assumptions of each model. We graphically compared cumulative proportions of disciplinary actions over time among those who did not take the MOC examination within 10 years of initial certification with those who took and failed the MOC examination in the study period and found them to be similar (Fig. 1), so we retained the primary comparison of those passing the MOC exam in the 10-year requirement window with those who did not pass either because they did not take, or took and failed the examination. Numbers and rates of disciplinary actions among subgroups of those who did not pass the IM MOC examination within 10 years were tabulated. Person-years in the study began from the study start date and continued until a disciplinary action occurred or until the study end date of 12/31/2015. Multiple imputation via the fully conditional specification model was used to replace missing values in covariates, assuming data were missing at random (see Table 1).
To correlate rates of disciplinary action with performance on first attempt IM MOC examination scores, we used exam Z-scores categorized into six, standard deviation, intervals. Within each interval, we calculated disciplinary actions per 10,000 person-years. We graphed disciplinary rates by Z-score intervals and used Kendall’s tau-b coefficient to gauge the magnitude of association.
We addressed the fourth hypothesis by using the Cochran-Armitage trend test to contrast the proportion of least severe to most severe disciplinary actions by MOC examination status among disciplined physicians.
To assess the possibility that observed associations of disciplinary actions with the IM MOC examination might be related to differences in baseline professionalism, we conducted a post-hoc sensitivity analysis incorporating residency program director ratings of professionalism into the full Cox regression model for all cohorts with complete ratings.
To assess whether initial certification examination score might be a better predictor of discipline than initial certification examination attempts, we conducted a post-hoc sensitivity analysis replacing initial certification examination attempts with the first attempt initial certification examination score in the full model.
P values were two-tailed. Cox regression coefficients were adjusted for multiple comparisons using a Holm stepdown correction. Analysis was conducted using SAS software version 9.4 (SAS Institute Inc., Cary, NC).
The Institutional Review Board of the University of Pennsylvania approved the study.
Demographic characteristics of physicians passing and not passing the IM MOC examination within 10 years of initial certification are displayed in Table 1.
Historical Cohort Study
Adjusted hazard ratios for discipline are displayed in Table 2. Among physicians who did not pass the IM MOC examination within 10 years of initial certification, the risk was more than double than that of those who did pass the examination (adjusted HR 2.09 (95% CI 1.83, 2.39)). Physicians who did not pass the IM initial certification examination on the first attempt had a 35% higher risk of discipline (adjusted HR 1.35 (95% CI 1.14, 1.60)) than those passing on the first attempt.
Hazard ratios for discipline also differed by demographic characteristics and were higher for physicians who were male versus female (adjusted HR 1.95 (1.67, 2.27)), older than 30 years versus 30 years or younger at training completion (adjusted HR 1.27 (1.08, 1.50)), living in the South (adjusted HR 1.78 (1.49, 2.14)) versus Northeast, and practicing in states with higher disciplinary rates (adjusted HR 1.13 (1.05, 1.22)). Increased state CME requirements were not associated with significant differences in hazard ratios for discipline (adjusted HR 1.02, (0.94, 1.09)).
Numbers and rates of disciplinary actions among subgroups of individuals not passing the IM MOC examination within 10 years of initial certification are shown in Table 3.
MOC Examination Score
We found an inverse relationship between IM MOC examination score and rates of disciplinary actions. (Fig. 2, Kendall’s tau-b coefficient = − 0.98 for trend, p < 0.001.)
Severity of Discipline
Among disciplined physicians, those passing the IM MOC examination within ten years of initial certification had a higher proportion of less severe disciplinary actions and a lower proportion of very severe actions than those who did not pass (45.0 vs 34.1% and 20.3 vs. 24.7%, respectively, Cochran-Armitage trend test statistic Z = 3.01, P = 0.003).
Not passing the IM MOC examination within 10 years of initial certification remained independently associated with higher disciplinary rates in both sensitivity analyses: specifically, controlling for baseline professionalism as measured by adding residency program director professionalism ratings to our full model (adjusted HR 2.1, 95% CI (1.80, 2.44)) and replacing initial certification examination attempts with initial certification examination score in the full model (adjusted HR 2.05, 95% CI (1.78, 2.35)).
Among physicians who had been certified for at least 10 years, there was a significant association between not passing the IM MOC examination and state medical board disciplinary actions; specifically, not passing the IM MOC examination within 10 years of initial certification was associated with more than twice the rate of disciplinary actions over time, compared to those who passed the examination. In addition, there was a “dose response” among those who took the IM MOC examination, with higher scores associated with significantly lower disciplinary rates. Finally, disciplinary actions were proportionally less severe among those who passed the IM MOC examination. These findings are similar to associations related to initial certification.15,16 That there is a differential association of disciplinary actions between physicians who did and did not pass the IM MOC examination is important information for patients and health care providers since it provides a marker for physicians who are less likely to have these actions.
About 2% of physicians in this population were disciplined, yet the absolute number receiving disciplinary actions was sizable (n = 949). Given that general internists may have panel sizes of 1000–3000 patients,20,21 the numbers of patients potentially cared for by physicians in this population with disciplinary actions could be conservatively estimated to be many hundreds of thousands to a few million. The large number of patients potentially affected further highlights the relevance of MOC to informing the public’s choice of physicians.22,23
In 2005, Papadakis et al. published the seminal work demonstrating association between professionalism in medical school and subsequent disciplinary actions quantified using Population Attributable Risk (PAR)24 at 26%.25 In comparison, the PAR, or percentage of total disciplinary actions in this population that can be attributed to not having passed the IM MOC exam is 35% (95% CI 28 to 39%).
Some have suggested the association between certification and disciplinary actions may relate more to the professionalism of those certified, compared to those who are not certified, than to medical knowledge.15,25,26 However, our sensitivity analysis suggested passing the IM MOC examination within 10 years of initial certification was associated with reduced disciplinary rates independent of prior professionalism ratings in residency. While it is true that residency ratings of core competencies are inter-correlated,27 our data add to growing evidence suggesting the amount of medical knowledge a physician has acquired, and maintains, is itself associated with better performance and care independent of other acts of professionalism.28,29,–30 This is particularly evident in the “dose response” of examination scores now demonstrated in the United States Medical Licensing Examination,31 certification15 and MOC (Fig. 2); i.e., it is not just passing vs failing that is associated with fewer disciplinary actions; rather, those with demonstrably more medical knowledge (i.e., higher scores) have significantly fewer disciplinary actions than those with lower scores distributed in a score-dependent gradient. Similarly, those who require more than one attempt to pass the IM initial certification examination are at greater risk for disciplinary actions (Table 2). Finally, those who fail the IM MOC examination have similarly higher disciplinary rates than those who do not take the IM MOC examination within 10 years of initial certification (Fig. 1). If it were merely an “act of professionalism” to complete the MOC process, then those who take but fail the examination should have disciplinary rates closer to those who take and pass the examination. However, these data demonstrate the opposite; supporting the hypothesis that more medical knowledge is associated with fewer disciplinary actions.
A close examination of published causes of disciplinary actions may reveal a rationale for the association between increased medical knowledge and fewer disciplinary actions. While it is true that Morrison et al. found in their large state sample of causes for discipline that most disciplinary actions might be categorized as lapses in professionalism, the single largest category was “negligence or incompetence” accounting for 34% of the causes of discipline.14 It is possible that physicians staying more current in knowledge and practice may well be more likely to recognize and provide appropriate care, thus avoiding actions considered as “negligence or incompetence.”
Some critics of MOC have suggested that CME is sufficient demonstration of staying current in knowledge and practice.13,32,33,–34 However, our study included differential CME amounts required by state medical licensing boards and found no difference in disciplinary rates associated with the amount of CME required for licensure, suggesting that completing CME alone does not reduce the risk of disciplinary action. By contrast, passing the IM MOC examination was the single strongest factor associated with fewer disciplinary actions in our analysis of this population.
Our study has limitations. First, this is a population-based cohort study, a strong study design for prognosis of outcomes like disciplinary actions, yet it is an observational study, and thus cannot determine causality. Second, while we did adjust for known and potentially associated variables, it is possible that factors for which we did not account may affect the risk of disciplinary actions. For example, our study may underestimate the association of not passing the IM MOC examination and disciplinary actions because some licensed physicians may have left the practice of medicine and, therefore, were no longer at risk for disciplinary actions. Published rates of attrition of general internists from the practice of medicine in the last decade were between 1735 and 23%36; whereas, there were only 16% in our population who never took the IM MOC examination, and this group had significantly higher rates of disciplinary actions than those who took and passed the MOC examination within 10 years (Table 3), indicating many were still practicing medicine. Insofar, as attrition from the practice of medicine may have occurred for some who never took the IM MOC examination, these already higher rates of disciplinary action would be underestimates of the true association. Also, we did not have a measure of patient volume per physician for our cohort. While some studies have demonstrated an association between patient volume and patient outcomes,3,4 others have not.37,38,39,–40 Even in those that did demonstrate an association, the magnitude of association was small; so, patient volume metrics, if added to our model, would be unlikely to change significantly the association of passing the IM MOC examination with disciplinary actions. Third, our study population included only general internists, the largest group of physicians certified. Future studies could assess similar associations in subspecialists. Fourth, our study excluded physicians disciplined within the first 10 years after initial certification, so our results are most generalizable to physicians who similarly were not disciplined in the first 10 years after initial certification, but this includes more than 98% of all physicians initially eligible for inclusion in our study. Fifth, we did not assess associations of disciplinary actions for physicians who were certified prior to 1990 because these physicians are not required to take the MOC examination to maintain their “time-unlimited” or “grandfathered” certificates.41 This may warrant future study given our finding that increased age is associated with increased risk of disciplinary actions.
In conclusion, this study provides important evidence to support the concept that passing a periodic assessment of medical knowledge among practicing physicians is associated with enhancing the quality of healthcare.
Baron RJ, Johnson D. The American Board of Internal Medicine: Evolving Professional Self-regulation. Annals of Internal Medicine. 2014;161(3):221–3.
Lipner RS, Hess BJ, Phillips RL. Specialty Board Certification in the United States: Issues and Evidence. Journal of Continuing Education in the Health Professions. 2013(S1):20–35.
Norcini JJ, Kimball HR, Lipner RS. Certification and specialization: do they matter in the outcome of acute myocardial infarction? Academic Medicine: Journal Of The Association Of American Medical Colleges. 2000;75(12):1193–8.
Fiorilli PN, Minges K, Herrin J, et al. Association of physician certification in Interventional Cardiology with in-hospital outcomes of percutaneous coronary intervention. Circulation. 2015;132(19):1816–24.
Gray BM, Vandergrift JL, Johnston MM, et al. Association Between Imposition of a Maintenance of Certification Requirement and Ambulatory Care-Sensitive Hospitalizations and Health Care Costs. JAMA: Journal of the American Medical Association. 2014;312(22):2348–57.
Holmboe ES, Wang Y, Meehan TP, et al. Association Between Maintenance of Certification Examination Scores and Quality of Care for Medicare Beneficiaries. Archives of Internal Medicine. 2008;168(13):1396–403.
Hayes J, Jackson JL, McNutt GM, Hertz BJ, Ryan JJ, Pawlikowski SA. Association between physician time-unlimited vs time-limited internal medicine board certification and ambulatory patient care quality. JAMA. 2014;312(22):2358–63.
Cook DA, Holmboe ES, Sorensen KJ, Berger RA, Wilkinson JM. Getting maintenance of certification to work: A grounded theory study of physicians’ perceptions. JAMA Internal Medicine. 2015;175(1):35–42.
Iglehart JK, Baron RB. Ensuring physicians’ competence—Is maintenance of certification the answer? The New England Journal of Medicine. 2012;367(26):2543–9.
Drazen JM, Weinstein DF. Considering Recertification. New England Journal of Medicine. 2010;362(10):946–7.
Levinson W, King TE. American Board of Internal Medicine Maintenance of Certification Program. New England Journal of Medicine. 2010;362(10):948–52.
Cook DA, Blachman MJ, West CP, Wittich CM. Physician attitudes about Maintenance of Certification: A cross-specialty national survey. Mayo Clinic Proceedings. 2016;91(10):1336–45.
Teirstein PS, Topol EJ. The role of maintenance of certification programs in governance and professionalism. JAMA. 2015;313(18):1809–10.
Morrison J, Wickersham P. Physicians disciplined by a state medical board. JAMA: Journal of the American Medical Association. 1998;279(23):1889–93.
Papadakis MA, Arnold GK, Blank LL, Holmboe ES, Lipner RS. Performance during Internal Medicine Residency Training and Subsequent Disciplinary Action by State Licensing Boards. Annals of Internal Medicine. 2008;148(11):869–76.
Lipner RS, Young A, Chaudhry HJ, Duhigg LM, Papadakis MA. Specialty Certification Status, Performance Ratings, and Disciplinary Actions of Internal Medicine Residents. Academic Medicine. 2016;91(3):376–81.
Johnson DA, Chaudhry HJ. Medical Licensing and Discipline in America: A History of the Federation of State Medical Boards. Lanham: Rowman and Littlefield; 2012.
Wolfe S, Williams C, Zaslow A. Public Citizen’s Health Research Group Ranking of the Rate of State Medical Boards’ Serious Disciplinary Actions, 2009–2011. 2012.
Federation of State Medical Boards of the United States: Continuing Medical Education Board by Board Overview: 2015.
Hong CS, Atlas SJ, Chang Y, et al. Relationship between patient panel characteristics and primary care physician clinical performance rankings. JAMA. 2010;304(10):1107–13.
Raffoul M, Moore M, Kamerow D, Bazemore A. A Primary Care Panel Size of 2500 Is neither Accurate nor Reasonable. Journal Of The American Board Of Family Medicine: JABFM. 2016;29(4):496–9.
Cassel CK. ABIM: Maintenance of Certification—For the public. 2010.
Brennan TA, Horwitz RI, Duffy FD, Cassel CK, Goode LD, Lipner RS. The Role of Physician Specialty Board Certification Status in the Quality Movement. JAMA: Journal of the American Medical Association. 2004;292(9):1038–43.
Fleiss J, Levin B, Paik M. Statistical Methods for Rate and Proportions. 3rd ed Hoboken: John Wiley & Sons; 2003:125–9.
Papadakis MA, Teherani A, Banach MA, et al. Disciplinary Action by Medical Boards and Prior Behavior in Medical School. New England Journal of Medicine. 2005;353(25):2673–82.
Papadakis MA, Hodgson CS, Teherani A, Kohatsu ND. Unprofessional behavior in medical school is associated with subsequent disciplinary action by a state medical board. Academic Medicine: Journal Of The Association Of American Medical Colleges. 2004;79(3):244–9.
Hauer KE, Clauser J, Lipner RS, et al. The Internal Medicine Reporting Milestones: Cross-sectional Description of Initial Implementation in U.S. Residency Programs. Annals of Internal Medicine. 2016;165(5):356–62.
McDonald F, Zeger S, Kolars J. Factors associated with medical knowledge acquisition during internal medicine residency. Journal of General Internal Medicine. 2007;22(7):962–8.
McDonald F, Zeger S, Kolars J. Associations of conference attendance with internal medicine in-training examination scores. Mayo Clinic Proceedings. 2008;83(4):449–53.
McCoy CP, Stenerson MB, Halvorsen AJ, Homme JH, McDonald FS. Association of volume of patient encounters with residents’ in-training examination performance. Journal of General Internal Medicine. 2013;28(8):1035–41.
Cuddy M, Young A, Gelman A, et al. Exploring the relationships between USMLE performance and disciplinary action in practice: A validity study of score inferences from a licensure examination. Academic Medicine. 2017;92(12):178–85.
Aboud KA, Ramesh V. Continuing Medical Education (CME): A reappraisal. Canadian Medical Education Journal. 2011;2(2):e91-e3.
Ahmed K, Wang TT, Ashrafian H, Layer GT, Darzi A, Athanasiou T. The effectiveness of continuing medical education for specialist recertification. Canadian Urological Association Journal—Journal De L’association Des Urologues Du Canada. 2013;7(7–8):266–72.
Balmer JT. The transformation of continuing medical education (CME) in the United States. Advances in Medical Education & Practice. 2013;4:171–82.
Bylsma W, Arnold GK, Fortna GS, Lipner R. Where have all the general internists gone? Journal of General Internal Medicine. 2010;25(10):1020.
Lipner RS, Bylsma WH, Arnold GK, Fortna GS, Tooker J, Cassel CK. Who Is Maintaining Certification in Internal Medicine--and Why? A National Survey 10 Years after Initial Certification. Annals of Internal Medicine. 2006;144(1):29–37.
Casalino LP, Pesko MF, Ryan AM, et al. Small Primary Care Physician Practices Have Low Rates Of Preventable Hospital Admissions. Health Affairs. 2014;33(9):1680–8.
Kralewski J, Dowd B, Knutson D, Tong J, Savage M. The relationships of physician practice characteristics to quality of care and costs. Health Services Research. 2015;50(3):710–29.
Pineault R, Provost S, Borgès Da Silva R, Breton M, Levesque J-F. Why Is Bigger Not Always Better in Primary Health Care Practices? The Role of Mediating Organizational Factors. Inquiry: A Journal Of Medical Care Organization, Provision And Financing. 2016;53.
Shortell SM, Schmittdiel J, Wang MC, et al. An Empirical Assessment of High-Performing Medical Groups: Results from a National Study. Medical Care Research & Review. 2005;62(4):407–34.
Lipner RS, Bylsma WH, Arnold GK, Fortna GS, Tooker J, Cassel CK. Who is maintaining certification in internal medicine -- and why? A national survey 10 years after initial certification. Annals of Internal Medicine. 2006;144(1):29–36.
The authors gratefully acknowledge the following individuals for their critical review of the manuscript: Vineet Arora, MD, MAPP; Richard J. Baron, MD; J. Taylor Hays, MD, Joseph C. Kolars, MD.
The Institutional Review Board of the University of Pennsylvania approved the study.
Conflict of Interest
All authors are employed by the American Board of Internal Medicine.
About this article
Cite this article
McDonald, F.S., Duhigg, L.M., Arnold, G.K. et al. The American Board of Internal Medicine Maintenance of Certification Examination and State Medical Board Disciplinary Actions: a Population Cohort Study. J GEN INTERN MED 33, 1292–1298 (2018). https://doi.org/10.1007/s11606-018-4376-z
- maintenance of certification
- disciplinary action