“That Was Pretty Powerful”: a Qualitative Study of What Physicians Learn When Preparing for Their Maintenance-of-Certification Exams

Abstract

Background

A key component of Maintenance of Certification (MOC) for family and internal medicine physicians is the requirement to pass a periodic examination of medical knowledge. Little is known about the effects of preparing for MOC exams on knowledge and practice.

Objective

To understand how MOC exam preparation can affect knowledge and practice.

Design

Qualitative, semi-structured interviews, 45–60 min each, conducted by telephone at participants’ convenience.

Participants

A total of 80 primary care physicians from the American Board of Family Medicine (ABFM) and the American Board of Internal Medicine (ABIM) who had recently taken an MOC exam; the sample purposefully selected to represent diversity of experiences with MOC exams and range of opinions about MOC, as well as diversity of participant backgrounds—gender, practice type, etc.

Approach

Close analysis of physicians’ accounts of what they learned when preparing for an MOC exam and how this knowledge affected their practice.

Results

Sixty-seven of 80 physicians stated they gained knowledge relevant to their practice. Sixty-three gave concrete examples of how this affected their practice, including direct changes to patient care (e.g., improved diagnosis or prescribing and reduced unnecessary testing) or less direct changes (e.g., improved ability to co-manage with other providers or communicate with patients). Physicians also described sharing what they learned with others, including peers and trainees.

Limitations

Interviews could have been affected by recall and/or social desirability bias, as well as researchers’ role as board staff. Although we followed a recruitment protocol designed to mitigate recruitment acceptance bias, our findings may not be generalizable to wider groups of physicians.

Conclusions

Most physicians from two primary care specialties interviewed reported ways in which studying for an MOC exam resulted in acquiring knowledge that was both relevant and beneficial to their patient care.

INTRODUCTION

The Maintenance of Certification (MOC) exam requirement is a highly debated aspect of physician board certification.1,2,3,4 Little is known about what physicians actually do to prepare for these exams and how preparation could influence patient care.

Past research suggests that imposition of the American Board of Internal Medicine’s (ABIM’s) MOC requirement, including an exam, resulted in meaningful improvements in practice, including decreased costs overall and increased beneficial mammography screenings.5, 6 Other studies report that both maintaining certification and better performance on MOC exams are positively associated with meeting process of care standards7, 8 and negatively associated with disciplinary actions.9 However, the mechanisms underlying these associations remain largely unstudied.

Our study aimed to inform debates around MOC by exploring how MOC exam preparation may have affected physicians’ knowledge and patient care, through interviews with 80 primary care physicians who had recently completed an MOC exam.

METHODS

Data Collection

We conducted semi-structured phone interviews, each lasting 45–60 min, with 40 primary care physicians each from ABIM and the American Board of Family Medicine (ABFM) who had recently completed an MOC exam (in 2016). We selected this group because they represent the largest body of primary care physicians in the USA.10 At the time of our study, ABFM required physicians to complete an MOC exam every 7 to 10 years; ABIM required an exam every 10 years for physicians initially certified after 1989.

Two anthropologists with expertise in interviewing physicians (BC, AE) led the interviews, which covered the following: physicians’ practice characteristics; approach to studying; materials used; impacts on personal and professional life; and what, if anything, they learned in the process. Additionally, some interviewees volunteered their opinions regarding the value of MOC in general (see Online Appendix sections 1 and 2). The interview guide (see Online Appendix section 3) was designed to review these topics through a natural-feeling conversation that also allowed interviews to go off-script to pursue new themes introduced by interviewees. Draft guides were piloted with 5 recent MOC exam takers to refine the questions and to verify all questions could be completed within the allotted time. Interviewers emphasized throughout that we sought interviewees’ honest and complete accounts of their MOC exam preparation experience.

While interviews covered a variety of topics, this paper focuses on interviewees’ responses to questions related to what they learned and how this may have affected patient care.

Recruitment and Sampling

The recruitment sample included physicians who reported spending most of their practice time in outpatient primary care settings (see Online Appendix section 4 for specifics). We recruited a diverse sample with a variety of perspectives, including different opinions about MOC and MOC exams, by emailing weekly batches of invitations to 20–60 physicians from the pool of possible participants, followed by email reminders over the subsequent week. The first set of invitations was random; later, we adjusted the mix to develop a representative overall sample in terms of attitudes toward the MOC exam, gender, in academic or solo practice, and whether interviewees had passed the exam on their first attempt. We ceased invitations after completing 40 ABIM and 40 ABFM interviews. It is commonly held that 15 to 20 interviews are generally sufficient to ensure thematic saturation11; we interviewed more than this to gather a broad range of experiences.

Data Analysis

A team of six coded transcripts, using the MAXQDA 2018 software, starting with a priori codes drawn from the guide and creating new codes as these emerged during analysis. To establish inter-rater agreement, all team members coded a set of interviews, then compared and reconciled results; next, all interviews were coded by at least two researchers who further compared coding throughout. We analyzed the coded interviews using the constant comparative method,12 then further analyzed selected portions, including a review of clinical content by physicians from each board (LL, LP).

All participants gave informed consent. The American Academy of Family Physicians IRB approved the study.

RESULTS

Overview

Interviewees described how exam preparation could affect patient care through the following: the types of information studied, knowledge acquired, and resulting direct or indirect changes to practice (see Fig. 1 for overview).

Figure 1
figure1

How MOC exam preparation can affect knowledge and patient care.

Demographics

As shown in Table 1, the interviewed group was very similar to the sampling frame. For example, the percentage of interviewees who said the MOC exam was unfair compared to the larger population was the same for ABFM physicians (25% in our sample and in the larger population) and very similar for ABIM physicians (27% in sample, 28% overall) (see Online Appendix section 2 for the analysis of selected quotes describing interviewees’ reasons for participating). Table 2 shows the years since initial certification.

Table 1 Characteristics of Sample Interviewed and Test Takers That Met the Study Inclusion Criteria
Table 2 Years Since Initial Certification

Preparation

Seventy-eight of 80 physicians devoted extra time and resources to studying different contents and using different study approaches than when not preparing for an exam. Sixty-six of 80 reported at least 2 months of studying, which could be concentrated (as in live review courses) or spread over months or years.

Physicians described how exam preparation differed from normal engagement with journals or conferences: studying could include reviewing material studied before, perhaps during residency, and could encompass the scope of their specialty (for more detail, see Chesluk et al.13).

Knowledge Relevant to Practice

Most physicians reported acquiring knowledge relevant to their practice (67 of 80) and/or provided specific examples of how knowledge acquired from studying benefited their practice (63 of 80).

Types of Knowledge

As shown in Table 3, knowledge relevant to practice could include information new to the physician, but also took other forms: information that was familiar, but could now be better comprehended and applied; information that validated or reinforced what they had already known well; and/or information they had once learned but had since neglected (e.g., basic science or topic areas far outside what they see in practice). Acquired knowledge sometimes integrated all these at once, as when interviewees described digging deeper into the basic science behind new studies they had seen previously but not fully investigated, which gave them better understanding of how to apply those advances to their practice. Exam preparation could require study of clinical information spanning their entire specialty area, and also potentially require reevaluation of their underlying scientific model of how to organize and apply their knowledge. Some described learning about the limits of their own knowledge: “[M]y take-home message from [studying a particular subspecialty area] was… how quickly you become uneducated in things that you don’t do… [I]t’s important to know… that maybe you should pull the trigger for [sub]specialty consults sooner on some things” (32 ABIM). (Throughout this paper and in the Online Appendix, we identify interviewees by a unique number between 1 and 80, and also indicate whether they are ABIM or ABFM physicians.)

Table 3 Physicians’ Descriptions of Types of Knowledge Acquired from Studying for an MOC exam

Physicians also described passing on information to trainees, fellow physicians, or other providers, as when one described interactions with a colleague exempt from the MOC exam requirement: “[M]y partner [is]… grandfathered in internal medicine… so he doesn’t have to take the exam… I actually find myself enjoying giving him what the board currently thinks about this and that, and he listens very attentively … because I have more refreshed information” (42 ABIM).

How Studying Affects Practice

As stated above, most (63) physicians gave concrete examples of how what they learned benefited their practice. They described many different types of beneficial changes to practice, grouped into several, overlapping categories (see Table 4; see also Online Appendix section 5 for an expanded set of examples, by physician and by theme). Some practice changes were focused on specific conditions, as one interviewee reported:

[Before studying] I had a patient who had had a [Transient Ischemic Attack (TIA)]… and she went on to have a stroke. I had a similar patient recently and, using the ABCD guidelines for TIA, I realized she was at high risk of progressing, so I put her in the hospital and was more aggressive about the management and [I] suspect her outcome was probably better than it might have been otherwise. (9 ABFM)

Table 4 Physicians’ Descriptions of Examples of Practice Change Resulting from Studying for an MOC exam

Others described changes that applied across various conditions. These broader changes to care could constitute a change in the scope of the care the physician could provide. One interviewee who cares for an indigent patient population said “I could probably reduce, significantly, the number of visits to the consultant… Having more in-depth understanding of diseases and theoretical knowledge helps me to reduce [subspecialist] visits and reduce the financial burden on [indigent] patients” (57 ABIM).

Physicians also recounted ways in which preparation affected their practice other than direct changes in patient care. Some described improved communication with subspecialists, including via more thorough, focused workups to accompany referrals. Some felt they could now better understand other providers’ treatment decisions, and so could explain those decisions to patients or coordinate or co-manage patients’ care with subspecialists. Physicians also described teaching what they had studied to trainees or sharing with colleagues, potentially catalyzing changes to others’ practices.

Interviewees additionally recounted changes to patient communication, saying studying enhanced their ability to explain their own decisions to patients, or educate them about clinical matters:

[O]ne of the things I learned when I was studying is… I practice a lot of defensive medicine. [Y]ou do it because… the patient wants to, but it’s not really clinically appropriate or it's not according to the guidelines. So now, I learned to… talk to patients more… [B]efore, I used to just… do what the patient wants, but I now talk to them more about the risks, the complications…. (7 ABFM)

Some used the MOC exam itself to win patients’ trust: “It just gives you confidence you’re doing the right thing, and that you could tell the patient about it… I would say, ‘You know, I just read about this on my board exam’… And that was pretty powerful. Otherwise, if I weren’t [studying at] that time, I don’t think I would have that conversation” (47 ABIM).

Our interviews suggest several mechanisms by which MOC exam preparation can affect practice:

  • Direct—new information can lead directly to a change in practice.

  • Reinforcing—information can increase physicians’ confidence in current practice patterns, e.g., by providing validating evidence.

  • Communicating—information can lead to improved communication with providers or patients, e.g., by enhancing understanding of subspecialists’ perspectives or improving credibility with patients.

  • Indirect—regardless of how information might apply to one’s own practice, physicians can spread it, formally or informally, to other providers or trainees.

Examples of Knowledge Seemingly Not Relevant to Practice

Twenty-five interviewees volunteered that they had studied content that seemed irrelevant to their practice. Some gave examples related to conditions they would rarely be called on to treat, particularly obstetrics for family physicians: “[T]hings that maybe weren’t as practical for me would be fetal monitoring strips for OBs, since I don’t do OB and I am not going to do OB…” (18 ABFM). Others described studying very rare conditions: “[T]here was an information bit [in the study material] about an infection from a sand fly only found in the Middle East. I thought, ‘What the hell is this doing there?’” (41 ABIM). Physicians also gave examples related to issues they thought would be exclusively handled by specialists: “[T]here was quite a bit of stuff in the subspecialty section that I just had no idea even existed… [L]ike, they did four different drugs, and I hadn’t heard of any of them. And I never really need to know that, because obviously a person would go to the specialist with that” (32 ABIM).

DISCUSSION

As we have shown, studying for MOC exams requires significant time and effort, as physicians engage in activities very different from what they normally do to stay current. Most interviewees said they acquired knowledge from MOC exam study that was relevant to their practice and could improve their patient care.

Our study suggests several mechanisms by which exam preparation can directly and/or indirectly affect physicians’ practices—both the practice of those studying for the exam, and also, potentially, the practices of fellow physicians, other providers, and trainees (see Fig. 1). These are not mutually exclusive; many of the examples quoted illustrate several in action together. Some of the benefits of exam preparation we identified may stem from the high-stakes, summative nature MOC exam requirement itself, as research on the “testing effect” reports learners in contexts that feature testing retain information better, independent of extra study time.14, 15

These mechanisms may, in part, explain the associations with costs and quality of care described in previous studies. For example, our interviews yielded several possible explanations for Gray et al.’s finding of an association between mammography screening and initiation of the MOC requirement,6 including raised awareness of new guidelines or increased understanding of the underlying research, which could, in turn, result in improved ability to communicate credibly with patients about these procedures. Beyond changes to physicians’ own practices, this knowledge could have been shared with other providers, multiplying the potential positive impact. Physicians in our study described experiencing all of these, in various combinations and related to numerous conditions and patient populations. In sum, these interviews suggest MOC exams may catalyze learning relevant to patient care, even in “the era of widely available instantaneous digital access to information and colleagues.”1

The limitations are as follows: (1) reliance on interviewees’ recall and perceptions of what was beneficial and/or relevant; (2) physicians’ narratives could be affected by social desirability bias, despite our efforts (described above), especially considering researchers’ roles as ABIM and ABFM staff—this also may have colored interviewing and analysis; (3) while we worked to ensure a sample as representative and diverse as possible, hypotheses about how our findings could apply more generally are tentative, both because we interviewed only primary care physicians from two specialties, and because those invited who chose to participate may differ from the invitees who did not—many interviewees expressed that they participated out of a strong desire to personally contribute to the current debates around MOC (see Online Appendix section 2).

Despite these limitations, we hope this paper contributes to debates around MOC, by exploring physicians’ descriptions of how exam preparation can benefit patient care. Further research can explore these connections between MOC exams and patient care to test whether they are generalizable to the broader population of physicians, both primary care and subspecialists; to understand any relationship between learning and factors such as years in practice, opinion of MOC, and exam performance; to assess whether learning leads to measurable improvements in care delivery or health outcomes; and to explore the impact of new types of MOC assessments16 on practice.

CONCLUSION

Most physicians interviewed from two primary care specialties reported ways in which MOC exam preparation resulted in acquiring knowledge that was relevant and beneficial to their patient care.

References

  1. 1.

    Teirstein PS, Topol EJ. The role of maintenance of certification programs in governance and professionalism. JAMA. 2015;313(18):1809–1810.

    Article  Google Scholar 

  2. 2.

    ABMS. Fact Sheet: American Board of Medical Specialties (ABMS) and ABMS Maintenance of Certification® (ABMS MOC®). 2013 [cited May 13, 2019]. Available from: http://www.abms.org/News_and_Events/Media_Newsroom/pdf/ABMS_Fact_sheet.pdf.

  3. 3.

    Cassel CK, Holmboe ES. Professionalism and accountability: the role of specialty board certification. Trans Am Clin Climatol Assoc. 2008;119:295–304.

    PubMed  PubMed Central  Google Scholar 

  4. 4.

    Lipner RS, Hess BJ, Phillips RL. Specialty board certification in the United States: issues and evidence. J Contin Educ Health Prof. 2013(S1):20–35.

  5. 5.

    Gray BM, Vandergrift JL, Johnston MM, et al. Association between imposition of a Maintenance of Certification requirement and ambulatory care-sensitive hospitalizations and health care costs. JAMA. 2014;312(22):2348–2357.

    CAS  Article  Google Scholar 

  6. 6.

    Gray B, Vandergrift J, Lipner R. Association between the American Board of Internal Medicine’s general internist’s Maintenance of Certification requirement and mammography screening for Medicare beneficiaries. Women's Health Issues. 2018;28(1):35–41.

    Article  Google Scholar 

  7. 7.

    Gray B, Vandergrift J, Landon B, Reschovsky J, Lipner R. Associations between American Board of Internal Medicine Maintenance of Certification status and performance on a set of Healthcare Effectiveness Data and Information Set process measures. Ann Intern Med. 2018;169(2):97–105.

    Article  Google Scholar 

  8. 8.

    Holmboe ES, Lynn L, Duffy FD. Improving the quality of care via Maintenance of Certification and the web: an early status report. Perspect Biol Med. 2008;51(1):71–83.

    Article  Google Scholar 

  9. 9.

    McDonald F, Duhigg L, Arnold G, Hafer R, Lipner R. The American Board of Internal Medicine Maintenance of Certification examination and state medical board disciplinary actions: a population cohort study. J Gen Intern Med. 2018;33(8):1292–1298.

    Article  Google Scholar 

  10. 10.

    Petterson S, McNellis R, Klink K, Meyers D, Bazemore A. The state of primary care in the United States: a chartbook of facts and statistics. Washington, DC: Robert Graham Center; 2018.

    Google Scholar 

  11. 11.

    Mason M. Sample size and saturation in PhD studies using qualitative interviews. Forum Qual Soc Res. 2010;11(3):Art. 8.

    Google Scholar 

  12. 12.

    Glaser BG. The constant comparative method of qualitative analysis. Soc Probl. 1965;12(4):436–436.

    Article  Google Scholar 

  13. 13.

    Chesluk B, Eden A, Hansen E, et al. How physicians prepare for Maintenance of Certification exams: a qualitative study. Acad Med. In Press.

  14. 14.

    Roediger HL 3rd, Karpicke JD. The power of testing memory: basic research and implications for educational practice. Perspect Psychol Sci. 2006;1(3):181–210.

    Article  Google Scholar 

  15. 15.

    Roediger HL, Karpicke JD. Test-enhanced learning: taking memory tests improves long-term retention. Psychol Sci. 2006;17(3):249–255.

    Article  Google Scholar 

  16. 16.

    ABMS, Vision for the Future Commission. Continuing board certification: vision for the future. February 12, 2019 [cited April 1, 2019]. Available from: https://visioninitiative.org/commission/final-report/.

Download references

Contributors

The authors wish to thank Elizabeth Bernabeo, Siddharta Reddy (including for the design of the figure), and especially Michele Johnson for their participation in this project. We also thank Rebecca Lipner for her help in defining the project and analyzing the data. Finally, we sincerely thank the physicians who generously gave of their time to participate in our interviews.

Funding

This study received support from the American Board of Family Medicine Foundation.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Benjamin Chesluk PhD.

Ethics declarations

The American Academy of Family Physicians IRB approved the study.

Conflict of Interest

At the time of this study, all authors and all those named above were employees of the American Board of Internal Medicine or the American Board of Family Medicine.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic Supplementary Material

ESM 1

(DOCX 70 kb)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Chesluk, B., Gray, B., Eden, A. et al. “That Was Pretty Powerful”: a Qualitative Study of What Physicians Learn When Preparing for Their Maintenance-of-Certification Exams. J GEN INTERN MED 34, 1790–1796 (2019). https://doi.org/10.1007/s11606-019-05118-z

Download citation

KEY WORDS

  • continuing professional development
  • continuing medical education
  • qualitative research