Background

The design [1], as well as the value of accreditation systems [2, 3], are topics that require additional research. In this study we reflect on the virtues of continuous quality improvement (CQI) in medical education.

There is an implicit belief that the accreditation of medical education programs ensures the quality of the education these programs provide and the graduates they produce [4,5,6]. The importance of the accreditation of medical education programs has been further heightened by a recent announcement by the Educational Commission for Foreign Medical Graduates. The announcement states that, by the year 2023, the only candidates who can be certified, and therefore considered for residency education in the United States, will be graduates of medical schools that have been accredited by organizations approved by the World Federation of Medical Education as working to appropriately formulated and applied standards [7]. Despite the respect that accreditation is accorded, little has been written about whether the accreditation process actually achieves its intended outcomes. In many jurisdictions, accreditation of medical education programs has traditionally been based on episodic evaluation [8]. In general terms, an episodic evaluation is an external review by an accreditation authority that includes on-site visits and is repeated at specified intervals [8, 9]. More recently, the focus of accreditation has been shifting from episodic evaluation to CQI [10,11,12,13]. CQI, which originated in industry as an approach to quality control for services or products [14,15,16,17], is a never-ending cycle of organizational improvement that uses quality indicators [18].

This paper, one of a series arising from the first World Summit on Accreditation Outcomes in Medicine, explores the role of CQI in systems traditionally based on episodic evaluation. As explained before, the evidence in literature for the benefits of a CQI system in medical education is insufficient, although CQI has been introduced in several jurisdictions. The result is a new bureaucratic instrument that is being used without sufficient empirical evidence. As such, it is important to investigate the pros and cons of CQI in medical education. In this utility-focused evaluation [19] we use a set of quality criteria for governance oversight to shed more light on these pros and cons.

Methods

We chose to perform a naturalistic utility-focused evaluation [19]. Three of the authors (LP, CC, and FS) were responsible accreditation officers in their distinct fields of medical education who wished to add to the evidence and inform decision making around investments in CQI. The fourth author (NA) is a certified lawyer, medical doctor and PhD student researching accreditation systems. At the time of this evaluation, there were no conflicts concerning CQI that might have influence their opinions.

We chose to use a set of quality criteria for government oversight that did not originate from the field of medical education, because a firm theoretical background in the accreditation literature for medical education was not available.

Our author team was assigned to the roles of both descriptor of the accreditation cases and evaluator of CQI based on the criteria. The authors discussed and evaluated each system using the criteria and searched for patterns with respect to the quality criteria. During these discussions, detailed questions about each system were answered by the author responsible for that system. This process was repeated iteratively until the authors reached consensus for each criterion.

Case examples: three accreditation systems

In the next section, the accreditation systems are briefly introduced to give the reader a context for our discussion about the pros and cons of CQI. A comprehensive description of the systems may be found on the websites of the accreditation authorities (afmc.ca; cacms-cafmc.ca; lcme.org; www.knmg.nl; www.royalcollege.ca) or obtained from the authors. LP, CC and FS provided detailed descriptions of the systems—both written and oral.

The American and Canadian undergraduate accreditation systems

In North America, two authorities accredit medical degree-granting programs: the Committee on Accreditation of Canadian Medical Schools (CACMS) for medical schools in Canada, and the Liaison Committee on Medical Education (LCME) for medical schools in the United States and Canada. The CACMS and the LCME are external bodies with respect to the programs they accredit. The accreditation cycle includes an on-site visit by a peer-review team; the interval between one full survey and the next is 8 years. The internal review process for the full survey requires approximately 2 years and involves a major expenditure of human and financial resources. In 2009, the Association of Faculties of Medicine of Canada (AFMC) approved a mandatory interim review process (IRP) for Canadian medical schools that includes a review of CACMS accreditation standards at the mid-point of the formal accreditation cycle, along with CQI to achieve and maintain compliance with standards throughout the cycle [8]. The AFMC IRP is currently independent of the CACMS, and the effectiveness of the process is the subject of ongoing review. The LCME, with the endorsement of CACMS, has introduced an accreditation standard requiring schools to have a CQI process to monitor compliance with accreditation standards; this came into effect during the 2015–16 academic year [5]. Research is needed to determine the efficacy of these mandatory requirements for CQI.

The Dutch postgraduate accreditation system

In the Dutch system there are two important external authorities for the accreditation of residency training: the CGS (Legislative College for Accreditation of Residency Training) and the RGS (Specialist Registration Committee). These are bodies of the Royal Dutch Medical Association (KNMG). The CGS is responsible for setting regulations (training requirements) for accreditation, while the RGS’s role is to monitor compliance with these regulations. The medical specialties in teaching hospitals are visited every 5 years by a survey team commissioned by the RGS. There are no specific mandatory internal quality management systems regarding residency training in teaching hospitals; however, sufficient instruments and systems are made available to conduct internal quality control. The postgraduate medical education programs are required to perform some form of CQI.

Continuous professional development accreditation in Canada

There are two basic types of continuous professional development (CPD) accreditation systems: provider-based accreditation and activity-based accreditation. In activity-based CPD accreditation systems, providers submit evidence to the accreditation organization about whether or not an activity they have developed complies with established administrative, educational, and ethical standards. If the accreditation authority agrees that the standards have been met, the activity is approved. Activity-based accreditation is used, for example, by the College of Family Physicians of Canada. Provider-based CPD accreditation systems recognize organizations as institutional providers of continuing education on the basis of their ability to demonstrate adherence to a set of standards. The accreditation process frequently includes an on-site visit in which a peer-review team has the opportunity to discuss specific standards with, among others, the CPD organization’s leadership, administrative staff, and participants. Once an organization is approved, all activities it develops during its accreditation cycle are automatically approved for credit. A provider-based accreditation system is used by the Royal College of Physicians and Surgeons of Canada (RCPSC) to focus on national and provincial CPD organizations, including national specialty societies, while the Committee on Accreditation of Continuing Medical Education (CACME) focuses on the university offices of continuing medical education (CME) in the faculties of medicine. These two provider accreditation systems will be used to illustrate the process by which continuous accreditation is expressed and promoted.

Criteria for oversight

Accreditation is actually a method of oversight. Oversight functions of government and accreditation processes in medicine may struggle with similar difficulties in achieving quality improvement [20, 21], even though different stakeholders are involved. The Netherlands Scientific Council for Government Policy (WRR) identified seven criteria as the basis for societally responsive oversight [22] (Table 1). Given their general applicability, we used these criteria to evaluate the pros and cons of CQI in undergraduate medical education in the United States and Canada; postgraduate medical education in the Netherlands; and CPD in Canada.

Table 1 Summary of the seven criteria and associated tasks
Full size table

Results

The results are summarized in Table 2 and reflect the authors’ evaluation of the pros and cons of CQI at the institutional level.

Table 2 CQI at the institutional level
Full size table

Discussion

Our findings show that an integrated system for accreditation of medical education may consist of external authorities and the local CQI system supporting timely reflection and action to serve the public interest. External authorities might even need less capacity due to local quality systems. The downside of CQI is the bureaucracy and costs involved, the risk of interference with other quality systems, the need for an open culture, and potential issues with impartiality and dependency of colleagues involved.

Our findings partially conflict with other literature. Since the 1980s CQI has been used in manufacturing and engineering, the financial sector, education, and health care. Various methods have been used to pursue CQI (e.g., the plan-do-check-act or plan-do-study-act cycle, the Six Sigma standard in manufacturing and other fields, “total quality management”) [23]. All of these share the concept of a “cycle of improvement.” In the manufacturing industry, CQI has been useful in reducing defects, lowering costs, and increasing quality and productivity [24, 25]. In the construction industry it has led to economic benefits, increased customer satisfaction, a reduced need to re-do work, and declining defects in housing projects [26, 27]. Further, CQI is readily applicable to standardized and repetitive processes [23]. In surgical care it has resulted in reduced infection rates and shorter delays in the operating room [23]. Comparing these positive literature reports with the evaluation of CQI in our contexts suggests a publication bias of positive results. It is difficult to measure economic effects and other outcomes in health care and medical education [28,29,30]. The cost–benefit ratio in our contexts may not always be positive. The difficulty in going through all the components of the plan-do-study-act cycle has been described before [30,31,32]. The bureaucracy and costs involved in CQI, the risk of interference with other quality systems, the need for an open culture, and potential issues with impartiality and dependency of colleagues involved are important findings. The criteria for oversight used in our analysis were instrumental to the detection of these issues.

Future research addressing CQI in medical education may address design variations [33], elaborate on various perspectives on the cost–benefit evaluation in different contexts [34], and quantitatively measure [35] the effects of CQI in medical education contexts. The gathering of evidence about accreditation systems and CQI in medical education seems to be still in its infancy [5, 9, 36].

The strength of this work is the use of criteria for oversight that provided a lens suitable to critically evaluate the use of CQI. The evaluation gains validity by using cases throughout the continuum of medical education and from three different countries. The cases warrant a naturalistic approach connected to real-life experiences. The evaluation would gain more reliability through the introduction of additional evaluators and cases. Moreover, a shift from qualitative research to a mixed methods design with an attempt to introduce quantitative elements would enhance the value of this kind of research.

Conclusion

We performed a utility-based evaluation in the hope of reducing complexity in decision making about the use of CQI. The result is one of both pros and cons. The introduction of CQI in the three cases studied may be regarded as an experiment. Further research is needed to find out the contexts in which itis worth investing in CQI. The use of criteria for oversight helped identify issues that will facilitate decision making around the introduction of CQI.