Background

Clinical supervisors have a key role in training future medical specialists. However, while valuable, their individual influence on the policies and criteria for accreditation of their training programs is usually limited, as is the influence of trainees [1,2,3]. Several studies have shown that clinical supervisors and trainees acknowledge the importance of standards in medical education [4,5,6], yet accreditation processes may collect data and information that does not always reflect the clinicians’ views on quality of training [7].

The system of accreditation of medical specialist training answers to the public and the medical society for the quality of the specialists produced [8]. Accreditation authorities set and monitor standards that oversee the training of future medical specialists [9]. In the context of this research accreditation is defined as a mandatory process by which an accreditation authority reviews the training program using a set of standards with a quality management approach [10]. The purpose of quality management is usually assessing, assuring and enhancing quality. It is a concept which originates in the manufacturing industry [11] and was defined as all activities designed to achieve and sustain high-quality output [12, 13]. These days, quality management of a product or service consists of three major approaches which could be used independently: quality assurance, quality improvement, and quality control [14]. In quality assurance compliance assessing (minimum) standards is ensured, whereas in quality improvement there is encouragement to move beyond the standards. Both above-mentioned quality approaches focus on the process of quality management, while quality control is focused on the output of the process. If needed standards should help accreditation authorities to address quality concerns. Ideally, following the standards should assure educational quality, promote (continuous) quality improvement and measure quality output and lead to high-quality medical specialist training.

Clinical supervisors and trainees have their own perspectives on what high-quality medical specialist training entails [15,16,17]. At the same time, external accreditation standards, which are usually the composite of the views of several societal and professional stakeholders, shape accreditation authorities’ views and assessment of the quality of training. Although accreditation standards often remain unchanged for a number of years, clinical training units are continuously subject to change under the influence of rapid developments in health care and medical education; clinical practice and societal developments have a more dynamic discourse than accreditation. This raises the question whether there is a gap between clinicians’ perspectives and the fixed perspectives behind the accreditation standards that clinicians follow.

The pilot study of Klessig et al. already showed that there might be a discrepancy between the existing accreditation requirements on the one hand and the quality indicators mentioned by clinical supervisors and trainees on the other hand [17]. Although this study indicates a potential discordance between these perspectives on quality of the training, research to date has not yet determined in which aspects the perspectives of clinicians differ from the accreditation standards.

Alignment of these perspectives may contribute to a more evidence informed accreditation design for continuous quality improvement of training [9, 18]. Therefore, this qualitative study aims to determine important quality aspects regarding postgraduate medical training according to clinicians’ perspectives, how these aspects relate to accreditation standards, and how clinicians encounter accreditation standards in practice.

Methods

Study design

This qualitative study was based on a case-study research approach, in which a phenomenon is examined within its real-life context [19]. This enabled a holistic view of specific, contextualized cases to understand complex situations and deepen knowledge [20]. We explored the variation and underlying issues in different stakeholders’ experiences of the quality and accreditation of medical specialist training in Australia, a new system established in 2002, and the Netherlands, a well-established system since 1961.

We considered the perspectives of clinical supervisors, trainees and accreditors on the quality and accreditation of medical specialist training by conducting in-depth interviews. In addition, accreditation standards from both jurisdictions were collected for document analysis.

Participants and procedure

The study included two groups of stakeholders. The group of clinicians consisted of clinical supervisors and trainees involved in daily clinical practice. We included a second group to deepen our understanding of accreditation standards, consisting of accreditors that contributed to the development of standards.

The criteria for participation were:

  1. a)

    Clinical supervisors and trainees from accredited medical specialist training programs in Australia and the Netherlands.

  2. b)

    Members of the Specialist Education Accreditation Committee (SEAC) from the Australian Medical Council (AMC, Australia), and accreditors from the Dutch Legislative College for Accreditation of Postgraduate Medical Education and Specialist Registration Committee (RGS, CGS, the Netherlands).

The participants in category (a) were not accreditors or surveyors in the accreditation of medical specialist training programs mentioned in category (b).

Recruitment of participants

Australian supervisors, trainees and accreditors were sourced through AMC affiliates. Dutch supervisors and trainees from different departments and hospitals were invited through the secretary of the hospital department, Dutch accreditors through the Secretary of the Dutch Board for Accreditation of Postgraduate Medical Education and Specialist Registration Committee (CGS, RGS the Netherlands).

Data collection

A total of 44 stakeholders were invited of whom 29 participated (Table 1). Semi-structured face-to-face or telephone interviews lasted 45–60 min. Interviews were conducted until data reached theoretical sufficiency [21]. Participants were asked the following: (1) what are the most important aspects of a high-quality training program? (2) How would you monitor these aspects if you were the accreditation authority?

Table 1 Participants
Full size table

The Australian and Dutch national standards were collected, respectively, the ‘Standards for Assessment and Accreditation of Specialist Medical Programs and Professional Development Programs by the Australian Medical Council 2015’[22] and ‘Standards for Postgraduate Medical Education Accreditation by the Legislative College 2016’ [23].

Data analysis

Data were analyzed in three phases. Firstly, the audiotaped interviews were transcribed verbatim. Anonymized transcripts were analyzed thematically and line-by-line using open coding to explore (1) all quality aspects of training (2) stakeholders’ experiences of using accreditation standards in practice. This resulted in code scheme 1 and 2, respectively.

In the second phase, accreditation standards of both jurisdictions were analyzed with open coding and were combined with the accreditor’s transcripts of code scheme 1, which resulted in code scheme 3. MaxQDA version 2018 was used for coding.

RM coded the data as second coder to ensure trustworthiness of the coding process and appropriate reflection of all ideas. Code schemes and themes were discussed by the researchers until consensus was reached.

In the last phase code scheme 1 and 3 were compared to determine how quality aspects identified by clinicians corresponded to or differed from the Australian and Dutch standards. Quality aspects mentioned by both, supervisors or trainees and the standards, were categorized as match. Standards lacking quality aspects that were mentioned by supervisors or trainees were categorized as mismatch.

Confidentiality and ethical considerations

This study was approved by the Dutch NVMO ethical review board on November 10, 2016, record number 798, and by the AMC Ethics Committee on December 9, 2016 using the National Statement on Ethical Conduct in Human Research. Data collection was conducted from December 2016 to November 2017.

Results

Two evident matches and two mismatches were noticed in the way stakeholders encounter accreditation standards in practice. In addition, analysis of trainee and supervisor perspectives identified 18 important quality aspects for inclusion in standards (Table 2). These aspects were compared with the standards of each country and identified two additional mismatches.

Table 2 Overview of the quality aspects mentioned by clinicians
Full size table

In the following sections, we will elaborate further on the above-mentioned two matches: (1) accreditation is necessary (2) trainees are the best source for quality measures, and four mismatches: (3) fundamental training aspects that accreditation standards do not capture (this section addresses two mismatches in quality aspects) (4) using standards lack dynamism and (5) quality improvement; driven by standards or intrinsic motivation of healthcare professionals.

Accreditation is necessary

All participants acknowledged the necessity of accreditation to evaluate quality of training, despite its substantial costs, time-consuming nature, and emotional burden. Many participants argued that without standards it would be difficult to assure a minimum level of quality.

“We have to guarantee a minimum quality for the society. (…) We, humans are fallible.” (AccreditorNL01)

Trainees mentioned the need for an impartial and objective perspective on training quality by an accreditation authority. In addition, supervisors find accreditation reports useful to demand changes or resources from the hospital administration.

“I recognize completely that there is a forcing function there. And I don’t think that is always a bad thing. (…) So, we were able to negotiate some things with hospital administrators that we’ve not been able to achieve traction on, because of the recommendations of the accreditation team (…).” (SupervisorAU04)

The perspectives between Australian and Dutch stakeholders were in contrast when they speak about accreditation as a lever for change. Australian stakeholders believed that accreditation was an enabler and driver of change. The accreditation authority had far more influence on the colleges than faculty members. Without the help of the accreditation authority, colleges were not likely to be able to eliminate differences between programs.

“So many hospitals are acting on historical, you know, eh- a custom. We always did it this way, you know.” (AccreditorAU02)

Dutch faculty members were more skeptical about the influence of accreditation. They were positive about accreditation driving reflection, however they claimed that they did not change their training method or vision after a site visit.

“I told you already, I am not fund of rules/regulations (…) I even do not know the rules (…) do not need to know them (…) I am a passionate supervisor for ten years now.” (SupervisorNL03)

Most supervisors emphasized that severe violation of standards by peers requires penalties, such as withdrawal of accreditation, replacing the program director, and decreasing self-regulation. In contrast, trainees did not consider punishment of non-compliant programs as beneficial, because this could establish a culture of fear and stop professionals from speaking up.

Although different participants believed that standards cannot induce radical change in or capture the culture of the training site, most supervisors and accreditors regarded standards as quite powerful for highlighting important aspects for education providers and training sites. Exclusion of subjects from standards could give the impression that these aspects of training are insignificant.

“And you can’t, you can’t set the standards for culture on a piece of paper. You can’t say, a department must have good culture.” (SupervisorAU04)

Trainees are the best source of quality measures

Participants considered trainees the best information source regarding training quality. Trainees’ information and feedback were seen as able to negate any window dressing by education providers. Questions were however raised about the conditions for collecting information from trainees, and particularly trainees’ capacity to speak up without assurances of confidentiality and anonymity.

“Keep your mouth, get the approval [successful accreditation].” (AccreditorNL03)

Some supervisors were concerned about trainees giving desirable answers to please accreditation surveyors. Despite the difficulty of the hierarchical environment, trainees were aware of their potential impact and felt responsible to give input.

“(…) it’s pretty hard I think for ehm, for trainees to speak out against stuff [harassment and bullying amongst other] like that, particularly where it’s hard to get into that specialty, and they feel like, you know, whistle-blowers in medicine are treated horrifically. So, ehm, I think, I think getting people to give you straight answers in an interview, that situation would be quite difficult (…).” (TraineeAU01)

Fundamental training aspects that accreditation standards do not capture

There were two evident mismatches in the quality aspects: service provision and training balance, and trainee empowerment. Neither of both standards covered these aspects, however their inclusion is supported, especially by trainees.

Trainees mention that service provision is often prioritized over training. Although they learn from providing care, learning from clinical work demands allocated time. The appropriate balance between workload and active and reflective learning needs to be secured.

“The training posts that I’ve now found most valuable are often the ones that have an appropriate workload, such the trainees are doing enough cases to gain clinical experience, but also spending some time discussing those cases with their consultants.” (TraineeAU04)

In the context of training aspects, trainees plead to be empowered and supported by their supervisors to speak up. Trainees are in a vulnerable and dependent hierarchical position; others taking advantage of them can impact their well-being, learning or performance. Mentoring may be beneficial for problems trainees experience during their training.

“(…) generally, people that enter into these things [training programs] are smart and motivated and will do what they need to do to succeed and it’s all about making sure that they are not taken advantage of in that time.” (TraineeAU01)

“We have to move towards a system where it is accepted to show vulnerability and a system dependent on trainees. Let them [programs] earn their trainees.” (TraineeNL01)

Using standards lack dynamism

Supervisors encountered too much rigidity with some accreditation standards. Ideally, the context of practice should be more emphasized. They would prefer more room for flexibility and negotiation, for example collegial peer review, in which clinical supervisors are encouraged to show best practices. Accreditors had similar perspectives; accreditation should be more dynamic to continuously adapt to changing circumstances, and accreditation authorities should not hinder, but support training programs.

“Visiting a program should not be an assessment, in my opinion it should be a collegial process to reflect together on the training program with the framework [standards] which we agreed upon. (…) So, if the visit is intended as a thorough check of my training program, then I’ll say well, that’s unsupportive and a sign of distrust.” (SupervisorNL01)

Improvement: driven by standards or intrinsic motivation of healthcare professionals

On the topic of quality improvement accreditors and clinicians have contrasting views. Accreditors in both jurisdictions believed firmly in standards being a driving force for quality improvement, although one Dutch accreditor questioned whether a high-stake summative assessment should be the approach.

Clinical supervisors and some trainees, however, mentioned that quality improvement should be driven by healthcare professionals and could not solely be achieved by an external authority. Many supervisors argue that setting standards is inherent to minimum acceptable norms and accreditation authorities should rather not go beyond their scope.

“But again, it’s about enabling the minimum standard and hopefully then ehm, allowing people to find, move beyond that to a higher standard.” (SupervisorAU04)

Discussion

This study explored the perspectives of trainees, clinical supervisors and accreditors on quality of training, accreditation and standards in Australia and the Netherlands. We explored the differences and similarities in their views and compared this to the accreditation standards of both countries. Two matches and four mismatches were identified. In this part, we will reflect on our findings using the three quality management approaches: quality assurance, quality control and quality improvement.

The participants’ perceived necessity of accreditation is a remarkable acceptance of accreditation considering the evidence-based era. Despite the lack of robust evidence for accreditation effectiveness and impact in the current literature, Brubakk et al. claimed that accreditation is not likely to be abandoned [24]. Efficacy of accreditation may not be linked to evidence; however, the social meaning may shed light on its efficacy. The clinicians’ perceived necessity of accreditation could originate in the legitimacy of the accreditation authority and therefor clinicians could respect its power [25]. It is also plausible that clinicians will comply with most standards even if there were no sanctions or legitimacy, because these standards fit their own ambition [26]. Social identity and motivation to protect the medical profession could be another explanation for requiring accreditation [27]. This may explain the supervisors’ desire for penalties in cases of severe violation of standards.

Participants acceptance of accreditation was regardless of the used quality management approach(es); however, the clinicians and the accreditors did not agree whether or not quality improvement should be driven by accreditation. Ideally, compliance with standards should lead to continuous quality improvement in which training programs strive for excellence [2, 28,29,30]. Although quality improvement is valued in many jurisdictions, quality improvement is not solely achieved by enforcing standards, since it requires a degree of (intrinsic) motivation of all concerned [31,32,33]. This was also confirmed by supervisors in this study. Research suggests that whether accredited persons’ intrinsic motivation is enhanced or undermined depends on how the accredited party perceives the accreditation authority’s standards and actions [31, 33]. If standards are perceived as fitting in a shared ambition this may increase the intrinsic motivation of the accredited party [31]. One study among Danish general practitioners (GPs) has shown that accreditation may even foster intrinsic motivation as long as GPs perceive accreditation as a quality improvement instrument [33]. Encouraging (continuous) quality improvement, means continuous (small) modifications and changes on the workplace. Change is challenging, but by merely developing standards and policy-making, without an implementation strategy for the clinical workplace it is hard to initiate willingness for change. This also aligns with leading change model of Kotter: a sense of urgency must be established, the vision must be created and communicated, empowerment to act on vision and improvement must be consolidated [34].

Australian standards covered more aspects than Dutch standards. This could be typical for a relatively new accreditation system which is based upon state of the art and societally responsible accreditation design and not constrained by a lot of historical influences. We believe that there are two other reasons for the differences: (re)current political debate and traditions. Access and selection are not included in the Dutch standards and equity and diversity are less prominent in the political debate than in Australia. Other differences like no standards for job satisfaction, outcome focused programs and community needs in the Netherlands could be explained by traditions, which are difficult to change. Nowadays, if an accreditation system was newly introduced in the Netherlands this would be most probably included in the standards. Trainee-centered programs are aligned with the strong educational influence from the Maastricht School with a large emphasis on student-centered education in the Netherlands [35].

If we compare Table 2 with the comprehensive WFME global standards for medical specialist training, the standards miss the following items: requirement for international or regional collaboration, allocated time for formal education, trainee empowerment, job satisfaction and trainee-centered programs [36]. In the development of accreditation, comparing international standards is an effective way of learning from the best practices of each other.

Most of the current fundamental quality aspects were captured by the standards. However, according to Koksma & Kremer quality is not an objective truth, but dynamic [37]. Quality aspects are subject to changing perspectives, for instance social responsibility is probably an emerging aspect. If we accept that quality is dynamic, the quite rigid accreditation standards must make room for flexible components. Supervisors and accreditors agreed upon flexibility regardless of the used quality management approach. We believe that accreditors may be key figures in adding more dynamics to accreditation standards by bridging between clinical and regulatory practice.

This study uses trainees’ views in measurement of quality and found that trainee feedback and information is the most important tool. This is relevant for setting up a quality control approach  that includes the output of the process. It is also common to search for consumers’ perspectives in evaluation. This study has taken trainees to be active participants in their learning rather than consumers. In the regulation of medical specialist training, public health services and patients’ interests are also at stake. While patients may be seen as primary consumers of outcomes of training, it is not always easy to embed their perspectives in training evaluation. Contrary to the Dutch accreditation processes, the Australian accreditation processes and standards do require education providers to engage health consumers and the community on defining educational purpose and program outcomes as well as in program evaluation. Australian specialist training providers are responding to these standards in multiple ways. The right instruments for patient and consumer feedback could enhance relevance and acceptability of their perspectives for training evaluation [38].

While trainee feedback is seen as a source of valuable information; the Australian and Dutch standards lack trainee empowerment. This could be a call for at least a quality assurance approach on the area of trainee empowerment. It is conceivable that the culture in which the training occurs is hierarchical and less likely to provide opportunities for feedback and criticism. Besides that, trainees are in dependent and vulnerable positions, which make them unwilling to speak up to seniors [39, 40]. By including trainee empowerment in standards, it does not necessarily lead to an immediate change, but at least it may highlight trainee empowerment as an important aspect of training.

Strengths and limitations

This study contributes to the scarce empirical knowledge about accreditation for medical specialty training. Participants were from different institutions in two countries. It is reasonable to assume that our findings will have some relevance and potential transferability to other settings; nevertheless, the relevance may be questioned in other contexts or cultures. This study is based on a socio-constructivist approach, quantitative proof for accreditation is still limited.

Suggestions for further research

Although we gained insights in clinicians’ perspectives about quality and accreditation of training, it would be interesting to explore the effect of standards addressing trainee empowerment in the clinical environment. Moreover, in order to improve accreditation systems, further research should focus on the value of flexible components in the accreditation design. For example, reducing the number of specific standards and focusing more on relevant themes, such as teaching and learning methods, or greater co-design of the process between the accreditors and the organization being accredited.

Practical implications

Our findings raise intriguing questions regarding the nature and extent of the current accreditation design. Future accreditation design should be more aligned with the dynamic concept of quality.

Conclusion

Accreditation standards change in a less dynamic fashion than the context and perspectives of clinicians, however clinicians did not dispute the utility of accreditation. Flexibility in accreditation standards and processes is needed to keep pace with the developments in and beyond the workplace. Empowering trainees as partners in learning and decision-making about training might add to the quality of training sites and needs to be addressed by the accreditation system. Improvement on trainee empowerment is required, especially since  trainees are considered the most valuable feedback source. Accreditation systems could also benefit from taking account of the internal drivers of clinicians.