Introduction

The aim of the Competency Based Training in Intensive Care Medicine collaboration (CoBaTrICE) endorsed by the national intensive care medicine (ICM) training organisations of 43 countries, including all those of the European region (“Appendix”), is to optimise the care of critically ill patients and their families through the development of common standards for high-quality postgraduate training and education [1–3]. We began this unique collaboration by creating an international ‘product specification’ for an intensivist through consensus development of intensive care competencies [4]. These competencies have been adopted by Croatia, Cyprus, France, Israel, Netherlands, Norway, Portugal, and the UK; and are in the process of adoption by Denmark, the Czech Republic, Greece, and Ireland. The second phase, which we report here, has been to develop internationally acceptable European standards for the ‘production process’—that is, the programmes of training in ICM.

We determined the need for formulating standards for programmes of training in ICM through a prior survey of national training organisations about the training environment for ICM in the 28 countries of the European region [5]. In this survey, standards for training were highlighted as a particular area for improvement, with only 18 (64%) countries having a formal national system for assuring the quality of ICM training, the majority using standards based on volume metrics (numbers of beds or admissions) and not the quality of the training experience. Across the 28 countries, only 14 (50%) assessed the trainees during routine clinical activities using structured assessment and formal documentation and only six countries (21%) recognised the work of trainers formally in a work plan or contract. Only one respondent expressed satisfaction with all aspects of training structures and processes. The paucity and variability of national training programme standards in ICM are potentially problematic because while variation in healthcare processes may be a strength [6] when the quality of the end-product is not in doubt, process standards become more important when the outcome in question is difficult to measure or complex [7, 8] as in the case of ‘an intensivist’. The absence of standards also makes it more difficult to develop baseline metrics for quality improvement of training, and impedes the European goal of professional mobility across international borders, consistent with the aims of the Bologna declaration [9] and the European Association for Quality Assurance in Higher Education [10].

This survey provided the rationale for developing internationally acceptable European training programme standards to support trainers, enhance the training experience, reduce undesirable variation, and increase and harmonise quality improvement efforts in ICM across Europe. We now report the development of these standards using consensus techniques.

Methods

We used a combination of a nominal (expert) group, and a modified Delphi process to develop and achieve consensus on quality standards for programmes of training in ICM, based on our prior experience with the development of the intensive care competencies [2]. The Delphi process has many forms [11] but the common component is iterative modification of the items under discussion until full consensus is achieved or no further changes are proposed [12, 13]. The number of iterations is therefore usually variable and not predefined. In order to achieve a balance between efficient use of time and resources while permitting maximum opportunities for interchange of ideas across Europe we employed group meetings, email, Web-based polling, and telephone calls in several stages (Fig. 1).

Fig. 1
figure 1

Consensus development process and outcomes

Full size image

The steering committee predefined consensus criteria and cut-points as follows. Standards achieving full consensus [100% of responding national coordinators (NCs) agreed or strongly agreed] were removed from further iteration and entered the final set. Standards were rejected if fewer than 75% of responding NCs agreed with their inclusion. Standards which achieved less than 100% but at least 75% agreement went through to the next iteration, and if full consensus could not be achieved these standards were then removed and categorised as suitable for future review. The final set of standards required formal approval by all 28 NCs. Proceedings were conducted in English.

Stage 1 of the study was the development of an initial draft set of standards by the expert group which consisted of nine participants and a moderator: of the nine participants, eight were European CoBaTrICE NCs representing their national ICM training programmes and one was a full-time senior healthcare executive who had previously been a senior academic intensivist. All participants were therefore highly experienced physicians and educators, nominated by their national training boards or societies. The group was moderated by an academic non-physician medical educationalist (“Appendix”). Preparatory work included asking participants to review their own national programme standards for plenary presentation. They met together for a 2-day workshop in June 2008, focussing on two questions: which aspects of training should be included in a quality assurance process? And, what minimum standards would be acceptable in your country? The expert group produced the initial standard-statements and grouped these into broad domains. After the meeting, members worked in pairs to clarify terminology; this then underwent further editing by the executive member of the working group, followed by revision by all members.

In stage 2 this first draft was circulated to all NCs in September 2008 for written comment, followed by a half-day meeting of all NCs or deputies at which detailed verbal comment was recorded by the research nurse and assimilated into the second revision by the project leads. This was then prepared as a Web-based survey for stage 3.

In stage 3 all 28 NCs reviewed the second iteration over a 2-month period, and rated each proposed standard using a Web-based structured feedback form. Respondents were asked to identify whether they agreed or disagreed with each proposed standard, and then to state whether it was achievable or not at present in their countries. They were invited to include their trainees in this process. The extent of agreement with each proposed standard was measured by using a four-point scale—agree strongly, agree, disagree, or disagree strongly. A ‘neutral’ option was excluded to force a decision. Against each response, respondents were able to add free-text commentary. Commentary from the NCs was used to clarify or modify standards where required. ‘Achievability’ was used as an indicator of likely difficulty with implementation, for example requiring investment of staff or resources for some countries.

In stage 4, the remaining, revised or rephrased standards were redistributed to all NCs by email and discussed at the next project meeting in March 2009, allowing further modification. At this stage wording was clarified or amended and items removed if they were overlapping, lacking in clarity or covered by existing legislation. Proposed standards achieving full consensus were again removed from further iteration and placed in the final set.

In order for the remaining standards to be included in the final iteration (stage 5), 100% consensus (agree or strongly agree) among NCs was required.

Results

Stage 1

The nominal group developed 52 standards, which were grouped into four domains.

Stage 2

Responses to the first draft from all NCs were collated, and discussed at a project meeting. This permitted terminological clarification, and the 52 proposed standards (electronic supplementary material) then went through to stage 3.

Stage 3

Of the 52 proposed standards, 23 were agreed by all 28 NCs and were therefore removed from further iteration and placed in the final set (Table 1). Of the remaining 29 standards, 27 achieved sufficient agreement (≥75%) to enter the next iteration, and two were excluded (<75% agreement). The two proposed standards which were excluded from further consideration were “ICUs with less than six beds can contribute to a clearly defined programme” (low agreement, high achievability), and “national examinations must be used in trainee selection processes” (low agreement, low achievability).

Table 1 The final set of standards
Full size table

Stage 4

The 27 standards retained for further iteration were recirculated to all NCs, of whom 13 met in person. At this stage, five standards achieved complete agreement and entered the final set (Table 1). Seven achieved at least 75% but less than 100% agreement, could not be modified to the participants’ universal satisfaction, and were reserved as suitable for future review (Table 2). Twelve were excluded from further iteration: four achieved less than 75% agreement, four were covered by existing legislation (e.g. training programmes must comply with the European working time directive), and four were captured in other standards and therefore redundant. Three standards remained for the next and final iteration.

Table 2 Standards for future review
Full size table

Stage 5

Three statements were recirculated to all NCs by email. One of these achieved 100% agreement. The other two achieved more than 75% agreement, but consensus was not possible, and they were reserved for future review. Both of these had a numeric component and although both were deemed important by the NCs, the minimum number or length of time could not be agreed upon. The first of these originally read: “The ICM training programme must provide access to >500 admissions per year” (89% agreed; 68% achievable). The number was reduced to “>300 admissions per year” which improved achievability but reduced agreement because of concerns about inadequate exposure. The second reserved standard was amended from “Each training programme must be accredited by national authorities at least every 5 years” to “at least every 7 years”, but this diminished agreement further. Concerns included the absence of national quality assurance systems, and a tension with the speed of scientific developments in ICM.

The agreed set of standards

The final set of 29 standards (Table 1) achieved 100% consensus (by definition): all 28 NCs either strongly agreed or agreed on their inclusion. The 100% consensus was achieved after stage 3 for 23 of these standards. After a further two stages, a further six propositions achieved 100% consensus. Only one standard was agreed strongly by all: that “the trainer must be a practising intensivist”.

The final set includes standards which may not be immediately achievable in some countries: for five of the standards, fewer than 75% of the NCs judged them to be achievable. The standard likely to present most challenges in the short term is 4.03b concerned with the contractual recognition of the supervisor’s role.

Standards for further review

Nine propositions not achieving 100% consensus were identified for further review and are shown in Table 2. Most of these achieved high levels of agreement in their final iteration but presented difficulties in determining metrics or setting values. The standard referring to nurses is an example: the group clearly felt that nurses were important to physician training in ICM, but were unable to identify a suitable internationally applicable metric. The two with numeric components were the number of admissions, and the period of accreditation, as described above. The remaining seven standards in Table 2 require further definition or exploration which could not be achieved within the project time frame.

Discussion

We have used consensus techniques to develop a preliminary set of international standards for programmes of training in ICM, approved by the representatives of the national training organisations of all 28 European region countries. This represents the first step in addressing perceived weaknesses in quality assurance of ICM training identified from the earlier European survey [5]. These standards are fully consistent with the Bologna declaration [9] and the recommendations of the European Association for Quality Assurance in Higher Education [10]. Used in conjunction with the clinical competencies, the standards will facilitate trainee and specialist professional mobility, and demonstrate to trainees, trainers, programme directors, government bodies, patients and the wider public that the programme as a whole—the ‘production process’—is of an acceptable standard.

The consensus method: outcomes and interpretation

The CoBaTrICE approach of developing standards using consensus amongst front-line staff who are also senior educators and representatives of their national training organisations enhances the viability and validity of these standards. Ongoing work will be required to review the standards themselves and their implementation. Once this preliminary set of standards has become embedded in national training programmes, the first review should involve more formal input from trainees and lay representatives. Although we have not set out to define “excellence”, commitment to quality by the NCs was evident by the expression of agreement with standards that were judged to be not (yet) achievable, and rejection of those where the level was too low. The final set (Table 1) thus includes examples of standards which will be aspirational for some countries, and may be used by NCs and others in their arguments for change and development as well as further research.

Two of the standards within the training centre domain relate specifically to size and are expressed in terms of volume measures (1.01 ‘6 beds or more’ and 1.02 ‘percentage of ventilator days’). These generated considerable discussion about the lack of evidence for a relationship between volume metrics and adequacy of training. Volume measures could exclude small but otherwise excellent ICU training centres, thereby preventing trainees from gaining experience in the type of hospitals where they may take up permanent specialist appointments. Arguably, having a sufficiently diverse case mix to enable acquisition of competencies (2.02a) is a sufficient indicator of adequacy. With specific reference to ventilator days, future medical advances might reduce the need for invasive ventilation, and thereby make it more difficult to achieve a specific volume measure. Volume-outcome metrics in education are an area for future research, and for this reason these standards should be kept under regular review. We should emphasise here that our focus has been on standards for programmes of training, and not solely on single training centres. Countries with modular programmes of training based on more than one centre will be able to meet volume measures without difficulty.

Missing or excluded components

The final set of standards does not include any which refer to examinations, competence acquisition, or duration of training. It would have been difficult to mandate a common standard for examinations given the diversity of approaches across Europe [1] and the acceptance of workplace-based assessment of competence as the most appropriate measure of educational outcomes. A common European ‘quality indicator’ examination in the form of the European Diploma of ICM is clearly desirable, but best achieved through evolution rather than imposition of standards. A similar argument applies to competence acquisition: the CoBaTrICE competencies are increasingly accepted as the international standard, and the method has now been adopted by the USA [14], so there was no need to include them as a separate standard. Instead, national coordinators chose to develop standards which provided the best environment for the acquisition of those competencies (see 2.02a). A standard for duration of training would be difficult to justify given a competence-based programme in which trainees may require differing periods of training.

Comparisons with other approaches and systems

There is considerable diversity in approaches to standards development and accreditation of medical education worldwide [15], resulting in calls for the development of global standards for undergraduate training and qualification [16, 17].

At specialist, graduate level, the current diverse approach to quality assurance of ICM training programmes in Europe is described in our recent survey [5]. In Australia it is the responsibility of the Australian Medical Council to accredit (specialist) training organisations [18] to undertake peer assessment. The USA and Canada have a dispersed and a centralised system, respectively [19]. Accreditation of graduate training programmes in the USA is the responsibility of a private not-for-profit organisation, the Accreditation Council for Graduate Medical Education (ACGME), which produces common standards for all training centres and speciality-specific standards for each training programme. The speciality-specific standards are monitored and reviewed by review committees with professional input requested by email. This means that critical care (a subspeciality of four primary specialities) has four different sets of programme standards [20–22]. Harmonisation may eventually occur, with the development of common competencies across some disciplines, using the CoBaTrICE approach [14]. Canada by contrast has a centralised universal standard linked to the CANMEDS competencies, and therefore more shared standards between specialities [23, 24].

Implications for quality improvement

Standards of education should be developed and audited to improve training and patient care [25, 26]. The programme standards presented here are a necessary, but not a sufficient, foundation for improving education and training in ICM. They must be combined with a dynamic quality assurance process delivered locally and managed by national training organisations [27, 28]. Regulatory authorities must recognise that quality assurance processes place significant demands on trainers, programme directors and professional organisations [29]. We propose therefore that these consensus standards should be seen as the foundation for a professionally owned, scientifically rigorous [30] dynamic process of quality improvement with the specific aim of improving the educational environment and hence creating better doctors delivering more reliable care.

Conclusion

The absence of training programme standards may result in a poor learning environment in which trainees fail to aspire to excellence. This first set of European standards focussed on the ‘production process’, coupled with the CoBaTrICE competencies should encourage national training organisations, trainers and trainees to drive up standards of training and clinical practice in ICM not only in Europe, but worldwide, with benefits for patients and families. If one accepts that effective regulation of medical education is vital because “social welfare is inextricably linked to the quality of the nation’s physicians” [31], and Donabedian’s perspective [32] that healthcare performance is influenced strongly by context, then common standards for training programmes will contribute to better patient care by a wider effect on the clinical environment.

The development of standards should be accompanied by research evaluation. Examples include surveys such as those organised in the UK by the General Medical Council [33], or the educational outcomes project in the USA supported by the Accreditation Council for Graduate Medical Education [34]. The work we have presented here provides the foundation for a formal European research programme evaluating the impact of quality assurance processes on vocational training, and for developing evidence-based medical education as a key component in improving healthcare outcomes.