Introduction

Following a rapid increase in the popularity of competency-based medical education [1, 2], the methods and concerns around the assessment of competence have been met with increasing interest [3, 4]. Assessment tools in a workplace that cannot be standardized and increased interest in the ‘softer’ skills pose challenges to the assessment procedures [5, 6]. One approach that has been suggested to operationalize the attainment of competencies is to determine whether or when a trainee can be trusted to execute a professional activity without supervision [7]. Trust in trainees requires observations that do not only draw on standardized skills and knowledge but take other facets of competence into account [8, 9]. Using a Delphi approach, we investigated the factors that educators in the Netherlands find important to consider when making entrustment decisions about medical trainees [10]. This yielded a list of 25 relevant factors or ‘facets of competence’ (FOCs) when entrusting trainees with clinical responsibilities.

The aim of the current study was to determine the external validity, i.e. the generalizability, of the factors that Dutch educators found essential. Our approach was to ask experienced educators in two countries to rank-order these FOCs and then to determine the level of agreement among the countries about the highest scoring FOCs.

The study was carried out among Dutch and German medical educators. The Netherlands and Germany differ in medical education culture, particularly in the sense that in the Netherlands education reform has dominated medical curricula throughout the country since the mid-1970s, while such processes in Germany have started only recently [11, 12]. It is fair to say that Dutch medical schools have ‘modern medical curricula’ (not necessarily ‘better’) and most German medical schools have predominantly ‘traditional medical curricula’, while a number of other countries in Europe have positions in between. Agreement among medical educators in the Netherlands and Germany about important FOCs would support the generalizability of those FOCs.

Methods

Participants

To find comparable groups in both countries we approached Dutch and German experts. We approached all 24 experienced clinicians in the Netherlands who met the following criteria: (1) holds an academic chair in medical education, (2) works in clinical practice and (3) supervises residents. The 36 German physicians we approached met the following criteria: (1) Master’s degree in medical education, (2) works in clinical practice and (3) supervises residents.

Questionnaire

The experts were invited by email to complete a questionnaire in an electronic format, after the German version had been translated from the Dutch language. To gain insight into the experts’ judgements about FOCs, they were asked to assign a score from 1 (‘least important’) to 5 (‘most important’) to each of the 25 FOCs that had resulted from the Delphi study. They were requested to give each possible score (1–5) five times (so, 5 FOCs had to get score 1; 5 FOCs had to get score 2, etc.).

Data analysis

We calculated means, medians and standard deviations for both the Dutch and German group of experts and a level of agreement, according to an adapted De Loe’s [13] procedure. De Loe developed a method to determine the amount of consensus, based on the percentages of answers in one or two contiguous categories of a rating scale (see Table 1). De Loe used this method for a 4-point scale, while we used the same method for a 5-point scale, which makes it a little more stringent.

Table 1 Levels of agreement according to De Loe [13]
Full size table

Results

Respondents

In total 8/24 Dutch and 8/36 German experts participated in the ranking study (response rates 33 and 22 %, respectively). In the Dutch group, 7 were male and the average age was 61 years (56–66 years). In the German group, the average age was 43 years (33–53 years) with 4 male and 4 female responders. The responders in both rounds represented a wide range of surgical and non-surgical disciplines (Dutch: cardiology, general practice, gynaecology, internal medicine, neurology, oncology and surgery; German: emergency medicine, gynaecology, internal medicine, psychiatry and surgery).

Results

The results of the ranking study are presented in Table 2. The medians and means indicate the ranking of the FOCs. For the Dutch group of responders, medians were between 1 and 5 and the means between 1.63 and 4.75 (SD 0.46–1.64). For the German group, medians varied between 1 and 5 and the means between 1.88 and 4.38 (SD 0.54–1.93). For both groups, the level of agreement varied from none to high.

Table 2 Rank-order, median scores, mean scores and level-of-agreement concerning the importance of 25 FOCs for entrustment decisions, as judged by Dutch and German clinical educators
Full size table

Discussion

The aim of this study was to determine which FOCs of medical trainees are considered most important to formulate entrustment decisions by experienced supervisors in residency training and to evaluate the agreement about them in different European countries, to expand their generalizability.

We found strong agreement between physician educators from the Netherlands and physician educators from Germany in ranking the competency facets relevant for entrustment decisions. The ‘top 10’ for the total group of responders, based on medians and means, appeared nearly the same as the ‘top 10’ for each of the two countries separately. The only FOC that substantially differed was ‘structure, work planning and priorities’, ranked 8 in the German group, but 20 among the Dutch.

The fact that two groups of physician educators from different medical education cultures highly agreed on the importance of certain FOCs for entrustment decisions strengthens the relevance of these FOCs. The top-10 align with Kennedy et al. [9]’s findings. In their grounded theory study, these authors also found ‘truthfulness’ (the absence of deception) and ‘conscientiousness’ (the thoroughness in data gathering and dependability) to be an important quality that supervisors value in trainees to determine their readiness for independent clinical work [9]. We had not explicitly included these as facets in our list, but it may be assumed that our respondents would score them highly if they had been included. Implicitly, they are reflected in our items that stress scientifically grounded working, openness, responsibility and coping with mistakes, all part of the top-10.

One limitation of our study is the low response, which also varied in the different parts of the study. However, the expertise of the participants and the fact that a wide range of disciplines are included adds to the relevance and the generalized nature of the findings, but the study cannot be viewed as conclusive. Another limitation is that we only included one other country. Future studies to establish a generalized nature of factors that affect entrustment decisions should include countries with other than the Western industrialized culture, as we cannot exclude that those countries would show different factors.

The background of making entrustment decisions deserves further empirical study, preferably with more experts and more countries and focused on a validation in practice.

As a practical outcome, our findings may be used as input for a frame of reference training [14] for clinicians who must regularly take entrustment decisions. Supervisors may be guided by their first impressions of trainees, which may be less accurate than they tend to think [15], and training can make them aware to take multiple facets of competence into account.

Conclusion

Our aim was to reveal what facets of competence are considered most important for entrustment decisions by supervisors of residents. We found high consensus between experts from the Netherlands and Germany, despite large differences in their curricula. Our findings are relevant for the development of assessment instruments to evaluate whether medical graduates are ready for clinical practice.

Essentials

  • There is high agreement among supervisors about what facets of competence are considered most important for entrustment decisions.

  • There is consensus between physician educators from two countries with different medical education climates (the Netherlands and Germany).