Introduction

In recent years, the landscape of medical education has significantly transformed due to increasing demands of public accountability and changing patient needs. In response to these evolving demands, competency-based medical education (CBME) has emerged. CBME has been gaining popularity in medical education programs [1]. In a CBME paradigm, medical curricula are structured based on predefined competencies that physicians should have acquired upon completion of the program [2, 3]. Despite the theoretical underpinnings of CBME, its implementation has encountered various obstacles [4]. Particularly, assessing competencies in real clinical environments has been a major barrier in the effective integration of CBME into medical education systems [5]. Recognizing this challenge, the concept of Entrustable Professional Activities (EPAs) has emerged.

EPAs are essentially tasks or activities that medical professionals should be able to perform competently and independently by the time they complete their training [6, 7]. EPAs are used to assess a learner’s ability to integrate and apply the necessary competencies in real-world clinical practice. They necessitate evaluating a learner’s progress and readiness for independent practice by observing their performance in these key professional activities in clinical practice [8]. The term “entrustable” indicates that, upon graduation or completion of a specific training period, a supervising physician or mentor should be able to entrust a medical graduate with these activities without direct supervision, considering them proficient and safe for the patients to perform these tasks independently [9, 10].

Considering the immense potential, integration and implementation of EPAs has gained rapid momentum, across various health professions and medical specialties [11, 12]. Despite this progress, a significant gap notably persists, when it comes to accommodating competing needs of curriculum designers and those of users in practice, namely trainers and trainees [13]. While the promise of EPAs in facilitating CBME is promising, there is lack of comprehensive evidence incorporating users’ perceptions during the design phase [8, 11, 14]. Therefore, the aim of this study was to design an EPA framework for workplace-based assessment by actively involving clinical educators, trainees and trainers throughout the process.

Methods

Setting and participants

This study took place in the interuniversity postgraduate General Practitioner’s (GP) Training, Belgium. To standardize GP Training across Flanders, four Flemish universities (KU Leuven, Ghent University, University of Antwerp, and the Flemish Free University of Brussels) collaboratively developed a postgraduate training program. This training program consists of three different training-phases and rotations, spread through three years, two rotations are in a GP practice, while one takes place at a hospital setting.

The GP Training is overseen by the Interuniversity Centre for GP Training (ICGPT). The ICGPT plays a pivotal role in coordinating and managing various aspects of the curriculum. Among its key responsibilities, the ICGPT oversees the allocation of clinical internships, conducts examinations, facilitates regular meetings between trainees and trainers, and maintains trainees’ learning electronic (e-) portfolios.

In 2018, the ICGPT initiated a shift towards CBME. The rationale of CBME was introduced in the curriculum by integrating first the CanMEDS roles. To facilitate this transition, two clinical competence committees (CCCs), comprising medical doctors and clinical educators from the four universities were appointed. These CCCs were tasked with coordinating workplace-based learning, and curriculum and assessment, respectively.

To align the curriculum with the patient needs in primary care, the two CCCs designated and defined ten different care contexts characteristic of primary care (i.e. short-term care, chronic care, emergency care, palliative care, elderly care, care for children, mental healthcare, prevention, gender related care, and practice management). Subsequently, in 2022, we initiated the process of designing specific EPAs for the GP Training. The EPAs aimed to facilitate and improve workplace-based assessment. These two CCCs participated in the design process, while trainers and trainees were invited to share their opinion as well.

Designing the EPA framework

The design of the EPA framework was based on participatory research design to engage different stakeholders [15]. Participatory research design is a community-based methodology aiming to create solutions for and with the people who are involved [15]. This iterative research approach encompassed three fundamental design-stages in a circular relationship, namely design, evaluation and refinement (Fig. 1). We executed two distinct iterative cycles, each with a specific group of stakeholders (Fig. 2). In cycle 1, we focused on CCCs, fostering discussions and validating the framework. In cycle 2, we involved clinical trainers and trainees, ensuring cross-validation. In the following section, we describe each iterative cycle, indicated as cycle 1 and as cycle 2, respectively.

Fig. 1
figure 1

Three design phases for designing the EPA framework

Full size image
Fig. 2
figure 2

Process for developing the EPA framework based on participatory design research

Full size image

In cycle 1, after reviewing relevant literature, we developed a blueprint of 60 EPAs corresponding to the ten different care contexts, already integrated in the curriculum [9, 10]. By doing so, we wanted to ensure practical applicability and relevance of our framework within the established educational environment. Afterwards, we linked all EPAs to the CanMEDS competency framework [16]. We defined competencies as broad statements that describe knowledge, skills and attitudes that GP trainees should achieve during the different training phases [17]. The CanMEDS framework identifies and describes different competencies for patient-centred care, and comprises seven different roles: medical expert, communicator, collaborator, leader, health advocate, scholar, and professional. By linking EPAs to CanMEDS, we constructed a matrix that served as a structured guide for integrating the EPAs in the workplace. Also, together with the CCCs we defined behavioural and cognitive criteria to anchor entrustment levels [9]. These criteria described required knowledge, skills, and attitudes in order for an EPA to be entrusted.

In cycle 2, we aimed at operationalising the EPAs, cross validating them by interviewing trainers and trainees, and deciding entrustment levels. Specifically, to operationalise the EPAs, we developed an assessment form, called Clinical Practice Feedback form (Fig. 3). We chose to link EPA assessments not only to direct and video observations, but also for case-based discussions. Additionally, we agreed upon entrustment levels and the entrustability scale. Entrustment was anchored on criteria that were defined along the EPAs. We decided to use the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) for validity and reliability reasons (Fig. 4) [18]. The Ottawa scale requires assessors to describe how much supervision they provided to trainees while performing a specific EPA. Concretely, the scale comprises five levels of performance ranging from trainers taking over the activity to trainees performing the activity without supervision (Fig. 3) [18].

Fig. 3
figure 3

Example of Clinical Practice Feedback form available in the e-portfolio

Full size image
Fig. 4
figure 4

Five levels of entrustment based on the O-SCORE scale [19]

Full size image

Data collection and analysis

In cycle 1, we evaluated the EPA blueprint by employing a modified Delphi methodology, with two rounds [19]. We invited members of the two CCCs (N = 14) to give feedback on the EPA blueprint via e-mail and during meetings, scheduled by the ICGPT. Members were asked whether they thought each EPA was necessary for workplace-based assessment and needed to be included in the framework. They were also encouraged to give feedback regarding the formulation of the EPAs. Once we gathered all the comments, we refined the blueprint and sent it back to the CCC members. In cycle 2, we interviewed two trainers and two trainees using semi-structured interviews and following the ‘think-aloud protocol’ [20,21,22], where we asked them whether each EPA was necessary and whether they were comprehensible for workplace-based assessment. Participants were required to articulate their thoughts while reading the EPA framework. This enabled us to gain insights into their thought processes and perspectives [22].

Data collection took place from February 2022 until September 2022. For quantitative data analysis we calculated descriptive statistics of consensus rates using SPSS 27 (IBM SPSS Statistics 27). We analysed qualitative data from CCCs members using content analysis on Microsoft Excel. For analysing data from the interviews with the trainers and trainees, we first verbatim transcribed the interviews, and, then, analysed the data using thematic analysis in NVivo (QSR International) [23, 24]. Qualitative data were analysed by two researchers separately to achieve triangulation, while a third researcher was consulted, when discrepancies arose [25].

Reflexivity and research team

The research team was composed of members with different backgrounds. Two members had a background in education, while the other two members had a background in biomedical sciences and general practice. All authors had research training and experience in medical education research. Methodological and design decisions were in line with the available literature. We predefined methodological steps before commencing the study. To ensure adherence to our design stages, we maintained a detailed logbook to document systematically progression and modifications from our initial protocol. We regularly discussed the results to ensure that our interpretations were close to the data.

Results

In cycle 1, fourteen members of the CCCs gave feedback on the list of 60 EPAs. In the first feedback round, all members agreed that all 60 EPAs were required in the framework. Twenty comments were given regarding the formulation of the EPAs and 16 adaptations were made based on the new suggestions. Comments regarding the formulation were about the use of certain words in order to make the framework understandable. In the second feedback round, consensus was reached on the formulation of the EPAs (Table 1).

Table 1 List of EPAs for the Flemish GP Training
Full size table

In cycle 2, we interviewed two trainers and two trainees. CCC members, trainers, and trainees agreed that all EPAs should be included in the framework. From these interviews, we identified three themes. Table 2 presents these three themes alongside their subthemes. Necessity of EPAs was the first theme and included shared mindsets about necessity of EPAs in order to improve workplace-based assessment and difficulties with interpreting the CanMEDS roles.

Table 2 Themes from the interviews with trainers and trainees
Full size table

“The EPAs are better than the CanMEDS. My trainer and I often do not know what we have to assess…He (the trainer) sometimes gives the same feedback for multiple roles.” (trainee 1).

Second theme was about the relevance of EPAs to clinical practice. Users thought that the EPA framework could easily be linked to their clinical work, promoting assessment and feedback opportunities. They agreed that EPAs were understandable and formulated in intuitive language for clinical work.

“I think that it (the EPA framework) is quite intuitive. I can see a lot of links between the EPAs and my daily practice.” (trainer 2).

I like the (EPA) framework. My trainer and I already discuss some of these (activities) during our weekly feedback session. (trainee 2)

Third theme included challenges in implementation of EPAs, regarding the large number of EPAs, perception of high-stakes assessment within an e-portfolio, and limitations inherent to the current e-portfolio. First, users expressed their concern regarding the large number of EPAs. They indicated that only a limited number might be feasible because of time constraints in the clinical workplace. Also, users thought that due to the large number of EPAs, trainees would “pick and choose” EPAs where they had performed well. Along with limited functionalities of the current e-portfolio, they indicated that EPAs might be used as showcasing performance instead for workplace-based assessment and feedback purposes. Mainly trainees expressed hesitation to document EPAs where they would need further improvement. They perceived the e-portfolio as a tool more suitable for high-stakes assessments rather than for feedback purposes.

“The list (of EPAs) is quite extensive… I do want to have a nice portfolio, so for sure I will try to include as many as possible. In case something happens (in my curriculum), I want to show how well I have been performing.” (trainee 1).

“I normally do not include patient cases that went wrong in my portfolio. Because different people have access to it (the e-portfolio).” (trainee 2).

Discussion

The aim of this study was to design an EPA framework by actively engaging and collaborating with different stakeholders. To be established as a “good” assessment framework, EPAs should be acceptable by the different stakeholders involved in the assessment process, such as curriculum designers, trainees and trainers [26, 27]. Incorporating their opinions and understanding their different needs must be integral to the design process. However, literature regarding EPAs design has mainly focused on experts’ opinion, neglecting users in practice [8].

From our findings, it becomes apparent that direct involvement and communication among diverse stakeholders are crucial for designing a useful for everyone EPA assessment framework. When various groups are involved in developing educational interventions, competing needs can be optimally addressed [28]. This optimization fosters a cohesive approach, ensuring high applicability rates and effectiveness, when the EPA framework is used in practice. The need for users’ involvement in the development process is currently demonstrated in the most recent EPA literature [29, 30]. Users’ involvement promotes common language and expectations, enhancing the clarity and effectiveness of EPA interventions, and, most importantly, empowers the users themselves by acknowledging their perspectives [31]. Ultimately, trainees and trainers are the ones using the EPA assessment frameworks during daily clinical practice, and are potentially confronted with unforeseen obstacles.

Additionally, users’ involvement in the process can help to identify potential implementation challenges [32, 33]. Our findings indicate differences in opinions regarding implementation of EPAs. In contrast to the CCC members, users expressed their concerns about the large number of EPAs included in the framework. They were particularly concerned about how to use sufficiently and adequately EPA assessments, while juggling clinical work. This concern echoes findings from other studies as well, related to the assessment burden [34]. In particular, when challenges in assessment processes arise in the clinical workplace, assessment is most probably not performed as intended [35].

Furthermore, our results illustrate tensions between assessment of learning and assessment for learning. Although the EPA assessments aim to better prepare trainees for clinical practice, users suggested that the purpose of the EPAs might not be explicit for everyone. Since EPAs are a form of assessment, they could potentially lead to strategic behaviours of documenting successful EPAs, and, therefore, creating a fragmented idea about trainees’ performance in clinical practice. Additionally, the use of the current e-portfolio for high-stakes assessments only adds to this tension. Especially, trainees were not comfortable with sharing performance evidence for improvement, because they perceived the stakes as high [36]. The dilemma between learning versus performing has been the Achilles point in workplace-based assessment [37]. The lines between assessment and feedback seem to be also blurred in EPAs [38, 39].

Involving users during the design process can lead not only to early adaptations and refinement of EPAs, but also to better allocation of resources. In order to ensure successful implementation of EPAs, it is essential to recognize the central role of both trainers and trainees. Future research should focus on training programs designed to equip faculty, trainers, and trainees with a profound understanding of EPAs. Users in practice need rigorous training covering EPA principles, assessment techniques, and feedback strategies [40]. Moreover, fostering a culture of interdisciplinary collaboration among stakeholder groups is imperative. Encouraging review of assessment tools and facilitating the exchange of opinions during designprocesses can significantly enhance the overall quality of EPA frameworks, and, even more broadly, of workplace-based assessment practices.

Although EPAs are a valuable framework for assessing competencies in workplace settings, integrating other assessment tools is crucial to capture the full spectrum of skills needed to meet patient needs. Future research should focus on combining EPAs with other assessment methods, such as simulation-based assessments, either with standardized patients or with virtual reality, that would allow trainees to demonstrate their clinical and interpersonal skills within safe, controlled environments that closely replicate challenging patient scenarios [41]. Additionally, incorporating multisource feedback and continuous portfolio assessments could offer a comprehensive view of a trainee’s performance across various settings and interactions [42, 43]. Together, these methods would enhance the EPA framework, ensuring a comprehensive assessment of all essential competencies that future physicians should acquire.

Limitations

We need to acknowledge several limitations in this study. First, in medical education research, users’ involvement prerequisites a degree of experience with a specific subject. In our study, we involved users in the early design process of the EPA framework. Although we are aware of this limitation, we intentionally and consciously chose a participatory research design. We believe that users are experts in their own experience, and that they hold the knowledge and capabilities to be involved as partners in the development process. Second, our study involved a low number of users due to difficulties in recruitment. This might have led to recruiting participants who are fully engaged in the educational practices of the GP Training. Nevertheless, our findings are rooted in two methodologies, namely a modified Delphi method and semi-structured interviews. Therefore, we used triangulation to verify our results [25]. Finally, although workshops are mostly commonly in co-design studies [44], our study coincided with the last COVID-19 lockdown, necessitating adjustments. To cope with these challenges and uncertainties, we opted for methods that were the most feasible for our participants at that moment. Despite these challenges, the contributions from all stakeholders were invaluable, particularly in exploring potential implementation and evaluation issues.

Conclusion

For EPAs to be successful, they need to be acceptable as an assessment framework by different stakeholders’ groups. Accommodation of competing stakeholders’ needs during the design process is crucial for enhancing acceptability and effectiveness during implementation. Our findings highlight the significance of collaborative efforts to design EPAs, emphasizing its potential to empower users, identify implementation barriers, and pinpoint unintended consequences. Through this collaborative approach, wherein diverse stakeholders contribute their perspectives, we can create effective educational solutions to complex assessment challenges.