BACKGROUND

The past 20 years has witnessed an exponential increase in the number of clinical practice guidelines produced by various national and international organizations. Simultaneously, there has been increasing pressure on guideline-producing organizations to adhere to more rigorous guideline development approaches. Resulting from the evidence-based medicine movement in the 1990s, there have been major advances in guideline development as underscored by the introduction and widespread adoption of the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach which has become the most common framework used by guideline-producing organizations in the USA and internationally.1 These developments have been paralleled by increased public scrutiny and desire for greater transparency, as witnessed by the establishment of standards for trustworthy guidelines by the Academy of Medicine (formerly Institute of Medicine) and Guidelines International Network, as well as the indexing of guidelines that met such standards in the National Guideline Clearing House (NGC).2, 3

The requirement for greater methodological rigor has increased the pressure on organizations invested in guideline development to include individuals with specific expertise in health research methods, clinical trial design, epidemiology, and evidence synthesis. Recent reports have highlighted the shortcomings of guidelines developed by specialty professional organizations and provide further rationale for strengthening the leadership role of methodologists within guideline panels.4, 5 Research has demonstrated that guideline development groups that include a methodologist are associated with a decreased rate of recommendations labeled as “expert opinion” and more transparent management of potential conflicts of interest.6, 7

Norris et al. have previously outlined four main tasks related to guideline development and provide a set of minimum skills and experience to perform these tasks which include development of key questions, assessment of certainty of effect estimates, development of recommendations, and teaching GRADE.8 Furthermore, other resources are available such as the Journal of Clinical Epidemiology series, which explains the GRADE approach, and the GIN-McMaster Guideline Development Checklist, which outlines the various steps in guideline development.9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28 However, the specific skillset and expertise needed by different members of a guideline development team have not been well defined and extend beyond learning resources. Furthermore, greater detail around the process of skill acquisition and an understanding of what expected behaviors a guideline methodologist must demonstrate to fulfill this role have also not been clearly delineated. To meet this unfulfilled need, we propose a theoretical framework of competency-based learning for individuals involved in guideline development.

OBJECTIVE

The primary objective of this manuscript is to outline and describe a theoretical framework of competencies and educational milestones that illustrate the requisite knowledge and skills that an individual should seek to acquire as part of a guideline development training program. We propose that such a framework will help (1) standardize the qualifications needed for individuals to serve on guideline panels, help lead guidelines, or serve as guideline methodologists; and (2) develop curricula for teaching and training of guideline panel members. This framework can also help enable organizations to identify guideline developers with the relevant and appropriate level of knowledge and skills.

METHODS

Competency-Based Medical Education Framework

We used a competency-based medical education (CBME) framework, commonly used in the U.S. medical education and graduate medical education system, to define specific competencies and milestones. Competency-based medical education is defined as “an outcomes-based approach to the design, implementation, assessment, and evaluation of a medical education program using an organizing framework of competencies” with competencies being defined as the essential minimal set of a combination of attributes, such as applied knowledge, skills, and attitudes, that enable an individual to perform a set of tasks to an appropriate standard efficiently and effectively.29, 30 Educational milestones are defined under the umbrella of each competency and provide narrative descriptors of the competencies and subcompetencies along a developmental continuum. In graduate medical education (GME), milestones, defined as “developmentally based achievements that residents are expected to demonstrate at established intervals as they progress through training,” are reported to the Accreditation Council for Graduate Medical Education (ACGME) and used to determine whether a trainee is meeting the requisite requirements to be deemed competent.31,32,33,34

Dreyfus Model

In accordance with the ACGME, we relied on the Dreyfus model to attribute competencies related to guideline methodology training. Originally developed by Hubert and Stuart Dreyfus who were commissioned by the U.S. Air Force to describe the development of the knowledge and skills of a pilot, this model consists of five stages: (1) novice, (2) advanced beginner, (3) competent, (4) proficient, and (5) expert.30, 35 They later identified a similar process of development in the chess player, the adult learning a second language, the adult learning to drive an automobile, and many others.35 When adapted to medicine, these levels outline progressive stages of educational growth and development through a dedicated educational curriculum with higher stages representing levels at which the learner has demonstrated mastery (Table 1).

Table 1 Example of Milestones for Levels 1–5
Full size table

Applying the CBME Framework and Dreyfus Model to Guideline Development

Six faculty from the U.S. GRADE Network (USGN; MHM, PD, RAM, RLM, SS, YFY) with extensive expertise as guideline methodologists helped develop the framework and competencies. The USGN is one of 16 established GRADE Working Group Networks and Centers around the world. It is the main organization in the USA focused on teaching guideline development using the GRADE approach through guideline development and systematic review workshops. Over the last 5 years, the USGN faculty has taught > 20 guideline development workshops attended by over 400 participants with a wide range of specialties (clinical, public health, environmental) and backgrounds (e.g., academic, health systems, policy) from a variety of professional societies and governmental entities.

An initial face-to-face meeting was held in 2017 and was attended by USGN members and two methodologist trainees (who have attended several training workshops). After consensus was reached to use a CBME model, two of the USGN members (SS and RAM) derived an initial set of competencies based on a review of GRADE guidance manuscripts developed by the GRADE Working Group (published in the Journal of Clinical Epidemiology), the GIN-McMaster Guideline Development Checklist, and the work by Norris et al..8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28 Subsequently, the group participated in focused discussions via teleconferences, and using an iterative process, the group developed a consensus-based set of proposed competencies, subcompetencies, and milestones.

RESULTS

Using the CBME model, we defined three core competencies: (1) facilitate the development of guideline structure and setup, (2) make judgments about the quality or certainty of the evidence, and (3) transform evidence to a recommendation. The competencies focused on standards necessary for ensuring the development of trustworthy guidelines, for example explaining the importance of multistakeholder input or developing clear and actionable recommendations. These competencies are further divided into 12 subcompetencies with proposed milestones (expectations of knowledge and skills), for each subcompetency (see Table 2).

Table 2 Preliminary Framework of Competencies for Guideline Panel Members
Full size table

Different Levels and Types of Competency May Be Necessary Depending on One’s Role in the Guideline Development Process

Acknowledging the different roles within a guideline panel, including that of the chair, methodologist, or panel members, some competencies may be more relevant and pertinent to fulfilling the expected role. Panel members may need to be most familiar with competencies related to transforming evidence to recommendations and limited competencies focused on making judgments about the certainty of evidence, whereas guideline methodologists may need to demonstrate most, if not all, of the core competencies. Guideline panel chair roles and expectations may vary depending on the organization’s guideline development process and may span across all three domains or may include a very small subset of competencies. As such, this framework has tried to capture as many of the potential behaviors and skills that different panel members may need, recognizing that different organizations may have variations in their guideline development processes with predefined roles and expectations that may be unique to their organization.

Not only will the set of competencies vary depending on the individuals’ role but the level of achievement may also be variable. To progress along the development continuum, individuals may need to attend workshops with dedicated training in guideline development, engage in self-directed learning activities, and/or demonstrate involvement in guideline efforts. At a given level, the expectation is that the learner substantially demonstrates that milestone as well as those in lower levels (if applicable). Level 1 represents the entry level characterized by expectations that focus on recognizing and acknowledging the importance of a specific skill or behavior relevant to the guideline development process. All panel members regardless of their role should reach level 1. We expect that the majority of learners who attend dedicated guideline development workshops or training programs are at level 1. The expectations for levels 2 and 3 require individuals to demonstrate progressive acquisition of knowledge and application of this knowledge. Individuals may demonstrate this behavior by their involvement in the development of “real-life” clinical practice or public health guidelines. Guideline panel members may only need to reach level 2 or perhaps level 3 for select competencies, whereas a guideline methodologist may be expected to demonstrate level 3 or 4 competency for most of the applicable competencies. For example, for the subcompetency of “formulate a healthcare question,” a guideline panel member who is the clinical or content expert may only need to demonstrate level 1 mastery (can explain how a standardized format can be used to structure healthcare questions in a guideline) or level 2 mastery (can explicitly describe the population, the interventions, and the comparators for each guideline question; can identify the outcomes (both benefits and harms); and rate the relative importance of the outcomes), whereas a guideline panel member serving as the methodologist may need to demonstrate mastery up to level 4 (can lead a guideline group to develop structured PICO questions with explicit considerations of population, intervention, comparator, and outcomes and in the selection and rating of outcomes) (Table 1). For guideline methodologists, progression to independent functioning (level 4) may only occur after working under, or alongside, a more experienced methodologist who can directly observe and account for the completed guideline product. Level 5 is the mastery or aspirational level and is applicable to a small subset of individuals who have advanced their own skills through their work on the development of a number of clinical practice guidelines across a spectrum of topics, are active in educational teaching activities or training workshops at a national or international level, or are dedicated to advancing the field through methodological and scholarly work in the practice of guideline development. Individuals at this level are expected to be master educators or mentors who can educate other learners who are at the beginning stages of training. In Figure 1, we outline the knowledge, skills, and expertise necessary for individuals to progress along a spectrum of performance levels.

Figure 1
figure 1

Progression through core competencies based on the Dreyfus model.

Full size image

This framework may be useful to various stakeholders including guideline panel members (by providing explicit expectations about knowledge and skills), organizations involved in guideline training (by facilitating training activities, curriculum development, and tools to competencies), and finally guideline-producing organizations (particularly those seeking guideline methodologists with the requisite relevant knowledge and skills to oversee guideline development) (Table 3).

Table 3 Value of Defining Competencies and Milestones for Guideline Panel Members
Full size table

DISCUSSION

We propose an initial theoretical framework for the development of a competency-based curriculum for learning and teaching guideline development. Competencies offer a common shared language for defining and specifying observable abilities, which can be expressed as actual behavior and can be assessed to ensure that learners have acquired them successfully.30 Implementation of competency-based education involves the identification of competencies, designing of curriculum and teaching programs that are mapped to specific competencies, and developing assessment tools that provide a valid and reliable evaluation of the outlined competencies.

As the majority of guideline methodology training is focused around attendance of guideline methodology workshops, development of competencies and operationalization of milestones can be used to structure learning, training, and assessment of guideline training activities. Although this framework describes the skills necessary for guideline development, every learner may not need to master every skill, behavior, or educational outcome. Depending on the role that an individual aims to take in the guideline development process, different levels of competency will be required. While we expect measurable progression of learners though the competency framework, we recognize that learners may advance at varying speeds or demonstrate competency levels that fall across a spectrum within each competency. Norris and colleagues have suggested three basic approaches to determine methodologist expertise level8: first, self-report of knowledge, skills, and experience using a standardized “guideline curriculum vitae (CV)”; second, demonstration of skills using worked examples; and third, a formal evaluation using a written or oral test. Additional assessment tools such as surveys or direct observation of skills within structured tutor trainee models of learning is likely to provide additional information.

Strengths and Limitations

To our knowledge, this is the first proposed framework for defining a set of competencies and milestones specific for guideline development akin to the five levels of milestones used in medical education.34 We view publication of this initial framework as a critical step toward establishing foundational competencies for various individuals involved in guideline development. Important limitations of this proposed framework are the lack of broad-based validation by a large and international group of established guideline methodologists and that the use of a competency-based framework may not be ideal for organizations that are not familiar with this educational framework. This framework was developed by members of the U.S. GRADE Network and relies on the GRADE approach for evaluating the certainty of evidence and strength of recommendations but also includes competencies that are considered standards for rigorous guideline development in general (irrespective of framework). For example, guideline setup, conflict of interest management, rating certainty in evidence, and developing clear and actionable recommendations are all critical steps in any guideline approach. We also acknowledge that the specific competencies for each of the panel members in their various roles have not been explicitly defined in this preliminary framework, but our goal was not to be overly prescriptive or impose this framework on organizations that may have already set processes and expectations for guideline panel members. However, we hope to conduct future studies to prioritize and build on this framework using the cumulative experience of national and international guideline developers, methodologists and learners and work toward synergy in establishing standards for competencies associated with training in guideline development. We envision additional studies to validate the competencies and gain consensus on the final working and operationalization of milestones. We will also seek to further refine the milestones and outline which set of competencies and what level of mastery should be expected from panel members in different roles. Future research may demonstrate that the initial set of competencies may be too specific or broad or that the described behaviors or knowledge outcomes may not be reflective of the specified competency. Despite this limitation, this framework can support capacity building by providing a set of skills and behaviors to structure a future curriculum or guide the educational experience of individuals involved in guideline development.

Future Directions

Future research efforts should focus on validation, evaluation, and further refinement of this framework. We encourage educators and other stakeholders to pilot test this framework to assess usability, ensure appropriate definitions of educational outcomes and behaviors, and improve the comprehensiveness of this framework. We also encourage the development of validated instruments to track the progress of learners and to assess if an individual has achieved educational milestones or mastered a competency. Additionally, future studies should be carried out to determine the minimal number of guidelines necessary to accomplish certain levels of progression using specific competencies, although the degree of guideline complexity (e.g., including diagnostic strategies and other additional guidance topics) will likely have an impact as well.

In conclusion, we propose a theoretical framework for a competency-based training program for individuals serving on guideline panels in various capacities including clinical or content experts, guideline methodologists, or chairs; this will help standardize and ensure adequate training by organizations that offer educational workshops, allow for a formal system to measure mastery of the competencies, and help guideline-producing organizations recognize the skill set needed to identify individuals with expertise to assist with guideline development in their respective fields.