Background

Roemer [1] defined a health system as the combination of resources, organizations, and financing and management bodies that culminate in the delivery of health services to a population. A redefinition of the term health system was proposed by the World Health Organization [2] to consist of all organizations, people, actions, and activities whose primary purpose is to promote, restore, and maintain health. This was further modified—to be more comprehensive and explicit—by a group convened by the World Health Organization’s (WHO) Alliance for Health Policy and Systems Research, to refer to delivery and financial and governance arrangements for health care and population health services and the broader context in which they are negotiated, implemented, and reformed [3].

Health systems guidance (HSG) are systematically developed statements produced at global, national, and regional levels to assist with decisions about appropriate options for addressing health systems challenges (including related changes in health systems arrangements), the implementation of these options, and the monitoring and evaluation of the implementation efforts [4]. The need to develop and use evidence-informed approaches to address national and global health system issues through the use of HSG has been articulated by many [5]. For example, the Millennium Development Goals (MDGs) reflect eight international goals (e.g., reduce child mortality, improve maternal health, and combat HIV/AIDS, malaria, and other diseases) and were developed by the United Nations in 2000. However, their attainment has been hindered by weak health systems and lack of system-specific recommendations [6, 7]. Formulating recommendations to address root causes has the potential to clear the path toward better health outcomes in line with health goals [8].

The terms guidelines and guidance both refer to any document containing a recommendation on a course of action. However, in contrast to the word “guidelines,” the term “guidance” is used in the health system context in order to make explicit the difference between the process of supporting evidence-informed judgments for health system issues from that of clinical judgments as would be expected with clinical practice guidelines (CPGs). As a policy-oriented product, HSG represents the whole body of knowledge that informs policy decisions on how health system issues should be classified or prioritized, appropriate health system governance, optimal financial arrangements, system organization, and the design and delivery of effective health programs and services [9–12].

The main bodies tasked with developing HSG are international and intergovernmental organizations (e.g., WHO, Pan American Health Organization), local ministries of health, and special national committees or agencies providing support to ministries of health. Their production is usually linked with these high-level health sector entities and less frequently in decentralized structures at the sub-national level. HSG may be aligned with specific goal-oriented taxonomies (e.g., WHO building blocks), unique national/regional governance, or tailored to financial/delivery arrangements with the common goal to optimize health care systems. However, while these organizations have demonstrated an interest in the development, appraisal, and reporting of this type of guidance as described in the WHO’s Handbook for Supporting the Development of Health Systems Guidance [13], for example, many of their members and collaborators (who are also members of the Appraisal of Guidelines for Research and Evaluation for Health Systems (AGREE-HS) team) contend that these intensions have not been matched by adequate action or results, in part, due to the lack of experience in developing HSG.

Indeed, in comparison to CPGs, the development of HSG pose different conceptual and methodological challenges related to the varied types of evidence that are relevant, valued, and considered; the complexity of health systems’ internal and external relations that must be addressed; and the pre-eminence of contextual factors that directly influence the design and adoptability of recommendations [4]. As with CPGs, HSG statements or recommendations should be justified by assessments of the quality of evidence supporting them, the potential for unintended consequences, and by discussions of implementation and contextual issues. For example, the full implementation of HSG recommendations can be further hampered by “bottlenecks” like health system fragmentation and capacity limitations; these limitations should also be addressed in the guidance documents [14, 15].

The potential for positive impacts by HSG as a decision tool to improve health systems is great. However, as with CPGs [16], this potential is only as good as the quality of the HSG. Indeed, the ability to impact and optimize health system performance and efficiency through the development and adoption of HSG is hampered by the dearth of tools to guide their development, appraisal, and reporting. As a consequence, this leaves knowledge users at a loss when choosing the highest quality and/or most appropriate guidance or in creating new guidance in circumstances where there is none or navigating circumstances where the existing guidance is not credible or of poor quality. In contrast to the development, appraisal, and reporting methods for CPGs, the development of HSG is still at a rudimentary stage [4]. There has been some work related to the use of “evidence briefs” to assist policymakers and stakeholders with working through a health system problem, options for addressing it, and key implementation considerations, informed by the best available data and research evidence [17]. However, there is a need for systematically and transparently developed guidance that can feed into such context-specific documents. With the rising trends that encourage bridging the gap between research and policy and practice, this is a significant research gap in both the science and practice of knowledge translation. HSG can provide this bridge between research synthesis and policy needs for evidence.

The creation of high-quality HSG requires tools to support the development and reporting of high-quality guidance and tools able to differentiate between high- and low-quality reports. There is also value in establishing acceptable quality thresholds and creating common methodologies and nomenclature among the HSG community (developers, users, and researchers). At present, however, there is no universally acceptable gold standard approach for appraising HSG, although there are some tools (for example the Handbook for Supporting the Development of Health System Guidance) to support their development [13] and reporting. To address these gaps, we are conducting a program of research with the international health systems guidance community that aims to create a tool to support the appraisal of HSG and further enhance HSG development and reporting.

The first step to this program of research, and the focus of this paper, was to conduct a knowledge synthesis of the published and grey literatures to determine concepts (items, criteria, or domains) related to HSG development, reporting, and quality. The results of the synthesis are to provide the foundational components of our HSG tool and to serve as a conceptual status report for the research community.

Methods

With the goal of generating a list of concepts that can be used to develop, appraise, and report HSG, we conducted an initial search to identify existing concepts, tools, templates, or checklists that have been used or could be used to describe, differentiate, or appraise the quality of HSG. The aim of this initial search was to determine if reviews on this topic exist, estimate an initial number of relevant papers available, highlight additional and useful search terms, and clarify inclusion and exclusion criteria and the most appropriate knowledge synthesis approach [18, 19].

After contemplating the variety of knowledge synthesis approaches available, a critical interpretive synthesis (CIS) approach was considered the most appropriate for three reasons. First, CIS is a systematic approach that facilitates the analysis of complex and diverse bodies of literature including qualitative, quantitative, and theoretical papers [20–26]. The available literature on HSG is highly heterogeneous and methodologically diverse, comprising of a mix of empirical qualitative and quantitative papers and non-empirical papers. A wider range of evidence and study designs are typical occurrences in health systems/health policy research and HSG is a nascent domain where good thinking is likely to be captured in expert opinion/views, editorial comment, policy documents, political statements, experiences of stakeholders, theoretical/discussion papers, and other colloquial forms of evidence. In contrast to CIS, conventional systematic reviews, for example, have been criticized for excluding forms of evidence traditionally considered as non-experimental [20, 27–30].

Second, the objective of a CIS is to develop new concepts and theories through a typically interpretive mode of inquiry. This is in contrast to more conventional systematic review approaches where the mode of inquiry is more aggregative and aimed at testing theories by collating, compiling, pooling, and summarizing common outcomes across a range of studies [19–21, 25–27, 31, 32]. Since our review aims at generating a candidate list of items, criteria, or domains for HSG development, appraisal, and reporting, this theory-generating approach that CIS promotes was deemed a good fit.

Third, CIS offers a more flexible, iterative, dynamic, and reflective approach requiring investigators to assess the extent to which new information or data are provided with each additional paper considered. It applies a relatively loosely defined set of processes for critically analyzing and synthesizing literature [19, 20, 23, 27, 28, 30]. This differs from the more conventional systematic review methods which have clearly stated study protocols, an exhaustive search of all available literature, standardized data extraction templates, and explicit quality appraisal checklists, as well as pre-determined focused questions, strict inclusion and exclusion criteria, and specified data boundaries [20, 23, 27–30].

With the unique conceptual and methodological challenges related to the development of HSG [4], it was essential to have a review method that was iterative, flexible, and dynamic. Our review was not simply aimed at summarizing this complex literature on HSG, so we sought to produce a logical and insightful interpretation of a purposefully sampled body of evidence. As such, the aim was to yield a comprehensive interpretation without requiring exhaustive identification of relevant items of literature as in a conventional systematic review [20, 29, 30].

Review question(s)

As per CIS methodological standards, our review questions served as compasses rather than anchors [20, 29, 33] allowing for the concepts of HSG to be derived from synthesis of the literature and constantly modifying them in an iterative manner throughout the review.

Our guiding review questions were as follows:

  1. 1.

    How have authors variously defined HSG quality in the literature?

  2. 2.

    How have authors interpreted and used criteria (or tools or instruments, checklists, systems, etc.): to describe or define HSG quality or reporting requirements, to appraise HSG quality, or to differentiate between HSG on the basis of quality?

  3. 3.

    What methods have authors and stakeholders used to develop these criteria?

  4. 4.

    What methods have authors and stakeholders used to address health system issues/challenges?

Literature search

In searching the literature, our goal was to select papers based on their potential relevance to HSG appraisal and quality, while including other papers that, though not directly relevant to health systems, were deemed important for the purpose of our review. Instead of including an exhaustive number of papers, our plan was to provide a comprehensive sampling frame of potentially relevant papers using emergent eligibility criteria [25, 32]. As per the standards of a CIS, the boundaries of our inclusion and exclusion criteria were modifiable, dynamic, and continuously shifting [20, 32, 34]. The goal was to populate the concept of HSG with new concepts rather than finding papers that reiterated ideas already captured in previously reviewed papers.

Eligibility criteria

  1. 1.

    Study content: (a) Papers that evaluated HSG or papers that report on criteria/tools that have been documented as important indicators of HSG quality and (b) papers that reported on methods for addressing health system issues/challenges.

  2. 2.

    Time frame: Papers are eligible if published in or after the year 2000 (when the first World Health Report on health systems [2] was published).

  3. 3.

    Context: Unrestricted. We sought papers that considered HSG evaluation and health system issues/challenges in various contexts (low-, middle-, and high-income countries).

  4. 4.

    Study design: Any study design.

  5. 5.

    Language: English, French, and Spanish

A combination of key and free text terms were used to search through various databases. Terms used were health systems, health policy, guidelines, guidance, health services arrangement, health services organization, health system issues, health system challenges, tools, instruments, criteria, items, domains, evaluation, appraisal, quality, and standards.

The search strategy included various electronic databases and sources: Cumulative Index to Nursing and Allied Health Literature (CINAHL), Cochrane Library, Excerpta Medica dataBASE (EMBASE), Google Scholar, Health Systems Evidence, Latin American and Caribbean Health Sciences Literature (LILACS), PubMed, Virtual Health Library, and Web of Science. We also searched subject-specific as well as regional electronic sources: Australia’s National Health and Medical Research Council, Evidence Best Practices for Public Health, University of Massachusetts National Guidelines Clearinghouses, National Institute of Health and Care Excellence (NICE), WHO’s Evidence Informed Policy Network (EVIPNet), WHO EURO’s Health Evidence Network, and Guidelines International Network (GIN) directories.

We also searched for other grey literature, including conference abstracts (Global Symposium on Health Systems Research, Canadian Association for Health Services and Policy Research, Health Systems and Process Improvement Conference, Canada’s Health Leadership Conference, Health Systems Reform in Asia Conference, International Society on Priorities in Health Care). Further, research reports, book chapters, unpublished data, dissertations, and policy documents that were nominated by members of the team or found in unique holdings of health sciences libraries in Canada were also included. Additional papers were identified by manually searching bibliographies, while more were obtained by hand searching some key journals (i.e., Health Policy and Planning, Health Services Research and Policy, Health Research Policy and Systems, Global Healthcare Systems, Health Systems and Reform, and Health Policy). This was a complementary search strategy to account for papers not included in electronic databases or with search terms that do not allow them to be easily identified [34, 35].

Finally, we also contacted (through emails, phone, Skype or in person) experts, colleagues, and members of our research team with a known interest on this topic to identify additional papers (published, unpublished, or ongoing). To increase the scope of these key informants, we also asked initial contacts to refer us to others who can provide more information. Relevant papers were then imported to Endnote bibliographic software.

A two-step sampling process was used. First, a purposeful sampling approach [36, 37] was used to investigate the literature to determine the range of unique candidate concepts—or theoretical domains—associated with HSG. We stopped sampling at the saturation point where looking at new literature no longer contributed additional concepts [25, 37]. Second, a theoretical sampling strategy was used to interrogate the literature relevant to each of the identified concepts. Theoretical sampling does not occur at a single point in the research process but is a recurrent feature, which aligns it well with the dynamic CIS methodology. Similar to the first approach, we stopped sampling at the saturation point where looking at new literature no longer contributed additional descriptions of the identified concepts. The intent here was to thoroughly capture the depth of the concept across the literature in order to generate and develop the theoretical underpinnings of the ideas rather than collecting numerous citations of identical concepts and/or descriptions. Screening titles, abstracts, and full text was completed independently and in duplicates (DAA and SA). Disagreements were resolved by consensus.

Quality appraisal

We did not consider it appropriate to undertake quality appraisal for three principal reasons: firstly, the literature on HSG is diverse and complex; secondly, the available literature could not be hierarchically ordered in terms of study design, importance, or relevance; and thirdly, very little consensus exists on how or whether to perform quality appraisal in a CIS methodology [27, 32, 38, 39]. Instead, we chose to evaluate papers for inclusion based on a judgment of their relevance and likely contribution to concept development and theory [23, 26, 32, 40]. Our assumption was that some methodologically weak papers are theoretically and conceptually pertinent [25, 29, 34]. The goal of our review was to generate key concepts relevant to the appraisal of HSG, so we applied a flexible relevance boundary in order to be as inclusive and comprehensive as possible and include papers that could contribute to this theory generation.

Data extraction and analysis

As a result of the nature of the data, we did not use a standardized data extraction template to retrieve the relevant elements of this review from all the retained papers. Instead, and as per CIS methodology, we gave latitude for a more narrative data retrieval approach. It was, however, possible to extract some key elements from all the papers including author(s), the year of publication, geographical location of the study or affiliation of the author, purpose of the study/paper, its relevance and a summary of its main conceptual contributions, and types of items, criteria, or domains considered. Data were extracted independently and in duplicate (DAA and SA) with disagreements resolved via consensus. The extracted data were then compiled in summary form.

Again due to the anticipated diversity of methodological designs and implementation of the different items, criteria, and domains that were extracted from the papers, a narrative approach to data synthesis was executed. The data extracted were sorted and categorized into groups with themes. We sought to collect similar concepts within the same group. This systematic method of recording themes, and making connections between themes and the data collected within a comprehensive category system [41, 42], is an advocated concept-development coding approach [43]. The information was imported into the analysis software NVivo version 9 in order to analyze groups of themes that depicted similar patterns in information. We did not organize the concepts in any hierarchical order but reported the concepts that were identified in each of the papers in this review.

Modeling from the health policy analysis triangle framework that was developed by Walt and Gilson [44], we were able to connect the concepts (constructs) together into meaningful domains. The framework considers all the essential elements that interact to shape policymaking by demonstrating that health policy should focus on the processes contingent on developing and implementing change, on the content of health policy reform, and on the context within which the policy is promulgated, as well as on the actors involved in the policy reform [44, 45]. The health policy analysis triangle framework is a highly simplified model of an extremely complex set of interrelationships between the different elements of the model (process, content, context, and actors) with each element influencing or being influenced by the other [45, 46]. Development, appraisal, and reporting of HSG play an important role in health policy making by providing options and recommendations to address a health systems issue. Therefore, similar to health policymaking, HSG development, appraisal, and reporting can be seen to occur in a series of discrete yet interconnected components of process, content, and context.

Results and discussion

No existing knowledge synthesis was found on the topic, and no existing HSG appraisal tool (draft or final version) was identified. We identified a total of 43 papers that met our eligibility criteria and reported on concepts (items, criteria, domains) considered directly or conceptually relevant to HSG and/or their quality (see Fig. 1 for a flowchart of the study selection).

Fig. 1
figure 1

Flowchart of study selection

Full size image

Forty-seven percent of the retained studies were technical reports, 32 % were concept papers, 13 % were quantitative studies, and 8 % used a mixed methods approach. The authors held affiliations at the following organizations: 33 % at universities, 25 % at the WHO, 23 % at research institutes, 12 % at government agencies (United Kingdom’s National Health Services [NHS], United States’ Centre for Disease Control [CDC], United States Agency for International Development [USAID], United Kingdom’s National Institute for Health and Care Excellence [NICE]), and 7 % at national ministries of health. Fifty-eight percent (58 %) of the papers were lead authored by an individual based in Europe, 30 % in North America, 8 % in Australia, 2 % in Asia, and 2 % in Africa. Over one third of the authors implicitly or explicitly identified the need for a high-quality tool aimed to systematically evaluate HSG and contribute to development and reporting of HSG. Thirty concepts were identified that are considered to be a good fit for, and may be relevant to, the development, appraisal, and reporting of HSG. The Appendix provides a description of the papers, their objectives, relevance to the HSG process, and the concepts extracted.

Process principles, content, and context principles

Ostrom [47] stated that identifying key concepts and relationships among them culminates in the generation of a theory, which connects these key concepts. We were able to organize the concepts and identify relationships within and across them. Through an iterative process, we clustered the concepts together into three meaningful domains: process principles, content, and context principles.

Process principles

Process principles represent the methodological elements and the defining principles that demonstrate the development integrity of the HSG. They refer to the ways in which the HSG is initiated, developed, and formulated and can be looked upon as the “who” and “how” of the guidance [44, 46]. Here, we find the procedures and principles that were employed by the HSG developers in coming up with the guidance recommendations. Process principles are critical in the development of HSG because the methodological strategy is crucial for guidance appraisal and for differentiating across HSG of varying quality. They are also important because they can articulate the tactics that could be utilized in the development and reporting of HSG to optimize quality. Process principles also depict the subjective and/or objective belief systems in place that represent the preferences (what ought to be) of individuals, groups, or populations on the course(s) of action for addressing a health systems challenge. Process principles also refer to how the guidance is made public in a way that is consistent and comparative as this can facilitate their comprehension, which will further enhance their uptake and aid in easing their application. Therefore, when developing, appraising, or reporting HSG, a conceptual understanding of the process is fundamental. Table 1 shows concepts for HSG process principles.

Table 1 Process principles
Full size table

We were able to point out patterns and draw linkages between the process principles concepts. We offer five examples here. First, guidance should be relevant to and developed for health system areas with a clearly demonstrated and documented need, and the feedback of appropriate stakeholders will further highlight priority areas, which may potentially lead to timely interventions. Second, systematic and transparent approaches have to be applied to search for and identify relevant evidence, which should also be available in a timely manner. Third, ensuring that stakeholders from all applicable health system and sub-system levels are involved in the HSG process, that relevant evidence is sought and that appropriate outcomes are chosen can enhance the comprehensiveness of the guidance recommendations. Fourth, having clearly defined and consistent outcomes and declaring and managing interests as well as engaging in a participatory approach that incorporates the various perspectives of multiple stakeholders is also important for transparency of HSG. Finally, the ethical lens applied will be impacted by the quality of the evidence available, the composition of the involved stakeholders, the outcomes/indicators selected, the health system levels/sector involved, and whether it is a priority area that is relevant to the setting.

Content

Content (the “what” of the HSG) represents the topics, subjects, and substance that demonstrate the content integrity of the guidance recommendations [44, 46]. Here, we find statements about the impetus of the endeavor that provides direction on the HSG objectives and goals. Keeping in mind that in the absence of an appropriate economic lens, even the best-designed HSG may not achieve adequate success; we also find here the myriad of economic factors that come with the HSG process and need to be taken into account. In addition, within the content of the HSG, it is also important to consider and reflect on the fact that not all guidance recommendations go into practice as planned and events may occur that may derail intended actions. We also find here assessment considerations, usually ongoing throughout the HSG process, and which refer to those elements that assist in determining whether the guidance process was properly followed and/or records the impact/outcomes of the HSG. The content of the HSG also articulates the operationalization of the proposed recommendations and carries information on how to best reach the target users of the guidance. Table 2 shows concepts for HSG content.

Table 2 Content
Full size table

As with the process principles, we were also able to point out patterns and draw linkages between the content concepts. For example, the way the problem is defined can provide direction on the costs and resources required for implementation of the solution, reveal some potential unintended consequences, and guide the operational plan. The operational plan can also provide hints on some potential unintended consequences. A clear problem definition will also inform the process and outcomes/impact evaluation of the HSG. Designing the operational options can inform the costs and resources required. The costs, resources, and operational plan will in turn influence whether the HSG will be effective and/or cost-effective. The updating plan is also contingent on the health system issue being addressed, the effectiveness of the guidance recommendation, and the resources required.

Context principles

Context principles represent those systemic factors like the local technical, situational, structural, institutional political, and socio-cultural components of the health policy environment that can impact HSG recommendations [44, 46]. Like health policymaking, HSG does not occur in a vacuum so it is important to pay attention to the variety of contextual factors that may have significant impact on how the guidance is developed, adapted, and implemented by the end users [48, 49]. The process principles and the content of the HSG all have to be contextualized; therefore, context principles impact these other clusters of the concepts. Context principles refer to the usability in context, and here, we find concepts related to those system components that make up the setting within which the HSG is to be used. A clear overview of the context or setting that will be impacted by the guidance recommendations is essential. This will provide an understanding of why some guidance recommendations may work in some settings and not others. We find here factors that will enhance and facilitate the adherence to HSG recommendations as per protocol. We also find here factors that represent the values and moral fabric of the society and how this can either facilitate or impede the HSG recommendations. Table 3 shows concepts for HSG context principles.

Table 3 Context principles
Full size table

We were able to point out patterns and draw linkages between the context principle concepts as well. We offer some examples here. For HSG to be feasible and sustainable, it should be affordable and resonate with local values (political and socio-cultural). Information on affordability, feasibility, and sustainability may also determine whether the guidance recommendations can be transferable to other settings. Providing socio-culturally appropriate recommendations may also indicate which other health systems can adapt and use the HSG. Also, the HSG should be flexible enough to accommodate shifts in values (e.g., political, socio-cultural). Additionally, HSG that reflects the socio-cultural preferences of the target population, that is affordable, feasible, sustainable, and transferable to comparable settings, can amass support from politicians. The external factors that originate from other institutional systems may impact the feasibility and sustainability of the guidance and may be influenced by the political and socio-cultural climate.

Synthesis across domains

As is expected of a Critical Interpretive Synthesis (CIS), we were able to point out relationships between the concepts across the three clusters as shown in Fig. 2. We highlight some relationships:

Fig. 2
figure 2

Framework of health systems guidance concepts

Full size image

  • ➢ Uptake of guidance can be enhanced if it is addressing a priority area for which evidence-based reports of effectiveness and cost-effectiveness exist.

  • ➢ Consultations with appropriate stakeholders are also crucial for feasibility of implementation and sustainability of the HSG because sometimes, the individuals tasked with implementing guidance recommendations may not be committed to them and this may influence adherence. Their input into the HSG process and support for the recommendations may alleviate this concern.

  • ➢ HSG outcome(s) chosen will influence the way the problem is defined and provides information that will be useful for evaluating the HSG.

  • ➢ Including an ethical viewpoint will also impact how the problem is defined and will influence cost-effectiveness thresholds.

  • ➢ Systematic and transparent processes may provide an impetus for donor involvement (affordability).

  • ➢ Information on affordability can determine whether the health system issues can be addressed in a timely manner and inform judgments on how to evaluate the process.

  • ➢ The HSG should be flexible enough to accommodate constantly evolving evidence and changing health system priorities.

  • ➢ The guidance should provide socio-culturally appropriate solutions that are relevant to the applicable levels/sectors of the health system.

  • ➢ HSG that is in line with ethical principles, addresses priority issues, and is timely will be appealing to politicians.

  • ➢ The external alignment of guidance may affect the effectiveness and impact the operational considerations.

  • ➢ The outcomes selected, the system level/sectors involved, or the ethical values in place may expose the external factors that may be pertinent.

  • ➢ Some determinants of generalizability of the guidance are the stakeholders involved, the evidence used, transparency of the process, ethical lens considered, and outcome/indicators selected

Conclusions

This project is the first phase of a multistage approach to create an internationally useful HSG tool, AGREE for Health Systems (AGREE-HS) that will inform the development, reporting, and appraisal of HSG. Modeling after the paradigm to create a tool for clinical practice guidelines [50], our first step was to conduct a review of the published and grey literatures to identify concepts related to HSG quality. In this vein, it was our expectation that the receptiveness, adoption, and diffusion of HSG recommendations depend on the perception of their quality, and with this study, we aimed to identify those core components of good quality HSG.

We found a total of 30 potential HSG appraisal concepts that have the capacity to discriminate between high- and low-quality guidance and direct their development and reporting. We found no existing tools to support HSG appraisal and found few studies describing concepts that were directly tested to appraise HSG. Indeed, the papers we examined reflected a variety of study designs and goals; none reflected methods used to develop appraisal methods. However, the data from our studies show a convergence of ideas in the HSG research community about what constitutes good and useable HSG. Together, these data can provide the foundation of a tool that guides HSG developers in the types of information important to report in a HSG document and tactics for optimal execution. These concepts also address issues of appropriateness and completeness as well as information, which are important components in the uptake of guidance recommendations.

One strength of this study is that we used a sound knowledge synthesis strategy, the critical interpretive synthesis (CIS) approach. We acknowledged the paucity of data related to HSG appraisal, as well as the diversity in the literature sources and types of available data. Also, given that HSG is not a highly bounded topic, we did not rely on a narrow initial research question; instead, we were refining it as the review progressed. The various methodological stages did not proceed as discrete entities as there was a constant to-ing and fro-ing throughout the review and synthesis process; at times, we were concurrently searching, sampling, critiquing, and analyzing. The CIS approach is both systematic and iterative with an interpretive approach to analysis and synthesis of data that allowed us to capture and critically analyze an in-depth depiction of how to differentiate between HSG on the basis of quality, among other considerations. Through CIS methodology, we were able to further identify and include additional papers that were not directly related to HSG but made valuable theoretical contributions to the process of appraising guidance [20, 27].

A weakness of our study is in the nature of the CIS strategy, which exposes us to the risk that certain concepts may have been missed (calling saturation too soon, for example). If we have missed important elements, we anticipate they will be identified in the second stage of our research. Also, the health policy analysis triangle framework is made up of four elements: process, content, context, and actors. But for our analysis, the original framework was modified as we included “actors” in the process principles. Actors (the who of policy) refer to local, national, or international individuals or groups (governmental and non-governmental) involved in the policy process [44, 45]. We found that actors could be incorporated under the process principles concept “participatory.” Additionally and at first glance, it appears that there is the risk that some concepts may lead to potential contradictions. For example, in some jurisdictions, achieving an ethical HSG may not be aligned with achieving a feasible HSG (e.g., in contexts where there is a bias toward citizens because of race or sexual orientation). Currently, the data provides no guidance on how to reconcile a situation like this.

In our analysis, we showed the role that the process principles, content, and context principles plays in the development, appraisal, and reporting of HSG and demonstrated the link and resonance within and between their various concepts. We pointed out patterns and drew linkages between the concepts in order to show that the concepts and domains do not typically occur in a linear, independent, or discrete manner. The interaction between these three clusters is an imperative consideration because they all influence the guidance process and can facilitate or impede the success of HSG recommendations.

Some of the concepts (criteria or items) and domains identified in this review may not be applicable to every jurisdiction or country so they are not intended to serve as a blueprint for all health systems to strictly apply. Guidance documents are administrative instruments that provide recommendations and implementation options typically in a step-by-step format but do not have force of law and, as such, allow for accommodations in approach [51, 52]. They are intended to provide recommendations on how to comply with governing statutes and regulations and assist staff and managers on how institutional mandates and objectives should be implemented in a manner that is fair, consistent, and effective [52]. Therefore, development of high-quality HSG will impact the type of recommendations being formulated, the degree to which they get implemented, the methods of dissemination, and the extent to which they impact on the usual operations of the health system [51].

Different players in the HSG process play different roles at different times and under different circumstances. Developers of guidance may only be looking at process principles and content, while end users in the field may focus more on usability of the guidance in their individual contexts. Thus, for example, to optimize this dual perspective and facilitate the division of labor, there is interest at the WHO to produce workbooks where the global guidance they develop (process principles and content) will be complemented by having a companion workbook for those at the receiving end (context principles). In this case, the role of the WHO will be the development of global guidance, while the country’s role (or health system’s role) will be to take this global guidance and then apply it to their local context. This makes our framework of health systems guidance concepts quite practical.

Our study has elicited 30 unique concepts. It is unclear the extent to which HSG stakeholders will view all concepts as equally important to our proposed tool, or, moreover, given the potential roles of the tool (development, appraisal, reporting), if there are varying levels of priority as a function of purpose. The next phase (stage 2) of our research program is aimed to create the beta version of the AGREE-HS by having international stakeholders prioritize these concepts and using those results to populate the tool. Our final phase (stage 3) will involve the usability testing of the beta version of the tool.