1 Background

The coronavirus disease 2019 (COVID-19) pandemic has led to renewed global interest in public health systems and their ability to address complex current and future public health challenges [1, 2]. In Canada, public health and publicly funded health systems are distinct, although they are closely related in that both are aimed at improving the health of Canadians. Public health initiatives, such as disease surveillance and health promotion, aim to prevent diseases and reduce the need for costly medical interventions, while publicly funded health services provide Canadians with access to the medical care they need when they are sick or injured, such as in hospital care [3].

Critical to understanding the ability of the public health system to meet its mandate are a performance framework and indicators that reliably measure whether or not the system is achieving what it set out to accomplish, allow for examination of performance amongst comparators, and encourage ongoing learning and quality improvement [4,5,6].

Within the health care sector, for example, hospitals and primary care [7], there is a long history of quality measurement to drive system performance. In an overview of countries in Europe, Williams and Nolte described six common domains of quality in health care, including (1) effectiveness, (2) efficiency, (3) access, (4) patient focus or responsiveness, (5) safety, and (6) equity [8]. However, quality measurement is still a developing field within public health [8]. Some relevant work to date includes that of the Public Health Quality Forum in the United States that developed a consensus statement in 2008 defining quality in public health as “the degree to which policies, programs, services and research for the population increase desired health outcomes and conditions in which the population can be healthy” [9]. They further described nine characteristics of quality in public health, including (1) population-centred, (2) equitable, (3) proactive, (4) health-promoting, (5) risk-reducing, (6) vigilant, (7) transparent, (8) effective, and (9) efficient [9]. More recently, O’Neill and colleagues conducted a qualitative study of public health system performance in Canada and identified conceptual, methodological, contextual, and infrastructure challenges in designing a performance measurement system for public health [6]. Some key challenges identified in this work included the mission and purpose of public health systems which includes a wide range of activities from health promotion to health protection and emergency response; the complexity of public health systems and actors, which involves multiple levels of government, organizations, and actors; the need for stable funding and foundational information technology to sustain a measurement system; and variation in governance and organizational structure of public health systems across jurisdictions, which make it challenging to design a performance measurement system that is flexible enough to accommodate these variations [6].

At present, there is no commonly accepted and used framework for measuring quality in public health systems. This scoping review aims to identify quality frameworks and describe key domains or components for public health systems in comparator countries.

2 Methods

We conducted a scoping review using the methods of Arksey and O’Malley [10] to identify quality frameworks for public health systems in selected comparator countries. The method consisted of six stages, including (1) identifying the research question; (2) identifying relevant literature; (3) selecting relevant literature; (4) extracting the data; (5) reporting the results. Arksey and O’Malley’s final step, consulting with stakeholders, was omitted due to capacity constraints during the COVID-19 pandemic.

2.1 Stage 1: Identifying the research question

The research question guiding the scoping review was: what frameworks have been used to assess the quality of the public health system at the national and provincial/territorial levels in Canada, and countries similar to Canada?

2.2 Stage 2: Identifying relevant literature

Public Health Ontario Library Services developed and performed the indexed and grey literature searches. MEDLINE (Ovid platform) was searched on May 16, 2022. The search strategy identified articles with at least one Medical Subject Headings (MeSH) Major Topic or title/natural language descriptor related to the concept of indicators/frameworks and at least one MeSH term or keyword in the title, natural language descriptor or abstract related to public health. Additionally, to focus the search during COVID-19, the identified articles contained at least one MeSH Major Topic or title/natural language descriptor related to health protection (i.e., control of communicable diseases, immunization, infection prevention and control, and emergency preparedness). Details of the search strategy are in Appendix A.

The grey literature was searched on May 30, 2022. The search strategy included web searches using a general search engine (Google) and two custom search engines programmed to search: 1) the websites of Canadian federal, provincial, and regional public health departments, and 2) the websites of selected international public health agencies. These programmable search engines assisted in providing customized Web searches by utilizing the Google search engine [11]. The grey literature strategies were adapted from the MEDLINE search strategy to accommodate the limitations of the search engines (e.g., 32-word limit per query), but the conceptual logic of the database search strategy was preserved. Only the first 50 results of web search queries (results sorted by relevance) were included for screening. The authors also hand-searched the reference lists of included grey literature and other articles as appropriate to identify any potentially relevant literature that had been missed in the search strategy. Details of the grey literature search strategy are in Appendix B.

Inclusion criteria included publication in English from 2012 to the time the search was performed in May 2022. We limited our searches to countries that have similar national contexts and public health systems to Canada through the use of a filter to exclude evidence from countries in Africa, the Caribbean, Central America, Latin America, South America, Asia, and Mexico, along with any articles tagged with MeSH terms for developing countries.

Additional exclusion criteria were frameworks focused solely on the health care system (e.g., hospital care); publications that took the form of an editorial, opinion, book, correspondence, or commentary; and publications that were highly focused on a specific area of public health work that were not more broadly relevant to public health systems.

2.3 Stage 3: Study selection

Included studies were determined using a three-step process. First, one reviewer (KR) screened articles and categorized them into two groups: (1) include or uncertain and (2) exclude. The documents in the include or uncertain group proceeded to the second step, where they underwent abstract or summary review by two reviewers (KR, MA). All articles that met the inclusion criteria or for which there was uncertainty proceeded to the third step of full-text screening. Any disagreement in the classification of articles at any step was discussed by the two reviewers (KR, MA). Disagreements were brought to a third reviewer (JH) to resolve through consensus-building. In addition, the second reviewer (MA) randomly reviewed 10% of all documents in the excluded group as a quality control measure.

2.4 Stage 4: Data collection and charting of the results

Data were extracted for included articles independently by two reviewers (KR, MA) and were collated in Microsoft Excel. Data extraction focused on descriptions of the frameworks, significant findings or themes, and methods if available.

2.5 Stage 5: Data summary, synthesis and report of results

Thematic analysis was performed on the included studies to identify common themes. One reviewer (KR) analyzed the information from each framework to identify common characteristics and grouped them into key themes. A second reviewer (MA) validated the findings independently by comparing the data extraction table to identified themes. Discrepancies were resolved through further discussion of all reviewers.

3 Results

The indexed literature search yielded 420 citations, with one duplicate publication identified and removed. Of the 419 publications that were included for the title and abstract screening, 34 were selected for full-text review. All of these articles were excluded primarily because they related to health care, as well as focused on quality improvement processes and were not generalizable to public health systems.

The grey literature search identified 1500 documents. After title screening, 50 documents were included for full-text review, and four documents met inclusion criteria. No additional papers for inclusion were found through reference list screening of relevant articles. Figure 1 is the PRISMA flow diagram for indexed and grey literature. The authors were not able to perform a critical appraisal of the included documents due to a lack of enough information about their methods.

Fig. 1
figure 1

PRISMA flow chart of the study selection process. *Indexed literature was de-duplicated. Grey literature was advanced directly to the screening stage and did not undergo the de-duplicating process due to a large number of records and lack of an automated process

Full size image

3.1 General characteristics of included documents

The four documents included originated from England (n = 1), Wales (n = 1), and the United States (US) (n = 2) [12,13,14,15]. All were published by national-level public health organizations, including two led or coordinated by a government [12, 13], one by an association [14], and one by a non-governmental organization [15]. The described goals of the public health quality frameworks were diverse and reflected the role of the entity producing the document. Public Health System Group for England and Public Health Wales were most similar in their goals of improving quality in public health with a focus on people receiving services, health equity, working with partners, and good administration/governance. Although the National Association of County and City Health Officials do not clearly describe the goal of their framework, it captures some components found in the government frameworks, such as having a clear purpose and identifying user needs. It additionally provides a toolkit to implement quality by linking needs assessment, planning and improvement efforts with a strong emphasis on indicators and reporting. The goal of the Public Health Accreditation Board quality framework was very different from the other three frameworks and is intended to be used as a guidebook to support accreditation requirements. Table 1 provides additional information on the included documents.

Table 1 Summary of source documents for included frameworks
Full size table

3.2 Themes

The analysis identified thirteen themes from the included frameworks. These themes are presented from most to least common in Table 2. The only theme described in all frameworks was “skilled workforce”. For example, the Public Health System Group for England described this theme as “investment in the public health workforce and their continuous development [12].”

Table 2 Major themes mapped to the included frameworks
Full size table

The next group of themes were found in the majority of frameworks (three of four). These themes included (1) strong leadership, (2) effective and timely service, (3) equity, (4) quality improvement, (5) close partnerships, (6) adequate resourcing, and (7) innovation. The strong leadership theme was linked to concepts of culture and effective governance. For example, Public Health Wales’ framework described this as “leadership and culture [13].” The effective and timely service theme contained references to responsiveness. The Public Health System Group for England also described the importance of responsiveness, with additional emphasis on this aspect across the life course [12]. The equity theme was commonly described as referring to access and usage. Both the Public Health System Group for England and Public Health Wales frameworks went beyond equity in access and described the importance of decreasing health inequities. Quality improvement was mentioned in all documents except that of the Public Health System Group for England. The Public Health Accreditation Board further made connections between some of the key themes and linked quality improvement, research and evaluation as drivers of innovation [15]. Interestingly, close partnerships were mentioned in all frameworks except that of the National Association of County and City Health Organizations. The partnership theme included concepts, such as collaboration, co-development, and public engagement. Concepts included in the adequate resourcing theme referred to financial sufficiency and sustainability, prioritization, and a strong organizational structure. The Public Health System Group for England also described the importance of prioritizing work in a deliberate way within the available resources [12]. Lastly, in this category, strong innovation was included in all the frameworks except that of the National Association of County and City Health Organizations.

The next group of themes was included in half of the frameworks (two of four): (1) assessment and evaluation, (2) measuring improvement, (3) safety, (4) evidence-informed decision-making, and (5) clear communication. Assessment and evaluation focused on community- and population-level work. Measuring improvement also overlapped with evaluation in the Public Health System Group for England framework and emphasized transparency [12]. Service user safety was mentioned in the Public Health System Group for England and Public Health Wales documents [12, 13]. Evidence-informed decision-making was present in the England and Wales frameworks [12, 13], with the Public Health System Group for England describing the importance of evidence generation and using need and evidence to inform decision-making [12]. Communication was mentioned in both the Public Health Wales and Public Health Accreditation Board frameworks [13, 15].

4 Discussion

This scoping review provides a comprehensive overview of the indexed and grey literature describing quality frameworks for public health systems. While there was no commonly used framework or approach found, we identified thirteen major themes. Derose and colleagues described a framework for local public health system quality assessment, including over-arching components of structure, process, and outcomes [27]. The thirteen themes identified in our study support their finding. The themes can be further categorized into three components, including the strength of the public health agency’s structural enablers (skilled workforce, strong leadership, adequately resourced); the process of work (quality improvement processes, close partnerships, innovation, assessment and evaluation, measuring improvement, evidence-informed decision-making); and the impact on those served (effective and timely service, equity, safety, communication) (Fig. 2). Additionally, there is significant overlap in the themes identified in our review of quality frameworks in public health and dimensions of quality frameworks described within the health care system. The US Institute of Medicine and Health Quality Ontario, Canada, both identified a number of areas of health care system quality which overlap with our findings [28, 29]. These include accessible, effective, safe, patient-centred (analogous to community needs and public engagement), equitable, efficient, and appropriately resourced.

Fig. 2
figure 2

Conceptual framework for quality in public health systems, included 13 themes

Full size image

This scoping review builds on the small body of literature describing quality frameworks for public health systems. A significant amount of knowledge on this subject is in the non-peer-reviewed literature. This scoping review benefits from a more comprehensive search of both the indexed and grey literature.

5 Limitations

This review was challenged by the paucity of literature. Additionally, the documents reviewed did not describe the methods for developing the frameworks (e.g., literature review, evaluation criteria for inclusion, who comprised decision-makers, etc.). Thus, we were unable to assess the risk of biases. This includes that quality and its underlying components may be different for practitioners, policy makers, communities, and individuals, and may also not reflect the views of those who are marginalized. We also limited our review to the English language literature and studies from countries comparable to Canada in terms of socio-demographic and public health system contexts. This may limit our findings of knowledge in this field. However, this may not reflect real-world implementation and interpretation of frameworks by different public health systems.

Future work should look to identify measures and targets of success within the identified public health quality framework themes that can be used to regularly report on public health systems.

6 Conclusions

There is an identified need for frameworks and indicators to measure and compare public health system performance within and between jurisdictions with similar socio-demographic and public health system contexts. This is especially true in the wake of the COVID-19 pandemic as public health systems support community recovery and prepare for the future. Our scoping review provides a starting point from which public health systems can develop indicators that support ongoing measurement of their quality of performance. Clear and transparent documentation of development processes and evaluation findings after implementation will further contribute to a learning public health system focused on quality.