Background

Pragmatic versus explanatory trials

Pragmatic trials aim to determine if an intervention works in real-world settings, so that results can be generalized to everyday practice and support decision-making by patients, providers, and health system leaders; contrastingly, explanatory trials aim to determine if and how an intervention works under well-defined and highly controlled conditions [1]. While differences between explanatory and pragmatic attitudes in trials were first highlighted in a seminal article by Schwartz and Lellouch five decades ago [2], interest in pragmatic trials has increased dramatically in recent years [3]. Accordingly, reporting guidelines for pragmatic trials have been recently published [4].

In practice, trials are seldom purely pragmatic or purely explanatory, but various design choices can make a trial more or less pragmatic. The PRagmatic Explanatory Continuum Indicator Summary (PRECIS-2) was developed to help trialists identify explicit design choices that can shift a trial towards being more pragmatic [5]. PRECIS-2 has nine dimensions along which a trial can be scored from very explanatory to very pragmatic. In brief, trials that are more pragmatic have broader eligibility criteria, recruit participants at the time of presentation, include a diverse range of settings that mirror real-world circumstances, do not require highly specialized training or research personnel, give healthcare providers flexibility in how the intervention is delivered, require no special strategy for monitoring protocol compliance, follow and monitor patients as in routine clinical practice, have clinically meaningful and patient-centered outcomes, and include all randomized patients in analysis.

Explanatory trials, more often conducted on innovative medical products and devices for regulatory purposes, usually do not fully explicate the benefits and harms of interventions relative to existing alternatives and hence do not address the central question of what is likely the best (among the available) options for particular patients. In contrast, pragmatic trials test a much wider range of interventions, including diagnostic, preventive, therapeutic, and delivery system interventions. They may test new interventions against current routine interventions or the comparative effectiveness of different routine interventions head-to-head. When designed appropriately, they may address not only whether an intervention works, but more importantly, for whom and under what conditions. They may test different quality and service improvement interventions as well as knowledge translation interventions. Pragmatic trials therefore offer an important opportunity to improve patient health and health system outcomes by reducing variations in care, improving uptake of evidence-based practice, and reducing costs. For these reasons, the need for more pragmatic trials has been identified as a priority by governments worldwide [6,7,8,9,10].

This manuscript presents the study protocol for a four-year, interdisciplinary, mixed methods research project with the ultimate goal to develop internationally accepted guidance for the ethical design and conduct of pragmatic trials. Although there are many innovative observational study designs which can be used to evaluate interventions and produce new knowledge that informs decision-making [11], here we focus exclusively on intervention studies that use randomization. We allow for a broad range of intervention types including diagnostic, preventive, therapeutic, knowledge translation, and delivery system interventions. See Table 1 for a brief glossary of terms used in this manuscript.

Table 1 Glossary of terms
Full size table

Ethical issues in pragmatic trials

Trials undertaken in real-world settings raise substantial ethical issues that have not yet been fully addressed [12,13,14,15]. These ethical issues arise not only from the push towards a greater degree of pragmatism (e.g. along the nine PRECIS-2 dimensions), but are closely tied to the types of interventions, as well as the choice of study design. In addition to established study designs which use patient randomization (see Example 1 below), pragmatic trials include some emerging designs and approaches which capitalize on methodological and statistical innovations as well as the availability of registries and routinely collected health data (e.g. cohort multiple randomized controlled designs [16], randomized registry trials [17], cluster cross-over trials [18], and stepped wedge cluster randomized trials [18, 19]). The cohort multiple randomized design and randomized registry trials are examples of pragmatic trial designs that facilitate the evaluation of usual care interventions embedded within routine settings, but their ethical implications remain unclear. Cluster randomization [20] may be used to randomize entire medical practices or hospitals to differing interventions and is a natural choice for evaluating service delivery or other health system level interventions; however, with the push towards more pragmatic trials, this design is increasingly being used to evaluate individual-level interventions (i.e. interventions that, in theory, could have been evaluated using traditional patient randomized designs). While the Ottawa Statement on the Ethical Design and Conduct of Cluster Randomized Trials [21] provides explicit guidance for cluster randomized trials, the use of this design in the case of usual care individual-level interventions raises additional ethical issues (see Example 2 below); moreover, the stepped wedge cluster randomized design raises its own unique ethical issues (Example 3).

Table 2 describes a preliminary framework of nine ethical issues; the pragmatic, intervention, and design characteristics that give rise to these issues; and their potential implications for researchers and research ethics committees. This framework was developed based on an initial scoping review [12] and will be further developed and refined during our project.

Table 2 Preliminary framework of ethical issues raised by pragmatic randomized controlled trials (pragmatic RCTs)
Full size table

Examples of pragmatic randomized controlled trials

For illustrative purposes, we present three examples of pragmatic trials that raise important ethical issues needing guidance.

Example 1: patient-randomized trial comparing usual care interventions (SUPPORT)

The SUPPORT trial [22] sought to determine the optimal level of supplemental oxygen on incidence of retinopathy of prematurity and mortality in preterm infants. Infants were randomized to high and low oxygen saturations, both of which fell within the range used routinely in practice. Lower oxygen levels were found to reduce retinopathy but increase mortality, leading the authors to urge caution in their use. In response to an anonymous complaint by parents, the US Office for Human Research Protections launched an investigation into the trial [23]. It determined that investigators failed to adequately inform parents of “reasonably foreseeable risks” as consent materials did not list visual impairment and death as research risks. This incited considerable debate in the literature [24,25,26,27] centered almost exclusively on considerations of risk and risk communication and reflecting a preoccupation with US regulations. A central ethical issue raised by the SUPPORT trial is: Should usual care interventions be considered part of research or clinical practice? Other issues raised are: What constitutes usual care? Can a study be considered “minimal risk” if its endpoints include serious impairments such as blindness or death? Do parents need to be informed that their child is participating in a trial? At what level of detail do the study interventions need to be disclosed to participants? What potential benefits and harms ought to be disclosed? What type of research ethics review is appropriate?

Example 2: cluster randomized registry trial comparing usual care individual-level interventions implemented as policy interventions at the cluster-level (FLUID)

The FLUID trial is a pragmatic cluster randomized cross-over trial comparing resuscitation with Ringer’s Lactate versus normal saline on death and hospital readmissions in hospitalized patients, with outcomes assessed using health administrative data [28]. Both fluids are considered usual care interventions that have been available for decades and are administered to many hospitalized patients. While fluid administration is an individual-level intervention, the study becomes feasible only when interventions are implemented as hospital policies (i.e. at the cluster level) with a waiver of patient informed consent. In particular, it is essential to have only one type of study fluid available throughout the hospital to minimize the risk of contamination due to a patient receiving both types of study fluids in different areas of the hospital. It would also be logistically challenging and very costly to recruit and randomize individual patients within all areas of participating hospitals. In contrast, the availability of routinely collected data available for all patients permits the conduct of a very cost-efficient trial involving hospitals across the province, without the need to recruit patients for data collection. However, the FLUID trial raises several ethical issues: What is an appropriate justification for adopting cluster randomization? When individual-level interventions are implemented institution-wide as a policy in a cluster randomized trial, may one proceed without patient consent? Do patients need to be notified about the trial; if so, how? Is consent required for the use of routinely collected participant data?

Example 3: stepped wedge trial of a quality improvement intervention (Surgical Checklist Trial)

The Surgical Checklist Trial was a stepped wedge trial randomizing surgical units in two hospitals in Norway to evaluate the impact of the World Health Organization Surgical Safety Checklist on morbidity, mortality, and length of hospital stay [29]. The stepped wedge design is characterized by the fact that clusters (here, surgical units) cross gradually and in random sequence from the control to the intervention condition, with all clusters exposed to the intervention by the end of the trial. As stated by the investigators, the 19-item checklist consists of an oral confirmation by surgical teams of the completion of the basic steps for ensuring safe delivery of anesthesia, prophylaxis against infection, effective teamwork, and other essential practices in surgery. Compliance with the checklist was assessed prospectively by nurses. All patient outcomes were collected from hospital administrative databases. The regional research ethics committee advised the investigators that the study was considered clinical service improvement and that research ethics approval and patient informed consent were not required. The Surgical Checklist Trial raises several ethical issues: Should trials evaluating quality and service improvement be considered research? What are the ethical implications if health system leaders conduct (potentially less robust) quality improvement studies rather than rigorous pragmatic randomized controlled trials or refrain from conducting important studies due to perceived ethical barriers? Should health professionals targeted by study interventions be considered research participants? If study participation poses no more than minimal risk, is patient or provider consent required? What is the harm–benefit balance of exposing all surgical units to the intervention by the end of the study?

Challenges to the conventional framework for research ethics

As illustrated by our three examples, pragmatic trials raise important ethical issues that have not yet been satisfactorily addressed. Existing ethical and regulatory frameworks were developed primarily for trials with explanatory aims, i.e. focusing on efficacy and safety of experimental interventions for marketing approval. With the move towards the conduct of more pragmatic trials, existing ethics guidance is becoming more difficult to interpret and apply and may not be sufficient to address these new ethical issues. On the one hand, the absence of clear guidance may put trials at risk of being accused of exposing participants to inadequate protections (real or perceived) (e.g. the SUPPORT trial [23]). It is essential to avoid undermining the trust that patient communities have in the research enterprise. On the other hand, strict enforcement of conventional rules is likely to pose unnecessary obstacles, undermine scientific quality or impede improvements in patient health and health system outcomes. Unless these ethical issues are addressed, important research with large potential healthcare benefits may be impeded or may go ahead without adequate safeguards.

Objectives

The overarching goal of our project is to develop guidance for the ethical design and conduct of pragmatic trials. Specific objectives are to:

  1. 1.

    Systematically identify ethical issues arising from pragmatic trials;

  2. 2.

    Document ethical practices in completed and ongoing pragmatic trials;

  3. 3.

    Elicit views and experiences of trialists, methodologists, chairs of research ethics committees, trial participants, and health system leaders;

  4. 4.

    Develop novel ethical solutions informed by ethical analyses;

  5. 5.

    Generate ethics guidance through a consensus process with an independent expert panel;

  6. 6.

    Disseminate tailored guidance to stakeholders.

Methods

Overview

Our project, summarized in Fig. 1, consists of both empirical and conceptual work and concludes with a consensus development process and knowledge translation. The empirical work, to be completed over the first three years, consists of five studies:

  1. 1.

    Key informant interviews with a small group of 24–40 experts;

  2. 2.

    Identification and review of a random sample of 300 completed and ongoing pragmatic trials;

  3. 3.

    Survey of trialists (investigators of the 300 pragmatic trials);

  4. 4.

    Survey of a random sample of chairs of research ethics committees in Canada, USA, UK, France, and Australia; and

  5. 5.

    Embedded focus group discussions and a community survey with trial participants (e.g. patients) and gatekeepers (i.e. those who have ability to allow or deny access to trial participants).

Fig. 1
figure 1

Overview of project phases

Full size image

All outputs will inform the ethical analysis in the conceptual work.

The conceptual work will be an ethical analysis of the identified ethical issues from the empirical work, resulting in a series of publications outlining proposed solutions with principles, policy options, and rationales. The empirical and conceptual work will proceed concurrently to allow the empirical and ethical analyses to be mutually informative.

The consensus process, occurring upon completion of the empirical and conceptual work, will involve identifying an international expert panel of trialists, ethicists, chairs of research ethics committees, regulators, funders, and community representatives and organizing a consensus conference attended by the panel, as well as invited researchers, representatives of funding agencies, regulators, journal editors, and patient and community groups. In addition, we will engage in e-consultation with the broader research community, funders, regulators, journal editors, and patient groups. The panel will produce the final ethics guidance document which will be disseminated to stakeholders by the research team.

Empirical studies

Study 1: key informant interviews

The objective of this study is to conduct interviews with a small group of pragmatic trial experts and stakeholders (trialists, ethicists, methodologists, chairs of research ethics committees, health system leaders, quality improvement experts, and patient representatives on research study teams) to generate a thorough understanding of types of ethical issues arising in the practice of pragmatic trials from a variety of perspectives. Informants will be selected using a purposive sampling strategy, augmented through snowball sampling. Potential interviewees will be selected across a broad range of jurisdictions and clinical areas to reflect a range of experiences, including lower- and middle-income countries. The sample size will be determined to ensure representation from all targeted stakeholders and by when saturation is reached (i.e. when new interviews cease to provide fresh information) [30,31,32,33,34]. Based on our prior experience with similar studies [35], we anticipate that 24–40 interviews will be required. Interviews are expected to last 1 h and will be audio recorded. A semi-structured interview guide will allow participants to respond freely, to illustrate concepts, and to present perspectives that the interviewer can probe further. To monitor the progress of the interviews and permit follow-up of issues that may emerge from the data, interviewing, transcription, and analysis will proceed concurrently. The interview guide may evolve as a typology of ethical issues begins to emerge. Recordings will be transcribed and verified before analysis. Data will be imported into a qualitative software package (NVivo 11) to facilitate thematic coding, evaluation, and analysis. The results will be used to formulate a typology of ethical issues arising from pragmatic trials, to be addressed in the conceptual work. It will also inform data extraction and questionnaire items for studies 2–5.

Study 2: review of published trials

The objectives of this study are to select and review a random sample of recently completed and ongoing trials that have more pragmatic (than explanatory) aims, to describe ethical characteristics, identify ethical challenges reported, the circumstances under which they arise, and how they are being addressed. We anticipate challenges in identifying a sample of “pragmatic” trials given wide variation in definitions and inconsistent and unreliable reporting of trial design. We will develop objective and reproducible criteria to characterize a trial as having pragmatic aims, as well as the conditions under which those criteria apply. We will work with an information scientist to develop a sensitive and specific electronic search strategy to identify a sample of trials. To develop and validate an electronic search strategy, we will use a multi-pronged approach to identify a gold standard set of trials meeting our criteria for testing the search strategy, including: (1) pragmatic trials conducted by the investigator team and our extensive networks; (2) pragmatic trials identified in the key informant interviews; (3) databases of funded pragmatic trials in Canada, the USA, UK, France, and Australia; (4) a database of pragmatic trials maintained by the PRECIS group [36]; (5) demonstration projects by the National Institutes of Health Collaboratory [37] and PCORnet [38]; and (6) trials included as exemplars in recent publications about pragmatic trials. The proportion of these pragmatic trials retrieved by the search strategy will be calculated and used to refine the search strategy if necessary as done in our previous work [39]. Once validated, the search strategy will be used to select a random sample of 300 pragmatic trials. As the ethical landscape may have been changing in recent years, we will include completed trials as well as study protocols for ongoing trials. Eligibility criteria, to be refined, will include trials or study protocols published in the past five years by investigators in Canada, UK, USA, France, and Australia, including trials conducted in lower- and middle-income countries. Items for extraction will be generated based on team discussion and key informant interviews. After pilot testing, two reviewers will independently extract data from each trial report. Discrepancies between reviewers will be identified and resolved by discussion with a third reviewer if required. Prevalence of ethical issues and practices arising from pragmatic design features, study design, and type of study interventions will be described overall and within subgroups of interest where feasible (e.g. over time, between countries, study sponsors). Methodological and reporting quality of the included trials will be collected and evaluated against major methodological and reporting criteria as done in our previous work [40, 41]. The preliminary ethics framework will be updated as extractions proceed.

Study 3: survey of trialists

The objective of this study is to gather more detailed information about practices and experiences identified in our review of pragmatic trials. After pilot testing, we will administer a survey consisting of open- and closed-ended items to corresponding authors of the sample of 300 trials. The primary mode of survey administration will be web-based, but alternatives (paper, telephone) will be considered to increase the response rate. A series of contacts (pre-notifications, notifications, and reminders) based on Dillman’s recommendations for the implementation of mail and Internet surveys will be used [42]. The survey will be used to characterize the ethical conduct, review, and reporting of pragmatic trials from the perspectives of trialists. We will offer respondents a $30 gift certificate or donation on behalf of the respondent in appreciation for their time. We will use questionnaire personalization—a previously published methodology developed by our team—to gather more detailed information about aspects of the published trial [43]. The anticipated response rate, based on previous experience with this population and methodology, is 65% [44]. Results from the survey will be compared to results from the published trial to assess adequacy of reporting of ethical issues and to describe implications of design choices (e.g. impact of alternative consent models on study recruitment and risks of selection bias). Potential non-response bias will be assessed by comparing characteristics of respondents and non-respondents using information in trial reports. We will describe the use of gatekeepers (i.e. individuals or bodies that represent the interests of community members, communities, or organizations [45]), use of consent waivers or alternative consent models (e.g. “streamlined consent” [12]), and details about information conveyed to participants in each study arm. We will describe the type of ethics review required, perceived impact on the timing of implementation, ethical and scientific quality of the trial, and uniformity of process and decisions in multicenter trials. We will explore the possibility of requesting informed consent documents and research ethics application forms and protocols for a subset of trials. Results will inform the ethics framework for analysis in the conceptual work.

Study 4: survey of research ethics committee chairs

The objective of this study is to gather information on the views, practices, and experiences of research ethics committee chairs in Canada, the USA, UK, France, and Australia. We will aim to select all research ethics committees that review clinical trials in Canada (approximately 200), the UK (approximately 100), Australia [46] (approximately 200), France (approximately 39) [47], and a random sample of 200 from over 9000 Institutional Review Board Organizations in the USA [48]. We chose these five countries primarily based on logistical considerations: our team members have connections with research ethics organizations in these countries which will help facilitate participation. Canada does not maintain a list of research ethics committees. We will use a strategy previously developed by our team to identify eligible committees [49]. It involves integrating internet searches with a list of Institutional Review Board Organizations maintained by the US Office of Human Research Protections. Given that biomedical and non-biomedical Institutional Review Boards are not differentiated in this list, we will use stratification by National Institutes of Health funding levels to increase the efficiency of identifying Boards with relevant experience reviewing clinical research.

Questionnaire items will be informed by the preliminary ethics framework, key informant interviews, and results from the trialist survey. Questionnaires will consist of open- and closed-ended items and include a series of scenarios. After pilot testing, a series of contacts (notifications and reminders) based on Dillman’s recommendations for Internet surveys will be used [42]. We will likely encounter challenges in ensuring an adequate response rate; based on previous experience with surveying this population, we expect a response rate of approximately 35% [49]. We will deliberately keep questionnaires short and adhere to recent recommendations for improving response rates [42]. Results will be summarized using descriptive statistics and compared across subgroups (e.g. country, size and type of committee, years of experience). Where feasible, potential non-response bias will be assessed by comparing characteristics of respondents and non-respondents using information on research ethics committee websites. Questionnaires will be prepared in both English and French. Open-ended responses will be analyzed thematically. Results will inform the ethics framework for analysis in the conceptual work and knowledge translation activities to research ethics committees.

Study 5: focus group discussions and community survey

The objective of this study is to gather information on the views and experiences of trial participants or prospective trial participants (e.g. patients), gatekeepers (organizational leadership, medical directors), and communities. Our research team members are involved in 15–20 pragmatic trials at any one time. We will identify one ongoing or recently completed pragmatic trial in each country, ensuring a range of types of pragmatic trials. We will conduct focus groups with eligible trial participants (with permission from the responsible research ethics committees and the chief investigator). We anticipate five focus groups with patients and five with gatekeepers. Focus groups will be 1–2 h in length and involve six participants per group. A semi-structured discussion guide will be used to gather information on participants’ experiences with the trial including recruitment, informed consent, perceived benefits and harms, any privacy concerns, and satisfaction with the trial. Among eligible prospective participants, we will explore potential reasons for non-participation. Discussions will be recorded, transcribed verbatim, and verified by the facilitator before analysis in NVivo 11 [50]. To monitor progress and permit follow-up of issues that may emerge from the data, discussions, transcription, and analysis will proceed concurrently [51]. For focus groups not conducted in French, interview guides will be developed in English by the study team and then translated into French by bilingual members of the research team. To verify the accuracy of the translation, all guides will be independently back-translated. All focus group discussions will be transcribed in the language in which the group was conducted. Non-English transcripts will be translated into English and then independently back-translated to the original language. A Canadian pragmatic trial will be used to design a survey of community members targeted by the trial (e.g. diabetes patients, hospital patients). Quantitative and qualitative analyses will be used to summarize results from the community survey. Results will be used to inform the ethical analysis in the conceptual work.

Ethical analysis

The ethical analysis will be an intensive process run concurrently with the empirical work and will extend over a period of three years. Conceptual work in bioethics is not amenable to the degree of a priori methodological specification that is expected of empirical research. Reproducibility is an indispensable feature of rigorous science, necessitating the clear statement of hypotheses and experimental methods upfront. Rigorous conceptual work in ethics begins with the articulation of clear and important questions and is realized in the construction of careful and clear analysis of the relevant concepts and of ethical arguments in peer-reviewed publications and policy reports [52]. The ethical analysis in this project will be based on an evolving framework of ethical issues developed using results from an extensive literature search conducted in preparation for this proposal (see Table 2) and revised using results from the five empirical studies. For each identified set of core ethical issues, an in-depth and written ethical analysis will be prepared. An extensive review of the scholarly literature will document and critically analyze arguments proffered for and against ethical positions. The ethical analysis will seek to synthesize foundational documents, regulations, and arguments in the literature into a coherent solution. Where disagreement among the various sources cannot be resolved by critical analysis, the contours of the ethical dispute will be documented. The ethical analysis will result in a series of background documents laying out proposed solutions with principles, policy options, and rationale and will also be submitted for peer-reviewed publication. These documents will be used as background materials in the consensus process.

Expert panel and consensus process

An international expert panel will be convened to develop ethics guidance. The composition of the panel is expected to be 4–6 pragmatic trialists and methodologists, 4–6 ethicists and chairs of research ethics committees, 2–4 quality improvement experts and health system leaders, 1–2 regulators, 1–2 journal editors, and 2–4 community members. No more than one-third of the members of the panel will be drawn from the research team and adequate representation from a broad range of countries including lower- and middle-income countries will be sought. The panel will be provided with output from the empirical studies as well as the documents prepared during the ethical analysis one month in advance of a three-day consensus conference which will also be attended by invited researchers and representatives from major funding bodies, regulators, journal editors, and community groups. The conference will consist of both open and closed sessions. At the open sessions, proposed ethics guidance with supporting ethical analysis will be presented by the research team and comments and discussion invited from attendees. The panel will then meet in closed sessions to discuss and formulate draft guidance. Based on previous experience with the consensus process [21], we anticipate that at the beginning of the meeting, the panel will set rules for debate, handling of disagreements, and how to achieve consensus. We do not expect to use a majority voting system but anticipate that agreement will be reached through discussion, with documentation of disagreements where they exist. After the meeting, a draft guidance document will be produced by a writing committee and refined after further discussion with the panel. An e-consultation process will be launched to invite comments from the broader research community, funders, regulators, journal editors, and community groups. Based on results from this process, the writing committee will make revisions and produce the final consensus guidance.

Discussion

The goal of our international, interdisciplinary collaboration is to develop, publish, and promote the uptake of guidance for the ethical design and conduct of pragmatic randomized controlled trials. We seek to create a novel approach to the ethics of pragmatic trials that improves upon the existing literature, which has been criticized for lacking convincing arguments grounded in ethical principles [53], including claims based on erroneous assumptions [54], appealing mainly to regulations in particular jurisdictions [55,56,57], offering speculative solutions with no guarantee of improvement over existing oversight procedures [58], and failing to reflect a broad vision of protecting the liberty and welfare interests of research participants [59]. The proposed process is informed and enriched by our previous experience with developing ethics guidance for cluster randomized trials [21, 60]. The planned output includes manuscripts, educational materials, and tailored guidance documents. Our proposed project is novel in that it: (1) involves close collaboration between clinical trialists, ethicists, and methodologists; (2) combines concurrent empirical and ethical analysis in a mutually informative approach; (3) integrates views and experiences of stakeholders (e.g. trialists, chairs of research ethics committees, health system leaders, community members); and (4) aims to generate guidance rooted in internationally accepted ethical principles rather than regulation specific to one jurisdiction. We expect that the study outputs will be of interest to a wide range of knowledge users including trialists, healthcare professionals, ethicists, research ethics committees, journal editors, regulators, health system leaders, research funders, and patient groups. Guidance will facilitate the conduct of research important to patients, clinicians, and the healthcare system, while upholding the highest ethical standards in research. While the scope of the planned guidance is intended to be international, some of the empirical studies (e.g. focus groups) will be geographically restricted based on logistical and feasibility considerations.

Our knowledge translation strategy will be guided by the Canadian Institutes of Health Research Guide to Knowledge Translation Planning [61]. We will use our considerable informal networks nationally and internationally to disseminate our findings. Work stemming from the research will be submitted for presentation at national and international conferences and meetings targeting specific stakeholder groups (e.g. journal editors, funders, regulators). The final ethics guidance document will be published in a major journal with summaries published simultaneously in other major journals. Educational material for researchers and research ethics committees will be developed. Social media will be used to communicate results to the public.