Background

As first articulated by Schwartz and Lellouch [1] , clinical trials can be conceptualized across a continuum from explanatory to pragmatic, with the former assessing if a practice, intervention, program, or treatment could work under ideal circumstances and the latter determining if it could work in routine practices. Both explanatory and pragmatic trials—and those that fall between the ends of the continuum—are essential for advancing research and practice in health care and public health. Pragmatic trials in particular are necessary for developing and testing interventions using the settings, resources, patients, and approaches to which they will ultimately be implemented, thus ensuring that research funds have the greatest potential to impact patient and population health. To this end, the United States (US) National Cancer Institute (NCI) leadership [2] has included the support and conduct of pragmatic trials as part of its recommendations for modernizing the clinical trial enterprise.

In recent years, there has been a significant increase in the number of pragmatic trials across a range of health areas, intervention types, delivery settings, and patient populations. Although the conceptualization of trials along the continuum has advanced over the years, there remains great variability in how pragmatic trials are defined, described, operationalized, and labeled. A recent study by Dal-Ré and colleagues [3] found that 36% (32) of self-labeled pragmatic trials (N = 89) more accurately reflected characteristics of explanatory trials (e.g., placebo-controlled, single-center). Similarly, in a review of recently published trials identified in the article title as pragmatic, 33 (45%) provided no justification in text for why their trial was pragmatic [4]. Although there is no single “correct” way to describe trials along the continuum, significant misclassification may provide misleading information to health care practitioners, policymakers, advocates, and patients about the applicability of the trial results to everyday settings and populations.

To remediate some of the confusion, and to help build capacity in pragmatic trials, workshops, short courses, tools, and online resources have been developed for researchers interested in conducting such trials. For example, in the USA, the National Institutes of Health (NIH) Health Care Systems Research Collaboratory has an online Living Textbook on pragmatic trials, which includes self-paced learning modules, factsheets, web-based presentations, podcasts, and starter kits on pragmatic trials geared toward investigators and research teams [5]. The Health Care Systems Research Collaboratory also hosts in-person training workshops on pragmatic trials for researchers with workshop materials available online. The ACCORDS Dissemination and Implementation Science Program of the Anschutz Medical Campus, University of Colorado, Denver, hosts an interactive, user-friendly training eBook for researchers, Pragmatic Trials: A Workshop Handbook [6]. The edX platform hosts a massive open online course, Pragmatic Randomized Controlled Trials in Health Care [7]. PragMagic is an online pragmatic trial resource designed to aid in the design, conduct, and evaluation of pragmatic trials, and includes the PragMagic Tool [8], a decision support tool for trial design.

One of the most commonly used resources is the Pragmatic Explanatory Continuum Indicator Summary (PRECIS) tool. First published in 2009, it was developed to aid researchers in considering the overall purpose and intent of their proposed trial, and the applicability of trial results to real-world settings [9]. Revised in 2015 [10], the PRECIS-2 tool consists of 9 key domains (i.e., eligibility, recruitment, setting, organization, flexibility [delivery], flexibility [adherence], follow-up, primary outcome, and primary analysis) that distinguish trials along the continuum from explanatory to pragmatic. Each domain represents a component of the trial that can make it more explanatory or more pragmatic. Domains are scored through discussion during the planning phase of the trial among the research team, with a score of 1 reflecting a more explanatory trial and a score of 5 reflecting a more pragmatic trial; scores are visually represented on a PRECIS-2 wheel, where wheels close to the hub reflect more explanatory trials and wheels close to the rim reflect more pragmatic trials. The PRECIS-2 tool is considered by many to be the gold standard for conceptualizing trials along the continuum. To date, and to the best of our knowledge, it is also the only tool that has demonstrated good inter-rater reliability and discriminant validity [11]. PRECIS-2 is often included as part of workshops and trainings on pragmatic trials. To facilitate trialists, an interactive web-based platform assists registered users in completing a PRECIS-2 wheel specific to their trial [12]. To date, over 700 international researchers have used this website software.

Despite the increasing number of resources for building capacity among investigators in pragmatic trials, few—if any—opportunities are designed to meet the unique needs of funding agency staff in medical research agencies. Across the NIH, program scientists are responsible for many scientific and administrative aspects of research. At the NCI, 1 of 27 institutes, centers, and offices of the US NIH, a leading medical research and funding agency, responsibilities of program scientists include (but are not limited to) the following: (1) assuming scientific leadership by defining short- and long-range goals for research programs, (2) stimulating interest in projects and special activities through communication with the scientific community, (3) identifying areas warranting either increased or decreased funding emphasis, (4) developing funding opportunity announcements to solicit applications from the scientific community, (5) consulting and advising grantees during the preparation of applications, and (6) managing funded research projects and research training programs [13]. The roles and responsibilities of program scientists differ from those of investigators and, accordingly, require different skill sets, training, and expertise, as their main responsibilities are tied to the goals of the funding agency in supporting and growing targeted areas of science rather than conducting independent research. Thus, program scientists are in a unique role of interacting directly with the research community while also creating research opportunities in targeted health areas.

This commentary describes an interactive, tailored workshop on pragmatic trials to meet the needs of program scientists and developed to complement existing training programs for other target audiences (e.g., academic researchers). The workshop focused on a specific health area—cancer—and included 29 program scientists working in 2 NCI divisions (i.e., Division of Cancer Control and Population Sciences and Division of Cancer Prevention), both divisions of which are heavily involved in developing and testing interventions across the cancer control continuum (i.e., etiology, prevention, detection, diagnosis, treatment, and survivorship) in real-world public health and cancer care delivery settings.

Pragmatic trials across the cancer continuum: an interactive workshop

The 6-h, in-person workshop consisted of a combination of didactic presentations from experts in the field and small-group activities customized to help program scientists with their everyday responsibilities. A workshop planning committee (SC, EK, AOM, NS, WN) was formed and included representatives from across the cancer control continuum in the Division of Cancer Control and Population Sciences and the Division of Cancer Prevention. The planning committee was responsible for developing the objectives, content, and structure of the workshop didactic portion and small-group activities (e.g., case studies, facilitator guides, group composition). Consistent with best practices in workshop development [14, 15], a pre-workshop online survey assessed participants’ general understanding of pragmatic trials, which in turn guided workshop content, scope, and selection of supplementary materials (e.g., list of key references and websites, PRECIS-2 toolkit, frequently asked questions document, and copies of seminal articles). The 1-day workshop was held in October 2018 at NCI in Rockville, MD, USA. A detailed copy of the workshop agenda can be found in Table 1.

Table 1 Agenda for pragmatic trials across the cancer continuum interactive workshop
Full size table

Presentations

The workshop began with an overview of the purpose and structure of the workshop (WN), followed by two 45-min expert presentations (MZ, KL), each including group discussion. The presentations introduced attendees to the concepts of explanatory and pragmatic trials and reviewed the PRECIS-2 tool for operationalizing trials along the continuum. The first expert discussed the strengths of randomized controlled trials (RCTs) in health [16], reviewed several limitations of RCTs (e.g., poor generalizability, applicability, and external validity [17, 18]), and introduced the conceptualization of RCTs along a multiaxial explanatory-pragmatic continuum [1, 19, 20]. The second presentation reviewed the most widely used tool, PRECIS-2, for operationalizing trials along the continuum during the design phase of the study [10]. Examples from cancer prevention and control trials were used to illustrate scores for each of the nine domains in the PRECIS-2 tool. Following the plenary component, participants attended two interactive sessions to enhance the application of knowledge.

Activity #1

The objective of the first group activity was to enhance attendees’ understanding of explanatory and pragmatic trials in cancer prevention and control, reflecting the range of trials that program scientists manage and advise on a daily basis. Attendees were divided into six groups consisting of a mix of individuals from epidemiology, behavioral science, health care delivery, implementation science, and symptom management/palliative care program areas. Each group was assigned one of three published articles on a cancer-focused trial: one trial that was more pragmatic, one trial that was more explanatory, and one trial that had aspects of both. The participating program scientists deliberated in small groups and (1) scored their trial using the PRECIS-2 tool, (2) identified relevant text excerpts reflecting their domain score, and (3) discussed their score for each domain. The PRECIS-2 toolkit, which includes the PRECIS-2 wheel, domain descriptions, and examples from published trials, all of which are downloadable on the PRECIS-2 website [12], served as template documents for the activity. Facilitators guided the discussion and asked probing questions, as needed, using a standardized “answer” sheet—a version of the PRECIS-2 tool that had been completed by the steering committee and consultants for each trial before the workshop. A blank PRECIS-2 wheel on a flip chart was available for each group to record their domain ratings. All attendees reconvened in plenary to discuss the groups’ consensus on the trial ratings and rationale. Two flip charts (one per group) were posted side by side for each of the three trials to visualize agreements and discrepancies in domain scores. Divergent scores and corresponding text were discussed, and domains that were particularly challenging to rate received extra attention. Discussion of the discrepancies provided an opportunity to emphasize the importance of a team-based approach for scoring domains as well as highlighting the reality that some variability in scores is expected, as there are no objectively correct or incorrect ratings.

Activity #2

The objective of the second small-group activity was to facilitate the program scientists’ ability to characterize the components of trials that would reflect a more pragmatic vs. a more explanatory design and thus be able to guide investigators should they be aiming for the former. This activity was designed to mirror the program scientists’ role in providing technical assistance to prospective grantees during the pre-submission process (e.g., feedback on overall study aims), which may include a discussion of research approach (e.g., designs, methods, theories, measures, outcomes), innovation, significance of the proposed study, and fit with overall program goals. To this end, participants across all groups received the same three documents: (1) a copy of an article of a relatively explanatory trial comparing modalities to increase colorectal cancer screening, (2) a copy of a PRECIS-2 scoring table that was pre-populated with domain scores and extracted text from that trial justifying the score, and (3) a copy of a blank PRECIS-2 scoring table. Again, the PRECIS-2 toolkit served as a template for the documents. With guidance from the facilitator (each of whom had a pre-scored version of the PRECIS-2 table), the groups were asked to provide suggestions for how to make the trial more pragmatic. As with the first activity, all participants reconvened and presented suggestions for how to make the trial more pragmatic, which led to an engaging discussion about the characteristics of pragmatic vs. explanatory trials, moderated by the two experts.

Evaluation

A paper-based questionnaire was administered at the end of the workshop and collected by non-NCI staff who assisted with the workshop logistics and preparation. The evaluation consisted of six items to assess the relevance and effectiveness of the workshop (e.g., “Activities were relevant to my professional role,” “Workshop was a good use of my time,” and “Presentations increased my understanding of the explanatory-pragmatic continuum,” all scored from 1 = strongly disagree to 5 = strongly agree). Participants were asked to indicate which, if any, resources or activities they would like as a follow-up to the workshop (e.g., interest group; working group to develop sample text on pragmatic trials for relevant funding opportunity announcements). Finally, attendees were asked to indicate what they liked most about the workshop, suggestions for improvement, and any additional feedback or comments (open-ended response option). Participants were given the option of providing their name at the end of the survey.

Nineteen of the 29 attendees (66%) completed the post-workshop evaluation. Overall, participants rated the workshop very positively. Most attendees were interested in follow-up activities to help educate researchers in pragmatic trials (e.g., hosting webinars, n = 13; providing training opportunities, n = 13), with relatively fewer interested in internal activities (e.g., NCI interest group, n = 11; trans-NIH interest group, n = 8). Most respondents (n = 14) reported the interactive exercises to be the most beneficial. As one respondent noted, “Activities were extremely helpful in contextualizing the material.” Another individual stated, “I really liked that we were able to apply what we learned in the slides to real-world examples.” A range of improvements were suggested, including additional time for the activities and group discussions, access to reading materials beforehand, and a greater variety of cancer-focused interventions as examples.

Conclusions

Building interest in and understanding of pragmatic trials is essential for expediting the application of trial results to usual practice settings and patient populations. To complement the existing trainings for other target audiences (e.g., academic researchers), we developed an interactive workshop tailored specifically to meet the needs of program scientists in their role at a major biomedical research funding agency. Feedback on the workshop was overwhelmingly positive. We anticipate that attendees will be more likely to incorporate knowledge gained from the workshop into a range of professional activities in the future.

This workshop may serve as a template for research funding agencies to adapt and use to build internal capacity in supporting the conduct of pragmatic trials, and to complement existing trainings for other target audiences. Collectively, such opportunities are likely to increase support for the conduct of pragmatic trials among funding agencies and researchers alike. Increasing the number and quality of pragmatic trials can help ensure that research investments directly impact health care practitioners, policymakers, and patients.