Contributions to the literature

  • The investigation documents that the PRECIS-2 and PRECIS-2-PS can be simultaneously used to feasibly and reliably rate randomized clinical trials with both patient and provider-level targets.

  • The combined use of the PRECIS-2 and PRECIS-2-PS tools may provide clarity for clinical trial planning and grant submission and review, and enhance the quality of clinical trial protocol and main outcome manuscript publications.

  • Future clinical trials research methods development could focus on continuum indicators/checklists that assess the extent to which clinical trials successfully catalyze health care system practice changes that are linked to pre-specified policy targets.

Background

Over the past decade, both pragmatic and implementation science clinical trial research methods have advanced substantially [1,2,3,4,5,6,7,8]. The US National Institutes of Health recently released an application request that jointly solicits proposals for embedded health care system pragmatic and implementation randomized controlled trials [9]. Pragmatic trials are characterized by a primary purpose of understanding the effects of an intervention under usual practice conditions [9]. Implementation studies aim to understand the behavior of practitioners and others working within health care systems as key influences on the adoption, implementation, and sustainability of interventions and practice guidelines, while implementation trials test strategies designed to improve use of and/or fidelity to effective interventions. Per the application request, pragmatic and implementation clinical trial results should optimally inform policy makers and other key stakeholders [9].

Pragmatic and implementation randomized clinical trials have natural areas of overlap. For example, multisite pragmatic and implementation trials grapple with similar regulatory issues [10,11,12]. Pragmatic and implementation trials may be designed to target both individual patient effectiveness and provider-level practice change outcomes [5, 7, 8, 13,14,15]. Also, cross-cutting commentary has encouraged the integration of policy targets into pragmatic and implementation clinical trial designs as an approach to accelerating health care system practice changes derived from clinical trial results [5, 14, 16,17,18,19,20,21]. Methodologic integration across pragmatic and implementation trials has the potential to provide clarity to clinical trial planning and grant submission and review and to improve the quality of clinical trial protocol and main outcome manuscript publications [9, 22]. A literature review, however, revealed few investigations that have directly addressed applied methodologic integration across the two fields while also incorporating policy perspectives.

Prior investigations straddling pragmatic and implementation clinical trial approaches have focused on the Pragmatic-Explanatory Continuum Indicator Summary (PRECIS), a tool for rating clinical trials on the pragmatic-explanatory spectrum [2, 3, 22,23,24,25,26]. The PRECIS tool allows clinical investigators, grant reviewers, journal editors, and others to assess multiple patient-level randomized clinical trial domains across the explanatory-pragmatic trial continuum. A pioneering investigation from a National Institute of Health, Health Care System Research Collaboratory investigative team, used the PRECIS-2 tool to evaluate and compare five pragmatic trials [24]. Along with quantitative PRECIS-2 ratings of the five trials, the investigation included notational comments regarding the feasibility of using the PRECIS-2 tool.

More recently, the PRECIS-2-Provider Strategies (PS) tool has been developed with the aim of better characterizing trials where participants are health care providers [3, 22]. The tool is intended to clarify the rating of trials that are targeting provider-level rather than patient-level outcomes [22]. A literature review revealed no published investigations that have simultaneously used the PRECIS-2 and PRECIS-2-PS tools to rate trials with both patient- and provider-level targets.

Over the past decade, members of the investigative team have developed and rolled out two multi-site cluster randomized clinical trials in US trauma care systems, the Disseminating Organizational Screening and Brief Intervention Services (DO-SBIS) and the Trauma Survivors Outcomes and Support (TSOS) investigations [27,28,29,30]. The DO-SBIS and TSOS trials trained a broad spectrum of front-line trauma center providers in behavioral interventions with the aim of improving the quality of mental health and substance use screening and intervention procedures and patient-level outcomes. Also, the DO-SBIS and TSOS trials aimed to use clinical trial results to impact national trauma care system policy. The American College of Surgeons Committee on Trauma (ACS/COT) has the capacity to invoke health care system-level regulatory policies that are guided by randomized clinical trial results. [14, 31,32,33]. Both the DO-SBIS and TSOS trials included as a core design element, a multistep planning process targeting national trauma care system policy impact, that included an end-of-study policy summit [27,28,29,30]. The policy summit agendas included a review of the pragmatic trial results augmented by stakeholder participant commentary. Participating stakeholders included clinician scientists, trauma surgical policy makers, patient injury survivors, frontline trauma center providers, and grant/contract program personnel.

The current investigation builds upon and extends these prior investigations by working to further the methodologic integration of pragmatic and implementation science clinical trial approaches while simultaneously incorporating policy research perspectives. The investigation used the PRECIS-2 and PRECIS-2-PS tools to rate patient and provider-level characteristics of the DO-SBIS and TSOS trials. The investigation examined whether both tools could be simultaneously used to feasibly and reliably rate the two trials. The study employed the Rapid Assessment Procedure Informed Clinical Ethnography (RAPICE) qualitative approach to systematically document observations regarding the feasibility of using the PRECIS-2 and PRECIS-2-PS to rate the trials. RAPICE was also used to make clinical trial-related policy observations. The RAPICE observations sought to identify PRECIS-2 and PRECIS-2-PS domains that overlapped with or were potentially impacted by College policy. Given that the original intent of the PRECIS tools did not include a policy focus, RAPICE also aimed to augment the PRECIS coding with policy relevant observations.

Methods

Design overview

The investigation combined quantitative patient-focused PRECIS-2 and provider-focused PRECIS-2-PS tool ratings of the DO-SBIS and TSOS clinical trials with the qualitative RAPICE approach. The TSOS and DO-SBIS clinical trials have been previously described in detail [27,28,29,30]. Briefly, conducted between 2007 and 2013, the DO-SBIS trial randomized 20 US level I trauma centers to the ACS alcohol screening and intervention mandate as usual care control (n = 10) versus study team alcohol screening and brief intervention training (n = 10) conditions. A policy summit was scheduled during the final portion of the DO-SBIS study. The TSOS trial was built upon methods developed in the DO-SBIS trial and was conducted between 2015 and 2020. The TSOS trial aimed to test the effectiveness of evidence-based screening and interventions for Post-traumatic stress disorder (PTSD) and mental health and substance use comorbidities at 25-level US Level I trauma centers nationwide. As with the DO-SBIS trial, an end-of-study ACS policy summit was proactively planned for the final years of the TSOS trial.

All DO-SBIS procedures were approved by the University of Washington Institutional Review Board prior to the initiation of the investigation. All TSOS procedures were approved by the Western Institutional Review Board before the initiation of patient recruitment. The RAPICE study procedures were approved by the University of Washington Institutional Review Board.

PRECIS rating procedures

Previously published PRECIS-2 and PRECIS-2-PS manuscripts and tool descriptions informed the rating procedures [3, 22, 24]. The original patient-focused PRECIS-2 evaluates trials on the pragmatic-explanatory continuum across nine domains, including eligibility, recruitment, setting, organization, flexibility-delivery, flexibility-adherence, follow-up, primary outcome, and primary analysis. The PRECIS-2-PS expands the coding to incorporate provider-focused implementation domains that include eligibility, recruitment, setting, implementation resources, flexibility of provider strategies, flexibility of intervention, data collection, primary outcome, and primary analysis. Each domain is scored on a 5-point Likert scale that ranges from 1—very explanatory (center of the wheel) to 5—very pragmatic (rim of the wheel).

The PRECIS raters included seven study team members at the University of Washington’s Harborview Medical Center (DZ, LW, KM, AE, LP, JR, JW). The raters volunteered to participate on the project and were recruited based on their availability and interest. The raters had varying familiarity with the DO-SBIS and TSOS trials. None of the raters had previously participated in PRECIS-2 coding exercises.

All raters received PRECIS-2 and PRECIS-2-PS coding training tailored to the acute care medical context in which the DO-SBIS and TSOS trials were conducted. The training consisted of an initial PowerPoint zoom call orientation with the lead investigator (DZ). The initial presentation reviewed the original PRECIS-2 and PRECIS-2-PS publications as well as previous manuscripts documenting PRECIS coding efforts. The raters were instructed to derive their PRECIS-2 and PRECIS-2-PS ratings from a review of the DO-SBIS and TSOS trials study protocol and main outcome manuscripts [27,28,29,30].

The initial coding effort raised questions for multiple raters regarding PRECIS-2 explanatory-pragmatic continuum domain scaling. A recalibration slide set that added example figures and published manuscripts to better articulate the PRECIS-2 explanatory-pragmatic continuum scaling was therefore developed [3, 22, 34,35,36,37]. The slide set was sent via email to the raters, and all raters were given the opportunity to discuss the recalibration training slide set with the lead investigator. Final data presented in the tables represent post-recalibration training scoring.

Quantitative data analyses

Consistent with other studies using the PRECIS, rater means, standard deviations, medians, and ranges were produced for each of the nine patient and provider domains for both trials [24]. The mean scores for each domain were plotted on the PRECIS-2 and PRECIS-2-PS wheel diagrams. Interrater agreement was measured with the intraclass correlation coefficient (ICC) and the Kappa statistics.

RAPICE data collection and analysis

The RAPICE qualitative approach has been described in detail in prior publications [13, 38, 39]. Core features of the RAPICE approach include the formation of an interdisciplinary team, use of multiple data sources, iterative data collection, analysis in real-time, and rapid completion of the project. RAPICE is designed to be adaptable to diverse real world research contexts and embeds participant observation within clinical research protocols. Investigators conducting clinical research are trained in RAPICE methods, including the logging and analysis of field notes. In the current study, RAPICE field notes were taken during PRECIS-2 training by the lead investigator (DZ); the field notes incorporated general observations of the training content, questions asked by the raters, and modifications required of the training materials to enhance feasibility of use of the tools. Also, RAPICE field notes were taken during the TSOS end-of-study policy summit by the investigation’s expert mixed methods co-investigator (LP).

Data analyses in RAPICE incorporate two different data analysis procedures. The immersion/crystallization style was used to provide interpretations of the data by both DZ and LP and consisted of a prolonged immersion into and experience with the text and then emerging, after concerned reflection, with an intuitive crystallization of the text [40]. The more time-intensive editing style utilized a methodology of focused thematic analysis [41]. Field notes were coded by LP to condense the data into analyzable units. Segments of text ranging from a phrase to several paragraphs were assigned codes based on a priori or emergent themes (also known as open coding) [13]. Following the open coding, codes were assigned to describe connections between and within categories (also known as axial coding). Codes were then combined into groups to generate themes through a process of constant comparison.

Results

Descriptive characteristics of the DO-SBIS and TSOS trials are presented in Table 1. Both trials were cluster randomized and included ≥ 20 trauma center sites. The trials each recruited ≥ 600 patients and trained a full spectrum of front-line trauma center providers in the respective intervention approaches. Both trials include a planned end-of-study policy summit.

Table 1 Characteristics of the DO-SBIS and TSOS randomized clinical trials
Full size table

The PRECIS-2 and PRECIS-2-PS wheels are displayed for both the TSOS and DO-SBIS trials in Fig. 1. All four PRECIS-2 wheels were rated to be more pragmatic than explanatory, with overall composite mean and median scores ≥ 4 (Table 2).

Fig. 1
figure 1

Disseminating Organizational Screening and Brief Intervention Services (DO-SBIS) and Trauma Survivors Outcomes and Support (TSOS), PRagmatic Explanatory Continuum Indicator Summary-2 (PRECIS-2) and PRagmatic Explanatory Continuum Indicator Summary-2 -Provider Strategies (PRECIS-2-PS), wheel diagrams

Full size image
Table 2 PRECIS-2 and PRECIS-2-PS ratings for the TSOS and DO-SBIS trials
Full size table

Across trials, ICCs ranged from 0.77 to 0.87, and Kappa values ranged from 0.25 to 0.37 (Table 2). For both trials, the primary analysis domain was consistently rated most pragmatic (mean = 5.0, SD = 0), while the follow-up/data collection domain was rated most explanatory (mean range = 3.14–3.43, SD range = 0.49–0.69). The greatest rater variability was observed for the TSOS patient flexibility (adherence, mean = 4.0, SD = 1.15) and DO-SBIS provider setting (mean = 4.29, SD = 1.11) and flexibility of provider-focused strategies (mean = 3.57, SD = 1.13) domains.

Analysis of the PRECIS-2 training and policy summit RAPICE field notes revealed two broad themes: (1) challenges to the feasible use of PRECIS and (2) benefits of combined use of PRECIS and RAPICE. Two challenges to the use of PRECIS-2 were identified, challenges associated with training raters and challenges associated with operationalization of the measure. Multiple raters noted a lack of attention to detailed domain scaling in the initial coding training (Table 3). Two raters noted a lack of detail in the protocol and main outcome papers regarding the selection and training of providers. Other themes identified by individual raters included the need for enhanced definitions of terminology required for domain coding and rater variation in knowledge of trial details.

Table 3 Rapid Assessment Procedure Informed Clinical Ethnography (RAPICE) observations and implications
Full size table

The benefits of the mixed methods use of both PRECIS-2 and RAPICE were also twofold. One benefit was convergence of findings from both methods, illustrated in the similar emphasis related to the rationale for rating some domains as more explanatory. Another benefit was complementarity due to the availability of information obtained from one method but not the other. This was illustrated in the underappreciated restrictive impact of ACS/COT national policy, specifically on the DO-SBIS patient, flexibility of delivery, domain, and the optimal approaches to the translation of clinical trial results into ACS/COT standards and associated sustainable US trauma care system practice changes revealed in the RAPICE analysis of policy summit field notes. Analysis of the RAPICE policy summit observations information revealed four subthemes. The first subtheme was the difference in perception of ACS/COT recommendations by trauma surgeons and the ACS/COT. The second subtheme pertained to the distinction between a top-down versus a bottom-up responsibility for implementation. The third subtheme pertained to discussions of the new psychological symptoms screening and referral requirement for US Trauma Care Systems. The fourth subtheme pertained to the evidence required for successful implementation derived from different stakeholder groups.

Discussion

The current study builds upon and extends prior investigations that have straddled implementation science and pragmatic clinical trial approaches [2, 3, 22, 23]. The study, for the first time, used the PRECIS-2 and PRECIS-2-PS tools to characterize two health care system patient- and provider-focused randomized trials that incorporated ACS/COT policy targets. Overall, coders rated both trials to be quite pragmatic, with composite mean and median PRECIS-2 and PRECIS-2-PS scores of ≥ 4. Although the DO-SBIS clinical trial temporally preceded the TSOS trial, the two investigations had multiple design similarities. Both trials trained a broad spectrum of providers at 20 or more trauma center sites to implement behavioral interventions with hundreds of patients. Both trials included planned intent-to-treat main outcome analysis and research follow-up assessments. Regarding complimentary aspects of the PRECIS-2 and PRECIS-2-PS wheels, coding with the two tools identified the similarities between the trials. Across trials and PRECIS-2 wheels, the intent-to-treat data analysis domain was consistently rated most pragmatic, while the follow-up/data collection domain was consistently rated most explanatory.

The investigation found fair-to-good interrater reliability on the PRECIS-2 and PRECIS-2-PS as assessed by the ICC and Kappa statistics [42, 43]. The PRECIS-2-PS was developed to capture domains relevant to provider-focused trials [22]. Other PRECIS investigations have characterized multiple pragmatic trials across patient-focused domains without detailed attention to provider characteristics [24,25,26]. The current investigation furthers the integration of pragmatic trial and implementation science methods by documenting that the PRECIS-2 and PRECIS-2-PS can be feasibly and reliably applied to characterize trials focused on patient- and provider-level outcomes. These findings suggest that the combined use of the two tools could enhance conceptual and methodological clarity in the area of clinical trial planning by facilitating an improved match between patient and provider-level design decisions and overall trial purpose [22]. The combined use of the two tools may also enhance conceptual and methodological clarity for clinical trial grant submissions and review, and clinical trial protocol and main outcome manuscript publications [9, 22].

Commentary spanning pragmatic and implementation science clinical trial approaches suggests that policy integration could accelerate the research-to-practice translation; the recent NIH combined pragmatic and implementation trial announcement emphasized the importance of clinical trial policy relevance [5, 9, 14, 16,17,18,19,20,21]. The ACS/COT has the capacity to invoke health care system level policy implementation strategies, including regulatory requirements linked to national trauma center verification site visits [44]. Both the DO-SBIS and TSOS trials directly targeted the advancement of national trauma care system regulatory guidelines and included a multistep policy planning process. The letters of support submitted in the original grant applications obtained up-front commitments from key US trauma care system policy stakeholders. A planned end-of-study ACS/COT summit that convened stakeholders for a review of policy-relevant clinical trial findings was a central element of both trials. The presentation of policy-relevant findings derived from the trials was planned to occur within a single grant cycle, effectively reducing the time delay between the production of clinical trial research results and widespread health care system practice changes [6, 14, 16, 21].

A common pragmatic trial investigative approach has been to work inductively from individual trials to identify potentially integrative methods [13, 14, 45]. One goal of the current investigation was to use RAPICE observations to identify PRECIS domains that overlapped with or were impacted by College policy. For example, RAPICE field observations noted that College requirements for specific alcohol screening and brief intervention procedures had the potential to delimit trauma center usual care, thus impacting the PRECIS-2-PS “flexibility of intervention” domain. In addition, a key study RAPICE observation was that the PRECIS-2 and PRECIS-2-PS domain coding did not capture many themes related to catalyzing the research-to-practice translation through a multiphasic policy planning process. Other pragmatic trials have engaged health care system policy makers up-front in trial design and roll-out to expedite the translation of clinical trial results; these prior pragmatic trial efforts have focused both on the end-of-study implementation of novel practices found to be beneficial and the de-implementation of practices found to be redundant/without benefit to patients [45,46,47,48]. This initial constellation of policy-focused health care system randomized clinical trials could be seen to constitute examples of the emerging “learning health policy systems” construct [17].

Future research could refine the learning health policy systems construct as it relates to pragmatic and implementation randomized clinical trials. Additional investigation could productively focus on the development of continuum indicators/checklists that assess the extent to which clinical trials successfully catalyze health care system practice changes that are temporally linked to pre-specified policy targets. Elements for continuum/checklist ratings might include incorporation of policy levers up-front in clinical trial design, ratings of policy makers’ initial buy-in and sustained commitment to incorporating trial results, and whether the trial included an explicit mechanism for the health care system integration of findings, such as an end-of-study policy summit. Of note, such checklists could be derived from previously identified policy-relevant implementation domains [49]. Ultimately a key retrospective indicator would be whether findings were incorporated into health care system practice changes. One hypothesis associated with this scaling would be investigations that include greater up-front buy-in, linkage of trial results to health care system policy levers, and end-of-study review will demonstrate diminished research-to-practice translation timelines [14, 21]. As with the combined application of the PRECIS-2 and PRECIS-2-PS, the use of novel learning health policy systems indicators could advance the quality of clinical trial planning, grant submission, grant review, and protocol and main outcome manuscript publications.

The investigation also contributes to an evolving literature on PRECIS coder training. Some [23, 25], but not all [24], prior PRECIS studies have described feasible use of the PRECIS tools as well as reliable coder ratings. As with previously described PRECIS rater training efforts, in the current study, RAPICE observation suggested that coders required iterative modifications to the training procedures [24, 25]. In the current investigation, these efforts focused on clarifying the explanatory-pragmatic continuum for acute care medical intervention trials. Also, the protocol and main outcome manuscripts used by coders to retrospectively derive scores may not have included adequate information regarding the full spectrum of PRECIS-2 and PRECIS-2-PS domains.

The investigation has additional limitations. Raters were drawn from a single study team, and only two trials were rated. This approach may have contributed to decreased variability in ratings across PRECIS-2 and PRECIS-PS domains and pragmatic trials; raters did, however, have varying degrees of familiarity with the two trials. Also, the study team acknowledges that the PRECIS coding occurred at the conclusion of a muti-year investigation; prospective ratings may prove beneficial in future investigations. The training required iterative modifications. An alternative approach would have been to develop a lengthier, more rigorous training up-front. However, as with previous PRECIS training efforts, the study team’s overarching goal was to develop a lower-intensity, more generalizable training procedure that could require context-specific iterative modifications [25]. One strength of this approach is the potential broader application of the PRECIS-2 tools across raters with disparate research and/or clinical backgrounds.

Beyond these considerations, this investigation also contributes to an evolving literature on the ability of the RAPICE approach to systematically augment quantitative data collected as part of larger-scale health care system trials [13, 50, 51]. A growing literature describes the systematic incorporation of qualitative observation to augment quantitative findings of pragmatic and implementation clinical trials. Prior studies included notational comments regarding the feasibility of using the PRECIS-2 tool across multiple health care system pragmatic trials [24]. The RAPICE approach builds upon and expands prior investigation by more systematically documenting qualitative observations related both to feasibility of use of the PRECIS-2 and PRECIS-2-PS tools as well as systematically documenting qualitative observations related to policy summit proceedings.

Conclusion

The investigation documents that the PRECIS-2 and PRECIS-2-PS can be simultaneously used to feasibly and reliably rate pragmatic clinical trial protocols. The quality of pragmatic and implementation clinical trial protocol design, grant review, and manuscripts submissions can be enhanced by the use of common metrics such as the PRECIS-2 and PRECIS-2-PS rating tools. Finally, to optimally address the research-to-practice gap, future integrative study could productively focus on the development of continuum indicators continuum indicators/checklists that assess the extent to which clinical trials successfully catalyze health care system practice changes that are linked to pre-specified policy targets.