A 2012 member survey conducted by the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) asked respondents about perceived barriers to using health economics and outcomes research in decision making [1]. The results were sobering. Among 130 self-identified decision makers responding to the survey, 45 % indicated that a perception among end users that research findings were methodologically flawed posed a ‘significant’ or ‘moderate’ barrier to using outcomes research in decision making [1]. In addition, 62 % of the decision makers indicated that a perception among end users that research findings were biased posed a ‘significant’ or ‘moderate’ barrier to using health economics and outcomes research in decision making [1]. The respondent sample size was small, raising questions regarding the generalizability of responses.

However, the respondents’ answers were consistent in spirit with results published 15 years ago [2] as well as those published more recently [3–7] identifying barriers to the incorporation of results regarding comparative and cost effectiveness (CCE) in decision making. These articles identified barriers related to the absence of: independent/unbiased/complete reporting, transparency, reproducibility and an appropriate comparator. Some of these concerns identified in the prior literature (e.g. ‘the absence of an appropriate comparator’) were not unique to non-randomized studies and could apply to randomized controlled trials (RCTs). Importantly, many of these concerns, including concerns related to reporting, transparency and reproducibility, could be targeted through appropriate signalling. The remainder of this article specifically discusses observational CCE studies utilizing patient-level data.

In a situation of asymmetric information, the producer (e.g. researcher) of the observational CCE study knows more about the characteristics of the research than the consumer or end user (e.g. private payer) of the observational CCE study and the producer’s challenge is to provide information that is of value to the consumer in assessing the quality of the research study and ultimately deciding whether or not to use the results to support coverage decisions. The development and availability of a statistical analysis plan (SAP) provides a vehicle for targeted signalling to decision makers. The focus here is on the development of SAPs for analytic, rather than descriptive, observational studies because decision makers are more often examining results from observational studies testing hypotheses related to CCE than observational studies strictly reporting descriptive statistics.

In the RCT setting, the SAP is made publicly available via the registration of the trial (at http://www.clinicaltrials.gov) and/or the publication of the analysis methods prior to completing the study. Several steps in the process just described are useful to note from a pure signalling standpoint: (a) the existence of an SAP; (b) the development of the SAP prior to conducting analyses; and (c) public availability of the version referenced in point (b). The existence of the SAP signals a commitment to transparency in the approach and to providing information needed to duplicate the CCE study. A priori (i.e. prior to study initiation) development of the SAP signals a commitment to the pre-specification of hypothesis-driven analyses. The public availability of the a priori version of the SAP signals confidence in the SAP as well as in its role in guiding the development of the final report.

Moving from RCTs to non-randomized observational studies of CCE, we may wonder whether the existence of an SAP, a priori development of the SAP and access to the a priori version is necessary. Given long-standing concerns among decision makers regarding the quality of CCE studies, some fundamental changes in the current process will be needed. It is argued here that all three components are important for improving confidence in hypothesis-driven observational CCE studies that are intended to support coverage decisions. In the absence of randomization, internal validity is of primary concern and investigators will propose associated analyses that can result in lengthy and detailed SAPs. These SAPs are generated for use by the internal study team and, increasingly, will be shared with the study sponsor. More sponsors require that study protocols and the attendant SAPs are developed and reviewed prior to commencing analyses.

Existence of the SAP is not enough, however. End users of observational CCE studies, such as hospital/health plan formulary decision makers and clinical providers, typically are not privy either to the study protocol or to the SAP. This lack of access to study details may lead to a potentially unsubstantiated but understandable lack of confidence in the results from observational CCE studies. Thus the second and third components (i.e. a priori development of the SAP and the public availability of this ‘a priori’ version) also may be needed to improve confidence in observational CCE studies.

It could be argued that the quality of observational studies more generally is already improving following the availability of research guidelines and that confidence in CCE studies will recalibrate accordingly without the need for public access to SAPs. It is true that familiarity with the tools necessary for conducting CCE studies has increased since guidelines regarding the conduct and reporting of observational studies were published [4, 5, 8–10] and specific guidance regarding the conduct and reporting of CCE studies for formulary decision making were made available [11]. Organizations have made a considerable effort to provide guidance regarding best practices (e.g. see ISPOR’s ‘Good Practices’ publications [22]). The availability of guidelines is an important but not sufficient development for the adoption of best practices.

How might the process leading to the availability of the a priori version of the SAP take shape? A professional organization can play a prominent role in the process by hosting a website to which SAPs for observational CCE studies employing non-randomized designs could be voluntarily submitted by researchers for public access. The prior availability of the SAP for an RCT is easily verified against the randomization date or another clearly established start date for patient enrolment. The start of the study is more difficult to establish for observational studies and could be anchored to the Institutional Review Board (IRB) protocol approval date. The a priori version of the SAP would include the planned date of the IRB protocol submission. At the conclusion of the study, a one-time update to the submitted SAP would confirm the date of the IRB approval and document the ‘methods history’ [12].

To reduce the administrative burden, the requirements for protocol amendments and/or reporting of deviations would not apply. The reduced reporting requirement could be justified by the non-interventional nature of observational CCE studies, as well as by the fact that important insights regarding the data generating processes of interest are not always circumscribed within ‘per-protocol’ analyses of observational data. For example, investigators concerned about the effect of treatment selection bias in a cost-effectiveness analysis using patient-level data may propose the creation of a propensity score-matched sample in the SAP. In the course of conducting the propensity score analysis, they may identify statistically and clinically significant disparities (e.g. based on age, race or severity status) in the receipt of treatment that could be useful for a better understanding of the sample to which the cost-effectiveness results generalize.

Voluntary submission of the study SAP by the research team members would allow the study team’s decision regarding granting public access to be made on a case-by-case basis, focusing on when it would be deemed beneficial without imposing an undue cost burden or violating data confidentiality clauses. ‘Voluntary submission’ rather than ‘registration’ avoids the formality (including administrative and researcher burden) associated with a process that is not unequivocally beneficial within the context of observational CCE studies. The advantages and disadvantages surrounding the registration of observational studies, more broadly, has fuelled a lively debate [13–20] regarding the utility and cost of public registration of observational studies. In the meantime, the public availability of SAPs for observational studies providing CCE information appears less daunting and, given persistent concerns among payers regarding bias, may be a worthwhile undertaking.

The structure outlined above would not address all concerns regarding public access to SAPs for observational CCE studies. There is the concern that the routine development of an SAP would remove the art from CCE research, clothe the analysis process in a straightjacket, and reduce the description of the statistical analysis process to a checklist. Current trends towards the development of SAPs for non-randomized study designs suggest that there is a willingness to develop rigorous pre-specified SAPs that are useful for guiding the conduct of analytic observational studies. As has occurred over time with IRB protocol templates that were initially structured to report exclusively on experimental studies, SAPs developed for observational CCE studies will need to reflect the goals of such studies. Specifically, SAPs developed for observational studies, including CCE studies, should discuss modification or recoding of original variables and describe statistical analyses to guard against model misspecification including methods to handle missing data, correlated data, bias and/or confounding [21].

The benefit to be gained from the public and a priori availability of the SAP assumes that end users at the local, regional or national level can review and independently assess the merits of the study based on the SAP. An informative review would require that statistical and research expertise is available to the end user. As has been noted [4], this expertise may not be available across local, regional and national decision makers. Lastly, there are benefits that are tangential to increasing transparency, but which are no less valued. The public availability of methods detailing study implementation encourages wider interest in these methods, fosters the standardization of measurement where useful, and provides incentives for formulary and health plan leadership to support the development of expertise in the methods of health technology assessment using observational studies.

Bold approaches targeting payers’ long-standing concerns with study transparency and reporting will be needed to boost confidence in CCE studies. The discussion here focuses on SAPs developed for observational CCE studies using patient-level data. The public availability of the SAP provides a targeted signal to payers that there is a commitment to transparency in analysis and reporting. This signalling can allay long-standing concerns and support the increased use of observational studies of CCE in decision making.