Patient-reported outcomes in randomized clinical trials: development of ISOQOL reporting standards

Abstract

Purpose

To develop expert consensus on a suite of reporting standards for HRQL outcomes of RCTs.

Methods

A Task Force of The International Society of Quality of Life Research (ISOQOL) undertook a systematic review of the literature to identify candidate reporting standards for HRQL in RCTs. Subsequently, a web-based survey was circulated to the ISOQOL membership. Respondents were asked to rate candidate standards on a 4-point Likert scale based on their perceived value in reporting studies in which HRQL was a study outcome (primary or secondary). Results were synthesized into draft reporting guidelines, which were further reviewed by the membership to inform the final guidance.

Results

Forty-six existing candidate standards for reporting HRQL results in RCTs were synthesized to produce a 40 item survey that was completed electronically by 161 respondents. The majority of respondents rated all 40 items to be either ‘essential’ or ‘desirable’ when HRQL was a primary RCT outcome. Ratings changed when HRQL was a secondary study outcome. Feedback on the survey findings resulted in the Task Force generalizing the guidance to include patient-reported outcomes (PROs). The final guidance, which recommends standards for use in reporting PROs generally, and more specifically, for PROs identified as primary study outcomes, was approved by the ISOQOL Board of Directors.

Conclusions

ISOQOL has developed a suite of recommended standards for reporting PRO results of RCTs. Improved reporting of PROs will enable accurate interpretation of evidence to inform patient choice, aid clinical decision making, and inform health policy.

Introduction

Patient-reported outcome (PRO) data, including health-related quality of life (HRQL), from randomised clinical trials (RCTs) may be used to inform clinical decision making, health policy, and reimbursement decisions [1]. PRO data can also be used to meet patients’ information needs and to establish treatment preferences [2–6]. However, the collection of PRO data in RCTs is not without costs, including patient and investigator time, and costs associated with questionnaire administration and analysis. For PRO data from RCTs to provide value, the RCTs and PROs need to be well designed, analyzed appropriately, and reported in a way that makes the results accessible and useful to end-users. Inadequate or poor-quality reporting limits the valid application of PRO findings in clinical practice [7–9] and can limit synthesis of trial results across studies, using meta-analytic or other synthesis approaches.

Although a number of publications provide guidelines for reporting HRQL outcomes, their implementation in RCTs remains suboptimal. In a recent review of 794 trials that reported HRQL outcomes of biomedical interventions across a range of clinical areas, less than 60 % provided a rationale for the selected outcome measure or a HRQL hypothesis, 33 % did not discuss the HRQL findings within the context of other trials outcomes, and only 28 % provided information on missing HRQL data [10].

Inadequacies in trial reporting in general have been addressed through the development of the Consolidated Standards of Reporting Trials (CONSORT) Statement [11]. This is ‘an evidence-based, minimum set of recommendations for reporting RCTs’ which ‘offers a standard way for authors to prepare reports of trial findings, facilitating their complete and transparent reporting, and aiding their critical appraisal and interpretation.’[11] The original CONSORT statement and its subsequent ‘extensions’ have been widely endorsed by a number of leading international journals [12]. CONSORT standards specific to PROs, and specifically HRQL, do not currently exist. Recognizing the need for improved reporting of quality of life outcomes, the International Society for Quality of Life Research (ISOQOL) formed a Task Force to lead the development of an official CONSORT extension. This work has been undertaken in collaboration with the CONSORT executive, MRC Midland and ConDuCT Hubs for Trials Methodology Research, leading international journal editors, policy makers, and patient representatives [11, 13, 14]. In this paper, we describe the process that was undertaken by the Task Force to establish an ISOQOL consensus on standards for PRO reporting and report the resultant suite of ISOQOL-recommended standards for PRO reporting. The suite of standards is intended to inform the process of establishing a CONSORT extension for reporting PRO outcomes of RCTs based on broader stakeholder input.

Methods

Systematic review

A comprehensive literature search was performed in OVID Medline in April 2011 to identify papers about HRQL reporting in RCTs. Search terms were specific to Clinical Trials as Topic/or Randomized Controlled Trials as Topic/and ‘Quality of Life’/or HRQL.mp. This search yielded over 4,000 hits; consequently, the search was limited to English-language review articles from 1985 to 2010.

Medline Search Strategy

1	Clinical trials as topic/or randomized controlled trials as topic/	222,848
2	‘Quality of Life’/or HRQL.mp	89,369
3	1 and 2	4,168
4	Limit 3 to (english language and ‘review articles’ and year = ‘1985–2010’)	1,871

Careful review of the resulting titles by two authors (MB and BB) reduced the number of potentially relevant papers to just over 300; abstracts of these papers were read and full papers obtained where the abstract illustrated potential relevance (n = 53). The resultant set of relevant publications was further supplemented with papers found in reference lists from seminal publications as well as those known to experts in the field. A search of the Cochrane database revealed no additional papers. All candidate standards for reporting HRQL outcome data in RCTs were abstracted from the papers identified by the comprehensive review of the literature and tabulated.

Survey development and administration

A panel of experts forming the ISOQOL Reporting Guidelines Task Force reviewed the candidate reporting standards (‘items’) to identify redundancies and omissions and created a comprehensive list of items. An online survey was then created based on this list of items. The survey asked respondents to rate the importance of each of the items for guiding the reporting of RCT HRQL results. Two contexts were provided to the respondents: HRQL as either a primary outcome or secondary outcome in a parallel-design phase III RCT. Respondents were asked to assume that no ‘expanded’ or secondary publication focused on RCT HRQL findings would be forthcoming. Four response categories were used: (essential, desirable, optional, and rarely necessary to include). An open text field was provided to capture respondents’ comments on each item. Finally, the survey requested demographic information relating to participant expertise and experience. Respondents could return to previous items by use of a back button and modify their answers before submitting their final results. Participants were advised that through submission of their results, consent to use their data in this research was implied. Ethical approval for the survey was obtained from the University of Birmingham Ethical Review Board (ERN_11-0225). The survey was developed using Survey Monkey [15].

ISOQOL members were invited via email to participate in the survey that remained open for a 7-week period during which time two email reminders were sent. Descriptive analyses were undertaken using Survey Monkey software statistical functions and Microsoft Excel [16]. Items were ranked according to the percentage of respondents that rated the item as essential. Exploratory subgroup analyses were undertaken to assess variation in response by self-reported expertise. Participants with similar expertise were grouped as follows: HRQL assessment/psychology/sociology/predominantly in psychometrics (grouped as Psychologist/Social Scientist); frequent journal reviewer/journal editor (as Journal Editor/Reviewer); clinical trials methods/analysis (as Methodologist); predominantly clinician/clinician-scientist (as Clinician); policy/public health/regulator/health administrator (as Policy Maker). Because survey participants were able to identify more than one area of expertise, the results of these exploratory analyses are for subgroups that are not mutually exclusive.

Development of the reporting standards

The results of the survey and free text comments were debated by the Task Force and synthesized into draft reporting standards. Items were placed into categories based on the organization of the existing CONSORT checklist of general reporting standards. Candidate items that were already contained within the main CONSORT 2010 checklist were flagged. The remaining items were grouped by the Task Force into: (1) those reporting standards recommended to be reported in any trial where HRQL is an outcome (primary or secondary) and (2) those recommended for clinical trials in which HRQL is specifically a primary outcome. Similar items were combined, resulting in a reduced draft set of reporting standards that was circulated to ISOQOL members attending the ISOQOL annual conference in Denver, CO in October 2011. Comments were invited through discussion at the Quality of Life in Clinical Practice Special Interest Group Meeting and following an oral presentation by Task Force Members. Further comments were encouraged and received by email. Feedback from the ISOQOL membership at the annual conference resulted in an important shift in the nature of the ISOQOL extension; the extension would no longer be exclusive to HRQL outcomes but rather would recommend reporting standards for all PROs. The final guidance document reflected feedback from both the ISOQOL Board and its broader membership.

Results

Systematic review

Fifteen papers by 14 unique first authors were identified that contained guidelines or standards for HQRL reporting in RCTs [7–9, 17–28]. From these papers, 46 HRQL RCT reporting standards were identified (Table 1). Removal of redundant items produced 40 final candidate reporting standards for use as survey items.

Table 1 Summary of Existing Guidance on Standards for HRQL Reporting in RCTs

Survey results

There were 161 respondents to the survey, of which 144 (97 %) were confirmed current members of ISOQOL. The survey link was potentially received by an estimated 480 members of ISOQOL. Although the number of invitations actually received could not be determined, the ratio of invitations to respondents implies a response rate of approximately 30 %. The majority (63 %) of respondents had an excess of 10 years experience in HRQL research (Table 2). Thirty-six of the 40 items (90 %) were rated as ‘essential’ by at least half of the respondents, 27 (68 %) were rated as ‘essential’ by at least two-thirds of respondents, and all items (100 %) were rated as either ‘essential’ or ‘desirable’ by over three quarters of respondents (Table 3). These rankings changed when HRQL was a secondary outcome; with 12 items (30 %) then rated as ‘essential’ by over one-half of respondents, and 26 (65 %) rated as either ‘essential’ or ‘desirable’ by over three quarters of respondents (Table 3). Results were fairly consistent across subgroups (by expertise) in exploratory analyses. Notably however, a higher proportion of clinicians rated the reporting of the clinical significance of results as ‘essential’ (Fig. 1).

Table 2 Survey respondent demographics

Table 3 Survey responses

Fig. 1

Survey response by participant expertise for selected reporting standards

ISOQOL-recommended PRO reporting standards for randomized clinical trials

The results of the survey and feedback from the ISOQOL membership were used to develop a suite of reporting standards. While the survey focused specifically on the development of standards for use in studies in which HRQL is an outcome, these standards were acknowledged to be appropriate for PROs in general, outcomes such as single-item symptom evaluation, patient-reported measures of utility, or assessment of patient adherence and satisfaction (Table 4) [29].

Table 4 ISOQOL-recommended PRO reporting standards for randomized clinical trials

Discussion

Synthesis of a systematic review of the literature and survey of the ISOQOL membership has informed the development of ISOQOL reporting guidelines for PROs in the primary publication of RCTs. More specifically, we provide reporting standards for RCTs reporting any PRO outcome and make additional recommendations for those trials in which PROs are the primary outcome. This work will be used to inform an official CONSORT extension for PRO reporting in RCTs.

While current CONSORT guidelines for trial reporting include several standards that apply to HRQL and PRO endpoints, the Task Force felt that further clarification was needed [11]. For example, the 2010 CONSORT guidance states that ‘a table showing baseline demographic and clinical characteristics for each group should be reported.’ However, neither the guideline nor the explanatory document explicitly recommends that publications report baseline values relating to the primary or secondary PRO endpoints with a measure of their variability. There is also a lack of clarity about the appropriate level of detail required for reporting missing outcome data, which is particularly critical when interpreting PRO trial endpoints.

The new ISOQOL reporting guidelines have important implications beyond improved documentation of trial outcomes if they also act to influence the design of future trials with PRO measures. Evidence for this type of impact is apparent in other areas; since the publication of the original CONSORT guidance, there is now work being undertaken to provide trialists and clinicians with standards for trial protocol development [30]. Future international collaborations among stakeholder groups could include similar efforts for informing trial design with respect to PROs.

The reality of journal word limits may constrain comprehensive reporting of all of the ISOQOL-recommended standards, particularly when the PRO is a secondary study outcome. A recent review of HRQL reporting in clinical trials across a range of clinical areas showed that HRQL was a secondary outcome in 75 % of studies (594/794) [10]. Without transparent reporting of these secondary outcomes, it may be impossible for readers or consumers to assess the quality of the results and identify any potential bias. Journal web appendices are a relatively new option that may, in part, alleviate this problem.

The process used to establish PRO reporting standards for RCTs has a number of strengths. The candidate reporting standards were identified through a formal systematic review, and the guidance was developed through a comprehensive iterative process with multiple opportunities for feedback from the experts within the ISOQOL membership who have a collective, extensive international experience in this field. A modified Delphi approach was used to assess member opinions and to reach consensus on a final list of reporting standards; this approach is consistent with the development of other CONSORT guidelines [31]. The consistency of endorsement of each standard across different areas of respondents’ expertise is an indication of the reliability of the survey data. One notable, but perhaps not surprising, exception to this consistency is that a higher proportion of clinicians rated reporting of the clinical significance of results as essential, demonstrating the potential importance of this item for translation of HRQL results into practice.

The interpretation of the survey results and the consensus recommendations are limited in some ways including the use of the online survey method to poll ISOQOL members. Although we attempted to invite all ISOQOL members to respond, it is unknown how many members actually received or viewed the email, and an accurate response rate cannot be calculated. Likewise, the ‘view rate’ of email invitations cannot be calculated due to the anonymous nature of the research method [32]. Further, it is likely that some ISOQOL members felt that they did not have sufficient expertise or interest in this academic topic to respond, and regrettably, we did not allow such a response option in our design. Likewise, it is possible that some respondents with only passing interest in the topic may have completed the survey without sufficient knowledge or experience in designing, analyzing, reporting, or interpreting RCTs. With regard to the latter possibility, however, we note that the majority of respondents did report having significant expertise in HRQL research. Again, the consistency of responses across expertise groups suggests reliability of the findings. Although there was clearly underlying variation in ISOQOL member opinions, particularly when HRQL is a secondary outcome, the systematic approach to the data by the Task Force led to final recommended standards that are in keeping with the majority view of the member survey results for each standard considered in the survey. Finally, while the literature search and survey focussed on HRQL standards, rather than all PROs, the consensus process endorsed the expanded scope.

Conclusions and implications

The results of this literature review and survey enabled an ISOQOL Task Force to arrive at consensus regarding recommended standards for reporting PROs in publications of randomized clinical trials. These recommendations were developed with input from the membership and endorsed by the ISOQOL leadership and have now been used in conjunction with information from other key stakeholders to inform a Consensus meeting and the development of a new CONSORT extension to guide the reporting of PROs [11, 31]. Knowledge transfer activities including endorsement from major journals will be crucial to the success of the CONSORT extension. This guidance will need to be evaluated with respect to its impact on the quality of PRO reporting, and we note that no formal evaluation of the past published recommended reporting standards has been undertaken. In keeping with promoting effective knowledge translation, the ISOQOL standards themselves will be available through the ISOQOL website and related publications and will serve as continued guidance for RCT reporting that reflects the ISOQOL consensus.

In sum, consumers of PRO data (including HRQL), who may be healthcare information providers, clinicians, policy makers, and patients themselves, should demand high-quality, well-reported data from RCTs. Likewise, organizations that fund clinical trials research should also be aware of issues relevant to PRO reporting and should require standardized reporting to facilitate use of PRO data in clinical practice. These consensus guidelines are an important component of establishing standards that will lead, through effective knowledge translation processes, to improved reporting practices.